Area(s) of responsibility
Key Responsibilities1. Data Solution Design & Architecture• Design end to end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.• Architect and implement scalable and secure Contemporary Data Warehouse (MDW) and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.• Develop data models, integration patterns, and reusable frameworks aligned with best practices and enterprise architecture standards.• Participate in requirement discussions, solution blueprinting, and technical feasibility assessments.2. Data Pipeline Development• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.• Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.• Build orchestration workflows in ADF, including pipelines, triggers, linked services,



integration runtimes, and parameterized datasets.3. Databricks & PySpark Engineering• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.• Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.• Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.• Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.4. Data Governance, Quality & Security• Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency.• Ensure compliance with organizational standards for data security, encryption, access control, and data lifecycle management.• Create and main

📌 Azure Databricks Developer (Pune)
🏢 Birlasoft
📍 Pune