Area(s) of responsibility

Key Responsibilities

1.

Data Solution

Design &

- Architecture
- Design end to end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.
- Architect and implement scalable and secure Up-to-date Data Warehouse (MDW) and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.
- Develop data models, integration patterns, and reusable frameworks aligned with best practices and enterprise architecture standards.
- Participate in requirement discussions, solution blueprinting, and technical feasibility assessments.

2.

Data Pipeline

Development

- Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
- Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
- Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
- Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
- Databricks &
- PySpark Engineering
- Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
- Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
- Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
- Collaborate with platform teams to manage Databricks clusters,



jobs, notebooks, and CI/CD integrations.
- Data Governance, Quality &
- Security
- Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency.
- Ensure compliance with organizational standards for data security, encryption, access control, and data lifecycle management.
- Create and maintain technical documentation, data flow diagrams, and operational support guides.
- Collaboration &
- Stakeholder Management
- Collaborate closely with BI, analytics, and business teams to understand data requirements and deliver reliable, production-ready solutions.
- Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
- Provide guidance and mentoring to junior engineers when required.

________________________________________

Mandatory Skills &

- Experience
- Minimum 7–10 years of experience in data engineering, with strong hands on exposure to Azure data ecosystem.
- At least 2 years of real project experience in Azure Databricks (not POCs).
- At least 2 years of hands-on experience building data pipelines using Azure Data Factory (ADF).
- At least 2 years of experience developing PySpark-based transformations in Databricks.
- Strong SQL programming experience, including writing complex queries, performance tuning, and handling large datasets.
- Experience with Azure Data Lake Storage (ADLS Gen2), data partitioning strategies, and file formats like Parquet/Delta/JSON.
- Knowledge of CI/CD pipelines (Azure DevOps preferred) for automated deployment of ADF/Databricks artifacts.

📌 Azure Databricks Developer (Pune)
🏢 Birlasoft
📍 Pune