Area(s) of responsibilityRole Summary

We are seeking a seasoned Senior Azure Data Engineer / Sr. Technical Lead with extensive expertise in Azure Databricks, PySpark, ADF, SQL, and modern data engineering practices. This role requires solid technical leadership, hands-on engineering capabilities, and the ability to design, architect, and deliver enterprise-scale data solutions. The ideal candidate will lead complex data initiatives, mentor engineering teams, and collaborate with cross-functional stakeholders to build a robust and scalable data ecosystem on Azure.

Key Responsibilities

1. Solution Architecture & Technical Leadership

- Lead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, ADF, and Azure Data Lake.
- Architect end-to-end Modern Data Warehouse (MDW) and Lakehouse solutions, ensuring scalability, performance, security, and cost optimization.
- Define technical standards, coding best practices, reusable frameworks, and architectural guidelines for engineering teams.
- Provide technical leadership across the project lifecycle-requirements analysis, solution blueprinting, estimation, development, and deployment.

- Data Pipeline Engineering

- Build, optimize, and maintain scalable, high-performance ELT/ETL pipelines to process large volumes of structured and unstructured data.
- Set up complex data ingestion frameworks, enabling seamless integration with on-premise systems, cloud services, APIs, and third-party sources.
- Ensure high availability, data reliability, and error-resilient orchestration workflows in Azure Data Factory.

- Azure Databricks & PySpark Expertise





- Design and implement advanced transformation logic using PySpark on Databricks, ensuring efficient data processing and code modularity.
- Utilize Delta Lake capabilities-ACID transactions, schema evolution, versioning, time travel-to manage enterprise-grade datasets.
- Perform cluster-level tuning, optimization of shuffle operations, caching, partitioning, and job parallelization.
- Manage Databricks job pipelines, notebooks, clusters, job scheduling, and integration with CI/CD pipelines.

Mandatory Skills & Experience

- 10-13 years of overall experience in data engineering and enterprise data platforms.
- Minimum 3 years of hands-on project experience in Azure Databricks (beyond POCs).
- Minimum 5 years of experience building and orchestrating pipelines in Azure Data Factory (ADF).
- Minimum 2+ years of strong PySpark experience with complex data transformation logic.
- 6+ years of ETL & Data Warehouse experience, including dimensional modelling, data partitioning, and performance optimization.
- Strong SQL expertise-complex queries, optimization, stored procedures, analytical functions.
- Proven experience working with Azure Data Lake Storage (ADLS), Delta Lake, and modern data processing patterns.

Good-to-Have Skills

- Experience working in Agile/Scrum environments, including sprint planning and backlog management.
- Knowledge of Azure Synapse, Event Hub, Databricks workflows, and Azure DevOps CI/CD.
- Databricks certifications (Associate/Professional) or Azure certifications (DP 203/DP 900).

Locations

- Country/Region: IN
- Requisition ID: 37467
- Work Model:
- Position Type:
- Salary Range:
- Location: INDIA - NOIDA- BIRLASOFT OFFICE

📌 Title: Azure Databricks Developer (Noida)
🏢 Birlasoft
📍 Noida