Job Title: Lead Azure Data Engineer

Location: Bangalore, Chennai, Hyderabad, Pune

Expereinece: 8-15 Years

Work Type: Contract (Extendable)

We are looking for an experienced Lead Data Engineer with 8 years of experience in designing, developing, and maintaining enterprise-scale data engineering solutions. The ideal candidate should have robust expertise in Microsoft Fabric, Databricks, PySpark, SQL/T-SQL, Delta Lake, Azure Data Factory, and Power BI. You will play a key role in modernizing legacy data platforms by building scalable Bronze, Silver, and Gold Medallion Architecture pipelines and migrating SQL Server ETL workloads to Microsoft Fabric.

Roles & Responsibilities

- Design, develop, and maintain scalable Bronze, Silver, and Gold data pipelines using Microsoft Fabric and Databricks.

- Implement Cascade 1.0 and Cascade 2.0 merge logic within the Silver layer, managing schema evolution, new attributes, and null value handling.

- Migrate legacy SQL Server objects including tables, views, stored procedures, and ETL processes into Microsoft Fabric Pipelines, Fabric Notebooks, and Databricks.

- Develop high-performance PySpark transformations and build optimized Delta Lake tables for enterprise data processing.

- Write and optimize complex SQL/T-SQL queries, stored procedures, views, functions, CTEs, and performance-tuned scripts.

- Design and develop Azure Data Factory (ADF)



pipelines for orchestrating enterprise data ingestion and transformation.

- Build and maintain scalable Lakehouse solutions using Microsoft Fabric and OneLake.

- Perform data ingestion, cleansing, transformation, validation, and reconciliation to ensure data quality and consistency.

- Establish data quality checks, lineage, reconciliation (apple-to-apple validation), and governance across enterprise datasets.

- Develop and optimize semantic models and support Power BI datasets for enterprise reporting and analytics.

- Collaborate with Data Engineers, BI Developers, QA teams, and business stakeholders to deliver high-quality data solutions.

- Optimize pipeline performance, resource utilization, and overall processing costs.

- Participate in Agile ceremonies including sprint planning, daily stand-ups, sprint reviews, retrospectives, and code reviews.

- Prepare technical documentation for data pipelines, migration activities, architecture, and operational support.

Required Skills

- Microsoft Fabric (Lakehouse, OneLake, Fabric Pipelines)

- Databricks

- PySpark

- SQL / T-SQL

- Delta Lake

- Azure Data Factory (ADF)

- Power BI (Datasets & Semantic Models)

- Data Engineering

- Medallion Architecture (Bronze, Silver, Gold)

- ETL Development

- Data Migration

- Data Quality & Data Lineage

- Agile / Scrum

📌 Lead Azure Data engineer (Bengaluru)
🏢 SWITS DIGITAL Private
📍 Bengaluru