Bangalore, Karnataka
Job Summary
We are seeking a highly skilled Azure Data Engineer to design, develop, and optimize enterprise-scale data platforms on Microsoft Azure. The ideal candidate will have robust expertise in building modern data lakes and lakehouse architectures, implementing batch and streaming data pipelines, data modeling, performance tuning, and cost optimization. The role involves working with large-scale datasets using Azure-native services and Databricks technologies to deliver reliable, scalable, and high-performance data solutions.
Key Responsibilities
Design, develop, and maintain scalable data ingestion and transformation pipelines using Azure Data Factory (ADF) and Azure Databricks .
Build and manage enterprise data platforms using ADLS Gen2 , Medallion Architecture (Bronze, Silver, Gold layers), and Lakehouse principles.
Develop robust ETL/ELT solutions using PySpark , Spark SQL , and Python .
Implement and optimize batch and streaming data processing solutions.
Design and maintain Delta Live Tables (DLT) for dependable and automated data transformation workflows.
Build streaming pipelines using Auto Loader , Structured Streaming, and Delta Lake technologies.
Establish and govern data assets through Unity Catalog for centralized security, governance, and metadata management.
Create and maintain dimensional and enterprise data models to support analytics and reporting requirements.
Develop and optimize Power BI datasets and semantic models for business intelligence and reporting.
Implement CI/CD pipelines for data engineering solutions using Azure DevOps or similar tools.
Monitor and improve data platform performance through workload optimization,
partitioning strategies, caching, and query tuning.
Drive cloud cost optimization initiatives across Azure Data Lake, Databricks, storage, and compute resources.
Ensure adherence to data governance, security, compliance, and best practices.
Collaborate with architects, business stakeholders, analysts, and application teams to understand requirements and deliver data solutions.
Skill Requirements
Azure Data Services
Azure Data Factory (ADF)
Azure Data Lake Storage Gen2 (ADLS)
Azure Databricks
Delta Lake
Unity Catalog
Data Engineering & Processing
Medallion Architecture
Batch Processing
Streaming Data Processing
Delta Live Tables (DLT)
Auto Loader
ETL/ELT Frameworks
Programming & Analytics
PySpark
Spark SQL
Python
Data Modelling
SQL
Reporting & Visualization
Power BI
Data Visualization
Semantic Models
DevOps & Automation
CI/CD Pipelines
Azure DevOps
Git
Infrastructure Automation
Performance & Optimization
Query Optimization
Spark Performance Tuning
Partitioning Strategies
Workload Management
Cost Optimization
Other Requirements
Experience with large-scale Azure Data Platform implementations.
Strong understanding of Lakehouse Architecture and Delta Lake.
Experience working in Agile/Scrum environments.
Knowledge of data governance and security frameworks.
Microsoft Azure Data Engineering certifications are highly desirable.
#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-
📌 Azure Senior Data Lead (India)
🏢 HCLTech
📍 India