Purpose/Objective

In this role, you will be instrumental in designing, developing, and optimizing data pipelines on the Microsoft Azure cloud platform, with a primary focus on Azure Databricks and Delta Lake.
You will work closely with cross-functional teams including Data Scientists, Analysts, and Software Engineers to deliver high-quality, reliable data solutions that drive business intelligence and machine learning initiatives.
The ideal candidate will play a key role in transforming raw enterprise data into trusted, analytics ready datasets that power reporting, advanced analytics, and AI/ML initiatives

Key Responsibilities of Role
- Data Engineering & Lakehouse Design- Design, implement, and optimize Azure Lakehouse architectures using Azure Data Lake Storage (ADLS Gen2) and Databricks (Delta Lake) - Implement medallion architecture (Bronze, Silver, Gold layers)



for structured and unstructured data - Develop scalable batch and streaming data pipelines using Databricks - Databricks Development- Build and maintain Databricks notebooks using PySpark / Spark SQL - Optimize Spark jobs for performance and cost (partitioning, caching, Delta optimizations) - Implement Delta Lake features: ACID transactions, schema evolution, time travel, and data versioning - Data Integration & Processing Develop, test, and deploy highly scalable ETL/ELT data pipelines using Azure Databricks (PySpark/SQL) and Azure Data Factory (ADF).
- Develop ingestion pipelines from multiple sources: - Relational databases - APIs - Files (CSV, JSON, Parquet) - Streaming sources (Event Hubs / Kafka, if applicable) - Handle incremental loads, CDC, and data quality validations - Azure Ecosystem Integration- Integrate Databricks with: - Azure Data Factory / Synapse Pipelines - Azure Functions & Logic Apps (where applicable) - Power BI and other BI tools - Implement secure access using Azure Active Directory, Key Vault, Managed Identities - Data Governance & Quality- Apply data validation, reconciliati

📌 Data Architect Engineer (Ahmedabad)
🏢 Adani Group
📍 Ahmedabad