Senior Data Lead - Azure Data Factory (ADF), Databricks

Experience: Not Available to Not Available years

Location: Chennai, India

Skills: Databricks, Apache Spark, PySpark, Scala, SQL, Delta Lake, Delta Live Tables, Azure, ADLS, Synapse, ADF, AWS, S3, Redshift, Glue, EMR, GCP, GCS, BigQuery, Dataflow, data lakes, lakehouse architectures, data modeling, Kafka, Event Hubs, Kinesis, MLflow, data orchestration tools, Airflow, Unity Catalog, CI/CD, Azure DevOps, Git, GitHub, Power BI, Tableau

Job Summary
Role Overview
The Databricks Engineer is responsible for designing, building, and optimizing scalable data processing and analytics solutions using Databricks, Apache Spark, and cloud data platforms. This role plays a key part in building modern lakehouse architectures, enabling advanced analytics, reporting, and AI/ML workloads.

Key Responsibilities

Key Responsibilities
Design, develop, and maintain ETL/ELT pipelines using Databricks and Apache Spark.
• Build and optimize batch and streaming data pipelines for large‑scale datasets.
• Implement Delta Lake solutions for reliable, performant, and scalable data management.
• Collaborate with Data Architects to implement lakehouse architecture.
• Work with cross‑functional teams including Data Scientists, BI teams, and Business stakeholders.
• Optimize Spark jobs for performance, cost, and scalability.
• Integrate data from multiple sources (databases, APIs, streaming platforms).




• Ensure data quality, validation, security, and governance standards are met.
• Support production deployments, monitoring, and performance tuning.
• Contribute to CI/CD pipelines and best practices for data engineering workflows.

Skill Requirements

Technical Skills & Experience
Mandatory
• Strong hands‑on experience with Databricks.
• Deep knowledge of Apache Spark (PySpark and/or Scala).
• Experience with Delta Lake, Delta Live Tables, and Databricks workflows.
• Strong SQL skills for data transformation and analytics.
• Experience working with cloud platforms: Azure (ADLS, Synapse, ADF) or AWS (S3, Redshift, Glue, EMR) or GCP (GCS, BigQuery, Dataflow).
• Experience building data lakes / lakehouse architectures.
• Understanding of data modeling (star/snowflake schemas).

Other Requirements
Positive to Have
• Experience with streaming technologies (Kafka, Event Hubs, Kinesis).
• Exposure to MLflow, feature stores, or ML pipelines on Databricks.
• Experience with data orchestration tools (ADF, Airflow).
• Familiarity with Unity Catalog and data governance frameworks.
• Databricks or Cloud platform certifications.

Tools & Technologies
Databricks, Apache Spark, PySpark / Scala / SQL, Delta Lake, Cloud Storage (ADLS / S3 / GCS), CI/CD (Azure DevOps, Git, GitHub), Visualization/BI tools (Power BI, Tableau – optional).

📌 Senior Data Lead Azure Data Factory (ADF), Databricks (Chennai)
🏢 HCL Technologies
📍 Chennai