Chennai, Tamil Nadu
Job Summary

Senior Data Engineer – Azure Databricks & ADF

Senior data engineer responsible for designing, developing, and optimizing scalable data pipelines using Azure Databricks and Azure Data Factory, with strong ownership in delivery, transition, and stabilization as part of offshore ramp-up.

Key Responsibilities

Key Responsibilities

- Design and develop end-to-end data pipelines using ADF and Databricks
- Build scalable transformation logic using PySpark / Spark SQL
- Implement and maintain Delta Lake architecture (Bronze/Silver/Gold layers)
- Translate business requirements into efficient data models and pipeline designs
- Lead complex data ingestion scenarios (structured, semi-structured, APIs)
- Actively support knowledge transition (KT) from incumbent team
- Drive performance tuning and optimization of pipelines and clusters
- Ensure data quality, validation, and reconciliation checks
- Contribute to CI/CD setup, code reviews, and release processes
- Mentor junior engineers and ensure delivery quality

Skill Requirements

Mandatory Skills

- 8+ years of data engineering experience
- Strong hands-on expertise in:

o Azure Databricks (PySpark, Spark SQL, Delta Lake)

o Azure Data Factory (pipeline orchestration, triggers, monitoring)

- Deep understanding of:

o Data pipeline design patterns

o Data modelling concepts

o Large-scale data processing

- Strong SQL expertise
- Experience working in complex enterprise data environments

Preferred Skills

- Prior experience in transition / offshore backfill engagements




- Familiarity with:

o Medallion architecture

o Data lakehouse concepts

- Exposure to:

o Azure DevOps / CI-CD pipelines

o Monitoring and logging frameworks

- Domain experience in Insurance / Financial Services

Soft Skills

- Robust ownership and accountability (critical for incumbent replacement)
- Ability to work independently with minimal supervision

Other Requirements

Mandatory Skills

- 8+ years of data engineering experience
- Strong hands-on expertise in:

o Azure Databricks (PySpark, Spark SQL, Delta Lake)

o Azure Data Factory (pipeline orchestration, triggers, monitoring)

- Deep understanding of:

o Data pipeline design patterns

o Data modelling concepts

o Large-scale data processing

- Strong SQL expertise
- Experience working in complex enterprise data environments

Preferred Skills

- Prior experience in transition / offshore backfill engagements
- Familiarity with:

o Medallion architecture

o Data lakehouse concepts

- Exposure to:

o Azure DevOps / CI-CD pipelines

o Monitoring and logging frameworks

- Domain experience in Insurance / Financial Services

Soft Skills

- Strong ownership and accountability (critical for incumbent replacement)
- Ability to work independently with minimal supervision

#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-

📌 Technical Specialist - Snowflake, Azure Data Factory (ADF), Databricks (Chennai)
🏢 HCLTech
📍 Chennai