Description
We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and optimization of our enterprise-scale data ecosystem. In this senior role, you will spearhead the build-out of high‑performance data pipelines, enforce rigorous data quality and validation frameworks, and ensure end‑to‑end reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our data engineering strategy, enabling advanced analytics and mission‑critical, data‑driven decision-making within our insurance-focused business domains.

Responsibilities
- Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta Lake, and related Azure data services.
- Build and optimize high‑volume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.
- Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog, security, and workspace governance.




- Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure Databricks Workflows.
- Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed and cost efficiency.
- Implement robust data quality and validation frameworks using tools like Outstanding Expectations, DLT expectations, or custom PySpark checks.
- Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and RBAC/ABAC policies.
- Architect end‑to‑end data solutions supporting analytics, ML, actuarial models, and business-critical reporting workloads.
- Evaluate new Databricks features, MLflow enhancements, Photon execution, serverless compute, and recommend adoption strategies.
- Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.

Qualifications
- 8-12 years experience on Data Engineering role working with Databricks & Azure Cl

📌 Lead Data Engineer (Gurugram)
🏢 EXL
📍 Gurugram