Azure Databricks Engineering Lead

Azure Databricks Engineering Lead

Position Summary

We are seeking an experienced Azure Databricks Engineering Lead to provide technical leadership and overall engineering ownership for an enterprise Azure Databricks data platform. This role will lead the design, development, optimization, and evolution of the Databricks ecosystem, while establishing engineering standards, reusable patterns, and best practices across the team.

The ideal candidate will bring 10–15 years of data engineering and data platform experience, with deep hands-on expertise in Azure Databricks, Apache Spark, Scala, SQL, ADLS, and batch/streaming architectures. This individual will serve as the senior technical authority for the Databricks team, driving technical decisions, mentoring engineers, guiding architecture, and ensuring delivery of scalable, secure, reliable, and high-performing data solutions.

Required Skills

Technical — Must Have

- Azure Databricks — advanced hands-on development and platform expertise
- Azure Data Lake Storage (ADLS)
- Apache Spark — architecture, core concepts, optimization, and performance tuning
- Scala
- SQL — advanced development and performance tuning
- Data Streaming — Spark Structured Streaming and enterprise streaming patterns
- Enterprise-scale data engineering and data platform architecture
- Experience establishing reusable frameworks, engineering standards, and development patterns

Domain — Preferred

- Healthcare experience, including claims, clinical, eligibility, provider, or member data
- HIPAA-aware data handling and security practices

Nice to Have

- Python / PySpark
- Broader Azure platform knowledge
- MongoDB
- CI/CD, DevOps, data governance, and data quality frameworks

Required Experience

- 10–15 years of progressive experience in Data Engineering, Data Warehousing, and/or enterprise data platforms.
- 5+ years of hands-on Databricks and/or Spark experience, preferably within Azure.
- Proven experience designing and delivering enterprise-grade ETL/ELT and data processing solutions at scale.
- Deep experience with Azure Databricks, Spark, Scala, SQL, Azure Data Lake, and batch/streaming architectures.
- Demonstrated experience serving as a technical or engineering lead for a data engineering team.
- Experience defining architecture, engineering standards, reusable frameworks, and development patterns.
- Proven ability to lead complex initiatives from strategy and design through implementation and production support.
- Robust communication skills and experience collaborating with architects, product owners, analysts, QA, and upstream/downstream teams.

Roles & Responsibilities

Technical & Platform Leadership

- Provide overall technical leadership and engineering ownership for the Azure Databricks team and capability.
- Establish the technical vision, architecture, engineering standards,



and development practices for the Databricks platform.
- Serve as the senior technical point of contact for Databricks architecture, design, development, performance, and operational decisions.
- Define and standardize reusable data engineering patterns, frameworks, and components.
- Establish technical guardrails around performance, scalability, security, data quality, monitoring, and operational readiness.

Architecture & Solution Design

- Lead the design of enterprise-scale, cloud-native data solutions focused on integration, transformation, governance, quality, scalability, and performance.
- Own solution and technical designs for complex data pipelines and platform capabilities.
- Define scalable ingestion and processing patterns supporting batch, streaming, and raw/curated/consumption data zones.
- Identify technical risks and drive appropriate architectural solutions.

Team Leadership & Delivery

- Lead and provide technical direction to the Azure Databricks engineering team, including design guidance, code reviews, mentoring, and technical coaching.
- Partner with product and business leade

Required Skills

Technical — Must Have

- Azure Databricks — advanced hands-on development and platform expertise
- Azure Data Lake Storage (ADLS)
- Apache Spark — architecture, core concepts, optimization, and performance tuning
- Scala
- SQL — advanced development and performance tuning
- Data Streaming — Spark Structured Streaming and enterprise streaming patterns
- Enterprise-scale data engineering and data platform architecture
- Experience establishing reusable frameworks, engineering standards, and development patterns

Domain — Preferred

- Healthcare experience, including claims, clinical, eligibility, provider, or member data
- HIPAA-aware data handling and security practices

Nice to Have

- Python / PySpark
- Broader Azure platform knowledge
- MongoDB
- CI/CD, DevOps, data governance, and data quality frameworks

Required Experience

- 10–15 years of progressive experience in Data Engineering, Data Warehousing, and/or enterprise data platforms.
- 5+ years of hands-on Databricks and/or Spark experience, preferably within Azure.
- Proven experience designing and delivering enterprise-grade ETL/ELT and data processing solutions at scale.
- Deep experience with Azure Databricks, Spark, Scala, SQL, Azure Data Lake, and batch/streaming architectures.
- Demonstrated experience serving as a technical or engineering lead for a data engineering team.




- Experience defining architecture, engineering standards, reusable frameworks, and development patterns.
- Proven ability to lead complex initiatives from strategy and design through implementation and production support.
- Strong communication skills and experience collaborating with architects, product owners, analysts, QA, and upstream/downstream teams.

Roles & Responsibilities

Technical & Platform Leadership

- Provide overall technical leadership and engineering ownership for the Azure Databricks team and capability.
- Establish the technical vision, architecture, engineering standards, and development practices for the Databricks platform.
- Serve as the senior technical point of contact for Databricks architecture, design, development, performance, and operational decisions.
- Define and standardize reusable data engineering patterns, frameworks, and components.
- Establish technical guardrails around performance, scalability, security, data quality, monitoring, and operational readiness.

Architecture & Solution Design

- Lead the design of enterprise-scale, cloud-native data solutions focused on integration, transformation, governance, quality, scalability, and performance.
- Own solution and technical designs for complex data pipelines and platform capabilities.
- Define scalable ingestion and processing patterns supporting batch, streaming, and raw/curated/consumption data zones.
- Identify technical risks and drive appropriate architectural solutions.

Team Leadership & Delivery

- Lead and provide technical direction to the Azure Databricks engineering team, including design guidance, code reviews, mentoring, and technical coaching.
- Partner with product and business leadership to establish technical roadmaps, priorities, estimates, and execution plans.
- Drive initiatives from architecture and planning through development, implementation, and production stabilization.
- Identify skill gaps and promote engineering excellence, automation, and continuous improvement.

Optimization & Operational Excellence

- Continuously improve the Databricks codebase and platform to drive performance, scalability, reliability, automation, developer productivity, and cost efficiency.
- Lead performance tuning and root-cause analysis for complex Spark and Databricks workloads.
- Establish effective monitoring, logging, alerting, data quality, and operational practices.
- Drive resolution of complex production issues and recurring operational challenges.

Quality & Production Readiness

- Ensure solutions meet enterprise standards for security, governance, data quality, documentation, monitoring, and supportability.
- Provide technical leadership throughout QA, UAT, deployment, and production implementation.
- Ensure solutions are production-ready and supported by appropriate CI/CD and engineering practices.

📌 Azure Databricks Engineering Lead 10+ Shruti (Hyderabad)
🏢 HCL INDIA
📍 Hyderabad