Lead Databricks Data Engineer
Role Summary
We are seeking a Lead Databricks Data Engineer to design, build, and operate scalable data platforms on Azure for our manufacturing business. The ideal candidate will lead end-to-end data solutions—data mesh, data lake, and data warehouse architectures—optimize Databricks workloads for performance and cost, enforce governance and security, and mentor engineering teams to deliver reliable, production-grade data pipelines.
Responsibilities
- Design and implement end-to-end data solutions on Azure Databricks, focusing on scalability, reliability, security, and cost optimization.
- Lead the design and rollout of data mesh, data lake, and data warehouse architectures tailored to manufacturing use cases and data domains.
- Translate business and analytics requirements into detailed technical designs, data models, and implementation plans.
- Develop, optimize, and maintain data ingestion and transformation pipelines using Databricks, Apache Spark, PySpark, and Azure Data Factory.
- Tune Databricks clusters, jobs,
and Spark workloads for performance and cost efficiency; enforce best practices for partitioning, caching, and resource allocation.
- Implement and enforce data governance, access controls, encryption, and disaster recovery processes across the Databricks setting.
- Produce and maintain technical design documents, runbooks, and operational runbooks that align with business objectives.
- Establish and promote CI/CD practices for data pipelines, notebooks, and infrastructure-as-code (IaC) deployments.
- Collaborate closely with data scientists, analytics, BI, operations, and domain teams to ensure data quality, lineage, and usability.
- Mentor and guide development teams on data engineering standards, Spark/PySpark patterns, testing, and observability.
Requirements
- 7+ years of professional experience as a Data Engineer or similar role working on cloud-based data platforms.
- 4+ years hands-on experience with Azure
📌 Lead Databricks Data Engineer (India)
🏢 AdAstra
📍 India