Provana - Technical Lead - Databricks Data Platform (India)

Provana - Technical Lead - Databricks Data Platform (India)

19 Nov
|
Provana
|
India

19 Nov

Provana

India

Role Overview :

We are seeking a high-caliber Technical Lead to spearhead the design and development on our enterprise-scale Databricks data platform. As a hands-on lead, you will be the primary authority on our Azure-based Lakehouse, ensuring that data pipelines are not just functional, but scalable, secure, and cost-optimized. You must have a proven track record of leading end-to-end implementations - from initial discovery and schema design to production deployment and monitoring.

Core Requirements :

- 10 - 12 years of experience in Data Engineering
- Azure Mastery: Minimum 3 years of lead experience specifically within the Azure cloud ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault, Azure functions).
- Databricks Expertise: Deep hands-on experience with Databricks (Spark/PySpark), specifically implementing Medallion Architecture (Bronze, Silver, Gold layers).
- Conversant with streaming and batch data processing.
- Proven Leadership: Must have led at least two full-lifecycle, end-to-end data platform implementations in a Tech Lead capacity.
- Proven Tech Lead experience on both Databricks and Azure
- Excellent understanding of Lakehouse architecture
- Very strong SQL skills and data modeling expertise is a must.
- Strong experience with GenAI-assisted development. Should have used GitHub Copilot, Claude Code, Codex or equivalent; AI-assisted code generation, refactoring, testing, documentation and debugging using GenAI tools.
- Should be conversant with LLM APIs, RAG, embeddings, vector databases, structured outputs, model selection, latency and token-cost optimization.
- Should be familiar with agentic development and loop engineering.

Key Responsibilities :

- Architectural Design: Lead the design and evolution of our Delta Lakehouse architecture, ensuring it supports both high-volume and complex batch processing.
- End-to-End Implementation:



Take full ownership of the data lifecycle: ingestion from disparate sources (APIs, RDBMS, NoSQL), complex transformations in PySpark, and final exposure via Gold tables or semantic layers.
- Technical Mentorship: Guide a team of 3 - 5 data engineers through code reviews, design workshops, and the establishment of "best-in-class" Python and SQL coding standards.
- Optimization: Conduct performance tuning of Spark clusters by exploring all available options like optimizing shuffle partitions and managing "Small File" problems to reduce Azure consumption costs.
- Data Governance: Implement and manage Databricks Unity Catalog for centralized fine-grained access control, lineage tracking, and data discovery.
- Handle production issues, root-cause analysis, and long-term fixes.

Technical Skills & Toolset :

- Languages: Expert-level Python (PySpark) and Advanced SQL. Familiarity with .net framework or Java is a plus.
- Orchestration: Advanced use of Azure Data Factory (ADF) and Databricks Workflows/Jobs for complex dependency management.
- DevOps/DataOps: Strong proficiency in Azure DevOps or GitHub Actions, specifically building CI/CD pipelines for automated Databricks notebook deployment and Infrastructure as Code (Terraform/Bicep).
- Observability: Implementation of data quality frameworks (e.g., Outstanding Expectations or Databricks Expectations) and monitoring via Azure Monitor/Log Analytics.

Preferred Qualifications :

- Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer Professional.
- AI/ML Integration: Experience supporting ML workloads by building feature stores and integrating with MLflow.
- Agile: Experience working in a high-velocity Scrum and Kanban environment.

Soft Skills :

- Strong ownership mindset and accountability
- Excellent communication with technical and non-technical stakeholders

📌 Provana - Technical Lead - Databricks Data Platform (India)
🏢 Provana
📍 India

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: provana - technical lead - databricks data platform (india) / india