19 Nov
|
Provana India Private
|
Noida
19 Nov
Provana India Private
Noida
Role Overview :We are seeking a high-caliber Technical Lead to spearhead the design and development on our enterprise-scale Databricks data platform. As a hands-on lead, you will be the primary authority on our Azure-based Lakehouse, ensuring that data pipelines are not just functional, but scalable, secure, and cost-optimized. You must have a proven track record of leading end-to-end implementations - from initial discovery and schema design to production deployment and monitoring.Core Requirements :- 10 - 12 + years of experience in Data Engineering- Azure Mastery: Minimum 3 + years of lead experience specifically within the Azure cloud ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault, Azure functions).- Databricks Expertise: Deep hands-on experience with Databricks (Spark/PySpark), specifically implementing Medallion Architecture (Bronze, Silver, Gold layers).- Conversant with streaming and batch data processing.- Proven Leadership: Must have led at least two full-lifecycle, end-to-end data platform implementations in a Tech Lead capacity.- Proven Tech Lead experience on both Databricks and Azure- Excellent understanding of Lakehouse architecture- Very strong SQL skills and data modeling expertise is a must.- Strong experience with GenAI-assisted development. Should have used GitHub Copilot, Claude Code, Codex or equivalent; AI-assisted code generation, refactoring, testing, documentation and debugging using GenAI tools.- Should be conversant with LLM APIs, RAG, embeddings, vector databases, structured outputs, model selection, latency and token-cost optimization.- Should be familiar with agentic development and loop engineering.Key Responsibilities :- Architectural Design: Lead the design and evolution of our Delta Lakehouse architecture, ensuring it supports both high-volume and complex batch processing.- End-to-End Implementation:
Take full ownership of the data lifecycle: ingestion from disparate sources (APIs, RDBMS, NoSQL), complex transformations in PySpark, and final exposure via Gold tables or semantic layers.- Technical Mentorship: Guide a team of 3 - 5 data engineers through code reviews, design workshops, and the establishment of "best-in-class" Python and SQL coding standards.- Optimization: Conduct performance tuning of Spark clusters by exploring all available options like optimizing shuffle partitions and managing "Small File" problems to reduce Azure consumption costs.- Data Governance: Implement and manage Databricks Unity Catalog for centralized fine-grained access control, lineage tracking, and data discovery.- Handle production issues, root-cause analysis, and long-term fixes.Technical Skills & Toolset :- Languages: Expert-level Python (PySpark) and Advanced SQL. Familiarity with .net framework or Java is a plus.- Orchestration: Advanced use of Azure Data Factory (ADF) and Databricks Workflows/Jobs for complex dependency management.- DevOps/DataOps: Robust proficiency in Azure DevOps or GitHub Actions, specifically building CI/CD pipelines for automated Databricks notebook deployment and Infrastructure as Code (Terraform/Bicep).- Observability: Implementation of data quality frameworks (e.g., Great Expectations or Databricks Expectations) and monitoring via Azure Monitor/Log Analytics.Preferred Qualifications :- Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer Professional.- AI/ML Integration: Experience supporting ML workloads by building feature stores and integrating with MLflow.- Agile: Experience working in a high-velocity Scrum and Kanban environment.Soft Skills :- Strong ownership mindset and accountability- Excellent communication with technical and non-technical stakeholders (ref:hirist.tech)
📌 Provana - Technical Lead - Databricks Data Platform (Noida)
🏢 Provana India Private
📍 Noida