Role Description

We are looking for robust Lead Databricks Developer with streaming expertise. Key Responsibilities:
Design, develop, and lead enterprise-scale data platforms and pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.
Build and optimize scalable batch and real-time data ingestion solutions using Kafka, Event Hubs, Auto Loader, Structured Streaming, and Delta Live Tables (DLT).
Develop and manage Lakehouse architectures leveraging Medallion (Bronze, Silver, Gold) design patterns and data engineering best practices.
Implement and govern enterprise data assets using Unity Catalog, including data lineage, metadata management, security policies, and access controls.
Collaborate with architects and cloud teams to build and deploy data solutions on AWS and Azure, leveraging services such as S3, ADLS, Event Hubs, IAM, and cloud-native security capabilities.
Drive data quality, observability, monitoring, and validation frameworks to ensure trusted, reliable, and business-ready data products.
Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency through workload tuning and FinOps best practices.




Lead cloud modernization and migration initiatives from legacy/on-premises platforms to up-to-date Databricks Lakehouse architectures.
Implement and support GenAI, RAG, AI Agents, Vector Search, Mosaic AI, MLflow, and Databricks Genie use cases to accelerate analytics and business insights.
Establish CI/CD, DataOps, Infrastructure as Code (Terraform), and DevOps best practices while providing technical leadership, mentoring team members, and ensuring delivery excellence.

Domain

Experience:
Experience in Healthcare Payer domain (Claims, Membership, Coverage)
Exposure to FHIR / healthcare data standards preferred
Understanding of data privacy, security, and governance requirements (PII/HIPAA)
Leadership Expectations
Operate at a Lead level driving design, development, and delivery
Mentor team members and ensure delivery quality
Own modules independently and collaborate across distributed teams Experience:
8+ years in Data Engineering
5+ years hands-on experience with Databricks
Prior experience in Tech Lead/Senior roles

Skills Databricks, PySpark, Apache Spark, AWS, Databricks, Apache Spark, PySpark

📌 Lead Ii Data Engineering Pune
🏢 UST
📍 Pune