Role Description

We are looking for robust Lead Databricks Developer with streaming expertise. Key Responsibilities:

- Design, develop, and lead enterprise-scale data platforms and pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.
- Build and optimize scalable batch and real-time data ingestion solutions using Kafka, Event Hubs, Auto Loader, Structured Streaming, and Delta Live Tables (DLT).
- Develop and manage Lakehouse architectures leveraging Medallion (Bronze, Silver, Gold) design patterns and data engineering best practices.
- Implement and govern enterprise data assets using Unity Catalog, including data lineage, metadata management, security policies, and access controls.
- Collaborate with architects and cloud teams to build and deploy data solutions on AWS and Azure, leveraging services such as S3, ADLS, Event Hubs, IAM, and cloud-native security capabilities.
- Drive data quality, observability, monitoring, and validation frameworks to ensure trusted, reliable, and business-ready data products.
- Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency through workload tuning and FinOps best practices.




- Lead cloud modernization and migration initiatives from legacy/on-premises platforms to modern Databricks Lakehouse architectures.
- Implement and support GenAI, RAG, AI Agents, Vector Search, Mosaic AI, MLflow, and Databricks Genie use cases to accelerate analytics and business insights.
- Establish CI/CD, DataOps, Infrastructure as Code (Terraform), and DevOps best practices while providing technical leadership, mentoring team members, and ensuring delivery excellence.

Domain

Experience:

- Experience in Healthcare Payer domain (Claims, Membership, Coverage)
- Exposure to FHIR / healthcare data standards preferred
- Understanding of data privacy, security, and governance requirements (PII/HIPAA)
- Leadership Expectations
- Operate at a Lead level driving design, development, and delivery
- Mentor team members and ensure delivery quality
- Own modules independently and collaborate across distributed teams Experience:
- 8+ years in Data Engineering
- 5+ years hands-on experience with Databricks
- Prior experience in Tech Lead/Senior roles

Skills Databricks, PySpark, Apache Spark, AWS, Databricks, Apache Spark, PySpark

📌 Lead II - Data Engineering (Pune)
🏢 UST
📍 Pune