We are looking for solid Lead Databricks Developer with streaming expertise.

Key Responsibilities

• Design, develop, and lead enterprise-scale data platforms and pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.

• Build and optimize scalable batch and real-time data ingestion solutions using Kafka, Event Hubs, Auto Loader, Structured Streaming, and Delta Live Tables (DLT).

• Develop and manage Lakehouse architectures leveraging Medallion (Bronze, Silver, Gold) design patterns and data engineering best practices.

• Implement and govern enterprise data assets using Unity Catalog, including data lineage, metadata management, security policies, and access controls.

• Collaborate with architects and cloud teams to build and deploy data solutions on AWS and Azure, leveraging services such as S3, ADLS, Event Hubs, IAM, and cloud-native security capabilities.

• Drive data quality, observability, monitoring, and validation frameworks to ensure trusted, reliable, and business-ready data products.

• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency through workload tuning and FinOps best practices.





• Lead cloud modernization and migration initiatives from legacy/on-premises platforms to modern Databricks Lakehouse architectures.

• Implement and support GenAI, RAG, AI Agents, Vector Search, Mosaic AI, MLflow, and Databricks Genie use cases to accelerate analytics and business insights.

• Establish CI/CD, DataOps, Infrastructure as Code (Terraform), and DevOps best practices while providing technical leadership, mentoring team members, and ensuring delivery excellence.

Domain Experience

• Experience in Healthcare Payer domain (Claims, Membership, Coverage) • Exposure to FHIR / healthcare data standards preferred • Understanding of data privacy, security, and governance requirements (PII/HIPAA) • Leadership Expectations • Operate at a Lead level driving design, development, and delivery • Mentor team members and ensure delivery quality • Own modules independently and collaborate across distributed teams Experience:

• 8+ years in Data Engineering • 5+ years hands-on experience with Databricks • Prior experience in Tech Lead/Senior roles

Databricks, PySpark, Apache Spark, AWS, Databricks, Apache Spark, PySpark

📌 Lead II - Data Engineering (Pune)
🏢 UST
📍 Pune