Key Responsibilities
- Design, build, and optimize end-to-end ETL/ELT pipelines in Databricks using Delta Lake, Delta Live Tables (DLT), Auto Loader, PySpark, and Spark SQL for high-volume, multi-format partner ingestion.
- Implement Medallion (zoned) architecture – Raw (bronze), Standardized (silver) with advanced validation, quarantine/reject logic, schema enforcement, and Curated (gold) consumer-ready datasets optimized for downstream COB/PI analytics.
- Leverage Unity Catalog for data governance, access control, lineage, and secure multi-tenant data management.
- Develop incremental processing, change data capture (CDC), backfill strategies, late-arriving data handling, and partitioning/optimization techniques (Z-Ordering, Liquid Clustering, Auto-Optimize) to eliminate performance bottlenecks.
- Build robust data quality frameworks using Delta constraints, expectations, and monitoring to ensure clean, reliable data for downstream consumption.
- Create production-grade Databricks Workflows, Jobs,



and orchestration for reliable batch and near-real-time processing using Spark Structured Streaming.
- Perform data profiling, mapping, reconciliation, and performance tuning of large-scale Spark jobs on Databricks clusters.
- Collaborate with Senior Data Architect and Data Modeller to translate target-state lakehouse design into implementable, testable increments.
- Deliver shippable, production-ready increments in Agile sprints within the implementation window, including CI/CD integration, unit/integration testing, and operational runbooks.
- Establish comprehensive observability using Databricks Lakehouse Monitoring, SQL Alerts, and dashboards for pipeline health and SLA compliance.

Requirements
Required Qualifications & Experience
- 8+ years of hands-on data engineering experience
- 5+ years building enterprise-scale solutions on Databricks (Unity Catalog, Delta Lake, Delta Live Tables)
- Proven track record delivering Medallion/zonal lakehouse architectures in prod

📌 Cloudera Data Engineer (Bengaluru)
🏢 McLaren Strategic Solutions
📍 Bengaluru