We are looking for a hands-on Senior Data Engineer with 7 to 9 years of experience to design, build, and optimize large-scale data pipelines and lakehouse solutions on Databricks. The ideal candidate is a solid individual contributor who stays current with the latest Databricks platform capabilities — including Unity Catalog, Lakeflow (Delta Live Tables), Lakebase, and Databricks' expanding AI/agent tooling — and can apply them to solve real-world data engineering problems at scale.
Key Responsibilities:
- Design, develop, and maintain scalable ETL/ELT pipelines on Databricks using PySpark, Spark SQL, and Delta Lake for batch and streaming workloads.
- Build and manage declarative pipelines using Lakeflow / Delta Live Tables (DLT), including expectations, data quality checks, and change data capture (CDC).
- Implement and manage data governance, access control, lineage, and data sharing using Unity Catalog across multiple workspaces and clouds.
- Optimize Spark jobs and Databricks clusters for performance and cost, leveraging Photon, serverless compute, auto-scaling,



and job clustering best practices.
- Design and implement medallion architecture (bronze/silver/gold) data models and lakehouse patterns for analytics and ML consumption.
- Work with Databricks Workflows (Jobs) to orchestrate multi-task pipelines, including dependency management, retries, and monitoring/alerting.
- Integrate Databricks with cloud-native services (AWS/Azure/GCP) such as S3/ADLS/GCS, Kafka/Event Hubs/Kinesis, Glue/ADF, and IAM/Entra ID for secure, automated data flows.
- Apply CI/CD practices for Databricks using Databricks Asset Bundles (DABs), Repos, and Git integration; automate deployments across dev/test/prod.
- Evaluate and adopt newer Databricks capabilities — Lakebase (serverless Postgres on the lakehouse), Unity Catalog Metrics, Genie/Agent Bricks, Mosaic AI, and real-time/streaming enhancements — and recommend where they add value to existing pipelines.
- Implement data quality

📌 DE&A - Sr. Data Engineer - Databricks (Pune)
🏢 Zensar
📍 Pune