Job Description Summary

We are seeking an experienced Data Engineer - Databricks to design, build, and operate scalable, high-performance data pipelines on the Databricks Lakehouse Platform. The role involves hands-on development using Apache Spark, Databricks notebooks, Delta Lake, and cloud-native services, along with close collaboration with analytics, AI/ML, and business teams.

Responsibilities
- Design, build, and maintain end-to-end data pipelines using Databricks (PySpark / Spark SQL)
- Implement batch and incremental data processing using Delta Lake and multi-hop architecture
- Develop and optimize Databricks notebooks, jobs, and workflows
- Ingest, transform, and curate large-scale structured and semi-structured datasets
- Support analytics, reporting, and downstream data consumption use cases
- Ensure data quality, reliability, lineage, and governance
- Collaborate with data scientists, analysts, and architects on AI/ML workloads
- Optimize Spark jobs for performance and cost efficiency
- Adhere to enterprise security, access control, and compliance standards




- Provide production support and troubleshoot data pipeline issues
- Document technical designs, data flows, and operational runbooks
- Mentor junior engineers and contribute to best practices

Mandatory skill sets
- 4+ years of experience as a Data Engineer with strong Databricks expertise
- Hands-on experience with Apache Spark, PySpark, and Spark SQL
- Robust knowledge of Delta Lake and Lakehouse architecture
- Advanced SQL skills
- Experience with ETL/ELT patterns and data warehousing concepts
- Exposure to at least one cloud platform (Azure / AWS / GCP)
- Understanding of distributed computing concepts
- Experience working in Agile teams

Preferred skill sets
- Experience with Unity Catalog and data governance
- Exposure to Auto Loader or streaming frameworks
- CI/CD for data pipelines
- Python for data engineering and automation
- Databricks certification (Associate / Professional)

Disclaimer

📌 IN_Senior Associate_Data Engineer Databricks_Data and Analytic (Hyderabad)
🏢 PwC Service Delivery Center
📍 Hyderabad