We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.
Key Responsibilities
- Design and develop scalable ETL/ELT pipelines.
- Develop PySpark and Spark SQL transformations.
- Build Bronze, Silver, and Gold layer data pipelines.
- Integrate SAP and enterprise source systems.
- Implement Delta Lake best practices.
- Optimize Spark jobs for performance and cost.
- Build reusable data engineering frameworks.
- Implement data quality and validation rules.
- Develop incremental and CDC pipelines.
- Collaborate with BI developers for optimized reporting datasets.
- Participate in code reviews and technical documentation.
- Support production deployments and troubleshooting.
Roles & Responsibilities
Ideal Candidate
- 812 years of Data Engineering experience.
- Strong hands-on experience with Azure Databricks.
- Expert in PySpark and Spark SQL.
- Experience with Delta Lake.
- Experience with Azure Data Factory.
- Knowledge of ADLS Gen2.
- Robust SQL skills.
- Experience with Git and CI/CD.
- Understanding of SAP data extraction techniques.
Preferred Certification
- Databricks Certified Data Engineer Professional.
- Experience with Structured Streaming.
- Knowledge of MLflow.
- Azure DevOps experience.
- Experience with Power BI datasets.
📌 Senior Databricks Data Engineer (Mumbai)
🏢 IITjobs
📍 Mumbai
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.