20 Nov
|
IITjobs
|
Mumbai
About the Role:
We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.
Key Responsibilities
Design and develop scalable ETL/ELT pipelines.
Develop PySpark and Spark SQL transformations.
Build Bronze, Silver, and Gold layer data pipelines.
Integrate SAP and enterprise source systems.
Implement Delta Lake best practices.
Optimize Spark jobs for performance and cost.
Build reusable data engineering frameworks.
Implement data quality and validation rules.
Develop incremental and CDC pipelines.
Collaborate with BI developers for optimized reporting datasets.
Participate in code reviews and technical documentation.
Support production deployments and troubleshooting.
Roles & Responsibilities
Ideal Candidate
812 years of Data Engineering experience.
Robust hands-on experience with Azure Databricks.
Expert in PySpark and Spark SQL.
Experience with Delta Lake.
Experience with Azure Data Factory.
Knowledge of ADLS Gen2.
Solid SQL skills.
Experience with Git and CI/CD.
Understanding of SAP data extraction techniques.
Preferred Certification
Databricks Certified Data Engineer Skilled.
Experience with Structured Streaming.
Knowledge of MLflow.
Azure DevOps experience.
Experience with Power BI datasets.
📌 Senior Databricks Data Engineer Mumbai
🏢 IITjobs
📍 Mumbai