Key Responsibilities

- Design and develop scalable data pipelines using Databricks and PySpark.
- Build and optimize ETL/ELT workflows for large-volume data processing.
- Develop data transformations using PySpark and SQL.
- Work extensively with Delta Lake for reliable and scalable data storage.
- Develop and maintain data pipelines using Databricks Workflows/Lakeflow.
- Perform data integration from multiple structured and unstructured sources.
- Optimize Spark jobs, queries, and pipeline performance.
- Implement data quality, validation, monitoring, and error-handling mechanisms.
- Collaborate with Data Architects, BI teams, and business stakeholders.
- Follow best practices for CI/CD, version control, and deployment.

Required Skills

- Strong hands-on experience with Databricks




- Strong PySpark and SQL skills
- Good understanding of Delta Lake
- Strong experience in Data Engineering and ETL
- Understanding of Data Warehousing and Data Modelling
- Experience with cloud platforms such as Azure / AWS / GCP
- Knowledge of Git, CI/CD and DevOps
- Good analytical and problem-solving skills
- Strong communication and stakeholder management skills

Valuable to Have

- Experience with Unity Catalog
- Databricks certifications
- Experience with Kafka / Azure Data Factory / Event Hubs
- Knowledge of Lakehouse architecture
- Experience in migrating legacy data platforms to Databricks

📌 Data Engineer and Senior Data Engineer (Mumbai)
🏢 Celebal Technologies
📍 Mumbai