We are looking for a Senior Developer Databricks Data Engineering with strong hands-on experience in Databricks, PySpark, Spark SQL, and Delta Lake to develop and maintain scalable data engineering pipelines. The ideal candidate should have experience building ETL/ELT pipelines, implementing incremental/CDC ingestion, optimizing Spark workloads, and working with Lakehouse architecture.
Key Responsibilities
- Develop and maintain PySpark and Spark SQL notebooks and jobs on Databricks.
- Build and manage scalable ETL/ELT pipelines across Bronze, Silver, and Gold Lakehouse layers.
- Design and maintain Delta Lake tables, including partitioning and schema evolution.
- Implement incremental and CDC ingestion using Delta MERGE and Auto Loader.
- Build and maintain dimensional data models for BI and analytics.
- Monitor, troubleshoot, and optimize Databricks jobs and Spark pipelines.
- Perform Spark and Delta Lake performance tuning to meet SLA and cost targets.
- Support production issue triage, root-cause analysis, and resolution.
- Develop reusable PySpark utilities and frameworks to improve engineering standards.
- Collaborate with technical leads and upstream/downstream teams on data pipeline requirements and changes.
Required Skills
- 4+ years of hands-on experience in Data Engineering.
- Strong experience with Databricks and Apache Spark.
- Proficiency in PySpark, Spark SQL, Python, and SQL.
- Strong knowledge of Delta Lake and Lakehouse architecture.
- Experience with Databricks Workflows / Jobs.
- Experience building ETL/ELT data pipelines.
- Knowledge of incremental and CDC ingestion using MERGE / Auto Loader.
- Experience with Spark performance tuning and query optimization.
- Understanding of dimensional data warehouse modeling.
- Experience with cloud storage such as ADLS or S3.
- Robust problem-solving and debugging skills.
Good to Have: Databricks Data Engineer certification, Kafka/Event Hubs experience, and Git/GitHub experience.
📌 Senior Developer - Databricks Data Engineering (Pune)
🏢 VISDIN Solutions
📍 Pune
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.