Position: Lead Data Engineer

Employment Type: Full time

Experience: 8+ Years

Location: Pune

Job Summary

We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and ETL. The candidate will be responsible for designing, building, and maintaining scalable data pipelines, modernizing legacy ETL processes, and delivering high-performance data engineering solutions on the Databricks platform.

Mandatory Skills
Databricks
PySpark
Python
SQL
ETL
Data Pipeline Development
Delta Lake
Data Modelling
Databricks Jobs & Workflows
Legacy ETL Modernization
Git
CI/CD
Agile Development
Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Qualified

Good to Have
Azure, AWS, or GCP
Structured Streaming
Workspace AI Agent
Data Governance and Security
DevOps Practices

Key Responsibilities
Design, build, and maintain scalable data pipelines on the Databricks platform.
Develop end-to-end data workflows from ingestion to transformation and consumption.




Refactor legacy code and ETL pipelines to PySpark and modern ELT patterns.
Optimize Spark jobs, clusters, and Delta Lake tables for performance.
Implement data quality checks, monitoring, and error handling.
Manage Databricks Jobs and workflow orchestration.
Follow software engineering best practices, including version control, testing, and CI/CD.
Troubleshoot and resolve production data pipeline issues.
Collaborate with architects, analysts, infrastructure, application, and cyber teams.
Mentor junior engineers and maintain technical documentation.

Required Certification (Mandatory)
Databricks Certified Data Engineer Associate OR
Databricks Certified Data Engineer Professional

Preferred Certifications
Databricks Certified Associate Developer for Apache Spark
Azure Data Engineer Associate
AWS Certified Data Analytics
Google Cloud Skilled Data Engineer

📌 Lead Data Engineer Pune
🏢 Experis
📍 Pune