Position: Lead Data Engineer

Employment Type: Full time

Experience: 8+ Years

Location: Pune

Job Summary

We are looking for an experienced Lead Data Engineer with strong expertise in Databricks, PySpark, Python, SQL, and ETL. The candidate will be responsible for designing, building, and maintaining scalable data pipelines, modernizing legacy ETL processes, and delivering high-performance data engineering solutions on the Databricks platform.

Mandatory Skills

- Databricks
- PySpark
- Python
- SQL
- ETL
- Data Pipeline Development
- Delta Lake
- Data Modelling
- Databricks Jobs & Workflows
- Legacy ETL Modernization
- Git
- CI/CD
- Agile Development
- Databricks Certified Data Engineer Associate OR Databricks Certified Data Engineer Professional

Good to Have

- Azure, AWS, or GCP
- Structured Streaming
- Workspace AI Agent
- Data Governance and Security
- DevOps Practices

Key Responsibilities

- Design, build, and maintain scalable data pipelines on the Databricks platform.
- Develop end-to-end data workflows from ingestion to transformation and consumption.




- Refactor legacy code and ETL pipelines to PySpark and modern ELT patterns.
- Optimize Spark jobs, clusters, and Delta Lake tables for performance.
- Implement data quality checks, monitoring, and error handling.
- Manage Databricks Jobs and workflow orchestration.
- Follow software engineering best practices, including version control, testing, and CI/CD.
- Troubleshoot and resolve production data pipeline issues.
- Collaborate with architects, analysts, infrastructure, application, and cyber teams.
- Mentor junior engineers and maintain technical documentation.

Required Certification (Mandatory)

- Databricks Certified Data Engineer Associate OR
- Databricks Certified Data Engineer Professional

Preferred Certifications

- Databricks Certified Associate Developer for Apache Spark
- Azure Data Engineer Associate
- AWS Certified Data Analytics
- Google Cloud Skilled Data Engineer

📌 Lead Data Engineer (Pune)
🏢 Experis
📍 Pune