Designation:
Lead Data Engineer (Databricks & PySpark)

Experience:
8to 14 years

Location:
Mumbai

Work Mode:
Hybrid

About the Role:

We are seeking a highly skilled and experienced Lead Data Engineer to design, build, and operate our next-generation data platform. In this role, you will champion Legacy Code Modernization efforts, migrating traditional ETL processes into contemporary, scalable ELT patterns on Databricks. As a technical leader, you will manage a cross-functional team, oversee end-to-end delivery, and collaborate with Infrastructure, Applications, and Cyber Security teams to drive data engineering excellence across the organization.

Key Responsibilities-

Data Pipeline Development & Operations

Design and Build:
Develop reliable, scalable end-to-end data workflows from ingestion through transformation to consumption on the Databricks platform.

Operations & Monitoring
: Implement robust error handling, alerting mechanisms, and monitoring to ensure pipeline performance and uptime.

Performance Tuning:




Optimize Spark job design and cluster configurations to maximize throughput and minimize cloud infrastructure costs.

Orchestration:
Manage complex multi-stage data workflows using Databricks Jobs and modern orchestration patterns.

Legacy Code Modernization

Code Refactoring:
Assess existing, traditional codebases and refactor legacy ETL workflows into highly efficient PySpark pipelines.

ELT Migration
: Transition legacy data structures to modern ELT lakehouse patterns on Databricks.

Risk Mitigation:
Maintain backward compatibility and data integrity during migrations, creating clear playbooks to minimize business disruption.

Data Engineering Excellence & Leadership

Governance & Quality:
Implement data quality verification frameworks and validation checks to protect data integrity.

Delta Lake Architecture:
Design and optimize Delta Lake tables utilizing advanced storage features, ACID transactions, and schema evolution.

Team Leadership:
Manage, mentor

📌 Lead Data Engineer (New Delhi)
🏢 Digitalcube Group
📍 New Delhi