Positions:1 Full Time

Experience

8 14 Years

Compensation Details

Variable pay: 10% performance bonus over and above fixed salary

Job Description

Data Pipeline Development Operations

Design, build, and operate scalable and reliable data pipelines on the Databricks platform

Develop end-to-end data workflows from ingestion through transformation to consumption

Implement robust error handling, monitoring, and alerting mechanisms

Ensure data pipeline reliability, performance, and maintainability

Optimize pipeline performance through efficient Spark job design and cluster configuration

Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization

Refactor legacy code and data pipelines to PySpark for improved performance and scalability

Migrate traditional ETL processes to up-to-date ELT patterns on Databricks

Assess existing codebases and identify opportunities for optimization and modernization

Ensure backward compatibility and data integrity during migration processes





Document refactoring approaches and create migration playbooks

Collaborate with stakeholders to minimize disruption during code transitions

Data Engineering Excellence

Implement data quality checks and validation frameworks

Design and maintain Delta Lake tables with appropriate optimization strategies

Develop reusable code libraries and frameworks for common data engineering tasks

Follow software engineering best practices including version control, testing, and CI/CD

Participate in code reviews and provide constructive feedback to team members

Troubleshoot and resolve data pipeline issues in production environments

Collaboration Knowledge Sharing

Work closely with data architects, analysts, and business stakeholders

Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams

Share knowledge and best practices with our Team

Mentor junior data engineers on PySpark and Databricks technologies

Document technical solut

📌 Lead Data Engineer 85103 (Pune)
🏢 Cephas Consultancy
📍 Pune