Data Pipeline Development & Operations

Design, build, and operate scalable and reliable data pipelines on the Databricks platform
Develop end-to-end data workflows from ingestion through transformation to consumption
Implement robust error handling, monitoring, and alerting mechanisms
Ensure data pipeline reliability, performance, and maintainability
Optimize pipeline performance through efficient Spark job design and cluster configuration
Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization

Refactor legacy code and data pipelines to PySpark for improved performance and scalability
Migrate traditional ETL processes to up-to-date ELT patterns on Databricks
Assess existing codebases and identify opportunities for optimization and modernization
Ensure backward compatibility and data integrity during migration processes
Document refactoring approaches and create migration playbooks
Collaborate with stakeholders to minimize disruption during code transitions

Data Engineering Excellence





Implement data quality checks and validation frameworks
Design and maintain Delta Lake tables with appropriate optimization strategies
Develop reusable code libraries and frameworks for common data engineering tasks
Follow software engineering best practices including version control, testing, and CI/CD
Participate in code reviews and provide constructive feedback to team members
Troubleshoot and resolve data pipeline issues in production environments

Collaboration & Knowledge Sharing

Work closely with data architects, analysts, and business stakeholders
Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams
Share knowledge and best practices with Team *
Mentor junior data engineers on PySpark and Databricks technologies
Document technical solutions and maintain comprehensive documentation

Essential Technical Skills

Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and da

📌 Senior Data Engineer (Pune)
🏢 ANRGI TECH
📍 Pune