Data Pipeline Development & Operations

• Design, build, and operate scalable and reliable data pipelines on the Databricks platform

• Develop end-to-end data workflows from ingestion through transformation to consumption

• Implement robust error handling, monitoring, and alerting mechanisms

• Ensure data pipeline reliability, performance, and maintainability

• Optimize pipeline performance through efficient Spark job design and cluster configuration

• Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization

• Refactor legacy code and data pipelines to PySpark for improved performance and scalability

• Migrate traditional ETL processes to modern ELT patterns on Databricks

• Assess existing codebases and identify opportunities for optimization and modernization

• Ensure backward compatibility and data integrity during migration processes

• Document refactoring approaches and create migration playbooks

• Collaborate with stakeholders to minimize disruption during code transitions

Data Engineering Excellence





• Implement data quality checks and validation frameworks

• Design and maintain Delta Lake tables with appropriate optimization strategies

• Develop reusable code libraries and frameworks for common data engineering tasks

• Follow software engineering best practices including version control, testing, and CI/CD

• Participate in code reviews and provide constructive feedback to team members

• Troubleshoot and resolve data pipeline issues in production environments

Collaboration & Knowledge Sharing

• Work closely with data architects, analysts, and business stakeholders

• Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams

• Share knowledge and best practices with Team *

• Mentor junior data engineers on PySpark and Databricks technologies

• Document technical solutions and maintain comprehensive documentation

Essential Technical Skills

• Data Engineering: Solid f

📌 Senior Data Engineer (Pune)
🏢 ANRGI TECH
📍 Pune