Job Description

Data Pipeline Development Operations
Design, build, and operate scalable and reliable data pipelines on the Databricks platform
Develop end-to-end data workflows from ingestion through transformation to consumption
Implement robust error handling, monitoring, and alerting mechanisms
Ensure data pipeline reliability, performance, and maintainability
Optimize pipeline performance through productive Spark job design and cluster configuration
Manage and orchestrate complex data workflows using Databricks Jobs and workflows

Legacy Code Modernization
Refactor legacy code and data pipelines to PySpark for improved performance and scalability
Migrate traditional ETL processes to modern ELT patterns on Databricks
Assess existing codebases and identify opportunities for optimization and modernization
Ensure backward compatibility and data integrity during migration processes
Document refactoring approaches and create migration playbooks
Collaborate with stakeholders to minimize disruption during code transitions

Data Engineering Excellence




Implement data quality checks and validation frameworks
Design and maintain Delta Lake tables with appropriate optimization strategies
Develop reusable code libraries and frameworks for common data engineering tasks
Follow software engineering best practices including version control, testing, and CI/CD
Participate in code reviews and provide constructive feedback to team members
Troubleshoot and resolve data pipeline issues in production environments

Collaboration Knowledge Sharing
Work closely with data architects, analysts, and business stakeholders
Collaborate with Infrastructure (Infra), Applications (Apps), and Cyber teams
Share knowledge and best practices with Team NCS
Mentor junior data engineers on PySpark and Databricks technologies
Document technical solutions and maintain comprehensive documentation

Essential Technical Skills
Data Engineering: Strong foundation in data engineering principles, ETL/ELT process

📌 Lead Data Engineer (Pune)
🏢 NCS Group
📍 Pune