​​​ ​​​​​​Role Summary

The Data Engineering Support Associate will be responsible for supporting, monitoring, troubleshooting, and enhancing data engineering pipelines built on Azure Data Factory (ADF), Azure Databricks, Python, and Azure Data Lake. The role involves ensuring smooth operation of data ingestion, transformation, validation, and reporting processes, along with supporting Data Platform, Master Data Management (MDM), Data Governance, and Analytics initiatives.

The platform includes Databricks-based data processing, data quality monitoring, lineage, governance, and Lakehouse capabilities.

Key Responsibilities

Production Support
- Monitor daily batch and scheduled data pipelines in ADF and Databricks.
- Perform incident analysis and resolution for failed jobs and data issues.
- Troubleshoot pipeline, notebook, cluster, and data integration failures.
- Analyze root causes and implement preventive actions.
- Support service requests, defect fixes, and data correction activities.
- Meet agreed SLAs and support targets.

Databricks Support




- Monitor and support Databricks jobs, workflows, and clusters.
- Troubleshoot PySpark notebook failures.
- Optimize Spark jobs for better performance and reduced execution time.
- Support Unity Catalog, data governance, and access management activities.
- Maintain logging, monitoring, and auditability of data platforms.

Python Development & Support
- Debug and maintain Python/PySpark scripts.
- Develop utility scripts for validation and automation.
- Implement exception handling and performance improvements.
- Support data quality checks and reconciliation activities.
- Manage code changes and deployments.

Azure Data Factory (ADF) Support
- Monitor ADF pipelines, triggers, linked services, and datasets.
- Resolve pipeline failures and connectivity issues.
- Support incremental and full data loads.
- Implement parameterized pipelines and workflow enhancements.

Data Quality & Governance Support
- Execute and monitor da

📌 Senior Engineer (Noida)
🏢 Topcoder
📍 Noida