​​​ ​​​​​​Role Summary

The Data Engineering Support Associate will be responsible for supporting, monitoring, troubleshooting, and enhancing data engineering pipelines built on Azure Data Factory (ADF), Azure Databricks, Python, and Azure Data Lake. The role involves ensuring smooth operation of data ingestion, transformation, validation, and reporting processes, along with supporting Data Platform, Master Data Management (MDM), Data Governance, and Analytics initiatives.

The platform includes Databricks-based data processing, data quality monitoring, lineage, governance, and Lakehouse capabilities.

Key Responsibilities

Production Support

- Monitor daily batch and scheduled data pipelines in ADF and Databricks.
- Perform incident analysis and resolution for failed jobs and data issues.
- Troubleshoot pipeline, notebook, cluster, and data integration failures.
- Analyze root causes and implement preventive actions.
- Support service requests, defect fixes, and data correction activities.
- Meet agreed SLAs and support targets.

Databricks Support

- Monitor and support Databricks jobs, workflows, and clusters.
- Troubleshoot PySpark notebook failures.
- Optimize Spark jobs for better performance and reduced execution time.
- Support Unity Catalog, data governance, and access management activities.
- Maintain logging, monitoring, and auditability of data platforms.

Python Development & Support

- Debug and maintain Python/PySpark scripts.
- Develop utility scripts for validation and automation.
- Implement exception handling and performance improvements.




- Support data quality checks and reconciliation activities.
- Manage code changes and deployments.

Azure Data Factory (ADF) Support

- Monitor ADF pipelines, triggers, linked services, and datasets.
- Resolve pipeline failures and connectivity issues.
- Support incremental and full data loads.
- Implement parameterized pipelines and workflow enhancements.

Data Quality & Governance Support

- Execute and monitor data quality rules and scorecards.
- Track completeness, validity, consistency, accuracy, and timeliness metrics.
- Support data remediation activities.
- Assist business users in investigating data issues.
- Maintain governance and compliance standards.

Documentation & Reporting

- Maintain support runbooks and operational procedures.
- Prepare weekly status reports and incident summaries.
- Document recurring issues and knowledge articles.
- Participate in customer governance and support review meetings.

Required Technical Skills

Skill Area Required Expertise
Azure Databricks PySpark, Notebooks, Workflows, Clusters
Python Core Python, Data Processing, Automation
Azure Data Factory Pipelines, Triggers, Datasets, Linked Services
SQL Complex Queries, Performance Tuning
Azure Data Lake ADLS Gen2, Data Management
Data Engineering ETL/ELT Processing
Monitoring Job Monitoring & Alerting
Source Control Git / Azure DevOps
Incident Management RCA, Ticket Resolution
Data Quality Validation & Reconciliation

Preferred Skills

- Unity Catalog and Data Governance concepts.
- Lakehouse Architecture.
- Delta Tables and Delta Lake.
- Azure Synapse Analytics.

📌 Senior Engineer (databricks and ADF) (Noida)
🏢 Topcoder
📍 Noida