The role focuses on ensuring reliable data movement, automated validation, data quality, and operational visibility across end-to-end data pipelines.
Responsibilities
- Perform end-to-end data pipeline validation covering source-to-target data flow and transformation accuracy.
- Design and maintain automated E2E test frameworks for data pipelines and ETL/ELT processes.
- Implement data quality checks for completeness, accuracy, consistency, uniqueness, validity, and timeliness.
- Validate business rules, reconciliation results, schema changes, and data transformations across pipeline layers.
- Automate regression and data validation testing and integrate tests into CI/CD pipelines.
- Perform observability validation including pipeline/job health, data freshness, volume, schema anomalies, and monitoring dashboards.
- Validate monitoring and logging using tools such as Amazon CloudWatch where applicable.
- Investigate data discrepancies and pipeline failures; perform root cause analysis (RCA) and coordinate defect resolution with Data Engineering teams.
- Document test scenarios, test results, defects, data quality metrics, and validation evidence.
- Work closely with Data Engineers, Data Quality Governance, and Platform teams to ensure reliable and trustworthy data pipelines.
Key Skills
- Data Pipeline Testing
- E2E Testing
- Test Automation
- Data Quality
- SQL
- ETL/ELT Validation
- Data Reconciliation
- Data Observability
- Amazon CloudWatch
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.