Data Engineer – QA

Experience: 5–6 Years
Employment Type: C2C
Location: Remote
Work Timings: 11:00 AM – 9:00 PM IST
Joining: Immediate / Short Notice Preferred

Job Summary

We are looking for an experienced Data Engineer – QA with 5–6 years of experience in data testing, data validation, automation, SQL, Python, PySpark, and cloud data platforms.

The ideal candidate will be responsible for validating data pipelines, source-to-target mappings, data transformations, data quality, reconciliation, and cloud-based data workflows. The candidate should have strong hands-on experience with Azure Data Factory, Azure Synapse Analytics, SQL Server, Python, and PySpark.

Key ResponsibilitiesData QA and Validation

- Analyze business and technical requirements and define appropriate test scenarios.
- Develop test plans and test cases based on business rules and data requirements.
- Perform end-to-end data validation and reconciliation.
- Validate source-to-target mappings and data transformations.
- Perform record count, data completeness, accuracy, and consistency checks.
- Validate schemas, file layouts, column sequences, and data formats.
- Analyze invalid records, exceptions, and rejected data.
- Perform production versus staging data comparisons.
- Investigate data issues and perform Root Cause Analysis (RCA).

Automation and Data Engineering

- Develop automated data validation and testing solutions using Python.
- Design and maintain data validation and reconciliation frameworks.
- Develop SQL queries for data analysis, validation, and troubleshooting.
- Build and maintain PySpark notebooks for data processing and validation.
- Develop, test,



and validate data pipelines using Azure Data Factory (ADF).
- Create automated reporting and data-quality utilities.
- Support data pipeline monitoring and troubleshooting.

Azure and Cloud Data Platforms

- Work with Azure Data Factory and Azure Synapse Analytics pipelines and notebooks.
- Validate data stored in Azure Storage Accounts and Containers.
- Work with Azure Synapse for data processing and validation.
- Validate data stored and processed through AWS S3.
- Perform Azure-to-AWS file transfer validation.
- Work with Azure Cosmos DB where required.
- Monitor and support data jobs and workflows.

Metrics and Reporting

- Extract and validate source-system metrics.
- Validate data within SQL Server metrics databases.
- Perform Power BI dashboard and report validation.
- Reconcile data across files, databases, and reporting dashboards.
- Identify data discrepancies and coordinate with relevant teams for resolution.

Collaboration

- Work closely with Data Engineers, Developers, Business Analysts, and business stakeholders.
- Participate in Agile ceremonies and contribute to sprint planning and testing activities.
- Communicate data-quality issues, defects, and validation results clearly.
- Maintain documentation related to test scenarios, validation rules, defects, and results.

Required Skills





- 5–6 years of experience in Data Engineering QA, Data Testing, or Data Validation.
- Strong hands-on experience with Python.
- Solid SQL and SQL Server / SSMS skills.
- Hands-on experience with PySpark.
- Experience with Azure Data Factory (ADF).
- Experience with Azure Synapse Analytics.
- Strong experience in data validation and reconciliation.
- Experience with large-scale data processing and data pipelines.
- Experience working with CSV, delimited, fixed-width, and Excel files.
- Experience with Azure Storage and AWS S3.
- Strong defect investigation and Root Cause Analysis skills.
- Good understanding of data quality, data transformation, and source-to-target validation.

Good to Have

- Azure Cosmos DB
- Azure Privileged Identity Management (PIM)
- Power BI dashboard validation
- Rally
- Microsoft Copilot or other AI-assisted development tools
- Experience building automated data testing and validation frameworks
- Knowledge of cloud-based data engineering environments

Technical Skills

Python | SQL | SQL Server | SSMS | PySpark | Azure Data Factory | Azure Synapse Analytics | Azure Storage | AWS S3 | Azure Cosmos DB | Power BI | Rally | Excel | Microsoft Copilot

Candidate Profile

The ideal candidate should be a hands-on Data QA / Data Engineer with strong analytical and problem-solving skills. The candidate must be comfortable working with large datasets, identifying data discrepancies, developing automated validation solutions, and collaborating with technical and business teams.

Pay: ₹60,000.00 - ₹110,000.00 per month

Experience:

- Data Engineering QA: 5 years (Required)

Work Location: Remote

📌 Data Engineer – QA (India)
🏢 Tech Next
📍 India