Experience: 6-10 years

Location: Hyderabad

Work Mode: Hybrid

Salary: Competitive

Employment Type: Full time

Role Overview

The Data Quality Engineer ensures the reliability, accuracy, performance, and scalability of data platforms and pipelines across batch and streaming workloads. The role focuses on validating end-to-end data ingestion, processing, enrichment, aggregation, and analytics on AWS, Kafka, Databricks, and Spark using SQL and Python. You will design and automate comprehensive test strategies, build reusable testing frameworks, and embed data quality checks throughout the lifecycle. The position partners closely with Data Engineering, Platform, Analytics, and DevOps teams to enforce SLAs/SLOs, secure data access, and maintain production readiness through CI/CD, monitoring, and incident response.

Required Qualifications

- 7+ years in QA, SDET, or Data Quality Engineering roles
- Expert SQL for complex data validation and reconciliation
- Proficiency in Python for automation and data testing frameworks
- Hands-on testing of ETL/ELT pipelines and data workflows
- Experience with streaming platforms (Apache Kafka or equivalent)
- Strong knowledge of AWS data services (S3, Glue, Redshift, Lambda, Athena, EMR, Step Functions)
- Databricks and Delta Lake testing experience
- Apache Spark (PySpark/Scala) validation experience
- Design and implementation of data quality checks (nulls, duplicates, schema drift, referential integrity)
- Experience with serialization formats (Avro, JSON, Protobuf) and schema evolution
- CI/CD integration for automated data tests and alerting
- Security and governance validation (IAM, permissions, encryption, lifecycle policies)
- Performance, load, and resilience testing for large-scale datasets
- Monitoring and observability using CloudWatch, Prometheus, Grafana (or similar)
- Strong debugging, analytical, and problem-solving skills

Responsibilities





- Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness
- Create and maintain automated data quality checks (nulls, duplicates, schema drift, referential integrity)
- Verify business rules and transformations using SQL-based validations
- Design, develop, and execute test strategies for Databricks-based pipelines and analytics workflows
- Establish data reconciliation and end-to-end traceability between source and downstream systems
- Test ETL/ELT pipelines on AWS services (Glue, Lambda, EMR, Step Functions)
- Validate SQL- and Python-based transformations
- Ensure quality across ingestion, enrichment, aggregation, and publishing layers
- Test reprocessing, backfills, and historical data loads
- Validate Spark-based pipelines (PySpark/Scala) in Databricks/Delta Lake
- Validate Kafka streaming pipelines for integrity, ordering, and delivery semantics (exactly-once/at-least-once)
- Test producer/consumer logic and serialization formats (Avro, JSON, Protobuf) with schema changes
- Validate topic configurations, partitions, offsets, retention, and schema evolution
- Simulate and test late arrivals, duplicates, and consumer failures
- Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB (or similar)
- Validate IAM roles, permissions, secure data access, encryption, and lifecycle policies
- Build and maintain automated data testing frameworks and reusable utilities in Python
- Create synthetic datasets and fixtures for deterministic testing
- Integrate data tests into CI/CD pipelines (pre-merge, scheduled, post-deployment) with automated alerts
- Validate pipeline performance, throughput, latency, and concurrency at scale
- Validate retry logic, error handling, idempotency, and recovery mechanisms
- Perform soak, regression, and failover testing
- Validate metrics, logs, and alerts using CloudWatch, Prometheus, Grafana (or equivalent)
- Define data SLAs/SLOs; participate in incident response, root-cause analysis, and postmortems

📌 Data Quality Engineer (Hyderabad)
🏢 Mobilution It Systems
📍 Hyderabad