We are seeking a Data Engineer Quality Engineer to ensure the reliability, accuracy, performance, and scalability of our data platforms and pipelines. This role focuses on validating end‑to‑end data ingestion, processing, streaming, and analytics workflows built on AWS, Kafka, SQL, Data Bricks and Python. You will work closely with Data Engineers, Platform Engineers, Analytics teams, and DevOps to embed quality into the data lifecycle—ensuring trusted data, resilient pipelines, and production‑ready systems. Key Responsibilities Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness. Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity). Verify business rules and transformations using SQL‑based validations. Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows Establish data reconciliation and end‑to‑end traceability between source and downstream systems. Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).



Validate transformations written in SQL and Python. Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers. Test reprocessing, backfills, and historical data loads. Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly‑once/at‑least‑once semantics. Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf). Validate topic configurations, partitions, offsets, retention policies, and schema changes. Simulate and test late arrivals, duplicate events, and consumer failures. Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services. Validate IAM roles, permissions, and secure data access. Verify data lifecycle policies, encryption, and storage optimizations. Build and maintain automated dat

📌 Sr. Data Engineer (Hyderabad)
🏢 ePace Technologies
📍 Hyderabad