Skills Required

Primary Skills

sql Python pyspark

Shift Timing: 1:00 PM – 10:00 PM

Key Responsibilities

Data Pipeline Testing:

· Validate end-to-end data ingestion, transformation, and consumption pipeline across bronze, silver, and gold layers.

· Perform source-to-target data validation for structured.

· Validate business rules and data transformations.

· Ensure data consistency, completeness, uniqueness and referential integrity.

Automation Testing:

· Design and develop automated test frameworks for data validation and regression testing.

· Automate ETL and data quality testing using python, SQL, PySpark, or testing frameworks.

· Build reusable validation scripts for schema validation, reconciliation, and data drift detection.

· Integrate automated test suites into CI/CD pipelines.

Databricks and Cloud Validation:

· Validate Databricks notebooks,



workflows, Delta Tables and job orchestration.

· Test Delta Lake functionalities include ACID compliance, schema evolution and time travel.

· Validate performance and scalability of Spark-based transformations.

Required Skills:

· SQL

· Python

· PySpark

· Databricks

· Batch Pipelines

· Incremental Loads

· CDC pipelines

· Knowledge of Medallion Architecture concepts

Preferred Qualifications:

· 4-6 Years of experience in QA/Data Testing.

· 2+ years of experience in Databricks/Data Lakehouse projects.

· Experience working in Agile/Scrum delivery models.

· Certification in Databricks, Cloud or Testing tool is an advantage.

📌 QA Automation Engineer – Data Engineering (India)
🏢 Sparix Global
📍 India