Job Summary

We are looking for an experienced ETL Automation Test Engineer (SDET) with strong hands-on expertise in Java, Apache Spark, Cucumber BDD, SQL, and Google Cloud Platform (GCP).

The candidate will be responsible for designing and developing automated testing frameworks to validate large-scale data pipelines, ETL transformations, and cloud-based data processing workflows.

Key Responsibilities

- Design, develop, and maintain ETL test automation frameworks using Java, Cucumber BDD, and JUnit/TestNG.
- Develop automated test scenarios for Apache Spark-based ETL pipelines running on GCP.
- Validate data ingestion, transformation, processing, and loading across multiple data sources.
- Perform source-to-target reconciliation and data integrity validation.
- Write complex SQL queries to validate transformations, joins, aggregations, and business rules.
- Automate data validation for Google BigQuery, Cloud Storage (GCS), and Dataproc.
- Test batch processing, incremental loads, and data synchronization workflows.
- Validate data quality, schema compatibility, duplicate handling, null values, and error scenarios.
- Integrate automated test suites into CI/CD pipelines.
- Analyze Spark job failures, data discrepancies, and GCP processing errors.
- Collaborate with data engineering teams to ensure reliable and scalable data processing.
- Utilize AI-assisted development tools to improve automation productivity and debugging.

Required Technical Skills

Technology

Expected Expertise

Java

Core Java, collections, streams, OOP, exception handling

Apache Spark

Spark SQL, DataFrames, transformations, distributed processing

Cucumber BDD

Gherkin, feature files, step definitions, hooks





ETL Testing

Source-to-target validation, reconciliation, transformation testing

GCP

BigQuery, Cloud Storage, Dataproc

SQL

Complex joins, aggregations, analytical queries

Test Frameworks

JUnit/TestNG, Maven

Data Formats

JSON, CSV, Parquet, Avro

CI/CD

Jenkins, GitHub Actions, or Cloud Build

Version Control

Git

GCP-Specific Testing Responsibilities

- BigQuery Validation: Automate data reconciliation, row-count comparisons, schema validation, and transformation checks.
- Cloud Storage Testing: Validate file ingestion, file formats, partitioning, and completeness of incoming data.
- Dataproc / Spark Testing: Validate Spark jobs, transformations, job execution results, and failure handling.
- Data Pipeline Testing: Verify end-to-end data movement between GCS, Spark, and BigQuery.
- Incremental Load Testing: Validate watermark logic, duplicate prevention, and correct processing of changed records.
- Performance Testing: Validate processing efficiency and identify bottlenecks in large-volume datasets.

Good to Have

- Experience with Google Cloud Dataflow and Apache Beam.
- Familiarity with Cloud Composer (Apache Airflow).
- Knowledge of Pub/Sub and streaming data pipelines.
- Experience with PySpark.
- Familiarity with Terraform and infrastructure automation.




- Knowledge of Master Data Management (MDM) and data governance.
- Experience using AI-assisted coding tools such as GitHub Copilot or Cursor.

Qualifications

- 4–8 years of experience in software testing, ETL testing, or automation engineering.
- Strong hands-on Java programming experience.
- Practical experience testing Spark-based data pipelines.
- Experience building Cucumber BDD automation frameworks.
- Strong SQL and data reconciliation skills.
- Hands-on experience with GCP data services, particularly BigQuery and Cloud Storage.

Expected Deliverables

- Scalable Java and Cucumber-based ETL test automation framework.
- Automated validation of Spark pipelines deployed on GCP.
- Source-to-target data reconciliation reports.
- Automated BigQuery and GCS data quality checks.
- CI/CD-integrated ETL regression test suites.
- Root cause analysis and reliable defect reporting.

Mandatory Skills for Screening

Skill

Priority

Java automation development

Mandatory

Apache Spark / Spark SQL

Mandatory

Cucumber BDD

Mandatory

ETL testing and data reconciliation

Mandatory

GCP – BigQuery and GCS

Mandatory

Advanced SQL

Mandatory

GCP Dataproc

Preferred

PySpark / Dataflow / Composer

Good to have

Ideal Candidate: A hands-on Java SDET with strong ETL testing expertise, capable of building Cucumber-based automation frameworks, validating Spark transformations, and testing end-to-end data pipelines on Google Cloud Platform.

Pay: ₹1,542,000.00 - ₹2,070,000.00 per year

Perks:

- Paid time off
- Provident Fund

Work Location: Hybrid remote in Chennai, Tamil Nadu

📌 ETL Automation Tester / Software Development Engineer in Test (SDET) (Chennai)
🏢 Predfut Technology Private
📍 Chennai