:

Key Responsibilities :

- Develop comprehensive test strategies, test plans, and test cases for AWS Glue inbound and outbound file processing data pipelines.
- Test data ingestion from sources such as :
- Amazon S3
- CSV, Excel, XML, fixed-length, variable-length, and delimited files
- Validate AWS Glue inbound and outbound file processing jobs developed using Python and PySpark.
- Validate Glue Workflows, triggers, job dependencies, schedules, retries, and failure handling.
- Test EventBridge-based scheduling and event-driven pipeline execution.
- Test StepFunction / Glue Workflows based file processing orchestration.
- Create comprehensive test cases for file and field level data validation/data quality checks for each of the file processing interface.
- Validate data transformations, joins, aggregations, filtering, sorting, lookups, enrichment, and business rules.
- Perform source-to-target data validation and reconciliation.
- Validate record counts, control totals, financial totals, hash totals, file counts, and data completeness.
- Test full-load, incremental-load, delta-load, and change-data-capture processes.
- Verify data types, field lengths, precision, scale, formats, null handling, default values, and mandatory fields.
- Test duplicate detection, referential integrity, orphan records, invalid records, and rejected-record processing.
- Test pipeline/interface restartability, rerun capability, idempotency, checkpointing, and recovery from failures.
- Validate error handling, exception management, dead-letter processing, and alert notifications.
- Test data movement between S3, Oracle, RDS, and other target systems.
- Validate CloudWatch logs, metrics, alarms, execution status, and operational audit information.
- Perform performance, volume, scalability, and batch-window testing for high-volume data pipelines.




- Identify data quality issues and work with developers to determine root causes.
- Create SQL queries and validation scripts for data comparison and reconciliation.
- Develop reusable test automation frameworks for data validation and regression testing.
- Validate security controls, including IAM permissions, encryption, access restrictions, and sensitive-data handling.
- Support system integration testing, regression testing, user acceptance testing, and production validation.
- Participate in Agile ceremonies, backlog refinement, defect triage, sprint planning, and release activities.
- Maintain test evidence, execution reports, defect reports, traceability matrices, and quality dashboards.

Required Technical Skills :

AWS Services :

- Familiarity with :
- AWS Glue Jobs
- AWS Glue Workflows
- Amazon S3
- Amazon CloudWatch
- AWS IAM
- Amazon EventBridge
- AWS Lambda
- Amazon SNS
- Amazon SQS
- AWS Secrets Manager
- AWS Step Functions
- AWS KMS
- Ability to review AWS Console configurations, job parameters, logs, metrics, and execution history.

SQL and Database Skills :

- Advanced SQL skills, including :
- Joins
- Subqueries
- Common table expressions
- Window functions
- Aggregations
- Data comparison queries
- Duplicate and exception identification
- Experience with Oracle, or similar databases.
- Ability to validate large datasets efficiently using SQL.

Programming and Automation :

- Working knowledge test automation and data validation.
- Experience with AWS Glue ETL/Pipeline test automation.
- Experience developing automated checks for :
- File availability
- File naming and file counts
- Schema validation
- Record counts
- Data completeness
- Data accuracy
- Duplicate records
- Control totals
- Source-to-target reconciliation
- Familiarity with pytest, unittest, Robot Framework, or similar testing frameworks

📌 AWS Glue Data Pipeline Tester (India)
🏢 Lancesoft India Private
📍 India