Job Description:
Perform end-to-end testing of Databricks-based data pipelines and solutions.
Validate data processing using Python, Apache Spark/PySpark and Databricks.
Perform data quality, data completeness, accuracy and integrity checks.
Develop and execute automated test cases/frameworks for data pipelines.
Validate data in JSON/XML formats and ensure correct data transformation.
Perform source-to-target validation and identify data discrepancies.
Work with Databricks Unity Catalog for data governance and access-related validation.
Validate ETL/data transformation logic and business rules.
Identify, troubleshoot and report data quality issues.
Collaborate with Data Engineers and other technical teams to resolve defects.
Contribute as an Individual Contributor, owning testing activities independently.