15 Nov
|
Dataart Technologies
|
Hyderabad
15 Nov
Dataart Technologies
Hyderabad
Responsibilities
- Own data quality for data moving from source systems into analytical datasets, from test design through execution, investigation, and reporting.
- Design, implement, and execute data quality tests, and publish quality metrics and dashboards that producing and consuming groups can use.
- Validate pipelines end to end, covering movement, transformation, completeness, history, freshness, and compliance.
- Write reconciliation, drift, and quality SQL on Snowflake layered models (raw, history, silver, and gold), including slowly changing dimensions and merge behavior.
- Validate PostgreSQL sources with consistent snapshot counts and checks that logical replication kept the expected data.
- Use logical and physical data models and data contracts to assess downstream impact and set quality coverage.
- Automate data comparison and reconciliation in Python with pytest, and keep the checks in GitHub.
- Trigger pipeline runs in Airflow and read task results into the quality verdict.
- Investigate quality issues and support remediation with the groups that produce and consume the data.
- Design failover, outage, and volume tests and judge the evidence, including recovery time and recovery point. The platform team executes destructive and chaos tests.
- Run read-only checks in pre-production and production.
- Integrate quality checks with CI/CD on GitHub Actions or an equivalent tool.
- Use AWS and Grafana to observe pipeline health, including CloudWatch metrics, secrets metadata, and container platform state.
- Check classification, masking, and retention against the stated data rules.
- Track defects to closure in Jira and keep test cases and results in TestRail.
- Provide quality evidence, risks, and readiness input for releases.
- Use an AI coding assistant to analyze data and author SQL, Python, and quality checks, and review generated output before it is used.
Requirements
- Senior-level data quality engineering experience in ETL testing and data validation.
- Robust SQL and Snowflake data-modelling experience, including layered raw, history, silver, and gold models, reconciliation and drift queries, and slowly changing dimensions and merge behavior.
- Experience with logical and physical data models and with assessing the downstream impact of schema and data-contract changes.
- Experience validating PostgreSQL sources, including consistent snapshot counts and logical replication.
- Hands-on Python automation for data comparison and reconciliation, using pytest or an equivalent framework.
- Experience running and reading orchestrated pipelines in Airflow or an equivalent scheduler.
- Experience validating data movement, transformation, completeness, history, freshness, and compliance from source systems into analytical datasets.
- Experience diagnosing data quality issues and supporting remediation.
- Experience integrating data quality checks with CI/CD using GitHub Actions or equivalent tools.
- Experience with test management and defect tracking in TestRail and Jira.
- Working knowledge of AWS and Grafana for pipeline observation, including CloudWatch, secrets management, and container platforms.
- Experience designing resilience tests and judging evidence for failover, outage, volume, and recovery objectives.
- Working knowledge of data classification, masking, and retention checks.
- Experience producing data quality metrics, reports, and dashboards.
- Experience with risk-based testing in Agile or Kanban teams.
- Hands-on use of AI coding assistants (Cursor, GitHub Copilot, Claude Code, Codex, or equivalent) to author SQL, Python, and quality checks, with human review of generated output.
- Ability to communicate clearly with engineering and business stakeholders.
- Experience providing quality evidence, risks, and readiness input for releases.
Nice to have
- Experience observing Snowflake Openflow or NiFi connector state well enough to recognize a stalled load.
- Experience with custom cross-table data quality checks that publish results for dashboarding.
- Familiarity with GenAI, LLMs, and agentic workflows for maintaining data quality checks.
- Familiarity with RAG and MCP concepts for QE tooling and for generating test data.
- Knowledge of AI-driven test optimization and coverage analytics (SeaLights or equivalent).
- Experience with service virtualization using WireMock or an equivalent tool, so checks can run when a dependency is unavailable.
- Experience specifying and reviewing dbt tests. Developers author those tests.
📌 Senior Data QE Engineer (Hyderabad)
🏢 Dataart Technologies
📍 Hyderabad