Job Summary

Were looking for a Senior SDET to own end-to-end quality for assigned product features across web apps and backend APIs, with a strong focus on evaluating AI/LLM-powered workflows. Youll build CI-gated automation and evaluation harnesses - covering UI, API, and non-deterministic AI agent behavior - set testing standards for the team, and drive a shift-left quality culture with engineering and product teams.

This position is open only to candidates currently based in Pune.

Responsibilities

- Design and maintain Python-based test automation frameworks for UI (Playwright) and REST APIs - reusable, scalable, and maintainable across the teams test suites

- Build automated evaluation suites for AI/LLM-powered features - tracking output correctness, hallucination rate, and regression across model/prompt versions - using tools such as Langfuse, Ragas, or equivalent

- Collaborate with engineering leads to define and integrate CI/CD quality checks in GitHub Actions - wiring test suites and evaluation runs into the pipeline and agreeing on pre-merge gates that block on failed tests, evaluation-score thresholds, or schema-compatibility issues

- Set testing standards and best practices for automation and AI evaluation, and mentor other SDETs on the team

- Contribute to acceptance criteria and flag quality risks early, working with cross-timezone engineering and product stakeholders

- Execute release verification for assigned features - feature-flag checks, smoke tests, and rollback validation

Qualifications

- 5+ years as an SDET with end-to-end quality ownership across web apps, backend APIs, and release processes





- Solid Python skills for test automation, API testing, and Pytest; hands-on experience with Playwright

- Experience evaluating AI/LLM-powered features - familiarity with metrics such as hallucination rate, output correctness, or LLM-as-judge approaches, and hands-on experience with at least one evaluation/tracing tool (e.g., Langfuse, Ragas)

- Experience integrating quality checks into CI/CD pipelines in GitHub Actions (or equivalent) - writing and maintaining pre-merge checks, test/quality gates, and pipeline configuration

- Track record of setting testing standards or mentoring other engineers/SDETs

- Strong communication skills; comfortable working across time zones

- Bachelors in CS, Engineering, or equivalent experience

Nice to Have

- SQL proficiency and experience validating ETL/batch transforms in Snowflake, Athena, or equivalent; familiarity with idempotency and failure recovery

- Experience testing access control and data security in multi-tenant systems (RBAC, dataset-level permissions); exposure to compliance frameworks such as HIPAA, SOC 2, or HITRUST

- Working knowledge of AWS (S3, ECS, CloudWatch) and tracing/logging tools

- Experience conducting performance, load, or stress testing (k6, JMeter)

- Familiarity with healthcare data infrastructure, the OMOP Common Data Model (CDM), or clinical vocabularies

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Sr SDET (Pune)
🏢 Verantos
📍 Pune