Home/Jobs/Senior Quality Engineer (AI-Augmented Testing)

Senior Quality Engineer (AI-Augmented Testing)

Lister Digital

Gurugram

6-10 years

Today

$48.2K–69.9K/yr

Full-time

Onsite

Skills Required

LLM

RAG

Gen AI

RAGAS

Prompt Testing

Hallucination Detection

AI Testing

Python

Selenium

Playwright

Test Automation

REST API Testing

DeepEval

LangSmith

TruLens

Description

Senior Quality Engineer role focused on traditional test automation and AI/GenAI quality assurance. The position emphasizes validating LLM outputs, RAG pipelines, and production-ready AI systems while embedding quality across Agile delivery.

Role: Senior Quality Engineer

Experience

- 6 to 10 years of experience
- 8-10 years of experience in quality engineering
- Proven hands-on experience testing LLM-powered applications and RAG pipelines
- Strong test automation expertise with Selenium, Playwright, or equivalent
- Solid API testing skills for REST, GraphQL, and gRPC
- Hands-on experience with LLM evaluation frameworks
- Advanced Python skills
- Strong CI/CD integration experience
- Experience testing cloud-native and microservices architectures
- Deep understanding of shift-left quality practices
- Excellent communication skills

Qualification

- ISTQB Advanced Level Test Automation Engineer or Test Manager certification
- ISTQB CT-AI Certified Tester AI Testing certification
- AWS, Azure, or GCP cloud fundamentals or associate-level certification

Responsibilities

- Design and implement testing strategies for GenAI applications
- Cover LLM output validation, RAG pipeline testing, prompt regression, hallucination detection, and responsible AI evaluation
- Build and maintain LLM evaluation frameworks and define metrics for relevance, faithfulness, precision, and recall
- Design prompt testing frameworks and validate prompt behavior across model versions
- Conduct adversarial testing and red-teaming for jailbreaks,



prompt injection, bias, and safety bypasses
- Validate AI guardrails and safety filters
- Test for model drift and performance degradation in production
- Develop synthetic test data generation pipelines using LLMs
- Collaborate with data scientists and ML engineers to validate ML pipeline quality
- Design, build, and maintain scalable test automation frameworks
- Develop API test automation suites covering service contracts and test coverage
- Implement AI-powered test automation
- Build and maintain BDD test frameworks
- Design and execute performance and load testing strategies
- Embed quality gates and automated test suites into CI/CD pipelines
- Champion shift-left testing practices
- Implement contract testing for microservices APIs
- Configure and maintain test reporting and observability
- Support shift-right testing practices
- Define and own the quality engineering strategy
- Lead test planning, estimation, and risk-based prioritisation
- Conduct and facilitate test design workshops
- Manage defect triage, root cause analysis, and quality reporting
- Contribute to and promote a Quality Engineering Centre of Excellence
- Mentor and coach junior and mid-level quality engineers

Additional Responsibilities

- Support production monitoring, synthetic monitoring, and canary testing strategies
- Develop reusable frameworks, shared tooling, best practice guidelines, and training resources
- Review test code and conduct knowledge-sharing sessions

Nice To Have





- Experience with chaos engineering and resilience testing
- Familiarity with multimodal AI testing
- Knowledge of AI red-teaming methodologies and adversarial ML attack patterns
- Exposure to MLOps platforms
- Experience with vector database validation
- Familiarity with AI observability platforms
- Experience in Financial Services, HealthTech, or other regulated industry environments

More Skills LLM Testing, RAG Pipeline Testing, CI/CD, BDD, Cucumber, Shift-Left Testing, JIRA, RestAssured, Karate, Postman, GraphQL, gRPC, Testim, Mabl, Applitools, Gherkin, JMeter, Gatling, k6, Locust, Pact, Spring Cloud Contract, Allure, ReportPortal, ExtentReports, Jenkins, GitLab CI, GitHub Actions, Azure DevOps, AWS, Azure, GCP, LitmusChaos, Gremlin, MLflow, Kubeflow, SageMaker, OpenSearch, Pinecone, FAISS, Arize AI, WhyLabs, Fiddler

Other

- High-impact role at the cutting edge of quality engineering
- Focus on enterprise applications and AI-powered platforms
- Work across Agile delivery teams
- Ensure AI systems are accurate, reliable, fair, and protected for production use
- Quality Engineering Centre of Excellence emphasis

Prepare for this role Recommended resources to build the skills for this position. Sponsored.

Top 50 RAG Interview Questions

Zenaique

Curated RAG interview questions and answers from Zenaique.

Top 10 RAG Quick-Prep Questions

Zenaique A shorter RAG interview prep set for quick review.

Top 20 LLM System Design Questions (Quick)

Zenaique A focused LLM system design interview question set.

More LLM jobs

AI Engineer - Forward Deployed

H2O.ai

India

Today

Gen AI - Lead

Iris Software

Noida

Today

Senior Agentic AI Engineer ID80140

AgileEngine

Bengaluru

Today

Consultant - AI Engineer

HCA Healthcare

Hyderabad

Today

Software AI Engineer

Tredence

Bengaluru

Today

Agentic AI Developer

EDMO

Pune

Today

📌 Senior Quality Engineer (AI-Augmented Testing) (Gurugram)
🏢 Top Gen AI Jobs
📍 Gurugram