About This Role

6 to 10 years of experience
Skills: Test Automation, Selenium, Playwright, AI Testing, LLM Testing, RAG Pipeline Testing, Python, REST API Testing, RAGAS / DeepEval / LangSmith, CI/CD, BDD / Cucumber, Prompt Testing, Hallucination Detection, Shift-Left Testing, JIRA

About the Role
We are seeking a Senior Quality Engineer with 8–10 years of experience and strong AI / GenAI testing expertise to lead and elevate quality engineering practices across enterprise applications and AI-powered platforms. You will bring deep test automation skills combined with hands-on experience validating LLM outputs, RAG pipelines, GenAI applications, and ML model performance — ensuring that AI systems are accurate, reliable, fair, and safe for production use. Beyond AI testing, you will champion shift-left quality practices, design robust automation frameworks, and embed quality as a first-class concern across Agile delivery teams. This is a high-impact role at the cutting edge of quality engineering — where traditional QE rigour meets the unique and evolving challenges of AI system validation.

Key Responsibilities

AI & GenAI Quality Assurance

- Design and implement comprehensive testing strategies for GenAI applications — covering LLM output validation, RAG pipeline testing, prompt regression, hallucination detection, and responsible AI evaluation.
- Build and maintain LLM evaluation frameworks using RAGAS, TruLens, DeepEval, or LangSmith — defining and tracking metrics for answer relevance, faithfulness, context precision, and recall across RAG and generative AI systems.
- Design prompt testing frameworks — systematically validating prompt behaviour, regression testing prompt changes, and assessing LLM output consistency and quality across model versions.
- Conduct adversarial testing and red-teaming of AI systems — probing for jailbreaks, prompt injection vulnerabilities, bias, and safety filter bypasses to ensure AI outputs meet safety and compliance standards.
- Validate AI guardrails and safety filters — testing content moderation, output filtering, and responsible AI controls across GenAI applications.
- Test for model drift and performance degradation — implementing monitoring-based quality checks that detect and alert on changes in AI model behaviour over time in production.
- Develop synthetic test data generation pipelines using LLMs — producing diverse, representative, and edge-case-rich test datasets for AI and non-AI system testing.
- Collaborate with data scientists and ML engineers to validate ML pipeline quality — testing data ingestion, feature engineering, model training outputs, and inference APIs across the ML lifecycle.

Test Automation & Framework Development

- Design, build, and maintain scalable, maintainable test automation frameworks using Selenium, Playwright, or equivalent — supporting functional, regression, and end-to-end testing across web and API layers.
- Develop API test automation suites using RestAssured, Karate, or Postman — covering REST, GraphQL,



and gRPC service contracts with comprehensive positive, negative, and boundary test coverage.
- Implement AI-powered test automation — leveraging tools such as Testim, Mabl, or Applitools for self-healing, visual AI, and intelligent test case generation and maintenance.
- Build and maintain BDD test frameworks using Cucumber and Gherkin — collaborating with product owners and business analysts to define clear, business-readable acceptance criteria and test scenarios.
- Design and execute performance and load testing strategies using JMeter, Gatling, k6, or Locust — validating scalability, throughput, and latency under realistic and peak load conditions.

Shift-Left & CI/CD Quality Integration

- Embed quality gates and automated test suites into CI/CD pipelines — ensuring every code change is validated through fast, reliable, and comprehensive automated checks before reaching production.
- Champion shift-left testing practices — engaging with development, architecture, and product teams early in the delivery lifecycle to define testability requirements and prevent defects at source.
- Implement contract testing using Pact or Spring Cloud Contract — validating microservices API contracts and preventing integration failures across distributed service boundaries.
- Configure and maintain test reporting and observability — Allure, ReportPortal, or ExtentReports — providing clear, actionable quality metrics and trend analysis for engineering and stakeholder audiences.
- Support shift-right testing practices — implementing production monitoring, synthetic monitoring, and canary testing strategies to detect and respond to quality issues in live environments.

Quality Leadership & Collaboration

- Define and own the quality engineering strategy across assigned programmes — establishing test approaches, automation roadmaps, and quality metrics aligned to programme goals and risk profiles.
- Lead test planning, estimation, and risk-based prioritisation — ensuring test coverage is comprehensive, well-targeted, and proportionate to business and technical risk.
- Conduct and facilitate test design workshops — working with development, BA, and product teams to derive thorough test scenarios from requirements, user stories, and acceptance criteria.
- Manage defect triage, root cause analysis, and quality reporting — providing transparent, evidence-based quality status updates to delivery teams and senior stakeholders.
- Contribute to and promote a Quality Engineering Centre of Excellence (CoE) — developing reusable frameworks, shared tooling, best practice guidelines,



and training resources for the broader QE community.
- Mentor and coach junior and mid-level quality engineers — conducting knowledge-sharing sessions, reviewing test code, and elevating QE capability across the team.

What We're Looking For

- 8–10 years of quality engineering experience with a strong and demonstrable track record across both traditional test automation and AI / GenAI system testing.

- Proven hands-on experience testing LLM-powered applications and RAG pipelines — including output validation, hallucination detection, prompt regression, and responsible AI evaluation.

- Strong test automation expertise — Selenium, Playwright, or equivalent — with experience designing and building scalable, maintainable automation frameworks from scratch.

- Solid API testing skills — REST, GraphQL, and gRPC — using RestAssured, Karate, Postman, or equivalent tools in production testing environments.

- Hands-on experience with LLM evaluation frameworks — RAGAS, TruLens, DeepEval, LangSmith, or equivalent — for structured AI output quality measurement.

- Advanced Python skills — for AI testing, automation framework development, test data generation, and LLM evaluation scripting.

- Strong CI/CD integration experience — embedding quality gates, automated test suites, and reporting into Jenkins, GitLab CI, GitHub Actions, or Azure DevOps pipelines.

- Experience testing cloud-native and microservices architectures on AWS, Azure, or GCP — including contract testing, distributed system validation, and container workload testing.

- Deep understanding of shift-left quality practices — engaging early in the delivery lifecycle to define testability, acceptance criteria, and quality requirements.

- Excellent communication skills — able to present quality metrics, risk assessments, and AI testing findings clearly to both technical and non-technical stakeholders.

Nice to Have

- Experience with chaos engineering and resilience testing — LitmusChaos, Gremlin, or equivalent — for validating AI system fault tolerance and recovery.
- Familiarity with multimodal AI testing — validating text, image, audio, and video AI model outputs.
- Knowledge of AI red-teaming methodologies and adversarial ML attack patterns.
- Exposure to MLOps platforms — MLflow, Kubeflow, or SageMaker — for ML pipeline quality validation and model governance testing.
- Experience with vector database validation — OpenSearch, Pinecone, FAISS — for RAG retrieval quality testing.
- Familiarity with AI observability platforms — Arize AI, WhyLabs, or Fiddler — for production AI model monitoring and quality tracking.
- ISTQB Advanced Level — Test Automation Engineer or Test Manager certification.
- ISTQB CT-AI — Certified Tester AI Testing certification.
- AWS / Azure / GCP cloud fundamentals or associate-level certification.
- Experience in Financial Services, HealthTech, or other regulated industry environments where AI governance and explainability are critical.

📌 Senior Quality Engineer (AI-Augmented Testing) (India)
🏢 Lister Digital
📍 India