Role Overview

We are seeking an experienced AI Test Lead to lead end-to-end testing and quality assurance of Generative AI applications. This role bridges AI quality engineering, test automation, and customer engagement. You will collaborate with QA Managers, AI/ML Engineers, Prompt Engineers, Full Stack Developers, and DevOps Engineers to validate production-ready, scalable AI applications against industry AI testing standards, while serving as the primary testing point of contact for customers.

Key Responsibilities

- AI Testing Strategy & Delivery Management
- Lead full lifecycle of AI testing activities: test strategy, planning, design, execution, defect management, and sign-off for GenAI applications
- Define and implement test strategies for LLM-powered applications including RAG systems, conversational AI, and AI agents
- Plan and manage functional, non-functional, and AI-specific testing: model evaluation, prompt regression, bias/fairness, robustness, hallucination detection, and explainability validation
- Establish test estimation, sprint-level test planning, milestones, and delivery timelines aligned with agile development cycles
- Track and report test coverage, quality metrics, and release readiness to QA leadership and stakeholders

Customer Engagement & Test Management

- Act as the primary testing point of contact for customers; lead requirement discussions, status reviews, and defect triage calls
- Translate customer business requirements and acceptance criteria into comprehensive AI test plans and test scenarios
- Manage day-to-day testing activities of the QA team: task allocation, progress monitoring, risk identification, and escalation management
- Present test results, quality dashboards,



and risk assessments to customer stakeholders in business-focused language
- Drive UAT support, release sign-off, and post-release quality monitoring in coordination with customer teams

AI Testing Standards & Quality Governance

- Implement AI testing practices aligned with industry standards: ISO/IEC 42001 (AI management systems)
- Apply responsible AI and risk frameworks such as NIST AI RMF and EU AI Act requirements to test planning and evidence collection
- Define quality gates, entry/exit criteria, and audit-ready test documentation for AI systems
- Establish evaluation benchmarks and metrics for LLM outputs: accuracy, relevance, groundedness, toxicity, and consistency

Test Automation & Tooling

- Design and build test automation frameworks primarily using Python (Pytest, Selenium, Playwright, Requests)
- Automate AI/LLM evaluation pipelines using frameworks such as DeepEval, Ragas, LangSmith, or promptfoo
- Integrate automated test suites into CI/CD pipelines (GitHub Actions, Jenkins) with reporting and alerting
- Drive API, integration, and performance testing for AI services and microservices
- Mentor QA engineers on Python scripting, automation best practices, and AI testing techniques

Required Skills & Qualifications

Experience

- 8+ years of overall experience in software testing and quality assurance
- 12 years of hands-on experience in AI/ML and GenAI testing activities




- Proven experience interacting with customers and managing end-to-end testing activities and QA teams
- Track record of testing production-grade AI applications through complete release cycles

Technical Knowledge

- Strong understanding of LLMs (GPT, Claude, Llama), prompt engineering, RAG architectures, and AI agents from a testing perspective
- Hands-on expertise in AI-specific testing: model evaluation, bias/fairness testing, adversarial/robustness testing, hallucination detection, and prompt regression testing
- Solid automation skills with Python scripting; experience with Pytest, Selenium/Playwright, and API testing tools (Postman, Requests)
- Familiarity with LLM evaluation tools and frameworks: DeepEval, Ragas, LangSmith, Hugging Face evaluate, or equivalent
- Working knowledge of cloud platforms (AWS, Azure, GCP), CI/CD pipelines, and test management tools (Jira, Zephyr, TestRail)

AI Testing Standards

- Knowledge of AI testing and quality standards: ISO/IEC 42001, and ISTQB CT-AI syllabus
- Understanding of responsible AI frameworks and regulations: NIST AI RMF, EU AI Act, and data privacy considerations in AI testing

Education

- Bachelor's or Master's degree in Computer Science, Engineering, AI/ML, Data Science, or related field
- ISTQB Certified Tester – AI Testing (CT-AI) or equivalent AI/QA certifications are a plus

What This Role Offers

- Lead cutting-edge GenAI testing programs with high visibility to customers and senior leadership
- Ownership of AI quality strategy from test planning through production sign-off
- Work with talented AI architects, engineers, and domain experts
- Continuous learning and growth in the rapidly evolving AI testing and quality engineering space

📌 Gen AI Test Lead (Hyderabad)
🏢 Qualizeal India
📍 Hyderabad