AI Test Lead

Experience: 7+ Years

Location: Remote

Position Summary

We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands-on exposure to AI/ML, Generative AI, LLM, and RAG-based applications .

The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.

Key Responsibilities

- Lead end-to-end testing activities for AI/ML and Generative AI applications.
- Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
- Design and implement automated testing frameworks for web, API, and AI-based applications.
- Test LLM, RAG, prompt-based applications, chatbots, and AI agents .
- Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
- Develop test datasets and evaluation methodologies for AI/GenAI applications.
- Perform API, functional, regression, integration, and end-to-end testing.
- Collaborate with developers, data scientists, product owners, and business stakeholders.
- Identify and troubleshoot defects and quality issues across application and AI components.
- Integrate automated tests into CI/CD pipelines .
- Mentor QA engineers and provide technical leadership to the testing team.
- Establish quality metrics, reporting, and continuous improvement practices.
- Ensure testing standards and best practices are followed across projects.

Required Skills





- 7+ years of experience in Software Testing / QA / Test Automation .
- Hands-on experience in AI/GenAI/LLM testing .
- Strong experience with Selenium, Playwright, Cypress , or similar automation tools.
- Strong programming skills in Python or Java .
- Experience in API testing using REST, Postman, Rest Assured, or similar tools.
- Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications .
- Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality .
- Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals , or equivalent.
- Good understanding of SQL and database testing.
- Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
- Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.

Valuable to Have

- Experience testing LangChain / LangGraph applications.
- Knowledge of vector databases and RAG architectures.
- Exposure to cloud platforms such as AWS, Azure, or GCP .
- Knowledge of AI security testing and responsible AI principles.
- Experience in performance and security testing.
- Previous experience leading or mentoring QA teams.

Education Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Candidate Profile The ideal candidate should have strong test leadership and automation experience along with practical exposure to AI/GenAI testing . The ability to work closely with engineering, data science, and product teams and translate AI quality requirements into effective testing strategies is essential.

📌 AI TEST LEAD (Bengaluru)
🏢 Vaisesika
📍 Bengaluru