GenAI Testing skilled with experience in validating Large Language Models (LLMs), Agentic AI systems, RAG applications, and AI-powered business solutions. The candidate will drive quality assurance for GenAI implementations, ensuring accuracy, reliability, security, and business alignment of AI-generated outcomes. Mandatory Skills GenAI Testing Concepts LLM Validation & Evaluation Prompt Engineering RAG Testing Functional & Automation Testing API Testing Python AI Model Evaluation Techniques Jira / Azure DevOps Agile Delivery Model Key Responsibilities Design and execute test strategies for GenAI, RAG, Copilot, and Agentic AI applications. Validate AI responses for accuracy, relevance, consistency, bias, toxicity, and hallucinations. Develop AI-specific test scenarios, prompt libraries, and evaluation frameworks.
Create automated validation approaches for LLM outputs. Perform prompt engineering and prompt optimization activities. Test Retrieval-Augmented Generation (RAG) workflows and knowledge-grounding mechanisms. Validate AI guardrails, compliance requirements, and security controls. Collaborate with Product Owners, Business SMEs, Data Scientists, and AI Architects. Define KPIs and quality metrics for AI solution evaluation. Support GenAI adoption initiatives and recommend QA best practices. Good to Have Azure OpenAI / OpenAI LangChain / LangGraph MCP (Model Context Protocol) Agentic AI Platforms GitHub Copilot AI Security Testing Python-based Automation Frameworks