Gen AI Quality Assurance

Summary

We are seeking an experienced Quality Engineer with 7 to 12 years of background to lead the validation of advanced Generative AI applications. This role focuses on ensuring the reliability and accuracy of diverse AI agents, including chatbots, Retrieval-Augmented Generation (RAG) systems, and various summarization tools. The successful candidate will architect and deploy robust automated testing frameworks utilizing tools like pytest, DeepEval, and RAGAS to assess agent performance. Proficiency in the Arize platform is highly desirable. The position involves rigorous evaluation of AI interactions via APIs, employing both standard automation protocols and cutting-edge LLM-as-a-Judge methodologies to measure critical quality indicators such as hallucination rates, faithfulness, contextual precision, and bias.

Location: Bangalore/Pune

Responsibilities
Execute comprehensive testing strategies for a wide range of AI agents, including conversational chatbots, RAG-driven systems, document summarizers, and call note analyzers.




Validate AI-generated outputs against key performance standards, ensuring accuracy, relevance, consistency, and completeness while identifying issues like hallucinations or bias.
Architect and maintain automated evaluation pipelines leveraging the DeepEval and RAGAS frameworks to create reusable test suites for RAG performance, summarization fidelity, and groundedness.
Develop and refine LLM-as-a-Judge evaluation protocols, establishing transparent rubrics and custom scoring mechanisms to objectively assess AI responses.
Engineer scalable automation scripts using pytest to validate API interactions, test prompt-response dynamics, and manage regression testing for AI models.
Integrate automated test suites with CI/CD pipelines, ensuring seamless execution in both local and production settings.
Capture and analyze detailed quality metrics, including toxicity levels, answer correctness, and overall response

📌 Hiring Gen Ai Quality Assurance Bengaluru
🏢 2COMs
📍 Bengaluru