We are looking for an AI QA Engineer to design, develop, and maintain a comprehensive testing framework for Generative AI and RAG (Retrieval-Augmented Generation) applications. The role involves validating AI-generated responses, evaluating retrieval quality, testing prompt behavior, and ensuring the reliability, security, and accuracy of AI-powered solutions developed by engineering teams.

Technical and Qualified Requirements

Mandatory
Robust understanding of Generative AI concepts, including:
Large Language Models (LLMs)
Retrieval-Augmented Generation (RAG)
Prompt Engineering
Embeddings
Vector Databases
Experience validating AI responses for:
Accuracy
Relevance
Groundedness
Consistency
Hallucination Detection
Citation Validation
Hands-on experience with AI evaluation tools such as:
Ragas
Giskard
Experience testing:
Prompts
Conversation Flows
Knowledge Sources
Document Retrieval Mechanisms
AI Integrations
Understanding of AI security testing, including:
Prompt Injection
Data Leakage




Unauthorized Information Disclosure
Solid analytical and problem-solving skills.

Preferred
Experience building automated testing frameworks for AI/LLM-based applications.
Exposure to Python programming and automation scripting.
Experience with vector databases such as Pinecone, Chroma, Weaviate, or FAISS.
Understanding of AI model evaluation metrics and benchmarking methodologies.
Experience working with cloud-based AI platforms and APIs.
Familiarity with CI/CD pipelines and automated quality assurance practices.

Job Responsibilities
Design and develop a scalable testing framework for validating RAG-based AI applications.
Create test strategies, test cases, and validation approaches for AI-powered solutions.
Evaluate AI-generated responses for accuracy, relevance, groundedness, consistency, and citation correctness.
Validate document retrieval quality and knowledge-source effectiveness.
Test prompts, con

📌 Ai Qa Engineer Pune
🏢 Cybage
📍 Pune