We are looking for an AI QA Engineer to design, develop, and maintain a comprehensive testing framework for Generative AI and RAG (Retrieval-Augmented Generation) applications. The role involves validating AI-generated responses, evaluating retrieval quality, testing prompt behavior, and ensuring the reliability, security, and accuracy of AI-powered solutions developed by engineering teams.

Technical and Skilled Requirements

Mandatory
- Strong understanding of Generative AI concepts, including:
- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Prompt Engineering
- Embeddings
- Vector Databases
- Experience validating AI responses for:
- Accuracy
- Relevance
- Groundedness
- Consistency
- Hallucination Detection
- Citation Validation
- Hands-on experience with AI evaluation tools such as:
- Ragas
- Giskard
- Experience testing:
- Prompts
- Conversation Flows
- Knowledge Sources
- Document Retrieval Mechanisms
- AI Integrations
- Understanding of AI security testing, including:
- Prompt Injection
- Data Leakage




- Unauthorized Information Disclosure
- Strong analytical and problem-solving skills.

Preferred
- Experience building automated testing frameworks for AI/LLM-based applications.
- Exposure to Python programming and automation scripting.
- Experience with vector databases such as Pinecone, Chroma, Weaviate, or FAISS.
- Understanding of AI model evaluation metrics and benchmarking methodologies.
- Experience working with cloud-based AI platforms and APIs.
- Familiarity with CI/CD pipelines and automated quality assurance practices.

Job Responsibilities
- Design and develop a scalable testing framework for validating RAG-based AI applications.
- Create test strategies, test cases, and validation approaches for AI-powered solutions.
- Evaluate AI-generated responses for accuracy, relevance, groundedness, consistency, and citation correctness.
- Validate document retrieval quality and knowledge-source effectiveness.
- Test prompts, con

📌 AI QA Engineer (Pune)
🏢 Cybage
📍 Pune