We are looking for an AI QA Engineer to design, develop, and maintain a comprehensive testing framework for Generative AI and RAG (Retrieval-Augmented Generation) applications. The role involves validating AI-generated responses, evaluating retrieval quality, testing prompt behavior, and ensuring the reliability, security, and accuracy of AI-powered solutions developed by engineering teams.

Technical and Skilled Requirements

Mandatory

- Strong understanding of Generative AI concepts, including:
- Large Language Models (LLMs)
- Retrieval-Augmented Generation (RAG)
- Prompt Engineering
- Embeddings
- Vector Databases

- Experience validating AI responses for:

- Accuracy
- Relevance
- Groundedness
- Consistency
- Hallucination Detection
- Citation Validation

- Hands-on experience with AI evaluation tools such as:

- Ragas
- Giskard

- Experience testing:

- Prompts
- Conversation Flows
- Knowledge Sources
- Document Retrieval Mechanisms
- AI Integrations

- Understanding of AI security testing, including:

- Prompt Injection
- Data Leakage
- Unauthorized Information Disclosure

- Strong analytical and problem-solving skills.

Preferred

- Experience building automated testing frameworks for AI/LLM-based applications.
- Exposure to Python programming and automation scripting.
- Experience with vector databases such as Pinecone, Chroma, Weaviate, or FAISS.
- Understanding of AI model evaluation metrics and benchmarking methodologies.
- Experience working with cloud-based AI platforms and APIs.
- Familiarity with CI/CD pipelines and automated quality assurance practices.

Job Responsibilities

- Design and develop a scalable testing framework for validating RAG-based AI applications.
- Create test strategies,



test cases, and validation approaches for AI-powered solutions.
- Evaluate AI-generated responses for accuracy, relevance, groundedness, consistency, and citation correctness.
- Validate document retrieval quality and knowledge-source effectiveness.
- Test prompts, conversation flows, and end-user interactions across multiple scenarios.
- Perform security testing focused on prompt injection, jailbreak attempts, and data leakage risks.
- Utilize tools such as Ragas and Giskard to measure and improve AI system quality.
- Collaborate with AI engineers and development teams to identify gaps and improve model performance.
- Define quality metrics, reporting mechanisms, and continuous evaluation processes for AI applications.
- Drive best practices and continuous improvement in AI testing methodologies. Cybage conducts recruitment only through authorized channels and official communications sent from email addresses bearing the @cybage.com domain. Cybage does not, under any circumstances, solicit and/or accept any form of payment, fee, deposit, or monetary consideration from candidates on the pretense of interviews, employment opportunities, offer letters, onboarding formalities, salary account opening, background verification, training, or any other recruitment-related activity. Any such request should be treated as fraudulent. Applicants are advised to verify the authenticity of all recruitment-related communications through Cybage's official channels and refrain from sharing personal, financial, or confidential information with unauthorized persons. Cybage disclaims any liability for losses arising from dealings with fraudulent individuals or entities misusing its name, brand, or employee identities.

📌 AI QA Engineer / Generative AI Test Engineer (Pune)
🏢 Cybage
📍 Pune