The Opportunity

We are seeking an experienced AI QA Engineer with 59+ years of professional experience and deep expertise in AI testing, quality engineering, validation frameworks, and enterprise AI solution assurance. The ideal candidate should possess strong hands-on experience validating AI systems, Agentic AI workflows, RAG applications, conversational AI solutions, and enterprise-scale AI platforms.

The individual will play a key role in ensuring the quality, reliability, security, safety, and performance of AI systems while collaborating with AI engineers, architects, developers, and business stakeholders to establish robust quality assurance practices across AI and Generative AI implementations.

Important note: This role requires professional implementation experience in relevant AI technologies. Personal portfolio projects, cloned demos, certifications, or tutorial-based builds are helpful, but are not sufficient on their own.

Application guidance: Candidates should be able to clearly explain at least two relevant AI implementations, including the business use case, their direct contribution, the technical approach, and the outcome delivered.

Your key responsibilities

- Design, develop, and execute comprehensive testing strategies for AI, GenAI, Agentic AI, and Agentic RAG systems.
- Validate Large Language Model outputs for accuracy, consistency, relevance, safety, and compliance requirements.
- Develop test frameworks and automation suites for AI services, copilots, AI assistants, and enterprise AI applications.
- Design and execute evaluation frameworks for RAG systems, retrieval pipelines, vector search solutions, and knowledge-based AI applications.
- Perform functional, integration, regression, performance, reliability, and scalability testing of AI-powered applications.
- Develop automated testing frameworks using Python to validate AI workflows, APIs, models, and orchestration pipelines.
- Validate prompt engineering approaches, agent workflows, tool usage patterns,



and reasoning capabilities in Agentic AI systems.
- Execute AI model testing across conversational AI, multimodal AI, and intelligent automation solutions.
- Perform AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and security validation activities.
- Assess AI solutions for bias, hallucinations, content quality, safety risks, and responsible AI compliance requirements.
- Validate integrations between AI systems, enterprise applications, APIs, databases, and cloud platforms.
- Utilize RAG evaluation frameworks such as Ragas to measure retrieval quality, answer faithfulness, relevance, and response effectiveness.
- Develop testing dashboards, quality metrics, defect reporting processes, and continuous quality improvement practices.
- Collaborate closely with AI engineers and architects to identify quality risks and improve operational reliability.
- Contribute to quality standards, governance controls, testing accelerators, reusable frameworks, and best practices.
- Stay current with advancements in AI testing, evaluation methodologies, Agentic AI, AI security, and emerging quality assurance technologies.

Skills and Attributes Professional Experience
- 59+ years of total experience, including 3+ years of directly relevant hands-on experience in AI/ML, applied AI, or GenAI systems.
- Proven experience building production, pre-production, or enterprise pilot AI applications.
- Solid hands-on depth in at least 3 of the following areas, with meaningful ownership in at least 2:
- LLM application development.
- Retrieval pipelines.




- Agent or tool orchestration.
- Evaluation, guardrails, or quality systems.
- Model adaptation or fine-tuning.
- Cloud deployment or optimization of AI applications.
- Conversational AI.
- Multimodal AI.
- Backend engineering for AI systems.

- Strong software engineering foundation with ability to build reliable, maintainable AI services.
- Able to clearly explain direct contributions to delivered AI implementations.

Educational Background
- Bachelors/masters degree in computer science, Data Science, Engineering, or related field.

Technical Skills
- Strong understanding of Large Language Models (LLMs), Generative AI, Agentic AI, and Agentic RAG systems.
- Hands-on experience validating and testing AI-powered applications and enterprise AI platforms.
- Experience developing AI testing strategies, validation frameworks, and automated test suites.
- Strong expertise in AI Model Testing, functional testing, regression testing, integration testing, and end-to-end quality validation.
- Experience performing AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and vulnerability assessments.
- Strong understanding of prompt engineering validation, reasoning evaluation, and response quality assessment techniques.
- Experience implementing evaluation frameworks and quality measurements for Agentic AI workflows and AI agents.
- Hands-on experience with Ragas and similar evaluation frameworks for assessing retrieval quality, answer correctness, faithfulness, relevance, and grounding.
- Experience validating Agentic RAG systems, retrieval pipelines, and knowledge-based AI applications.
- Experience testing AI integrations across APIs, enterprise applications, databases, and cloud services.
- Strong understanding of AI observability, evaluation metrics, monitoring, and performance benchmarking.
- Experience working with Microsoft Azure AI Platform and Azure AI

📌 Senior GenAI Consultant (Mumbai)
🏢 EY
📍 Mumbai