Job Position : Senior AI Quality Assurance (QA) Engineer
Location : Bengaluru
Experience : 7+ Years
Required Skills:
We are looking for an experienced Senior AI Quality Assurance (QA) Engineer to ensure the quality, reliability, and production readiness of enterprise-grade AI applications.
This role extends beyond traditional software testing and focuses on the evaluation and quality engineering of LLM applications, RAG systems, and AI agents. You will own automated testing, AI evaluations, benchmark creation, observability, prompt regression testing, and end-to-end validation of AI workflows. Working closely with AI Engineers, Product Managers, and Platform teams, you will establish measurable quality standards and ensure every release meets enterprise-grade expectations for accuracy, reliability, performance, and scalability.
Key Responsibilities:
AI Evaluation & Benchmarking
Design benchmark (golden) datasets and build automated evaluation pipelines for LLM applications. Define quality gates and continuously evaluate prompts, models, retrieval pipelines,



and agent behaviour using metrics such as hallucination rate, tool selection accuracy, execution accuracy, precision, recall, latency, and cost.
RAG & Agent Quality Validation
Validate Retrieval-Augmented Generation (RAG) pipelines and AI agent workflows by testing retrieval quality, context relevance, tool invocation, reasoning flow, memory, and end-to-end task completion. Design evaluation scenarios covering ambiguous queries, multi-turn conversations, retrieval failures, and edge cases.
Python Automation & API Testing
Develop and maintain scalable automation frameworks using Python and Pytest for unit testing, integration testing, API testing, regression testing, and end-to-end validation. Build reusable test utilities and integrate automated quality checks into CI/CD pipelines.
Frontend Automation
Develop automated UI test suites using Playwright to validate AI-powered user journeys, conversational interfaces

📌 Senior AI QA Engineer (Bengaluru)
🏢 DATAMAXIS
📍 Bengaluru