Role Summary

As the AI Quality Engineer, your primary responsibility is to ensure that AI-driven and agentic systems behave correctly, reliably, safely, and in alignment with defined business intent.

Rather than focusing on manual test execution or traditional QA cycles, you will design and operate continuous evaluation systems that assess the quality, accuracy, performance, and business effectiveness of AI-driven workflows and outputs.

You will define how system behaviour is measured, how failures are detected, and how quality is continuously validated and improved through automated evaluation frameworks, feedback loops, and real-world operational signals.

You will work closely with the AI Product Owner and AI Engineers to ensure that delivery systems are not only functional, but reliable, measurable, and continuously improving in production use.

Detailed Responsibilities

AI Evaluation & Quality Engineering

- Design and operate evaluation frameworks for AI-driven workflows, outputs, and agent behaviours.
- Define measurable standards for correctness, usefulness, safety, and business alignment of AI outputs.
- Build automated and semi-automated evaluation systems including scoring models, confidence thresholds, and regression testing for AI behaviour.
- Ensure AI systems consistently align with business intent and expected outcomes.
- Continuously refine evaluation methods based on observed system performance and failure patterns.

AI Testing, Validation & Reliability
- Design AI-native testing strategies, including synthetic test generation and evaluation-driven approaches.
- Develop evaluation harnesses for AI-generated code, decisions, workflows, and RAG-based systems.
- Identify edge cases, failure modes, and adversarial scenarios to test system robustness.
- Detect and manage behavioural drift, hallucinations, and performance degradation over time.
- Ensure reliable system behaviour across varying inputs, contexts, and usage patterns.

Observability & Continuous Feedback
- Implement observability for AI systems including logging, tracing, performance metrics,



and evaluation outputs.
- Build dashboards and reporting to monitor system quality, reliability, cost, and latency.
- Establish feedback loops between production behaviour and ongoing system improvements.
- Enable rapid detection and resolution of AI system issues or degradations.

AI Safety & Governance
- Ensure AI systems operate within defined security, compliance, and governance boundaries.
- Identify and mitigate risks such as hallucinations, data leakage, and inconsistent behaviour.
- Define escalation paths for low-confidence or ambiguous outputs.
- Support appropriate human-in-the-loop controls where required.

Collaboration & Continuous Improvement
- Work closely with AI Engineers to embed evaluation into AI workflows from the outset.
- Partner with the AI Product Owner to translate business intent into measurable evaluation criteria.
- Provide continuous feedback to improve prompts, agents, and orchestration logic.
- Contribute reusable evaluation patterns and quality frameworks across the AI Engineering Pod.
- Support scaling of AI quality practices as the capability expands.

Skills & Experience

Essential

- Experience in software testing, quality engineering, or test automation within complex distributed systems.
- Strong understanding of modern software delivery lifecycles, CI/CD pipelines, and quality assurance principles.
- Experience designing automated test frameworks, validation pipelines, or system verification approaches.
- Strong analytical skills with the ability to define expected system behaviour and detect deviations or anomalies.
- Experience working with API-based systems, cloud-native architectures, and distributed systems.




- Ability to interpret system outputs (logs, traces, metrics, responses) to identify inconsistencies, errors, or behavioural issues.
- Experience applying AI-assisted tools or techniques (e.g. ChatGPT, Claude, Gemini, GitHub Copilot, or similar) to support testing, analysis, or debugging workflows.

Desirable

- Experience evaluating AI/LLM-based systems, including:
- Prompt and response testing
- Hallucination detection
- Reasoning consistency checks
- Retrieval accuracy evaluation in RAG systems
- Familiarity with AI observability and evaluation tooling (e.g. prompt logging, LLM tracing, evaluation dashboards, or similar approaches).
- Exposure to Retrieval-Augmented Generation (RAG) systems, vector databases, or semantic search systems.
- Experience designing synthetic test data, adversarial test cases, or scenario-based evaluation frameworks.
- Experience with advanced testing approaches such as performance testing, chaos engineering, or resilience testing.
- Understanding of data quality, validation techniques, and production monitoring in data-driven or AI-enabled systems.

Personal Attributes

- Strong attention to detail with a structured, analytical approach to problem solving.
- Passionate about system reliability, correctness, and continuous improvement.
- Comfortable working in fast-moving, highly agile environments where systems evolve rapidly.
- Curious and open to AI-driven and emerging engineering approaches.
- Able to clearly communicate quality risks, issues, and system behaviour to both technical and non-technical stakeholders.
- Collaborative and supportive, working effectively with engineering and product teams.
- Pragmatic, balancing delivery speed with system quality and operational risk.
- Confident in challenging assumptions and surfacing risks early to improve outcomes.

You'll also carry out other duties, within the broad scope and spirit of your role, as requested by your manager. Our business continuously evolves, so your role will too.

📌 AI Quality Engineer (Bengaluru)
🏢 Kaplan
📍 Bengaluru