Overview

Mission

Own release readiness for agents whose outputs go into client-facing RFPs, data grids and pitch decks. Test that answers are grounded and correctly cited, that users never see content they are not entitled to, that multi-agent handoffs behave, and that generated files are correct. Then turn this into automated gates that every agent must pass.

Responsibilities

Key Responsibilities
Define quality standards and release gates per agent (groundedness, citation accuracy, hallucination, task completion, latency, cost) aligned to the PRD's decision-grade vs valuable-enough split.
Build golden datasets with Global BD SMEs from real RFPs, grids and prior submissions; run UAT cycles with named business testers.
Test citations by checking every claim links to a real, permission-appropriate source, and that gaps and contradictions are flagged rather than invented.
Test permissions using personas with different access to prove security trimming holds across retrieval and live tool calls.
Test agent behaviour including tool selection,



execution correctness, failure handling and multi-agent handoffs / downstream triggers.
Test document outputs for grid cell accuracy, template fidelity, and tracked-change style proofing suggestions rather than silent edits.
Run adversarial tests for prompt injection (including via uploaded RFP files), data leakage and safety boundaries.
Automate regression for model, prompt, embedding and retrieval changes inside the CI/CD pipeline.

Qualifications

Experience & Working Style
5–8 years in QA / test engineering, with 2+ years testing LLM, RAG or agent systems.
Has built an automated LLM evaluation suite that gated real releases.
Can sit with business reviewers, capture why an output is wrong and turn it into a repeatable test.

Not required (don't screen out for these): Traditional manual-only QA backgrounds are not sufficient. Deep ML modelling knowledge is not required.

📌 Ai Qa Engineer Maharashtra
🏢 Publicis Groupe Holdings
📍 Maharashtra