Overview

Mission

Own release readiness for agents whose outputs go into client-facing RFPs, data grids and pitch decks. Test that answers are grounded and correctly cited, that users never see content they are not entitled to, that multi-agent handoffs behave, and that generated files are correct. Then turn this into automated gates that every agent must pass.

Responsibilities

Key Responsibilities

- Define quality standards and release gates per agent (groundedness, citation accuracy, hallucination, task completion, latency, cost) aligned to the PRD's decision-grade vs positive-enough split.
- Build golden datasets with Global BD SMEs from real RFPs, grids and prior submissions; run UAT cycles with named business testers.
- Test citations by checking every claim links to a real, permission-appropriate source, and that gaps and contradictions are flagged rather than invented.
- Test permissions using personas with different access to prove security trimming holds across retrieval and live tool calls.
- Test agent behaviour including tool selection,



execution correctness, failure handling and multi-agent handoffs / downstream triggers.
- Test document outputs for grid cell accuracy, template fidelity, and tracked-change style proofing suggestions rather than silent edits.
- Run adversarial tests for prompt injection (including via uploaded RFP files), data leakage and safety boundaries.
- Automate regression for model, prompt, embedding and retrieval changes inside the CI/CD pipeline.

Qualifications

Experience & Working Style
- 5–8 years in QA / test engineering, with 2+ years testing LLM, RAG or agent systems.
- Has built an automated LLM evaluation suite that gated real releases.
- Can sit with business reviewers, capture why an output is wrong and turn it into a repeatable test.

Not required (don't screen out for these): Traditional manual-only QA backgrounds are not sufficient. Deep ML modelling knowledge is not required.

📌 AI - QA Engineer (Mumbai)
🏢 PUBLICIS GROUPE
📍 Mumbai