Overview

Mission

Own release readiness for agents whose outputs go into client-facing RFPs, data grids and pitch decks. Test that answers are grounded and correctly cited, that users never see content they are not entitled to, that multi-agent handoffs behave, and that generated files are correct. Then turn this into automated gates that every agent must pass.

Responsibilities

Key Responsibilities

• Define quality standards and release gates per agent (groundedness, citation accuracy, hallucination, task completion, latency, cost) aligned to the PRD's decision-grade vs positive-enough split.

• Build golden datasets with Global BD SMEs from real RFPs, grids and prior submissions; run UAT cycles with named business testers.

• Test citations by checking every claim links to a real, permission-appropriate source, and that gaps and contradictions are flagged rather than invented.

• Test permissions using personas with different access to prove security trimming holds across retrieval and live tool calls.

• Test agent behaviour including tool selection,



execution correctness, failure handling and multi-agent handoffs / downstream triggers.

• Test document outputs for grid cell accuracy, template fidelity, and tracked-change style proofing suggestions rather than silent edits.

• Run adversarial tests for prompt injection (including via uploaded RFP files), data leakage and safety boundaries.

• Automate regression for model, prompt, embedding and retrieval changes inside the CI/CD pipeline.

Qualifications

Experience & Working Style

• 5–8 years in QA / test engineering, with 2+ years testing LLM, RAG or agent systems.

• Has built an automated LLM evaluation suite that gated real releases.

• Can sit with business reviewers, capture why an output is wrong and turn it into a repeatable test.

Not required (don't screen out for these): Traditional manual-only QA backgrounds are not sufficient. Deep ML modelling knowledge is not required.

📌 AI - QA Engineer (India)
🏢 Publicis Groupe Holdings
📍 India