Job Description

QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi-
agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use,
orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks. You will
design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM-
driven systems.

Responsibilities:

• Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows,
including agent routing, tool/function calls, and multi-turn conversations.

• Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and
completeness, using both human review and automated methods.

• Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated
regression suites that run as agents, prompts,



models and tools change.

• Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,
tool-call failures, context loss, looping or runaway behavior.

• Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.

• Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and
quantify them with metrics.

• Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool
and orchestration causes.

• Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
​​​​​​​
• Document test strategy, eval results and quality trends clearly for global stakeholders.

📌 Senior QA Engineer, AI Agents & Agentic Workflows (Pune)
🏢 QAD
📍 Pune