Company Description
QAD is building a world-class SaaS company, and we are growing. We are looking for talented individuals who want to join us on our mission to help solve relevant real-world problems in manufacturing and the supply chain.
We are a virtual first company and your primary work experience will be virtual / working from your home. Occasional travel to a physical office may be required to enhance working relationships, collaboration, design, strategy and alignment.
Job Description
QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi-
agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use,
orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks. You will
design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM-
driven systems.
Responsibilities:
• Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows,
including agent routing,
tool/function calls, and multi-turn conversations.
• Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and
completeness, using both human review and automated methods.
• Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated
regression suites that run as agents, prompts, models and tools change.
• Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,
tool-call failures, context loss, looping or runaway behavior.
• Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
• Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and
quantify them with metrics.
• Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool
and orchestration causes
📌 Senior QA Engineer, AI Agents (Pune)
🏢 QAD
📍 Pune