Role & responsibilities

- Validate the entire agentic lifecycle, starting from user intent and prompting through agent reasoning and routing.
- Ensure accurate MCP tool execution and conduct thorough end-to-end workflow validation.
- Utilize solid observability skills to monitor and assess the effectiveness of workflows.
- Employ automation techniques to streamline testing and validation processes.
- Conduct AI quality assessments to ensure optimal functionality and performance of the system.
- Collaborate with cross-functional teams to refine and enhance lifecycle validation methodologies.

Mandatory skill sets:

- Strong experience in Agentic AI testing and validation.
- Must have hands-on knowledge of MCP testing, including tool invocation validation, payload/schema validation, orchestration, error handling, retries, and API/integration testing.
- Experience with AI observability and tracing tools such as LangFuse, Phoenix,



or similar platforms for debugging, root cause analysis, and agent behavior validation.
- Strong understanding of AI quality metrics including hallucination detection, accuracy, correctness, groundedness, consistency, determinism, and latency.
- Experience designing and executing prompt testing strategies, including multi-turn, multi-hop ambiguous, negative, and guardrail/jailbreak scenarios.
- Understanding of intent recognition, routing validation, agent selection, context retention, memory validation, and reasoning/planning validation.
- Ability to create test strategies, evaluation rubrics, golden datasets, and quality metrics for AI systems.
- Experience with API testing and automation frameworks (e.g., Postman, Katalon, Playwright, or equivalent).

📌 Agentic AI Testing Engineer (Kolkata)
🏢 PwC India
📍 Kolkata