Description
Support the design, execution, and scaling of evaluation and annotation programs for agentic AI systems, with a focus on defining metrics, schemas, rubrics, and quality frameworks for multi-step reasoning, tool use, task completion, policy adherence, and secure agent behavior.
Job Description
As a Senior Associate in Consumer & Community Banking, you will support the development and operationalization of evaluation frameworks for agentic AI systems. This role will focus on how AI agents plan, reason, use tools, follow policies, recover from errors, and complete tasks across multi-turn, multi-step workflows. You will partner with data science, machine learning engineering, product, architecture, tech, and Linguistics to define what “good” agent behavior looks like and translate that into measurable evaluation criteria. You will design annotation schemas, create rubrics for agent trajectories, train annotators, lead calibration exercises, maintain gold and challenge datasets, and help ensure evaluation outputs are consistent, scalable, auditable, and actionable.




The ideal candidate has hands-on experience evaluating LLM-based or agentic systems, including tool-calling behavior, planning quality, action sequencing, grounding, error recovery, and human-in-the-loop review workflows. This role requires someone who can go beyond language annotation and directly support the measurement and improvement of agentic AI performance.
Job Responsibilities
- Define and operationalize evaluation metrics for agentic AI workflows, including task success, step-level correctness, tool-use quality, policy adherence, recovery behavior, escalation decisions, and safe failure outcomes.
- Build and maintain agent-specific gold sets, challenge sets, and regression suites to assess planning quality, action sequencing, grounding, compliance boundaries, hallucination risk, loop detection, and release readiness
- Design annotation schemas, rubrics, taxonomies, and labeling guidelines fo

📌 Data Domain Architect Senior Associate (Bengaluru)
🏢 JPMorganChase
📍 Bengaluru