About The Opportunity

Avenue Code is looking for an AI Quality Analyst to work with Fanatics as part of its AI/ML Platform team.

The AI/ML Platform team builds and supports Fanatics’ internal AI platform, including the gateway, agent runtime, registry, retrieval, and evaluation infrastructure used by teams developing solutions with LLMs and AI agents.

As the team’s dedicated AI Quality Analyst, you will be responsible for evaluating whether AI agents are performing as expected. You will run evaluation sets, review outputs against established rubrics, maintain high-quality ground-truth datasets, identify failure patterns, and provide actionable findings to the engineering and product teams responsible for each agent.

This is a 12-month contract prospect based in Hyderabad, India, following a hybrid work model.

Responsibilities

Run evaluation sets against production and pre-release AI agents on a regular cadence and following meaningful changes to prompts, models, tools, or retrieval systems.
Analyze and report evaluation results, including pass rates, regressions, trends,



and recurring failure patterns.
Review AI agent outputs against defined quality rubrics covering correctness, grounding, tool-use accuracy, safety, tone, and output format.
Score outputs consistently, document edge cases, and identify areas where evaluation rubrics need clarification or improvement.
Curate and maintain ground-truth datasets used to evaluate AI and LLM-based applications.
Create, review, and validate golden answers and expected outputs for evaluation cases.
Classify failures by root cause, including prompt issues, retrieval failures, incorrect tool usage, or model-related errors.
Identify and retire outdated, ambiguous, or low-quality evaluation cases.
Compare LLM-as-a-judge evaluation results against human review and identify discrepancies between automated and human scoring.
Provide agent owner teams with reproducible examples, severity assessments, and suspected root caus

📌 AI Quality Analyst (India)
🏢 Avenue Code
📍 India