We are hiring detail-oriented professionals to evaluate AI agent performance on real-world, multi-step tasks. You will review agent traces, compare two AI responses, verify facts against tool results, and select the better response.

Responsibilities

- Review AI agent traces, including tool calls, tool results, and final responses.
- Compare two responses and select the one that better completes the task.
- Verify numbers, facts, and conclusions against actual tool outputs.
- Identify hallucinations, skipped steps, incorrect data, and instruction violations.
- Write short, clear justifications (23 sentences) for your decision.
- Apply evaluation guidelines consistently and flag unclear or ambiguous cases.

Requirements

- Bachelors degree; Masters preferred. Relevant experience accepted.
- Strong English reading and writing skills.
- Strong attention to detail and analytical reasoning.
- Comfort with JSON, tables, basic SQL, and API responses.




- Ability to distinguish accurate responses from fluent but incorrect ones.
- Availability for 4–5 flexible hours daily.

Preferred

- Experience in AI annotation, RLHF, LLM evaluation, or model evaluation.
- Background in QA, data analysis, technical writing, or research.
- Familiarity with Claude, ChatGPT, Cursor, Codex, or other AI tools.
- Understanding of tool calling, prompts, or agentic AI workflows.

Selection Process

Shortlisted candidates will complete a short assessment with a user task, an agent trace, and two responses. You will select the better response and provide a brief justification.

Note: This is a freelance project-based engagement, not full time employment. Only candidates based in India should apply.

📌 Agentic Evaluation Specialist Freelance (India)
🏢 Innodata India
📍 India