Hiring: Agentic AI Evaluation Specialist | Remote | India

Work Mode: Remote

Engagement Type: Freelance

Location: India

Qualification : Master's degree in Computer Science, IT, Engineering, Data Science, or a related field.

About the Role

We are looking for detail-oriented professionals to evaluate the performance of AI agents on complex, real-world tasks. This role goes beyond reviewing chatbot responsesyou will assess how AI agents execute multi-step tasks, use tools, interpret data, and arrive at their final answers.

You will compare AI-generated responses, verify their accuracy against actual tool outputs, identify errors or unsupported claims, and provide clear, evidence-based feedback to improve AI systems.

Key Responsibilities

- Evaluate AI agent workflows, including tool calls, tool outputs, and multi-step execution traces.
- Compare two AI-generated responses and select the one that best satisfies the task requirements.
- Verify facts, calculations, and conclusions against source data and tool results.
- Identify hallucinations, reasoning errors, missed steps, and instruction-following issues.
- Write concise justifications explaining evaluation decisions.
- Apply consistent quality, accuracy, safety, and instruction-adherence criteria.

Requirements

- Master's degree in Computer Science, IT,



Engineering, Data Science, or a related field. Candidates from other disciplines with relevant experience are also welcome.
- Strong English comprehension and written communication skills.
- Experience in AI/LLM evaluation, data annotation, software QA, API testing, data analysis, or similar analytical work.
- Ability to interpret structured data such as JSON, tables, basic SQL queries, and API responses.
- Robust attention to detail, analytical thinking, and evidence-based decision-making.

Preferred Qualifications

- Experience with RLHF, AI model evaluation, response rating, or side-by-side comparison tasks.
- Familiarity with AI agents, tool calling, multi-step workflows, or AI automation.
- Experience using Claude Code, Cursor, Codex, or similar AI-powered coding tools.
- Background in software testing, data validation, technical research, or quality assurance.

Note: Prior experience with agentic AI is an advantage, but candidates with strong analytical and technical validation skills are encouraged to apply.

Selection Process

Shortlisted candidates will complete a brief screening assessment involving an AI task, its execution trace, and two candidate responses to evaluate.

📌 Agentic AI Evaluation Specialist (India)
🏢 Innodata India
📍 India