Role Overview

We are seeking technically skilled professionals to evaluate AI agents performing complex, multi-step tasks. The role involves reviewing agent traces, tool calls, technical outputs, and final responses to determine whether the agent completed the task accurately and appropriately.

Position Details

- Location: India Remote
- Engagement: Freelance / Independent Contractor
- Commitment: 45 hours daily, flexible schedule
- Duration: 1 month, with potential extension
- Language: English
- Assessment: Mandatory Tier 2 AgentX assessment

Key Responsibilities
- Review AI-agent traces, including tool calls, tool inputs, outputs, and final responses.
- Verify whether the agent used appropriate tools and followed the correct sequence.
- Identify hallucinations, incorrect calculations, unsupported claims, and skipped or unnecessary steps.
- Compare two responses and select the more accurate and instruction-aligned response.
- Write concise, evidence-based rationales for evaluation decisions.
- Flag ambiguous, incomplete, or technically unclear cases.
- Apply evaluation guidelines consistently.

Minimum Qualifications

- Bachelors degree completed or in progress in Computer Science, AI, Robotics, Computational Linguistics, Software Engineering, Data Science, Mathematics, or a related field.
- Solid written English and analytical reasoning.
- Comfortable reading code, logs, JSON, API responses,



tables, and technical traces.
- Ability to evaluate structured, multi-step AI workflows.
- Availability for 45 hours daily.
- Successful completion of the Tier 2 AgentX assessment.

Required Technical Exposure Exposure through coursework, self-study, or projects to at least one of the following:

- Automated planning, MDPs, heuristic search, or decision theory.
- Reinforcement learning, reward modelling, RLHF, or RLAIF.
- Distributed systems, state management, fault tolerance, or asynchronous execution.
- NLP, LLM architecture, attention mechanisms, context windows, or function calling.

Hands-on experience with at least one of the following is required:
- Languages: Python, Rust, C++, or TypeScript.
- Frameworks: LangGraph, AutoGen, CrewAI, DSPy, Temporal, or Prefect.
- Storage: Qdrant, Pinecone, Milvus, PostgreSQL, Redis, or Neo4j.
- Infrastructure: Docker, E2B, Modal, Kubernetes, or Asyncio.

Preferred Experience

- AI data labelling, RLHF, LLM evaluation, or model evaluation.
- Software QA, API testing, data validation, or technical quality assurance.
- Experience with tool calling, agentic workflows, logs, APIs, or structured data.
- Familiarity with ChatGPT, Claude, Cursor, Codex, or similar AI tools.
- Exposure to multiple technologies across the listed categories.

📌 Freelancer - RLHF Specialist (Noida)
🏢 Innodata India
📍 Noida