Senior AI Evaluation & RLHF Specialist

Experience: 2+ Years

Employment Type: Contract 3 Months

Language Requirement: C1+ English

Location: Work from Office

We are hiring Senior AI Evaluation & RLHF Specialists to work on advanced AI/LLM evaluation and alignment projects . The role involves evaluating AI-generated responses, preference ranking, RLHF data generation, multi-turn reasoning evaluation, and improving model outputs.

Key Responsibilities
Evaluate and rank AI/LLM responses using detailed evaluation rubrics.
Perform Side-by-Side (SxS) evaluations for factuality, helpfulness, safety, reasoning, coherence, and instruction-following.
Identify errors and provide clear, fact-based evaluation rationales.
Rewrite and improve AI-generated responses.
Evaluate mathematical, logical, reasoning, and tool-use outputs.
Review multi-turn conversations and agentic AI workflows.




Create golden responses and edge-case examples.
Participate in calibration sessions and maintain high inter-rater reliability.
Follow evolving project guidelines and meet quality and productivity SLAs.

Required Skills
2+ years of hands-on experience in AI data annotation, AI model evaluation, RLHF, preference scoring, or AI training data.
Experience working on complex AI/LLM evaluation projects.
Solid analytical and critical-thinking skills.
Excellent written English communication ( C1 or above ).
Ability to provide objective, evidence-based evaluation rationales.
Strong attention to detail and ability to follow complex guidelines.
Comfortable working in a rapid-paced, SLA-driven workplace.

📌 Senior Ai Evaluation & Rlhf Specialist Noida
🏢 Innodata
📍 Noida