AI Evaluation & Rubric Reviewer (India)

AI Evaluation & Rubric Reviewer (India)

19 Nov
|
Hirely
|
India

19 Nov

Hirely

India

We are looking for experienced AI evaluation professionals to review and improve evaluation tasks used to assess advanced AI agents.

You will examine AI-generated task prompts, grading rubrics, and model responses to determine whether evaluation criteria are accurate, fair, and consistent. Your work will help improve the reliability and quality of AI evaluation systems.

Responsibilities

- Review AI evaluation tasks, prompts, and grading rubrics.
- Analyze lengthy AI agent transcripts to identify errors and reasoning failures.
- Determine whether model failures are genuinely justified by the evaluation criteria.
- Identify ambiguous, inaccurate, or unsupported grading requirements.
- Correct unfair or poorly structured rubric criteria.
- Flag evaluation tasks that cannot be reasonably fixed.
- Provide concise, evidence-based explanations for review decisions.

Requirements

- Substantial experience writing, reviewing, or applying evaluation rubrics for LLMs.
- Previous experience in AI evaluation, RLHF, model training, or AI quality assurance.
- Robust analytical and critical-thinking skills.
- Excellent written English and attention to detail.




- Ability to review lengthy transcripts and assess model behavior objectively.
- Ability to independently verify information rather than blindly relying on reference answers.

Relevant Experience

Experience with projects involving

- LLM evaluation and benchmarking
- RLHF and human feedback
- AI response evaluation
- Rubric design and quality assurance
- Prompt and instruction evaluation
- AI agent testing and assessment

Applicants should list relevant AI evaluation projects and explain their specific responsibilities.

Contract Details

- Compensation: $40-$75/hour
- Work arrangement: Fully remote
- Schedule: Flexible
- Payment: Weekly via Stripe or Wise

Join a team working on the evaluation and improvement of next-generation AI systems.

About Hirely

Hirely connects qualified professionals with remote opportunities in AI research, business, software engineering, and technology. We are recruiting candidates on behalf of a hiring partner, which manages final candidate selection, onboarding, project assignments, and engagement terms.

📌 AI Evaluation & Rubric Reviewer (India)
🏢 Hirely
📍 India

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: ai evaluation & rubric reviewer (india) / india

Subscribe to this job alert:

Get the latest job offers by email for: ai evaluation & rubric reviewer (india) / india