Hirely is recruiting AI Quality Analysts on behalf of a hiring partner for an AI evaluation project focused on personalized AI interactions.

You'll design multi-turn prompts and evaluate how effectively AI models use relevant personal context to produce grounded, natural, relevant, and helpful responses .

What You'll Do

- Design creative multi-turn prompts based on realistic personal scenarios
- Execute conversations of typically 1-5 turns to test AI personalization
- Evaluate whether AI responses appropriately use relevant context
- Assess responses across dimensions including Grounding, Integration, and Helpfulness
- Identify incorrect personalization, unsupported inferences, and hallucinations
- Evaluate whether personal information is integrated naturally without unnecessary repetition
- Compare two AI responses side-by-side and determine which response is more helpful and natural
- Identify subtle differences in response quality, usability, and naturalness
- Write clear and defensible rationales explaining model rankings
- Reference specific conversation turns when documenting evaluation decisions
- Provide detailed annotations and constructive feedback
- Extract and verify relevant debugging information from model outputs
- Follow strict data-hygiene procedures throughout evaluation work

Who We're Looking For

- Solid English reading and writing skills
- Bachelor's degree or equivalent experience in a relevant field
- Strong analytical and critical-thinking skills
- Ability to evaluate nuanced and ambiguous AI responses




- Strong written communication and ability to explain evaluation decisions clearly
- Ability to design creative prompts and multi-turn conversations
- Understanding of personalization, contextual reasoning, and AI response quality
- Experience in AI evaluation, data annotation, content moderation, or related work is strongly preferred
- Background in Computer Science, Linguistics, Journalism, Policy, Law, Ethics, or another analytical field is valuable
- Desktop/laptop with a reliable internet connection

Compensation & Terms

- Contractor engagement
- At least 4 hours/day, up to 40 hours/week
- Fully remote

Application Process

1. Submit your application and profile
2. Shortlisted candidates receive a Job Interest Form
3. Complete the assessment within 24 hours
4. Successful candidates proceed to pre-onboarding requirements

Important Requirement This project requires candidates to use their primary personal Google account and enable relevant personal data sources for the evaluation. Candidates should only apply if they are comfortable with the project's stated data-access and evaluation requirements.

About Hirely

Hirely connects professionals with remote opportunities from hiring partners across artificial intelligence, software engineering, cloud, data, research and other specialised fields. Hirely acts as a recruitment and talent-sourcing partner for the hiring companies it represents. Final hiring decisions, onboarding, compensation and project assignments are determined by the hiring partner.

📌 AI Quality Analyst - English (India)
🏢 Hirely
📍 India