Role Summary

The GenAI Specialist Rating evaluates AI-generated responses against defined quality rubrics

to determine whether responses meet gold-standard expectations. Specialists assess factual

accuracy, grounding, completeness, helpfulness, safety, and user satisfaction while providing

transparent rationale for every decision.

Their work directly contributes to high-quality evaluation datasets used for model training,

benchmarking, and calibration.

Key Responsibilities
Evaluate AI-generated responses using standardized evaluation rubrics
Assess factual accuracy of geographic and local information
Verify that responses are properly grounded in available map and place data
Evaluate:

Factuality

Grounding

Helpfulness

Completeness

Relevance

Clarity

Safety

Instruction following
Identify hallucinations and unsupported claims
Distinguish between critical and minor defects
Provide detailed justification for ratings
Escalate ambiguous or policy-sensitive cases
Maintain consistency across evaluation tasks
Participate in calibration exercises




Contribute feedback to improve evaluation guidelines

Core Skills

Generative AI Evaluation
Understanding of LLM behavior
Prompt-response evaluation
Hallucination detection
Response quality assessment

Analytical Thinking
Critical reasoning
Evidence-based decision making
Attention to detail
Pattern recognition

Maps Knowledge
Geographic reasoning
Navigation concepts
POIs (Points of Interest)
Local search behavior
Route interpretation

Language Skills
Excellent written English
Reading comprehension
Ability to interpret nuanced prompts

Secondary Skills
Search verification techniques
Knowledge of local businesses
Travel and navigation terminology
Familiarity with Maps products
Basic data annotation experience
Spreadsheet proficiency

Ancillary Skills [Valuable to know]
SQL (basic)
Annotation platforms
Jira
Documentation tools
AI evaluation platforms

Qualifi

📌 Ai Specialist/data Annotation Hyderabad (India)
🏢 Tech Mahindra
📍 India