Domain Expert (Music) | LLM Evaluation

Experience:
3–6 Years

Work Shift

7:30 PM – 12:30 AM IST

*Remaining 4 hours are flexible.*

Role Overview

We are looking for a highly qualified
Music Domain Expert
to help evaluate and improve state-of-the-art
Large Language Models (LLMs)
.

In this role, you will use your expertise in music to create challenging evaluation prompts, assess AI-generated responses, identify factual and reasoning errors, and provide evidence-based feedback to improve AI model quality.

Key Responsibilities
- Create advanced, domain-specific prompts covering
music theory, genres, composers, artists, music production, and instruments
.
- Evaluate AI-generated responses for
factual accuracy, reasoning, completeness, and nuance
.
- Identify hallucinations, logical inconsistencies, outdated information, and edge cases.
- Develop challenging benchmark datasets and adversarial test cases.
- Compare and rank multiple AI-generated responses based on predefined quality criteria.
- Provide clear, evidence-based feedback supported by reliable references.




- Collaborate with AI researchers and evaluation teams to improve model performance.
- Maintain high standards of annotation quality, consistency, and documentation.

Day-to-Day Activities
- Write music-related prompts across varying levels of difficulty.
- Review and compare multiple AI responses.
- Determine whether responses are accurate, relevant, and well-reasoned.
- Explain why a response is correct or incorrect using authoritative sources.
- Identify ambiguous questions and suggest improved versions.
- Document model weaknesses and recurring error patterns.

Minimum Qualifications
- Master's degree or higher
in Music or a closely related discipline.
- 3+ years of professional, research, teaching, or industry experience
in Music.
- Excellent written
English communication
skills.
- Solid analytical and critical-thinking abilities.
- Exceptional attention to detail and accuracy.

Preferred Qualificat

📌 Music Professional (India)
🏢 Sourcebae
📍 India