We are hiring a QA Analyst to independently verify the outputs of this evaluation pipeline against real call audio, identify gaps in current detection logic and rubrics, and help continuously improve the accuracy of both our voice AI agents and the systems that evaluate them.
This role requires a minimum of 2+ years of relevant experience in QA, annotation, or evaluation work, and is well suited to a candidate who can work independently, exercise sound judgment on ambiguous cases, and communicate findings clearly to a technical team.
Key Responsibilities
Call Annotation
- Review and annotate real customer calls, including transcription review, issue tagging, and labeling, using tools such as Label Studio
- Apply and help refine annotation guidelines for a multilingual environment spanning Hindi, English, and Hinglish
LLM-as-Judge Verification
- Review calls flagged by the automated evaluation pipeline and determine whether the model's assessment was correct (for example, verifying whether a detected barge-in genuinely occurred by reviewing the audio)
- Document agreements and disagreements with transparent supporting reasoning to enable ongoing measurement and improvement of model accuracy
Quantitative Signal Verification
- Validate system-flagged quantitative signals against actual call audio (for example, confirming whether a detected agent interruption occurred at the reported timestamp)
- Identify and report false positives and false negatives to support refinement of detection logic
- Surface edge cases not adequately addressed by current rubrics or detection systems
Proactive Quality Review
- Conduct regular review of calls outside the flagged queue to identify emerging issues not yet captured by existing rubrics or detection systems
- Communicate identified patterns to the team to inform ongoing improvements to rubrics, prompts, and detection logic
Clarion.ai specializes in delivering enterprise-grade AI solutions focused on operational excellence and real-world impact. We bridge the gap between AI potential and production-ready implementation.