About the Role
As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues — so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.
What You'll Do
• Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety.

• Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively.

• Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear,



trackable quality metrics over time.

• Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end-to-end testing of AI-powered features and the surrounding product.

• Partner with product managers and engineers to translate ambiguous quality bars ("is this positive enough to ship?") into measurable, repeatable evaluation criteria.

• Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering.

• Maintain traceability and reporting on eval/QA results for compliance-sensitive workflows, given the regulated nature of the data we handle.

What We're Looking For
• 5 plus years of experience in QA/test engineering, ML evaluation, or a related quality-focused engineering role.

• Hands-on experience testing or evaluating AI/LLM-powered features — building eval sets, scoring rubrics, or ben

📌 AI Evaluation Engineer (Mangalore)
🏢 Vinyl Equity
📍 Mangalore