Home/Jobs/AI Evaluation Engineer — Evals & Systems Verification AI Evaluation Engineer — Evals & Systems Verification Meraki-Labs Bengaluru 3-6 years Today $25.3K–39.8K/yr Full time Onsite Skills Required LLM ML evaluation Python TypeScript Playwright Cypress API testing Description AI Evaluation Engineer focused on evaluation scaffolding and the quality layer for AI-native systems. The role emphasizes verification, reliability, and release readiness in production.

Role: AI Evaluation Engineer — Evals & Systems Verification Location: HSR, Bengaluru (On-site) Experience 3–6 years

Background in SDET/QA automation or ML evaluation

Proven ownership of test or evaluation infrastructure, not just executing tests

Strong programming skills in Python or TypeScript

Experience in API-level testing

Highly autonomous; able to define workflows and processes for evaluation and verification

Familiarity with LLM applications or demonstrated interest in evaluating non-deterministic systems Responsibilities Build and maintain evaluation harnesses for AI-facing features

Measure and tune system quality, including capture quality, retrieval quality, and guidance quality

Own end-to-end and API test infrastructure

Support a continuous, daily-release cycle

Design and execute host-behavior probes across diverse AI assistants

Ensure product behavior aligns with expectations across hosts and continual changes

Gate production releases through UAT, regression analysis,



and quality reporting More Skills API-level testing, LLM applications, evaluation harnesses, test infrastructure, release gates, UAT, regression analysis, quality reporting, host-behavior probes, scripted user sessions, AI assistants Other Work on monitoring product behavior across AI assistants such as Claude and ChatGPT

Focus on proving reliability in production for AI-native codebases

Verification is the key bottleneck for scaling capabilities Follow for Daily AI Tutorials Prepare for this role Recommended resources to build the skills for this position. Sponsored. Generative AI with Large Language Models Coursera Comprehensive LLM course covering transformer architecture, fine-tuning, RLHF, and deployment.

Python for Everybody Specialization Coursera Learn Python from scratch — variables, data structures, web scraping, and databases. Python 3 Programming Specialization Coursera Intermediate Python covering classes, inheritance, APIs, and data processing. More LLM jobs Senior Analyst, AI Engineer Cardinal Health United States Today Software Engineer II, AI Enablement (Remote) Symetra USA Today Junior Software Engineer (AI-Forward) Texas Sports Academy Austin Today LLM Engineer (Large Language Models) Fospe Software Private Limited Bengaluru Today Software Engineer - AI Agentic Product Dev Team (US) Eightfold Santa Clara Today Full Stack Developer- Generative AI Recruitment Hub 365 Pune Today

📌 AI Evaluation Engineer — Evals & Systems Verification (Bengaluru)
🏢 Top Gen AI Jobs
📍 Bengaluru