Role: AI Evaluation & Harness Engineer Healthcare AI

Experience: 7 to 11 Years

Location: Bangalore / Mumbai

Role Overview:

We are looking for an AI Evaluation & Harness Engineer to design and operate both halves of this system: the evaluation infrastructure that proves our AI is trustworthy, and the production harness prompts, tool contracts, guardrails, context management that makes our agents perform reliably in the first place. This is a hands-on technical role for someone who has actually built and evaluated LLM or agentic systems in production, not only tested them from the outside. You will set technical direction for how AI quality is measured and gated, build reusable evaluation datasets and judges, and work directly with the engineers building our agents to close gaps between what ships and what's actually been proven safe.

Responsibilities

- Design and build evaluation harnesses for LLM-powered and agentic workflows, covering prompt-level regression testing, structured-output/schema validation, retrieval quality (context precision/recall), faithfulness, and answer relevancy, using frameworks such as RAGAS and DeepEval to operationalize these metrics rather than building scoring logic from scratch each time.
- Build and maintain LLM-as-judge evaluation pipelines with rubric design, calibration against human review, and inter-rater reliability checks, distinguishing dimensions that require deterministic assertions from those that genuinely require semantic judgment.
- Own evaluation and testing for agentic systems built on frameworks such as Google's Agent Development Kit (ADK), LangChain/LangGraph, or comparable orchestration frameworks including trajectory evaluation, tool-call accuracy (tool selection vs. parameter construction, evaluated independently), agent handoff accuracy in multi-agent workflows, and failure-recovery testing via deliberate failure injection.
- Design and maintain the regression-gated evaluation pipeline that runs on every prompt change, model version bump, or context update functioning as the AI equivalent of a CI/CD quality gate and isolate model changes from prompt changes when both occur together rather than validating only at the end-to-end layer.
- Own model lifecycle and drift management: treat prompts as versioned behavioral specifications, run continuous (not just release-time) regression suites against golden datasets,



and detect silent model drift introduced by upstream provider updates before it surfaces as a user-facing quality problem.
- Build and evolve the production harness around our agent Lambdas/services: structured-output contracts, prompt/context versioning, guardrails against unsafe or out-of-scope input, escalation logic for low-confidence outputs, and the observability/tracing layer that captures every input prompt model response downstream action as an inspectable record.
- Build a structured failure taxonomy specific to our systems grounding/hallucination failures, reasoning and instruction-drift failures, safety/adversarial failures, coordination failures in multi-agent handoffs, and stochastic/consistency failures and use it to drive systematic test coverage rather than ad hoc test-writing.
- Design human-in-the-loop review workflows for low-confidence or high-stakes outputs, including sampling strategy, escalation paths, and feedback loops that route confirmed failures back into evaluation datasets.
- Convert every significant production incident into permanent evaluation coverage: identify why existing evaluations didn't catch it, close the gap, and confirm the fix before redeployment.
- Establish evaluation datasets, judges, rubrics, and regression baselines as shared, versioned infrastructure usable by other project teams, rather than infrastructure rebuilt per project.
- Produce audit-ready documentation of evaluation methodology, results, and drift trends to support compliance and clinical-safety review.

Qualifications

Required

- Bachelor's degree in Computer Science, Machine Learning, or related technical field, plus 3+ years of software engineering experience with 1+ years specifically building, evaluating, or productionizing LLM or agentic AI systems or equivalent experience.
- Direct, hands-on experience integrating LLM APIs (OpenAI, Anthropic, Gemini, or similar) into production systems — this must be applied AI/ML engineering experience,



not testing AI features from a pure QA vantage point.
- Practical experience with evaluation frameworks or techniques such as RAGAS, DeepEval, LLM-as-judge scoring, or comparable evaluation tooling.
- Strong Python engineering skills, including experience with structured-output/function-calling patterns and prompt versioning.

Preferred

- Experience with agent orchestration frameworks such as Google's Agent Development Kit (ADK), LangChain/LangGraph, or similar.
- Cloud platform experience — GCP (Vertex AI, Cloud Functions), AWS (Lambda, Bedrock), or Azure (AI Foundry) — is a strong plus, particularly for serverless AI workloads.
- Experience evaluating RAG pipelines or multi-agent systems in a regulated or safety-critical domain (healthcare, finance).
- Familiarity with HIPAA/PHI handling requirements and healthcare data governance.
- Experience designing failure taxonomies, trajectory/handoff test scenarios, and severity-tiered quality gates for agentic systems.
- Exposure to observability/tracing tooling for multi-step AI pipelines (e.g., LangSmith, Datadog LLM Observability, or equivalent).

About the AI Evaluation Practice Our AI Evaluation practice sits at the point where AI ambition meets AI accountability — every agent we ship, every model we adopt, and every prompt we change passes through the systems this role builds. If you've been in the room building and debugging LLM systems and want to make that discipline rigorous and repeatable at scale, this is that role.

If you are interested in this chance, please provide the following details along with your updated profile to [email protected]:

Primary Skill : SRE SME – Site Reliability Engineering Subject Matter Expert

Total Experience

Principal Employer/Client Company

Payroll Company in case of Contract

Current Location

Work Location

LWD in case of Notice Period In case of Long Notice Period, is there any possibility to negotiate the days? If so, in how many days?

Current CTC

Expected CTC

Any Offer in Pipeline (Provide Details)

Have PF Deductions and Form 16 Yes / No

I will be reaching out to you over a call or email as soon as possible.

Thanks & Regards,

Harshavardhan Ponnaganti

Talent Acquisition Executive

ZettaMine Labs Pvt Ltd.

Mobile: +91 (phone hidden)

📌 AI Evaluation & Harness Engineer (Bengaluru)
🏢 ZettaMine Labs
📍 Bengaluru