About The Role

Shipping AI products is not the same as shipping software. The model layer changes every few weeks. The same prompt produces different answers tomorrow. A "small" model upgrade can silently break the feature your top customer depends on. The discipline that catches all of this - before customers do - is evals.

PingAura is hiring our first Evals Engineer to own that layer end to end. Our AI Coworker ships to paying enterprise customers and runs across OpenAI, Google Gemini, and Anthropic. We need someone who treats evals as a product, not a script - someone who can build the bench, run it as a gate on every release, and turn quality into a measurable curve.

We are pre-seed and backed by 14 CXO angel investors, Google for Startups, and AWS. The product is live. The evals problem is real and ours to solve.

This is not a prompt-tweaking role. It is for someone who can build the systems around evals: datasets, graders, traces, dashboards, regression gates, and the reliability loops that decide whether an AI feature is ready to ship.

Responsibilities

• Own the eval bench.



Design eval datasets, write graders (including LLM-as-judge), and turn customer escalations into permanent regression tests
• Gate every model upgrade, prompt change, and agent behavior change on eval pass. No eval, no merge
• Measure quality, latency, and cost across model versions and providers. Surface the tradeoffs the team can act on
• Detect regressions across releases - when something breaks, your bench catches it first
• Partner with the AI Agent Engineer on tool-use, structured output, and long-horizon agent evaluation
• Build the data pipeline: capture production traces on Langfuse, label them, and feed them back into the bench
• Make evals first-class internally - dashboards, alerts, weekly quality reports the whole team reads
• Instrument human-in-the-loop labeling for the cases evals alone cannot judge

You may be a valuable fit if

• You have 2+ years working with large lan

📌 Evals Engineer (Mumbai)
🏢 PingAura.ai
📍 Mumbai