JD For AI Assurance / AI Evaluation Engineer

Role Description

Join the AI Trust Assurance (AITA) team to evaluate, test, and assure AI applications, AI agents, GenAI solutions, and multimodal AI systems. The role focuses on validating the quality, safety, reliability, performance, and compliance of AI systems using industry-standard evaluation frameworks, tools, and methodologies.

About the Role

We are looking for professionals with experience in AI, Machine Learning, Data Science, Generative AI, Agentic AI, or Test Automation who are passionate about building trust in AI systems by evaluating AI systems through Metrics scoring approach.

You will help evaluate LLMs, RAG applications, AI agents, voice agents, multimodal applications, and AI-powered business solutions by designing evaluation frameworks, datasets, benchmarks, and automated validation processes.

Key Responsibilities

- Develop Python-based frameworks, utilities, and automation for AI evaluation.
- Evaluate AI applications, LLMs, RAG systems, AI agents, and multi-agent workflows.
- Evaluate Text-to-Text, Voice-to-Voice, Voice-to-Text, Video-to-Text, and Multimodal AI applications.
- Design and execute prompt testing and response validation.
- Create and maintain evaluation datasets, benchmarks, and golden datasets.
- Assess agent reasoning, planning, memory, tool usage, workflow execution, and task completion.
- Validate AI systems for hallucinations, grounding, safety, bias, robustness, and compliance.
- Identify appropriate evaluation metrics, scoring thresholds, and benchmarks for AI agents. Interpret evaluation results, explain scoring rationale, and provide data-driven recommendations to AI developers to improve model and agent performance.




- Monitor AI applications using observability and tracing tools.
- Prepare and analyse evaluation reports, dashboards, and insights for stakeholders.
- Perform continuous post-production evaluation and monitoring of AI applications to identify model drift, performance degradation, safety risks.
- Collaborate with AI Engineers, Data Scientists, ML Engineers, Developers, and Business Teams.

Required Skills

- Bachelor's or masters degree in computer science, AI & ML, Data Science, Engineering, or a related field.
- AI Assurance / AI Evaluation Engineer with Experience: 5-12 Years
- Python Programming
- AI, Machine Learning, and Generative AI concepts
- AI Evaluation Frameworks: DeepEval, RAGAS, Hugging Face
- AI Evaluation Platforms: Industry standard platforms like Azure AI Evaluation, Confident AI, Promptfoo, TruLens, Credo AI, Holistic AI etc.
- AI Observability & Tracing: LangSmith, LangChain, OpenTelemetry, LangFuse etc.
- Responsible AI (RAI), EU AI Act, NIST AI RMF
- Understanding of AI Agents, Agentic AI, RAG Applications, LLM-based Systems, ML-based Systems, SDLC Agents, and Business AI Agents
- AI Evaluation Metrics (Hallucination, Faithfulness, Groundedness, Tool Calling Accuracy, Agent Trajectory Accuracy, Task Completion Rate)
- NLP (Natural Language Processing)
- Prompt Engineering
- APIs and Integrations
- Data Quality, Data Classification, Data Validation, and Benchmarking
- CI/CD and Test Automation
- Experience with Conversational AI, Voice AI, and Multimodal AI systems
- Experience evaluating autonomous and multi-agent workflows

Mandatory Skills

- AI Evaluation Tools & Metrics (Hallucination, Faithfulness, Groundedness, Tool Calling Accuracy, Agent Trajectory Accuracy, Task Completion Rate)
- Python Programming

📌 AI Assurance / AI Evaluation Engineer (Mumbai)
🏢 LTM
📍 Mumbai