Job SummaryRole Summary

Responsible for validating AI, GenAI, Agentic AI, and Intelligent Document Processing (IDP) solutions by creating ground truth datasets, defining evaluation frameworks, executing test scenarios, and assessing the accuracy, reliability, and business relevance of AI-generated outputs before production deployment.

Key Responsibilities

- Define AI testing strategy, test plans, and acceptance criteria.
- Create, maintain, and curate ground truth datasets for AI model evaluation.
- Validate AI-generated outputs against business rules, expected outcomes, and ground truth data.
- Design and execute functional, non-functional, and regression tests for AI applications.
- Evaluate LLM responses for accuracy, completeness, relevance, consistency, and hallucinations.
- Develop test cases for RAG systems, AI Agents, IDP, and workflow automation solutions.
- Measure AI performance using evaluation metrics such as precision, recall, accuracy, and response quality.
- Identify defects, prompt issues, retrieval gaps, and model performance concerns.
- Collaborate with AI engineers and business SMEs to refine prompts, workflows, and evaluation criteria.
- Support UAT, model validation, and production readiness assessments.
- Produce AI quality reports, scorecards, and recommendations for release approval.

Skill Requirements





- 5+ years of QA/Test Engineering experience.
- Experience testing AI/ML, GenAI, or Intelligent Automation solutions.
- Proven experience creating and managing ground truth datasets and evaluation benchmarks.
- Robust understanding of AI evaluation methodologies and quality metrics.
- Experience with test automation frameworks and API testing.
- Strong analytical skills with the ability to compare AI outputs against expected business outcomes.
- Familiarity with Python, SQL, and data validation techniques.

Preferred

- Experience testing Agentic AI, RAG, and Intelligent Document Processing (IDP) solutions.
- Experience with AI evaluation tools such as Promptfoo, LangSmith, DeepEval, Ragas, TruLens, Mlflow or similar platforms.
- Claims domain experience and validation of Claims AI solutions is a strong plus.
- Experience evaluating document extraction, claims adjudication, fraud detection, and customer service AI use cases.

Key Selection Criteria

- Proven ability to establish and maintain AI ground truth datasets.
- Experience validating AI-generated outputs at scale.
- Strong understanding of hallucination detection, response evaluation, and AI quality assurance.
- Ability to define measurable quality gates for production deployment of AI and Agentic solutions.

Other Requirements

📌 Test Lead (India)
🏢 HCL Technologies
📍 India