Job Summary

Test Automation + Agent/LLM Evaluation (Accuracy, Hallucination, Regression) Role Summary: Responsible for designing, developing, and executing automated testing frameworks for AI/LLM-based applications and intelligent agents. The role focuses on validating model performance, response accuracy, hallucination detection, regression testing, and overall reliability of AI solutions.

Responsibilities

- Develop and maintain automated test frameworks for AI agents and LLM-powered applications.
- Evaluate LLM responses for accuracy, relevance, completeness, and consistency.
- Design test cases to identify and measure hallucinations, incorrect outputs, and bias.
- Perform regression testing to ensure model updates do not impact existing functionality.
- Create evaluation datasets, benchmarks, and test scenarios for agent workflows.
- Monitor key AI quality metrics such as precision, recall, response quality, and task success rate.
- Collaborate with AI engineers, data scientists, and product teams to improve model performance.
- Generate testing reports, insights,



and recommendations for continuous improvement.

Required Skills

- Experience in test automation using Python, Selenium, Playwright, PyTest, or similar tools.
- Understanding of Generative AI, LLMs, AI agents, and prompt engineering.
- Knowledge of LLM evaluation frameworks such as LangSmith, DeepEval, Ragas, Promptfoo, or similar.
- Experience with API testing and automated validation pipelines.
- Solid analytical and troubleshooting skills.

Preferred Qualifications

- Experience with OpenAI, Azure OpenAI, Anthropic, Gemini, or other LLM platforms.
- Knowledge of MLOps, AI model monitoring, and CI/CD integration.
- Familiarity with AI governance, responsible AI, and model quality assessment.

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Test Lead - API Automation (Mumbai)
🏢 Hexaware Technologies
📍 Mumbai