Title: AI/ML Test Engineer – GenAI

Location - Hyderabad

Experience - 1-3 years

Technical Skills -

• Solid experience in Generative AI, LLMs, and Agentic AI systems

• Hands-on expertise with AI evaluation frameworks (RAGAS, DeepEval, TruLens, LangSmith, Promptfoo, etc.)

• Proficiency in Python and AI/ML development libraries

• Knowledge of Prompt Engineering, prompt testing, and optimization

Ability to define and track evaluation metrics such as accuracy, relevance, groundedness, hallucination rate, latency, and user satisfaction

• Experience in creating automated evaluation pipelines and benchmarking frameworks

• Strong understanding of AI safety, guardrails, bias testing, and responsible AI practices

• Familiarity with REST APIs, JSON, vector databases, and knowledge retrieval systems

• Experience in A/B testing, human-in-the-loop evaluation, and red teaming

• Strong experience in Manual Testing of AI/GenAI applications, including functional, exploratory, UAT, regression, and end-to-end testing





• Expertise in validating Agent Reasoning, Tool Calling, Workflow Execution, and Response Quality

• Hands-on experience in Automation Testing using Python frameworks

Key Responsibilities -

- Design, execute, and automate evaluation strategies for Agentic AI applications.
- Develop evaluation datasets, test cases, and benchmark suites.
- Measure and improve agent performance, reasoning quality, tool usage, and workflow effectiveness.
- Analyze model outputs and identify hallucinations, biases, safety risks, and failure patterns.
- Collaborate with AI Engineers, Product Teams, and Domain Experts to improve agent quality and reliability.
- Generate evaluation reports, dashboards, and actionable recommendations.

Skills:- Generative AI, Python, Manual testing, Automation, Prompt engineering, RESTful APIs, JSON and A/B Testing

📌 AI/ML Test Engineer – GenAI (Hyderabad)
🏢 Improving
📍 Hyderabad