We're Hiring | Agentic AI Tester

We are looking for an experienced
Agentic AI Tester
to join our growing team. This role goes beyond traditional QA and focuses on testing autonomous AI agents powered by Large Language Models (LLMs). The ideal candidate will have hands-on experience in AI testing, LLM evaluation frameworks, and agentic AI concepts to ensure reliability, safety, and performance in real-world applications.

Job Details
- Role:
Agentic AI Tester
- Location:
Mumbai (Goregaon)
- Experience:
5–7 Years
- Work Mode:
Hybrid (3 Days Work From Office)
- Employment Type:
Full-Time

Key Responsibilities
- Design and execute test strategies for Agentic AI systems.
- Evaluate LLM outputs for correctness, consistency, reasoning quality, and safety alignment.
- Develop and refine LLM-as-Judge frameworks for automated AI evaluations.
- Validate multi-step reasoning, planning, tool usage, and decision-making capabilities of AI agents.
- Define evaluation rubrics, golden datasets, and testing criteria.
- Test edge cases including hallucinations, prompt injections, tool misuse,



and failure recovery.
- Analyze agent traces, logs, and decision paths to identify quality gaps.
- Collaborate with engineering teams to improve prompts, guardrails, and evaluation pipelines.
- Contribute to continuous evaluation systems and CI/CD processes for AI applications.

Required Skills
- Strong understanding of LLM fundamentals including prompting, tokens, temperature, and context windows.
- Hands-on experience with LLM-as-Judge or automated AI evaluation techniques.
- Knowledge of Agentic AI concepts including:
- Planning
- Memory
- Tool Usage
- Reflection and Self-Correction
- Experience testing NLP, ML, or AI-driven systems.
- Strong QA and test strategy design experience.
- Ability to define evaluation rubrics and scoring mechanisms.
- Experience with Python scripting for evaluation frameworks.
- Familiarity with OpenAI, Azure OpenAI, Anthropic, or similar LLM platforms.

Positive to Hav

📌 Agentic AI Tester (Mumbai)
🏢 Simplrize Global Technologies
📍 Mumbai