Responsibilities ● Design and execute test plans for agent behavior across reasoning, tool usage, multi-turn flows, and goal completion. ● Create prompt regression suites and synthetic test data for 50+ use cases. ● Build and maintain automated testing frameworks for agent workflows using Python. ● Evaluate agent output using a combination of rubric-based, LLM-based, and scenario-based techniques. ● Test for hallucinations, prompt injection, bias, PII leakage, and response safety. ● Work with observability tools (e.g., LangSmith, Trulens, logging pipelines) to analyze agent execution. ● Collaborate with engineers to iterate on prompt design, tool chaining, and fallback logic. ● Contribute to the development of agent test harnesses and benchmarking pipelines. ️ Skills & Qualifications Required: ● 3+ years in QA, software testing, or NLP/ML product evaluation. ● Strong Python scripting and test automation experience. ● Experience testing REST APIs, workflows, or backend logic. ● Familiarity with LLMs and basic concepts in prompt engineering.



● Understanding of hallucination, context limits, and RAG-based agents. Preferred: ● Experience with LangChain, CrewAI, or custom agent frameworks ● Experience with LLM evaluation tools like LangSmith, Promptfoo, Trulens, Ragas ● Exposure to guardrails, moderation APIs, or adversarial testing ● Prior experience in generative AI/LLM-based applications or chatbots ? Bonus Points ● Experience with synthetic data generation ● Familiarity with CI/CD, version control (Git), and Agile QA practices ● Ability to write or grade evaluation prompts using GPT-4 or similar models ?

What We Offer
● Opportunity to work on cutting-edge GenAI platform used across 50+ enterprise use cases ● Work with cross-functional AI-first teams (MLEs, PEs, PMs) ● Competitive compensation and advantages ● A collaborative, fast-paced environment where quality is a first-class citizen

📌 Senior Test Automation Engineer(AI Testing + python) (India)
🏢 Quantiphi
📍 India