Responsibilities
● Design and execute test plans for agent behavior across reasoning, tool usage, multi-turn flows, and goal completion.
● Create prompt regression suites and synthetic test data for 50+ use cases.
● Build and maintain automated testing frameworks for agent workflows using Python.
● Evaluate agent output using a combination of rubric-based, LLM-based, and scenario-based techniques.
● Test for hallucinations, prompt injection, bias, PII leakage, and response safety.
● Work with observability tools (e.g., LangSmith, Trulens, logging pipelines) to analyze agent execution.
● Collaborate with engineers to iterate on prompt design, tool chaining, and fallback logic.
● Contribute to the development of agent test harnesses and benchmarking pipelines.

️ Skills & Qualifications
Required:
● 3+ years in QA, software testing, or NLP/ML product evaluation.
● Strong Python scripting and test automation experience.
● Experience testing REST APIs, workflows, or backend logic.
● Familiarity with LLMs and basic concepts in prompt engineering.




● Understanding of hallucination, context limits, and RAG-based agents.

Preferred:
● Experience with LangChain, CrewAI, or custom agent frameworks
● Experience with LLM evaluation tools like LangSmith, Promptfoo, Trulens, Ragas
● Exposure to guardrails, moderation APIs, or adversarial testing
● Prior experience in generative AI/LLM-based applications or chatbots

? Bonus Points
● Experience with synthetic data generation
● Familiarity with CI/CD, version control (Git), and Agile QA practices
● Ability to write or grade evaluation prompts using GPT-4 or similar models

? What We Offer
● Opportunity to work on cutting-edge GenAI platform used across 50+ enterprise use cases
● Work with cross-functional AI-first teams (MLEs, PEs, PMs)
● Competitive compensation and advantages
● A collaborative, rapid-paced environment where quality is a first-class citizen

📌 Senior Test Automation Engineer(AI Testing + python) (India)
🏢 Quantiphi
📍 India