AI/ML Test Engineer –

Minimum 7 years of experience

–

1. Evaluate agent decision‑making: Assess whether an agent’s scope decisions, planning steps, and tool‑calling sequences are correct and safe
2. Strong Playwright experience and understanding
3. Agentic AI experience and understanding
4. API
5. Audit AI‑generated tests: Review assertions and test cases produced by agentic testing systems to ensure they verify the right behaviours.
6. Supervise agentic testing systems: Tune and correct what the AI testing system chooses to cover, rather than writing every test manually
7. Verify orchestration reliability: Validate the orchestration framework (e.g., Agents SDK) and ensure tool/function‑calling reliability.
8. Check guardrails and autonomy levels: Test how much autonomy agents have in production and whether guardrails prevent unsafe actions
9. Build independent verification layers: Maintain human‑owned checks that validate AI‑generated outputs.
10. 6+ Years of Total IT experience in QA/SDET and Testing, with 1to 2 years in the field of AI/ML domain.
11. Test Agentic AI applications and AI-Driven Workflows
12. Validate agent workflows complete flow from start to end
13. Test LLM-based applications for accuracy, relevance, consistency, and hallucinations.
14. Test multi-agent workflows and communication between agents.
15. Test RAG (Retrieval-Augmented Generation) workflows and knowledge retrieval.
16. Validate agent memory and context handling.
17. Perform API testing using Postman or similar tools.
18. Perform database testing and backend data validation using MongoDB/PostgreSQL.
19. Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector
20. Perform basic prompt injection and AI security/guardrail testing.
21.



Analyse agent execution logs and identify the root cause of failures.
22. Create and maintain test cases, test scenarios, bug reports, and QA documentation.
23. Work closely with AI/ML and development teams to improve overall agent reliability.

Required Skills
1. Strong knowledge of Software Testing / QA fundamentals.
2. Experience with Manual Testing and API Testing.
3. Understanding of Generative AI, LLMs, prompts, tokens, context, and hallucinations.
4. Understanding of Agentic AI concepts and agent workflows.
5. Knowledge of AI agents, tool/function calling, planning, execution, and memory.
6. Basic understanding of RAG and vector databases.
7. Ability to design test cases for non-deterministic AI outputs.
8. Strong analytical, debugging, and problem-solving skills.
9. Perform database testing and backend data validation using MongoDB/PostgreSQL.
10. Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector.
11. Python knowledge and test automation experience.
12. Selenium, Playwright, PyTest, or similar automation tools.
13. Knowledge of AI evaluation techniques such as groundedness, relevance, faithfulness, and response quality.
14. Experience with Azure/AWS/GCP and AI services.
15. Git, CI/CD, and Docker knowledge.
16. Experience testing multi-agent or AI orchestration systems.
17. Strong QA/SDET engineering background
18. Understanding of agentic orchestration frameworks
19. Ability to evaluate AI generated code, tests, and decisions
20. Risk based thinking and judgment-heavy evaluation

Key Skills 6+ years as QA/SDET, 2+ years in the field of AI/ML, Agentic AI Testing, Generative AI Testing, LLM Testing, RAG, Prompt Testing, AI Testing, Python, Test Automation, Use case generation and Management. Robust QA and Testing Skills, PGVector, Chroma DB, Mongo DB, PostgreSQL

📌 AIML Test engineer (Mumbai)
🏢 Mphasis
📍 Mumbai