17 Nov
|
Mphasis
|
Mumbai
AI/ML Test Engineer –
Job Description
Minimum 7 years of experience
Job Description
–
• Evaluate agent decision‑making: Assess whether an agent’s scope decisions, planning steps, and tool‑calling sequences are correct and safe
• Strong Playwright experience and understanding
• Agentic AI experience and understanding
• API
• Audit AI‑generated tests: Review assertions and test cases produced by agentic testing systems to ensure they verify the right behaviours.
• Supervise agentic testing systems: Tune and correct what the AI testing system chooses to cover, rather than writing every test manually
• Verify orchestration reliability: Validate the orchestration framework (e.g., Agents SDK) and ensure tool/function‑calling reliability.
• Check guardrails and autonomy levels: Test how much autonomy agents have in production and whether guardrails prevent unsafe actions
• Build independent verification layers: Maintain human‑owned checks that validate AI‑generated outputs.
• 6+ Years of Total IT experience in QA/SDET and Testing, with 1to 2 years in the field of AI/ML domain.
• Test Agentic AI applications and AI-Driven Workflows
• Validate agent workflows complete flow from start to end
• Test LLM-based applications for accuracy, relevance, consistency, and hallucinations.
• Test multi-agent workflows and communication between agents.
• Test RAG (Retrieval-Augmented Generation) workflows and knowledge retrieval.
• Validate agent memory and context handling.
• Perform API testing using Postman or similar tools.
• Perform database testing and backend data validation using MongoDB/PostgreSQL.
• Knowledge on Key Value Caching,
Vector Databases like Chroma DB/PGVector
• Perform basic prompt injection and AI security/guardrail testing.
• Analyse agent execution logs and identify the root cause of failures.
• Create and maintain test cases, test scenarios, bug reports, and QA documentation.
• Work closely with AI/ML and development teams to improve overall agent reliability. Required Skills
• Robust knowledge of Software Testing / QA fundamentals.
• Experience with Manual Testing and API Testing.
• Understanding of Generative AI, LLMs, prompts, tokens, context, and hallucinations.
• Understanding of Agentic AI concepts and agent workflows.
• Knowledge of AI agents, tool/function calling, planning, execution, and memory.
• Basic understanding of RAG and vector databases.
• Ability to design test cases for non-deterministic AI outputs.
• Solid analytical, debugging, and problem-solving skills.
• Perform database testing and backend data validation using MongoDB/PostgreSQL.
• Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector.
• Python knowledge and test automation experience.
• Selenium, Playwright, PyTest, or similar automation tools.
• Knowledge of AI evaluation techniques such as groundedness, relevance, faithfulness, and response quality.
• Experience with Azure/AWS/GCP and AI services.
• Git, CI/CD, and Docker knowledge.
• Experience testing multi-agent or AI orchestration systems.
• Strong QA/SDET engineering background
• Understanding of agentic orchestration frameworks
• Ability to evaluate AI generated code, tests, and decisions
• Risk based thinking and judgment-heavy evaluation
📌 Quality Assurance Engineer (Mumbai)
🏢 Mphasis
📍 Mumbai