- Test and validate AI agents to ensure they perform actions accurately and as designed.
- Develop and execute test cases for different conversational and task-based AI workflows.
- Analyze test results, identify bugs or performance issues, and collaborate with the engineering team for fixes.
- Document testing procedures, outcomes, and recommendations for improvement.
- Ensure that AI agents handle prompts, contexts, and actions correctly.
- Contribute to the creation of automated QA processes for AI systems.
Candidate Requirements:
- Robust QA Engineer profile with AI / GenAI agent testing experience.
- 2 years of experience in QA, software testing, or AI testing roles.
- Experience testing and validating AI agents to ensure they perform actions accurately and as designed.
- Ability to develop and execute test cases for conversational and task-based AI workflows.
- Ability to analyze test results, identify bugs or performance issues, and collaborate with engineering for fixes.
- Good understanding of Generative AI, LLM agents, or autonomous AI workflows.
- Understanding of DeepEval and other AI testing frameworks.
- Manual testing experience and familiarity with automation tools.
- Strong problem-solving, analytical, and communication skills.
- Ability to collaborate with developers, prompt engineers, and data scientists.
Preferred Skills:
- Exposure to LLM frameworks (LangChain, CrewAI, AutoGen, etc.).
- Basic programming or scripting knowledge (Java or Python).
- Understanding of AI evaluation metrics and model performance tuning.