AI QA Engineer – Agentic AI / GenAI

Experience: 7+ Years

Employment Type: Full time/Contract

Immediate/Early Joiners Preferred

Role Overview

We are looking for an AI QA Engineer with strong Python and automation expertise to validate and automate testing for Agentic AI, GenAI, and LLM-based systems.

Key Responsibilities

- Develop Python-based test automation frameworks using Pytest / unittest.
- Test and evaluate LLM outputs for accuracy, relevance, consistency, and hallucinations.
- Validate multi-agent workflows, AI pipelines, and integrations.
- Build and implement evaluation metrics such as Precision, Recall, and F1-score.
- Perform API, integration, functional, and regression testing.
- Integrate automated tests into CI/CD pipelines and ensure high test coverage.
- Identify, debug, and report issues across AI/ML systems.





Required Skills
- Strong Python programming and test automation.
- Hands-on experience with Pytest / unittest.
- Good understanding of Generative AI, LLMs, and Agentic AI concepts.
- Experience in API testing, debugging, and integration testing.
- Knowledge of AI/ML evaluation and testing methodologies.
- Experience with CI/CD and Git/version control.

Preferred
- Experience testing LLM applications, RAG systems, AI agents, or multi-agent frameworks.
- Exposure to LLM evaluation tools/frameworks and prompt testing.

? Interested candidates can share their updated resume at [email protected]

📌 AI QA Engineer (India)
🏢 Runmap Infotech Private
📍 India