Functional AI Tester - GenAI- - - - - - - - - - - -
We are seeking a Quality Assurance (QA) Engineer focused on testing Generative AI (GenAI) applications with a strong emphasis on Python-based test automation, GenAI evaluation, and ETL/data quality validation. You will design and execute end-to-end test strategies that ensure our AI solutions are accurate, reliable, safe, and compliant.
About the Role
You will be involved in QA for GenAI features including Retrieval-Augmented Generation (RAG), conversational AI and Agentic evaluations. The role centers on:
- Systematic GenAI evaluation (qualitative and quantitative metrics)
- ETL and data quality testing for the data flows that feed AI systems
- Python-driven automated testing
This position is hands-on and collaborative, partnering with AI engineers, data engineers, and product teams to define measurable acceptance criteria and ship high-quality AI features.
Key Responsibilities
- Test strategy and planning
- Define risk-based test strategies and detailed test plans for GenAI features.
- Establish explicit acceptance criteria with stakeholders for functional, safety, and data quality aspects.
- Python test automation
- Build and maintain automated test suites using Python (e.g., PyTest, requests).
- Implement reusable utilities for prompt/response validation, dataset management, and result scoring.
- Create regression baselines and golden test sets to detect quality drift.
- GenAI evaluation
- Develop evaluation harnesses covering factuality, coherence, helpfulness, safety, bias, and toxicity etc.
- Design prompt suites, scenario-based tests, and golden datasets for reproducible measurements.
- Implement guardrail tests including prompt-injection resilience, unsafe content detection, and PII redaction checks.
- Track quality metrics over time.
- RAG and semantic retrieval testing
- Verify alignment between retrieved sources and generated answers.
- Verify adversarial tests.
- Measure retrieval relevance, precision/r
📌 Functional AI Tester (Pune)
🏢 Michelin
📍 Pune