Functional AI Tester - GenAI- - - - - - - - - - - -
We are seeking a Quality Assurance (QA) Engineer focused on testing Generative AI (GenAI) applications with a solid emphasis on Python-based test automation, GenAI evaluation, and ETL/data quality validation. You will design and execute end-to-end test strategies that ensure our AI solutions are accurate, reliable, safe, and compliant.

About the Role
You will be involved in QA for GenAI features including Retrieval-Augmented Generation (RAG), conversational AI and Agentic evaluations. The role centers on:
Systematic GenAI evaluation (qualitative and quantitative metrics)
ETL and data quality testing for the data flows that feed AI systems
Python-driven automated testing

This position is hands-on and cooperative, partnering with AI engineers, data engineers, and product teams to define measurable acceptance criteria and ship high-quality AI features.
Key Responsibilities
Test strategy and planning
Define risk-based test strategies and detailed test plans for GenAI features.




Establish explicit acceptance criteria with stakeholders for functional, safety, and data quality aspects.
Python test automation
Build and maintain automated test suites using Python (e.g., PyTest, requests).
Implement reusable utilities for prompt/response validation, dataset management, and result scoring.
Create regression baselines and golden test sets to detect quality drift.
GenAI evaluation
Develop evaluation harnesses covering factuality, coherence, helpfulness, safety, bias, and toxicity etc.
Design prompt suites, scenario-based tests, and golden datasets for reproducible measurements.
Implement guardrail tests including prompt-injection resilience, unsafe content detection, and PII redaction checks.
Track quality metrics over time.
RAG and semantic retrieval testing
Verify alignment between retrieved sources and generated answers.
Verify adversarial tests.
Measure retrieval relevance, precision/r

📌 Functional Ai Tester Pune
🏢 Michelin
📍 Pune