Job Title: Gen AI Tester / AI QA Engineer

Experience: 6–10 Years

Location: [Add Location]

Employment Type: Full time

Job Overview

We are looking for a skilled Gen AI Tester to validate, evaluate, and ensure the quality and reliability of Generative AI applications and LLM-based solutions. The ideal candidate will have experience in AI/ML testing, prompt testing, model performance evaluation, automation, and validating RAG and LLM workflows.

Key Responsibilities
Test and validate LLM-based applications, Gen AI workflows, and chatbot interactions
Perform prompt testing, scenario testing, hallucination detection, and response validation
Validate RAG pipelines including data ingestion, embeddings, vector search, and retrieval accuracy
Execute functional, non-functional, and usability testing for Gen AI systems
Evaluate model performance using precision, recall, BLEU, ROUGE, and other metrics
Create test plans, test cases, and defect reports for AI applications
Test APIs, microservices, and backend integrations involving AI models




Work closely with data engineers, ML engineers, and developers to improve model output
Perform automation testing using Python, PyTest, or similar frameworks
Validate guardrails, content safety, and ethical AI compliance

Must-Have Skills
Solid experience in AI/ML application testing
Hands-on knowledge of LLMs, Gen AI tools (OpenAI, Azure OpenAI, Gemini, etc.)
Experience with prompt engineering and prompt evaluation
Understanding of RAG architecture, embeddings, vector DBs (FAISS, Pinecone, Chroma)
Solid knowledge of Python for automation/testing
Experience in testing REST APIs
Ability to identify hallucinations, bias, toxicity, and response inconsistencies

Good to Have
Knowledge of Azure ML, AWS Sagemaker, or GCP AI Platform
Experience with MLOps tools
Familiarity with test automation frameworks
Understanding of NLP concepts
AI/ML related certifications

Education
Ba

📌 Gen Ai Tester Bengaluru (India)
🏢 TalentOla
📍 India