Job Title: Gen AI Tester / AI QA Engineer

Experience: 6–10 Years

Location: [Add Location]

Employment Type: Full-time

Job Overview

We are looking for a skilled Gen AI Tester to validate, evaluate, and ensure the quality and reliability of Generative AI applications and LLM-based solutions. The ideal candidate will have experience in AI/ML testing, prompt testing, model performance evaluation, automation, and validating RAG and LLM workflows.

Key Responsibilities

-
Test and validate LLM-based applications, Gen AI workflows, and chatbot interactions

-
Perform prompt testing, scenario testing, hallucination detection, and response validation

-
Validate RAG pipelines including data ingestion, embeddings, vector search, and retrieval accuracy

-
Execute functional, non-functional, and usability testing for Gen AI systems

-
Evaluate model performance using precision, recall, BLEU, ROUGE, and other metrics

-
Create test plans, test cases, and defect reports for AI applications

-
Test APIs, microservices, and backend integrations involving AI models

-




Work closely with data engineers, ML engineers, and developers to improve model output

-
Perform automation testing using Python, PyTest, or similar frameworks

-
Validate guardrails, content safety, and ethical AI compliance

Must-Have Skills

-
Solid experience in AI/ML application testing

-
Hands-on knowledge of LLMs, Gen AI tools (OpenAI, Azure OpenAI, Gemini, etc.)

-
Experience with prompt engineering and prompt evaluation

-
Understanding of RAG architecture, embeddings, vector DBs (FAISS, Pinecone, Chroma)

-
Strong knowledge of Python for automation/testing

-
Experience in testing REST APIs

-
Ability to identify hallucinations, bias, toxicity, and response inconsistencies

Good to Have

-
Knowledge of Azure ML, AWS Sagemaker, or GCP AI Platform

-
Experience with MLOps tools

-
Familiarity with test automation frameworks

-
Understanding of NLP concepts

-
AI/ML related certifications

Education

-
Ba

📌 Gen AI Tester (Bengaluru)
🏢 TalentOla
📍 Bengaluru