Job Title: Gen AI Tester / AI QA Engineer

Experience: 6–10 Years

Location: [Add Location]

Employment Type: Full-time

Job Overview

We are looking for a skilled Gen AI Tester to validate, evaluate, and ensure the quality and reliability of Generative AI applications and LLM-based solutions. The ideal candidate will have experience in AI/ML testing, prompt testing, model performance evaluation, automation, and validating RAG and LLM workflows.

Key Responsibilities

- Test and validate LLM-based applications, Gen AI workflows, and chatbot interactions

- Perform prompt testing, scenario testing, hallucination detection, and response validation

- Validate RAG pipelines including data ingestion, embeddings, vector search, and retrieval accuracy

- Execute functional, non-functional, and usability testing for Gen AI systems

- Evaluate model performance using precision, recall, BLEU, ROUGE, and other metrics

- Create test plans, test cases, and defect reports for AI applications

- Test APIs, microservices, and backend integrations involving AI models

- Work closely with data engineers, ML engineers,



and developers to improve model output

- Perform automation testing using Python, PyTest, or similar frameworks

- Validate guardrails, content safety, and ethical AI compliance

Must-Have Skills

- Strong experience in AI/ML application testing

- Hands-on knowledge of LLMs, Gen AI tools (OpenAI, Azure OpenAI, Gemini, etc.)

- Experience with prompt engineering and prompt evaluation

- Understanding of RAG architecture, embeddings, vector DBs (FAISS, Pinecone, Chroma)

- Solid knowledge of Python for automation/testing

- Experience in testing REST APIs

- Ability to identify hallucinations, bias, toxicity, and response inconsistencies

Good to Have

- Knowledge of Azure ML, AWS Sagemaker, or GCP AI Platform

- Experience with MLOps tools

- Familiarity with test automation frameworks

- Understanding of NLP concepts

- AI/ML related certifications

Education

- Bachelor's degree in Computer Science, IT, or related field

📌 Gen AI Tester (Bengaluru)
🏢 TalentOla
📍 Bengaluru