Role Overview

We are seeking a Senior LLM/GenAI Testing Engineer to ensure the quality, reliability, and safety of AI-powered applications such as chatbots, copilots, RAG systems, and AI agents. The role involves designing test strategies, building automated evaluation frameworks, validating AI outputs, and ensuring production readiness.

Key Responsibilities
Test and evaluate LLM/GenAI applications for accuracy, relevance, hallucinations, and instruction-following.
Develop automated evaluation frameworks using Python and GenAI testing tools.
Validate RAG systems, chatbots, and agent-based workflows.
Perform AI security, safety, and prompt injection testing.
Test API performance, latency, and response quality.
Create benchmark datasets, regression suites, and quality metrics.
Collaborate with AI, Product,



and Engineering teams to define release quality gates.

Required Skills
5+ years in QA/Test Automation; 2+ years in LLM/GenAI testing.
Robust Python, PyTest, API Testing, and SQL skills.
Hands-on experience with LLMs, GPT, GenAI, RAG, Chatbots, or Conversational AI.
Experience with evaluation tools such as DeepEval, Ragas, Promptfoo, LangSmith, or similar.
Knowledge of CI/CD, test automation, and quality assurance best practices.

Preferred Skills
Experience with AI security testing, red teaming, and OWASP LLM.
Knowledge of vector databases, embeddings, and Responsible AI practices.
Exposure to prompt engineering and GenAI evaluation methodologies.

📌 Senior Llm/ Gen Ai Test Engineer Hyderabad (India)
🏢 The Ksquare Group
📍 India