Role Overview

We are seeking a Senior LLM/GenAI Testing Engineer to ensure the quality, reliability, and safety of AI-powered applications such as chatbots, copilots, RAG systems, and AI agents. The role involves designing test strategies, building automated evaluation frameworks, validating AI outputs, and ensuring production readiness.

Key Responsibilities
- Test and evaluate LLM/GenAI applications for accuracy, relevance, hallucinations, and instruction-following.
- Develop automated evaluation frameworks using Python and GenAI testing tools.
- Validate RAG systems, chatbots, and agent-based workflows.
- Perform AI security, safety, and prompt injection testing.
- Test API performance, latency, and response quality.
- Create benchmark datasets, regression suites, and quality metrics.
- Collaborate with AI, Product,



and Engineering teams to define release quality gates.

Required Skills
- 5+ years in QA/Test Automation; 2+ years in LLM/GenAI testing.
- Robust Python, PyTest, API Testing, and SQL skills.
- Hands-on experience with LLMs, GPT, GenAI, RAG, Chatbots, or Conversational AI.
- Experience with evaluation tools such as DeepEval, Ragas, Promptfoo, LangSmith, or similar.
- Knowledge of CI/CD, test automation, and quality assurance best practices.

Preferred Skills
- Experience with AI security testing, red teaming, and OWASP LLM.
- Knowledge of vector databases, embeddings, and Responsible AI practices.
- Exposure to prompt engineering and GenAI evaluation methodologies.

📌 Senior LLM/ Gen AI Test Engineer (Hyderabad)
🏢 The Ksquare Group
📍 Hyderabad