Role Description

We are looking for an experienced QA Engineer – Generative AI & Agentic AI to lead quality assurance activities for GenAI and Agentic AI solutions.

This role goes beyond traditional functional testing. The successful candidate will be responsible for validating the correctness, reliability, quality, and production-readiness of AI-generated outputs, including multi-step agent workflows, Text-to-SQL pipelines, Retrieval-Augmented Generation (RAG) systems, and LLM-based applications.

The ideal candidate will have hands-on, production-grade experience testing GenAI/Agentic AI applications end-to-end and should have successfully supported at least one production release of a GenAI or Agentic AI solution.

Key Responsibilities

• Design, develop, and execute comprehensive test strategies for GenAI and Agentic AI applications, covering functional, non-functional, integration, and AI output-quality testing.
• Test multi-step agentic workflows, validating agent orchestration, tool/function calls, task completion, chained decision-making, reasoning consistency,



and outputs across individual agent hops.
• Evaluate AI-generated outputs for correctness, relevance, coherence, hallucination, consistency, and reliability.
• Test Text-to-SQL workflows, including validation of:
• Natural language to SQL generation accuracy
• SQL syntax and logic
• Schema/table/column alignment
• Query execution
• Result accuracy against Snowflake data models

• Validate RAG pipelines, including:
• Retrieval relevance
• Context grounding
• Chunking effectiveness
• Context utilization
• Faithfulness of generated responses to retrieved information

• Apply GenAI evaluation metrics and methodologies such as:
• Embedding similarity
• ROUGE
• BLEU
• LLM-as-a-Judge
• Other relevant GenAI quality and accuracy metrics

• Design and maintain LLM-as-a-Judge evaluation frameworks and scoring rubrics for scalable and automated assessment of GenAI outputs.
• Create and maintain golden datasets,

📌 QA Engineer – Generative AI (Thiruvananthapuram)
🏢 UST
📍 Thiruvananthapuram