We are seeking an experienced AI Test Engineer with 6+ years of overall testing experience and a minimum of 2 years of hands-on experience in AI/GenAI testing. The ideal candidate will be responsible for validating AI-powered applications, evaluating LLM responses, testing RAG-based solutions, ensuring Responsible AI compliance, and establishing AI quality assurance processes.

Role- AI Test Engineer

IT Exp- 6+

Location- Hyderabad

Key Responsibilities

AI Response Evaluation & Validation
Evaluate AI/LLM responses using:
Human Evaluation
Automated Evaluation
LLM-as-a-Judge techniques
Validate AI outputs for:
Accuracy
Relevance
Grounding
Consistency
Completeness
Design and execute test scenarios for RAG-based applications.
Identify and analyze:
Hallucinations
Bias
Toxicity
Inconsistent responses
Perform functional, regression, and validation testing for AI features and AI-powered applications.

AI Evaluation Frameworks & Tooling
Apply and utilize AI evaluation frameworks such as:
RAGAS
LangSmith (Positive to Have)




Similar AI evaluation and observability tools
Define and monitor AI quality metrics including:
Precision
Recall
Faithfulness
Relevancy
Toxicity
Bias
Support integration of AI evaluation processes into CI/CD pipelines.

Responsible AI & Governance
Validate Responsible AI principles including:
Fairness
Safety
Transparency
Bias Detection
Test system prompts, guardrails, and prompt templates.
Execute prompt validation and jailbreak testing.
Ensure compliance with organizational AI governance standards.

Test Data & Dataset Management
Design and curate:
Golden Datasets
Adversarial Datasets
Edge-case Datasets
Prepare datasets for:
Model Validation
AI Evaluation
Regression Testing
Maintain dataset traceability and version control.

Collaboration & Quality Assurance
Collaborate with AI Engineers, Product Owners, Business Teams, and SMEs.
Analyze AI quality issues and provi

📌 Walk In Ai Test Engineer Hyderabad
🏢 Tata Consultancy Services
📍 Hyderabad