We are seeking an experienced AI Test Engineer with 6+ years of overall testing experience and a minimum of 2 years of hands-on experience in AI/GenAI testing. The ideal candidate will be responsible for validating AI-powered applications, evaluating LLM responses, testing RAG-based solutions, ensuring Responsible AI compliance, and establishing AI quality assurance processes.

Role- AI Test Engineer

IT Exp- 6+

Location- Hyderabad

Key Responsibilities

AI Response Evaluation & Validation
- Evaluate AI/LLM responses using:
- Human Evaluation
- Automated Evaluation
- LLM-as-a-Judge techniques
- Validate AI outputs for:
- Accuracy
- Relevance
- Grounding
- Consistency
- Completeness
- Design and execute test scenarios for RAG-based applications.
- Identify and analyze:
- Hallucinations
- Bias
- Toxicity
- Inconsistent responses
- Perform functional, regression, and validation testing for AI features and AI-powered applications.

AI Evaluation Frameworks & Tooling
- Apply and utilize AI evaluation frameworks such as:
- RAGAS
- LangSmith (Positive to Have)




- Similar AI evaluation and observability tools
- Define and monitor AI quality metrics including:
- Precision
- Recall
- Faithfulness
- Relevancy
- Toxicity
- Bias
- Support integration of AI evaluation processes into CI/CD pipelines.

Responsible AI & Governance
- Validate Responsible AI principles including:
- Fairness
- Safety
- Transparency
- Bias Detection
- Test system prompts, guardrails, and prompt templates.
- Execute prompt validation and jailbreak testing.
- Ensure compliance with organizational AI governance standards.

Test Data & Dataset Management
- Design and curate:
- Golden Datasets
- Adversarial Datasets
- Edge-case Datasets
- Prepare datasets for:
- Model Validation
- AI Evaluation
- Regression Testing
- Maintain dataset traceability and version control.

Collaboration & Quality Assurance
- Collaborate with AI Engineers, Product Owners, Business Teams, and SMEs.
- Analyze AI quality issues and provi

📌 Walk-in || AI Test Engineer (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad