PFB JD for AI Test Engineers

Key Responsibilities
AI Response Evaluation & Validation
• Evaluate AI/LLM responses using multiple approaches including:
• Human Evaluation
• Automated Evaluation
• LLM‑as‑a‑Judge techniques
• Validate AI outputs for accuracy, relevance, grounding, consistency, and completeness.
• Design and execute test scenarios for RAG‑based applications, ensuring response faithfulness and reduced hallucinations.

AI Evaluation Frameworks & Tooling
• Apply or integrate AI evaluation frameworks such as:
• RAGAS
• LangSmith (valuable to have)
• Define evaluation metrics (precision, recall, faithfulness, toxicity, bias, etc.) and analyze results.
• Collaborate with engineering teams to embed AI evaluation into CI/CD pipelines where applicable.

Responsible AI & Governance
• Ensure compliance with Responsible AI guidelines, including:
• Fairness
• Bias detection
• Safety
• Transparency
• Validate system instructions, prompt templates, and guardrails.
• Support system instruction extraction, prompt testing, and jailbreak prevention strategies.





Test Data & Dataset Management
• Understand business and functional requirements to:
• Design and curate high‑quality datasets (golden datasets, adversarial datasets, edge‑case datasets).
• Prepare datasets for training validation, evaluation, and regression testing.
• Maintain dataset versioning and traceability.

Collaboration & Quality Advocacy
• Work closely with AI engineers, product owners, and domain SMEs to understand AI behavior and risks.
• Provide actionable insights on AI quality issues, improvements, and risks.
• Contribute to AI testing best practices, standards, and reusable assets.

Required Skills & Qualifications
Mandatory Skills
• Strong understanding of AI/LLM testing concepts and AI quality assurance.
• Experience in AI response evaluation techniques (human, automated, LLM‑as‑judge).
• Ability to analyze AI outputs and identify:
• Hallucinations
• Bias
• Inconsistencies
• Hands‑on experience in te

📌 AI Test Engineer (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru