We are looking an experienced AI Test Engineer to drive quality assurance for AI-powered applications, including LLM models, conversational bots, autonomous agents, and MCP-enabled systems. The ideal candidate will have strong expertise in Playwright with .NET, API/UI automation, AI/LLM validation, and CI/CD automation. The role involves building scalable test frameworks, validating AI responses, testing agent workflows, and ensuring the reliability, security, and performance of GenAI solutions.

Role :AI Test Engineer

Location: PAN India

Experience Range: 5 to 12years

Key Responsibilities

- Design and execute test strategies for LLM-powered applications, AI chatbots, conversational AI, and autonomous agents.
- Develop and maintain Playwright (.NET) automation frameworks for UI and API testing.
- Validate AI model outputs for:
- Accuracy
- Relevance
- Hallucinations
- Consistency
- Safety and Guardrails
- Test MCP (Model Context Protocol) endpoints and MCP-enabled agent workflows.
- Automate API testing and integrate test suites into CI/CD pipelines.
- Create test datasets, evaluation metrics, and AI validation frameworks.




- Perform regression, functional, integration, and end-to-end testing for AI systems.
- Collaborate with developers, data scientists, and product teams to ensure quality delivery.
- Monitor and troubleshoot AI system failures, agent interactions, and model behavior.
- Implement quality gates within Azure DevOps/GitHub Actions/Jenkins pipelines.

Required Skills:

AI / GenAI Testing

- LLM Testing
- Prompt Validation
- AI Response Validation
- Conversational Bot Testing
- Autonomous Agent Testing
- MCP (Model Context Protocol)
- Hallucination Detection
- AI Safety & Guardrail Testing
- RAG Validation

Automation Testing
- Playwright with .NET (C#)
- UI Automation
- API Automation
- Postman / REST APIs
- Selenium (Valuable to Have)

Programming
- C#
- .NET Core / ASP.NET
- JavaScript / TypeScript (Good to Have)

DevOps & CI/CD
- Azure DevOps
- GitHub Actions
- Jenkins
- Docker (Preferred)

Other Skills
- Agile/Scrum
- SQL Databases
- Test Framework Design
- Version Control (Git)

📌 Software Engineer (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad