AI / Quality Assurance Engineer

Experience: 4-7 Years

Role Type: QA Engineering / Test Automation / AI Quality

Primary Stack: Azure | .NET APIs | Next.js | Postman | GitHub | Docker | AI/LLM Testing

Position Overview

We are seeking a hands-on AI / Quality Assurance Engineer to test modern cloud applications and AI-enabled solutions built on Microsoft Azure. The ideal candidate will have robust experience in functional, API, integration, regression, and automated testing, along with an understanding of AI/LLM applications. The role will validate traditional application functionality as well as AI agents, RAG, MCP integrations, model responses, security, and end-to-end AI workflows.

Key Responsibilities

Application & API Testing

· Develop and execute functional, regression, integration, API, and end-to-end test scenarios.

· Test C#/.NET ASP.NET Core APIs using Postman and automated testing frameworks.

· Validate Next.js/TypeScript application workflows.

· Create test plans, test cases, test data, and acceptance criteria.

· Identify, document, track, and validate defects through resolution.

· Validate authentication, authorization, error handling, API contracts, and enterprise integrations.

· Support UAT, release validation, and production-readiness testing.

AI & Agent Testing

· Test Azure OpenAI/OpenAI applications, AI agents, multi-step workflows, and tool/function calling.

· Test Model Context Protocol (MCP) client/server interactions, tool execution, authorization, and failure handling.

· Validate RAG pipelines including ingestion, retrieval, filtering, grounding, and citations.

· Test structured AI outputs and response schemas.





· Create AI evaluation datasets and regression scenarios.

· Evaluate response accuracy, groundedness, relevance, citation quality, latency, and reliability.

· Test negative scenarios including prompt injection, unauthorized tool access, hallucination, and sensitive-data exposure.

Automation & DevOps

· Develop automated API, integration, regression, and UI tests.

· Integrate automated tests and AI evaluations into GitHub CI/CD pipelines.

· Work within GitHub Flow, pull-request, code-review, and tagged-release processes.

· Test applications in Docker-based environments using WSL and VS Code Dev Containers.

· Validate deployments on Azure App Service and future Azure Container Apps environments.

· Use Claude Code Enterprise appropriately to improve test development, analysis, and documentation.

· Collaborate with developers, architects, and program teams using Microsoft Teams.

Required Qualifications

· 4-7 years of software QA / quality engineering experience.

· Strong functional, regression, integration, and end-to-end testing experience.

· Strong REST API testing experience and hands-on experience with Postman.

· Experience with automated testing using C#, TypeScript/JavaScript, Python, Playwright, Selenium, or comparable frameworks.





· Experience testing modern web applications and APIs.

· Experience with Git/GitHub and CI/CD.

· Understanding of Docker and containerized applications.

· Strong debugging, defect analysis, and problem-solving skills.

Preferred AI Skills

· Azure OpenAI / OpenAI application testing.

· AI agents and agentic workflow testing.

· Model Context Protocol (MCP) testing.

· RAG and retrieval evaluation.

· Prompt and model-response testing, groundedness, and citation validation.

· AI evaluation frameworks and test datasets.

· Prompt-injection and AI security testing.

· Azure AI Search and Microsoft Foundry evaluation capabilities.

Technology Environment

Area

Technology

Cloud / Hosting

Azure / Azure App Service / Azure Container Apps

Backend

C# / .NET / ASP.NET Core Minimal API

Frontend

Next.js / TypeScript / pnpm

AI

Azure OpenAI / AI Agents / MCP / RAG

API Testing

Postman

Test Automation

Playwright / Selenium / C# / TypeScript / Python

Containers / Development

Docker Desktop / WSL / VS Code Dev Containers / Windows 365

DevOps

GitHub / GitHub Flow / CI/CD / Tagged Releases

Tools

Claude Code Enterprise / Microsoft Teams

Ideal Candidate The ideal candidate understands that testing AI applications goes beyond deterministic application behavior. They should be comfortable validating the complete flow: Next.js -> .NET APIs -> AI Agent -> Azure OpenAI -> MCP/RAG -> Enterprise Systems, and testing both software correctness and AI quality, including accuracy, grounding, citations, security, reliability, and failure scenarios.

📌 AI Quality Assurance Engineer (Pune)
🏢 Palo Alto Labs
📍 Pune