The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows.

The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.

Job Responsibilities

Write automation alongside development to test unit level, API and e2e UI
Implement backend services supporting agent runtimes, tool execution, and memory layers
Integrate LLMs into business workflows and internal systems
Contribute to orchestration logic connecting tools, APIs, and external services
Write clean, maintainable, well-tested code
Participate in architectural discussions
Collaborate with AI engineers to productionize experimental features
Contribute to monitoring, logging, and performance optimization

Requirements

Robust hands-on experience testing LLM-based or agentic AI applications




Expertise in Python automation (PyTest, Playwright, Postman)
Understanding of prompts, embeddings, vector stores, tokenization
Experience testing RAG pipelines and grounding data workflows
Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP)
Ability to evaluate unstructured AI output using structured scoring techniques
Strong analytical and debugging skills in complex AI-driven workflows

Desirable Skills

Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.;
Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI
Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming
Knowledge of model fine-tuning, RLHF, and observability pipelines
Exposure to distributed architecture and enterprise platforms
Ability to mentor QA engineers on AI testing methods

Qualifications

5+ years of professional software engineering experience
Hands-on wo

📌 Senior Test Analyst (Mumbai)
🏢 SmartStream
📍 Mumbai