Description
The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows.

The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.

Job Responsibilities
- Write automation alongside development to test unit level, API and e2e UI.
- Implement backend services supporting agent runtimes, tool execution, and memory layers
- Integrate LLMs into business workflows and internal systems
- Contribute to orchestration logic connecting tools, APIs, and external services
- Write clean, maintainable, well-tested code
- Participate in architectural discussions
- Collaborate with AI engineers to productionize experimental features
- Contribute to monitoring, logging, and performance optimization

Requirements
- Robust hands-on experience testing LLM-based or agentic AI applications.




- Expertise in Python automation (PyTest, Playwright, Postman).
- Understanding of prompts, embeddings, vector stores, tokenization. • Experience testing RAG pipelines and grounding data workflows.
- Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP).
- Ability to evaluate unstructured AI output using structured scoring techniques.
- Strong analytical and debugging skills in complex AI-driven workflows.

Desirable Skills
- Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.;
- Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI.
- Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming.
- Knowledge of model fine-tuning, RLHF, and observability pipelines.
- Exposure to distributed architecture and enterprise platforms.
- Ability to mentor QA engineers on AI testing methods.

Qualifications
- 5+

📌 Senior Test Analyst (Mumbai)
🏢 SmartStream
📍 Mumbai