About the Role

We're looking for an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You'll design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

Responsibilities
Design and execute test plans for LLM, RAG, and agentic workflows
Build evaluation datasets (golden sets, edge cases, adversarial prompts)
Assess output quality: accuracy, relevance, hallucination, tone, and consistency
Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing
Validate tool-calling, multi-step agent behavior, and guardrails
Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses)
Track metrics such as latency, cost per request, and failure rates
Log, triage, and reproduce defects with transparent prompts, inputs, and model versions
Partner with ML engineers, product owners,



and developers to improve prompts and models
Document findings and contribute to release go/no-go decisions

Required Qualifications
4+ years in QA, test automation, or data/ML evaluation
Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings)
Experience testing APIs and writing automated tests (Python preferred)
Solid analytical skills and attention to detail
Transparent written communication for documenting non-deterministic behavior

Preferred Qualifications
Experience with evaluation tools (e.g.,LangSmith, Ragas, DeepEval)
Familiarity with RAG pipelines, vector databases, and agent frameworks
Knowledge of AI safety, responsible AI, and compliance practices
Cloud platform experience (Azure)
Background in test automation frameworks

📌 Ai Test Engineer Llm And Ai Qa Ahmedabad (India)
🏢 Synoptek
📍 India