About the Role

We're looking for an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You'll design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

Responsibilities

- Design and execute test plans for LLM, RAG, and agentic workflows
- Build evaluation datasets (golden sets, edge cases, adversarial prompts)
- Assess output quality: accuracy, relevance, hallucination, tone, and consistency
- Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing
- Validate tool-calling, multi-step agent behavior, and guardrails
- Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses)
- Track metrics such as latency, cost per request, and failure rates
- Log, triage, and reproduce defects with clear prompts, inputs, and model versions
- Partner with ML engineers, product owners,



and developers to improve prompts and models
- Document findings and contribute to release go/no-go decisions

Required Qualifications

- 4+ years in QA, test automation, or data/ML evaluation
- Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings)
- Experience testing APIs and writing automated tests (Python preferred)
- Strong analytical skills and attention to detail
- Transparent written communication for documenting non-deterministic behavior

Preferred Qualifications

- Experience with evaluation tools (e.g.,LangSmith, Ragas, DeepEval)
- Familiarity with RAG pipelines, vector databases, and agent frameworks
- Knowledge of AI safety, responsible AI, and compliance practices

- Cloud platform experience (Azure)
- Background in test automation frameworks

📌 AI Test Engineer (LLM and AI QA) (Ahmedabad)
🏢 Synoptek
📍 Ahmedabad