Experience: 8+ years IT (solid SDET foundation) + 3+ years dedicated AI, LLM, and Agentic system evaluation.

Experience in a product-led organization or AI research lab is preferred.Role Overview A strategic role driving AI, Advanced Quality Engineering, and Thought Leadership. Define evaluation models for LLM and Agentic workflow safety and reliability, combining traditional QA foundations with next-gen AI testing paradigms to lead niche skills groups.

Key Responsibilities

- Frameworks &
- Strategy: Build automated evaluation frameworks for LLMs, RAG pipelines, and Agentic workflows. Implement "LLM-as-a-judge" benchmarks, HITL, and tools like LangSmith, TruLens, DeepEval, Phoenix, and Ragas into CI/CD.
- Cloud &
- Agentic Infra: Deploy agentic frameworks (Perception/Reasoning/Action) in AWS, GCP, and Azure using containerization, service governance, and cloud orchestration.
- AI Evaluation: Validate functional output quality (hallucinations, relevance, tool-calling precision)



and non-functional vectors (red-teaming, prompt injection, jailbreaking, toxicity, latency, cost).
- Upskilling &
- Presales: Mentor QA/SDETs into niche AI testing specialists. Support presales and drive market authority through articles and open-source benchmarks.

Required Skills &

- Qualifications

- Core SDET Mastery: Expert-level Java, Python, and Node.JS scripting. Advanced hands-on automation with Selenium, Playwright, RestAssured, Mobile testing (iOS/Android), and strong Manual testing foundations.
- AI/ML Expertise: Transformer architectures, Vector DBs (Pinecone, Milvus), fine-tuning, and frameworks like LangChain, LlamaIndex, or AutoGen.
- Metrics &
- MLOps: Experience with BLEU, ROUGE, G-Eval, BERTScore, Context Precision, and Faithfulness. Familiar with MLflow, Weights &
- Biases, and cloud infrastructure-as-code.

📌 Lead Software Engineer (Chennai)
🏢 Virtusa
📍 Chennai