Job Title: Senior QA Automation Engineer - AI &

- LLM Evaluation

Location: HYD Campus, Hyderabad

Years of Experience: 10-15 Years

Job Summary: We are seeking a highly experienced Senior QA Automation Engineer with a strong foundation in Software Development Engineering in Test (SDET) and extensive experience in AI, LLM, and Agentic system evaluation. This strategic role will drive AI initiatives, Advanced Quality Engineering, and Thought Leadership within our organization, focusing on the development of evaluation models for LLM and Agentic workflows.

Responsibilities

- Frameworks &
- Strategy: Build automated evaluation frameworks for LLMs, RAG pipelines, and Agentic workflows. Implement benchmarks such as 'LLM-as-a-judge', HITL, and tools like LangSmith, TruLens, DeepEval, Phoenix, and Ragas into CI/CD processes.
- Cloud &
- Agentic Infra: Deploy agentic frameworks (Perception/Reasoning/Action) in AWS, GCP, and Azure utilizing containerization, service governance, and cloud orchestration.
- AI Evaluation:



Validate functional output quality including hallucinations, relevance, and tool-calling precision, as well as non-functional vectors such as red-teaming, prompt injection, jailbreaking, toxicity, latency, and cost.
- Upskilling &
- Presales: Mentor QA/SDETs into niche AI testing specialists. Support presales efforts and enhance market authority through articles and open-source benchmarks.

Mandatory Skills

- 8+ years of IT experience with a strong SDET foundation.
- 3+ years of dedicated experience in AI, LLM, and Agentic system evaluation.
- Proficiency in AI/ML technologies.

Preferred Skills

- Experience in a product-led organization or AI research lab.
- Familiarity with advanced testing paradigms and tools.

Qualifications

- Bachelor's degree in Computer Science, Engineering, or a related field.
- Robust analytical and problem-solving skills.
- Excellent communication and leadership abilities.

📌 Architect (Hyderabad)
🏢 Virtusa
📍 Hyderabad