Testing Specialist - LLM Evaluation & Quality

Design and implement model evaluation frameworks using RAGAS, custom scorers, and automated accuracy benchmarking to validate agent outputs against structured specifications and Gherkin acceptance criteria.

Develop adversarial prompt testing suites that probe agent behaviour for hallucinations, prompt injection vulnerabilities, and edge-case failures across the

Refinement, Decision, and Coding Agents.

Build and maintain regression test suites that continuously verify agent behaviour consistency across model updates, prompt changes, and skill-file modifications within the LangGraph-based architecture.

Integrate AI-specific test tooling into CI/CD pipelines using GitHub Actions,

ensuring every agent deployment is gated by automated quality, fairness, and reliability checks aligned with EU AI Act and DORA requirements.

Define and track quantitative quality metrics, confidence thresholds, retrieval precision, response latency, and drift detection, providing actionable dashboards to the engineering team.

Collaborate closely with AI/ML Engineers and DevSecOps teams in Germany and Hyderabad to establish testing standards, review evaluation results,



and continuously raise the quality bar for production agent releases.

Bachelor's degree in Computer Science, Engineering, or a related field, with demonstrated experience in testing AI/ML systems or LLM-based applications.

Robust programming skills in Python with hands-on proficiency in pytest, test automation frameworks, and scripting for evaluation pipelines.

Practical experience with LLM evaluation methodologies and tools such as

RAGAS, custom scoring functions, accuracy benchmarking, and adversarial/red

team testing techniques.

Solid understanding of CI/CD integration using GitHub Actions or comparable platforms, with the ability to embed AI test gates into automated deployment workflows.

Familiarity with agentic architectures such as LangGraph or LangChain, including knowledge of retrieval-augmented generation (RAG) patterns and multi-agent orchestration concepts. Nice-to-have: Experience with regulatory compliance testing (EU AI Act, DORA),

familiarity with vector database validation and containerization

(Docker/Kubernetes).

📌 AI Testing Specialist LLM Evaluation & Qualit (Hyderabad)
🏢 Hucon Solutions
📍 Hyderabad