Role & responsibilities

We are seeking an accomplished Test Architect with deep expertise in AI quality engineering and agentic AI systems. In this role you will own the end-to-end quality strategy for AI-powered products from LLM evaluation frameworks and multi-agent orchestration testing to scalable test infrastructure. You will operate at the intersection of software architecture, AI/ML, and enterprise quality engineering, shaping both the tools teams use and the standards they follow.

Architecture & Strategy

- Design and own the enterprise-wide Test Architecture for AI products including LLM-based applications, RAG pipelines, and autonomous agentic systems.
- Define quality gates, evaluation metrics, and acceptance criteria aligned with AI model behaviour, safety, and regulatory requirements.
- Architect scalable, cloud-native test infrastructure on Azure / AWS / GCP supporting continuous evaluation of non-deterministic AI outputs.
- Lead the selection and integration of AI testing toolchains into CI/CD pipelines (e.g. Playwright, Cypress, LangSmith, Evals frameworks).
-

Agentic AI & LLM Testing

- Build evaluation harnesses for multi-agent systems — covering agent reasoning chains, tool-use accuracy, memory recall, and goal completion rates.
- Design test strategies for agentic workflows built on frameworks such as AutoGen,



LangGraph, CrewAI, or custom orchestration layers.
- Develop adversarial testing suites: prompt injection attacks, jailbreak probes, hallucination detection, and bias evaluation.
- Establish ground-truth benchmarking datasets and model regression pipelines to track LLM quality across fine-tuning cycles.

Product & Platform Engineering

- Drive design and delivery of internal testing products: AI Eval Platforms, automated quality dashboards, and AI observability tooling.
- Contribute to the product roadmap for quality tooling — writing PRDs, shaping backlogs, and aligning engineering with business outcomes.
- Integrate performance testing strategies (K6, JMeter) for AI API endpoints, latency SLAs, and throughput benchmarks.

Leadership & Enablement

- Lead, mentor, and grow a team of 8–15 QE engineers across automation, AI eval, and performance disciplines.
- Establish CoE standards: coding conventions, test design patterns (POM, BDD), and documentation norms across QE squads.
- Partner with AI Product Managers, Data Scientists, and MLOps engineers to co-own quality across the full model lifecycle.
- Present architecture decisions and quality health metrics to senior leadership and external stakeholders.

📌 AI Test Architect (Hyderabad)
🏢 Centific
📍 Hyderabad