Architecture &
Strategy
Design and own the enterprise-wide Test Architecture for AI products including LLM-based applications, RAG pipelines, and autonomous agentic systems.
Define quality gates, evaluation metrics, and acceptance criteria aligned with AI model behaviour, safety, and regulatory requirements.
Architect scalable, cloud-native test infrastructure on Azure / AWS / GCP supporting continuous evaluation of non-deterministic AI outputs.
Lead the selection and integration of AI testing toolchains into CI/CD pipelines (e.g. Playwright, Cypress, LangSmith, Evals frameworks).

Agentic AI &
LLM Testing
Build evaluation harnesses for multi-agent systems — covering agent reasoning chains, tool-use accuracy, memory recall, and goal completion rates.
Design test strategies for agentic workflows built on frameworks such as AutoGen, LangGraph, CrewAI, or custom orchestration layers.
Develop adversarial testing suites: prompt injection attacks, jailbreak probes, hallucination detection, and bias evaluation.




Establish ground-truth benchmarking datasets and model regression pipelines to track LLM quality across fine-tuning cycles.

Product &
Platform Engineering
Drive design and delivery of internal testing products: AI Eval Platforms, automated quality dashboards, and AI observability tooling.
Contribute to the product roadmap for quality tooling — writing PRDs, shaping backlogs, and aligning engineering with business outcomes.
Integrate performance testing strategies (K6, JMeter) for AI API endpoints, latency SLAs, and throughput benchmarks.

Leadership &
Enablement
Lead, mentor, and grow a team of 8–15 QE engineers across automation, AI eval, and performance disciplines.
Establish CoE standards: coding conventions, test design patterns (POM, BDD), and documentation norms across QE squads.
Partner with AI Product Managers, Data Scientists, and MLOps engineers to co-own quality across the full model lifecycle.
Present architecture decisions and quality health metrics to senior leadership and external stakeholders.

📌 Test Automation Architect Hyderabad
🏢 Centific
📍 Hyderabad