Responsibilities

Testing Strategy & Design
- Define testing strategies for AI agents, conversational interfaces, and agentic workflows
- Design behavioral test suites for non-deterministic outputs—where "correct" isn't binary
- Build evaluation frameworks that measure groundedness, factuality, relevance, and task completion
- Identify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, drift
- Develop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows

Test Automation & Infrastructure
- Build automated evaluation pipelines that run as part of CI/CD
- Create test harnesses for LLM-based systems—mocking, fixtures, and reproducible test scenarios
- Develop regression suites that detect quality degradation when prompts, models, or data change
- Build monitoring and alerting for production agent quality (accuracy, latency, error rates)
- Maintain test infrastructure: test data management, workplace setup, reporting dashboards

Evaluation & Metrics




- Define quality metrics for each solution—what to measure and what thresholds matter
- Build and maintain evaluation datasets (ground truth, reference outputs, edge case collections)
- Conduct systematic prompt evaluation when prompts or models change
- Track quality trends over time and identify when re-evaluation is needed
- Report quality metrics to the team and stakeholders in clear, actionable terms

Collaboration & Quality Culture
- Partner with AI Solutions Engineers to define testability requirements during design
- Work with AI Solutions Analysts to translate acceptance criteria into test scenarios
- Review solution designs from a quality and testability perspective
- Advocate for quality practices across the team—testing isn't an afterthought, it's part of delivery
- Contribute to incident response by diagnosing quality failures and building regression tests

Qualifications

Required
- 3–7 years of softwar

📌 AI/ML Quality Engineer (Bengaluru)
🏢 Allegis Global Solutions (AGS)
📍 Bengaluru