Were looking for an AI Agent Quality Analyst to monitor and evaluate the performance of our AI agents in production. This role identifies recurring issues and failure patterns, diagnoses likely causes, and works closely with product and engineering to drive improvements. It sits at the intersection of product and engineering, translating technical findings into clear, actionable insight for both sides.

Key Responsibilities

- Continuously review AI agent outputs to identify issues, with a focus on recurring patterns rather than isolated incidents
- Investigate outputs, logs, and correction history to determine where and why agents underperform
- Use Python to extract, organize, and analyze the data needed to support findings
- Communicate issues and patterns clearly to both technical and non technical stakeholders
- Recommend likely root causes and directions for improvement
- Maintain an ongoing record of issues and patterns to track trends over time
- Partner with the product team to understand intended agent behavior and business context
- Work with engineering to ensure findings are well understood and actionable
- Distinguish between data or coverage gaps and genuine model or logic failures

Required Skills Experience

- Strong analytical skills with demonstrated ability to identify patterns in data
- Working proficiency in Python for data extraction and analysis (e.g. pandas, basic scripting against APIs or databases)
- Comfortable working with structured data from a relational database (PostgreSQL or similar)
- Excellent written and verbal communication skills, with the ability to translate technical findings for non technical audiences
- Strong cross functional collaboration skills
- Disciplined and consistent documentation habits

Nice to Have





- Understanding of RAG (retrieval augmented generation) concepts, including retrieval and ranking
- Prior experience reviewing AI or LLM generated content

What Success Looks Like

- A clear, current picture of common failure patterns across AI agents
- Faster diagnosis-to-fix cycles for engineering
- Clear visibility for product into data gaps versus model behavior issues
- A reliable historical record of issue trends over time

Who we are:

At CIED Group, we are on a mission to make the global food supply chain more efficient, intelligent and transparent. We design technologies that build data trust and drive productivity across the agri-food supply chain, from the farm to the consumers plate.

Our flagship product, RightOrigins, is an agentic AI platform that drives automation throughout the supply chain workflows. We also provide traceability services and build custom audit management tools for food certification bodies, simplifying regulatory compliance and audit inspection processes with a digital, collaborative approach.

At CIED Group, every idea is built around one core belief: when data drives credibility, trust follows naturally.

What we offer:

- Friendly environment
- Chance to own responsibility and demonstrate accountability
- Chance to develop creative skills ideas
- International horizons - work with international teams and clients from all corners of the world
- Company parties, team events and the chance to work with great colleagues from all over the world
- Medical Insurance

Industry: IT

Employment Type: Full-time

Location: Currently Remote/Hybrid

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 AI Agent Quality Analyst (Kochi)
🏢 CIED Technologies
📍 Kochi