Roles & Responsibilities

- Design and develop Agentic AI / GenAI applications and autonomous workflows.
- Build and optimize agent orchestration, reasoning, planning, memory and context-management systems.
- Develop RAG pipelines, embedding/retrieval strategies and enterprise knowledge grounding.
- Integrate LLM-powered applications with APIs, enterprise systems, data platforms and GCP.
- Own AI quality evaluation, failure analysis, observability and continuous improvement of agent behaviour.
- Implement testing strategies and evaluation pipelines to assess response quality, reliability and system performance.
- Monitor AI systems across quality, latency, cost and performance.
- Apply Responsible AI, security, privacy, governance and compliance principles.
- Contribute to production troubleshooting, RCA, reusable components and engineering best practices.

Key Skills





Agentic AI | Generative AI | LLMs | AI Testing & Evaluation | RAG | Prompt Engineering | Agent Evaluation | Failure Analysis | LangChain | LangGraph | AutoGen | CrewAI | Multi-Agent Systems | Vector Databases | Embeddings | AI Observability | MLOps | CI/CD | GCP | Responsible AI

We are especially interested in candidates who:

Have a solid inclination towards AI Testing & Evaluation
Enjoy identifying failure modes and improving AI/agent behaviour
Have experience evaluating LLMs, RAG or agentic systems
Are interested in AI quality, reliability, safety and observability
Can go beyond API integration and understand model behaviour, reasoning and algorithms

📌 Agentic App Engineer- Testing and Evaluation (Pune)
🏢 EY
📍 Pune