Role Overview

Lead the agentic AI engineering function, responsible for the design, architecture, and quality of all AI agents on the platform. Define engineering standards for prompt design, agent evaluation, and LLM integration across the team.

Key Responsibilities

- Architect the agent engineering framework, including orchestration patterns, prompt governance, and tool-calling standards.
- Lead design reviews of all agent implementations across the team.
- Own the RAG pipeline architecture, including embedding strategy, retrieval optimization, and accuracy benchmarks.
- Define the agent evaluation framework, including golden datasets, confidence calibration, and regression testing.
- Set prompt engineering standards and chair the prompt review board.
- Mentor JL1 engineers on agent development and evaluation.
- Monitor and optimize agent performance in production, focusing on latency, cost, and accuracy.

Required Skills & Experience





- 10+ years of experience in AI/ML engineering, including 5+ years working with production LLM systems.
- Deep expertise in prompt engineering, agent design, and LLM integration.
- Experience architecting RAG pipelines at scale.
- Experience with agent evaluation, including hallucination testing, confidence scoring, and regression frameworks.
- Expert-level Python skills; familiarity with TypeScript/Node.js is a plus.
- Experience leading AI/ML engineering teams.
- Understanding of LLM cost management, token budgeting, and latency optimization.

Nice to Have

- Experience with Model Context Protocol (MCP) or tool-calling standards.
- Background in enterprise AI platform development.
- Knowledge of ITSM or IT operations domains.

📌 AI Architect (Pune)
🏢 BAPON Technologies
📍 Pune