Lead the agentic AI engineering function responsible for the design, architecture, and quality of all AI agents on the platform. Define engineering standards for prompt design, agent evaluation, and LLM integration across the team.

WHAT YOU'LL DO
Architect the agent engineering framework: orchestration patterns, prompt governance, and tool-calling standards
Lead design review of all agent implementations across the team
Own the RAG pipeline architecture: embedding strategy, retrieval optimization, and accuracy benchmarks
Define the agent evaluation framework: golden datasets, confidence calibration, regression testing
Set prompt engineering standards and chair the prompt review board
Mentor JL1 engineers on agent development and evaluation
Monitor and optimize agent performance in production: latency, cost, and accuracy

REQUIRED SKILLS & EXPERIENCE





10+ years in AI/ML engineering; 5+ years with production LLM systems Deep expertise in prompt engineering, agent design, and LLM integration

Experience architecting RAG pipelines at scale Agent evaluation: hallucination testing, confidence scoring, regression frameworks

Python expert; TypeScript/Node.js familiarity a

plus Experience leading AI/ML engineering teams

Understanding of LLM cost management, token

budgeting, and latency optimization

NICE TO HAVE
Experience with Model Context Protocol (MCP) or tool-calling standards
Background in enterprise AI platform builds
Knowledge of ITSM or IT operations domains

📌 Ai Ml Engineer Pune
🏢 Futran Tech Solutions
📍 Pune