Role & responsibilities
- Design, develop, and enhance AI/LLM applications using Python, LangChain, RAG, and Agentic AI frameworks.
- Build and optimize RAG pipelines covering document ingestion, chunking, embeddings, vector search, retrieval, reranking, and LLM response generation.
- Develop and integrate LLM/agent observability to monitor application health, performance, reliability, and AI quality.
- Implement tracing and monitoring across LLM calls, agent workflows, tools, retrieval pipelines, and APIs.
- Track key AI metrics such as latency, token usage, response quality, hallucination/error rates, retrieval accuracy, failures, and cost.
- Build dashboards, alerts, and monitoring mechanisms to identify LLM failures, degraded retrieval, latency issues, and production anomalies.
- Work with tools such as Azure Application Insights, Azure Monitor, OpenTelemetry, LangSmith, Azure AI Foundry/evaluation capabilities, or equivalent observability platforms.
- Implement LLM evaluation and RAG evaluation using appropriate metrics and feedback mechanisms.




- Monitor and troubleshoot multi-agent workflows, tool calls, prompt execution, and model responses.
- Integrate observability into FastAPI/microservices and cloud-native AI applications.
- Deploy and support AI workloads on Azure, with exposure to Docker/Kubernetes being an advantage.
- Establish logging, error handling, audit trails, and production monitoring for enterprise AI systems.
- Collaborate with AI engineers, data engineers, architects, and business stakeholders to improve AI reliability, performance, and user experience.
- Continuously identify opportunities to improve model performance, retrieval quality, scalability, latency, and operational cost.

Preferred candidate profile

Must-Have Skills
- Python robust hands-on development
- RAG – strong practical experience
- LangChain – strong hands-on experience
- LLMs / Generative AI
- Agentic AI / LangGraph
- Azure OpenAI
- Azure AI Search / Vector Search
- Embe

📌 Artificial Intelligence Engineer (Bengaluru)
🏢 Allegis Group
📍 Bengaluru