Role & responsibilities
Design, develop, and enhance AI/LLM applications using Python, LangChain, RAG, and Agentic AI frameworks.
Build and optimize RAG pipelines covering document ingestion, chunking, embeddings, vector search, retrieval, reranking, and LLM response generation.
Develop and integrate LLM/agent observability to monitor application health, performance, reliability, and AI quality.
Implement tracing and monitoring across LLM calls, agent workflows, tools, retrieval pipelines, and APIs.
Track key AI metrics such as latency, token usage, response quality, hallucination/error rates, retrieval accuracy, failures, and cost.
Build dashboards, alerts, and monitoring mechanisms to identify LLM failures, degraded retrieval, latency issues, and production anomalies.
Work with tools such as Azure Application Insights, Azure Monitor, OpenTelemetry, LangSmith, Azure AI Foundry/evaluation capabilities, or equivalent observability platforms.
Implement LLM evaluation and RAG evaluation using appropriate metrics and feedback mechanisms.
Monitor and troubleshoot multi-agent workflows, tool calls, prompt execution, and model responses.




Integrate observability into FastAPI/microservices and cloud-native AI applications.
Deploy and support AI workloads on Azure, with exposure to Docker/Kubernetes being an advantage.
Establish logging, error handling, audit trails, and production monitoring for enterprise AI systems.
Collaborate with AI engineers, data engineers, architects, and business stakeholders to improve AI reliability, performance, and user experience.
Continuously identify opportunities to improve model performance, retrieval quality, scalability, latency, and operational cost.

Preferred candidate profile

Must-Have Skills
Python robust hands-on development
RAG – robust practical experience
LangChain – robust hands-on experience
LLMs / Generative AI
Agentic AI / LangGraph
Azure OpenAI
Azure AI Search / Vector Search
Embeddings, semantic search and document retrieval
FastAPI / REST APIs
Prompt engineering and LLM evaluation
Production deployment of AI applications

📌 Artificial Intelligence Engineer Hyderabad
🏢 Allegis Group
📍 Hyderabad