Role & responsibilities

- Design, develop, and enhance AI/LLM applications using Python, LangChain, RAG, and Agentic AI frameworks.
- Build and optimize RAG pipelines covering document ingestion, chunking, embeddings, vector search, retrieval, reranking, and LLM response generation.
- Develop and integrate LLM/agent observability to monitor application health, performance, reliability, and AI quality.
- Implement tracing and monitoring across LLM calls, agent workflows, tools, retrieval pipelines, and APIs.
- Track key AI metrics such as latency, token usage, response quality, hallucination/error rates, retrieval accuracy, failures, and cost.
- Build dashboards, alerts, and monitoring mechanisms to identify LLM failures, degraded retrieval, latency issues, and production anomalies.
- Work with tools such as Azure Application Insights, Azure Monitor, OpenTelemetry, LangSmith, Azure AI Foundry/evaluation capabilities, or equivalent observability platforms.
- Implement LLM evaluation and RAG evaluation using appropriate metrics and feedback mechanisms.
- Monitor and troubleshoot multi-agent workflows, tool calls, prompt execution, and model responses.




- Integrate observability into FastAPI/microservices and cloud-native AI applications.
- Deploy and support AI workloads on Azure, with exposure to Docker/Kubernetes being an advantage.
- Establish logging, error handling, audit trails, and production monitoring for enterprise AI systems.
- Collaborate with AI engineers, data engineers, architects, and business stakeholders to improve AI reliability, performance, and user experience.
- Continuously identify opportunities to improve model performance, retrieval quality, scalability, latency, and operational cost.

Preferred candidate profile

Must-Have Skills

- Python strong hands-on development
- RAG – robust practical experience
- LangChain – strong hands-on experience
- LLMs / Generative AI
- Agentic AI / LangGraph
- Azure OpenAI
- Azure AI Search / Vector Search
- Embeddings, semantic search and document retrieval
- FastAPI / REST APIs
- Prompt engineering and LLM evaluation
- Production deployment of AI applications

📌 Artificial Intelligence Engineer (Hyderabad)
🏢 Allegis Group
📍 Hyderabad