Role: LLM Engineer
Experience: 4+ Years
Location: Pune

Role Overview
We are seeking an experienced LLM Engineer to build, deploy, and scale production-grade Generative AI systems spanning agentic workflows, Retrieval-Augmented Generation (RAG), evaluation, and backend infrastructure. The role focuses on reliably serving LLM and agentic applications at scale while optimizing quality, cost, and latency.

Key Responsibilities

Agentic Systems
Build agent workflows using frameworks such as LangGraph, CrewAI, AutoGen, or OpenAI Agents SDK.
Implement function calling and tool use to enable agents to reliably interact with APIs, databases, and internal services.
Use MCP (Model Context Protocol) or similar standards to connect agents with external tools and data sources.
Implement guardrails and fallback mechanisms for predictable and secure multi-step agent behavior.

Retrieval-Augmented Generation (RAG)
Build and optimize RAG pipelines, including chunking strategies, embedding model selection, and re-ranking.
Deploy, scale,



and evaluate vector databases such as Pinecone, Qdrant, FAISS, pgvector, ChromaDB, Milvus, or Azure AI Search.
Diagnose hallucinations and poor responses by identifying retrieval-related issues and improving the underlying RAG pipeline.

Evaluation & Quality
Build and maintain evaluation datasets, golden datasets, and regression suites covering retrieval accuracy, answer quality, and agent task completion.
Use evaluation frameworks such as RAGAS, DeepEval, TruLens, LangSmith, or LangFuse.
Track metrics including faithfulness, context precision/recall, answer relevance, and groundedness.
Monitor quality metrics across models and prompt versions to identify regressions and support model/provider decisions.
Combine automated evaluation with human-in-the-loop reviews for nuanced correctness, tone, and safety scenarios.
Backend Engineering & Deployment
Build and maintain APIs and microservices using FastAPI or Flask to expose ag

📌 Llm Engineer Nagpur
🏢 Nice Software Solutions
📍 Nagpur