Role: LLM Engineer
Experience: 4+ Years
Location: Pune

Role Overview
We are seeking an experienced LLM Engineer to build, deploy, and scale production-grade Generative AI systems spanning agentic workflows, Retrieval-Augmented Generation (RAG), evaluation, and backend infrastructure. The role focuses on reliably serving LLM and agentic applications at scale while optimizing quality, cost, and latency.

Key Responsibilities

Agentic Systems
- Build agent workflows using frameworks such as LangGraph, CrewAI, AutoGen, or OpenAI Agents SDK.
- Implement function calling and tool use to enable agents to reliably interact with APIs, databases, and internal services.
- Use MCP (Model Context Protocol) or similar standards to connect agents with external tools and data sources.
- Implement guardrails and fallback mechanisms for predictable and secure multi-step agent behavior.

Retrieval-Augmented Generation (RAG)
- Build and optimize RAG pipelines, including chunking strategies, embedding model selection, and re-ranking.
- Deploy, scale, and evaluate vector databases such as Pinecone,



Qdrant, FAISS, pgvector, ChromaDB, Milvus, or Azure AI Search.
- Diagnose hallucinations and poor responses by identifying retrieval-related issues and improving the underlying RAG pipeline.

Evaluation & Quality
- Build and maintain evaluation datasets, golden datasets, and regression suites covering retrieval accuracy, answer quality, and agent task completion.
- Use evaluation frameworks such as RAGAS, DeepEval, TruLens, LangSmith, or LangFuse.
- Track metrics including faithfulness, context precision/recall, answer relevance, and groundedness.
- Monitor quality metrics across models and prompt versions to identify regressions and support model/provider decisions.
- Combine automated evaluation with human-in-the-loop reviews for nuanced correctness, tone, and safety scenarios.

1. Backend Engineering & Deployment
- Build and maintain APIs and microservices using FastAPI or Flask to expose ag

📌 LLM Engineer (Nagpur)
🏢 Nice Software Solutions
📍 Nagpur