Role Overview

We are looking for an experienced LLM Scientist to research, design, and prototype advanced Generative AI systems. This role is experimentation- and coding-heavy, with a robust focus on Agentic AI design, RAG architecture, LLM fine-tuning, and rapid Python-based prototyping of new GenAI techniques ahead of production handoff.

Key Responsibilities

Agentic AI & GenAI Prototyping

- Design and prototype Agentic AI / Multi-Agent systems using LangGraph, LangChain, and MCP.
- Research and experiment with emerging LLM and Agentic AI techniques.
- Rapidly prototype and validate new GenAI approaches using Python.
- LLM Fine-Tuning & Experimentation
- Fine-tune LLMs using LoRA/QLoRA.
- Conduct experiments comparing models, prompting strategies, and architectures.
- Run controlled experiments, including A/B tests and ablation studies.
- Apply statistical rigor when designing experiments and interpreting results.

- RAG Architecture
- Architect and iterate on RAG pipelines.
- Experiment with chunking strategies, embeddings, hybrid search, and reranking.
- Evaluate retrieval quality and optimize retrieval strategies.

1. Evaluation & Benchmarking

- Build evaluation and benchmarking frameworks for GenAI systems.
- Evaluate retrieval quality, groundedness, hallucination, and relevance.
- Implement observability and tracing using tools such as LangFuse.
- Develop robust evaluation methodologies and avoid metric gaming.

Machine Learning & Statistical Analysis

- Apply statistical concepts including hypothesis testing, confidence intervals, significance testing, and sample-size considerations.
- Design and evaluate supervised, unsupervised, and semi-supervised learning approaches.
- Analyze bias-variance tradeoff, regularization, and overfitting/underfitting.
- Apply classical ML algorithms and evaluation metrics to support GenAI system development.




- Collaboration & Research
- Track emerging LLM and Agentic AI research and assess applicability to business problems.
- Partner with LLM Engineers to hand off validated prototypes for production scaling.
- Research and evaluate new approaches to improve GenAI system performance.

Required Skills ML & Statistical Foundations

- Strong understanding of statistics and probability, including distributions, hypothesis testing, confidence intervals, and significance testing.
- Strong understanding of supervised, unsupervised, and semi-supervised learning paradigms.
- Knowledge of bias-variance tradeoff, regularization, and overfitting/underfitting diagnosis.
- Experience with model evaluation metrics such as Precision, Recall, F1, and ROC-AUC.
- Practical experience with classical ML algorithms including regression, classification, clustering, and ensemble methods.
- Understanding of optimization fundamentals such as gradient descent, learning-rate tuning, and convergence behavior.

GenAI & Engineering Skills

- Strong Python coding skills with experience building experimental pipelines and tooling from scratch.
- Deep understanding of LLMs, Transformer architecture, and Generative AI fundamentals.
- Hands-on experience with Agentic AI, RAG, and LLM fine-tuning.
- Proficiency in LangChain, LangGraph, and MCP (Model Context Protocol).
- Experience with vector databases such as FAISS, ChromaDB, and Pinecone.
- Strong understanding of GenAI evaluation methodologies, including groundedness, hallucination detection, and relevance scoring.

Good to Have

- Knowledge of NLP techniques such as BERT, embeddings, and sentiment analysis.
- Experience with classical ML techniques such as XGBoost and anomaly detection.
- Research background or publications in NLP/LLMs.
- Experience working in regulated enterprise domains.
- B.Tech/M.Tech in Computer Science, IT, or a related field.

📌 LLM Scientist (Nagpur)
🏢 Nice Software Solutions
📍 Nagpur