Role & responsibilities
Mandatory Skills
1. Advanced Prompt Engineering.
- Experience with structured outputs (JSON mode, function/tool calling) and system prompt design.
- Ability to systematically test, version, and optimize prompts for quality and token efficiency.
- Knowledge of prompt injection risks and mitigation techniques.
2. LLM Architecture Design
- Solid understanding of transformer architecture, attention mechanisms, tokenization, and context windows.
- Ability to design scalable LLM system architectures, including agents, memory, tool use, and orchestration layers.
- Experience making trade-off decisions on model selection, latency, cost, and accuracy.
3. RAG System Implementation
- Hands-on experience building production RAG systems end-to-end.
- Proficiency with vector databases such as Pinecone, Weaviate, FAISS, Chroma, Milvus, Qdrant, or pgvector.
- Knowledge of embedding models, chunking strategies, hybrid search (semantic + keyword), re-ranking, and query transformation.
- Experience evaluating RAG pipelines using tools like RAGAS, TruLens, or custom metrics.
4. Fine-tuning Strategy
- Ability to determine when fine-tuning is the right approach compared with prompting or RAG.
- Hands-on experience with PEFT techniques (LoRA, QLoRA), instruction tuning, and alignment methods (RLHF, DPO).
- Experience with dataset curation, cleaning, and preparation for fine-tuning.
- Familiarity with Hugging Face Transformers, PEFT, TRL, Axolotl, or Unsloth.
5. LangChain / LlamaIndex (Advanced)
- Deep, hands-on experience building complex chains, agents, and multi-step workflows.
- Proficiency with LangGraph, LCEL, custom tools, retrievers, memory modules, and callbacks.
- Experience with LlamaIndex indexing strategies, query engines, and data connectors.
- Ability to extend frameworks with custom components when needed.
6. Multi-model Integration
- Experience integrating and orchestrating multiple LLM providers and open-source models.
- Ability to implement model routing, fallback mechanisms, load balancing, and cost optimization.
- Exposure to multimodal models (text, image, audio, vision-language).
- Familiarity with gateways and abstraction tools such as LiteLLM, OpenRouter, or similar.
7. Production Deployment
- Proven experience deploying LLM applications to production at scale.
- Proficiency with model serving frameworks such as vLLM, TGI, Ollama, Triton, or TensorRT-LLM.
- Strong knowledge of Docker, Kubernetes, CI/CD, and cloud platforms (AWS, Azure, or GCP), including services like Bedrock, Azure OpenAI, or Vertex AI.
- Experience with LLMOps and observability tools such as LangSmith, MLflow, Weights & Biases, Langfuse, or Arize.
- Understanding of caching, batching, quantization, rate limiting, and latency optimization.
📌 LLM Engineer - Manager (Pune)
🏢 PwC
📍 Pune