15 Nov
|
Covalense Digital
|
Bengaluru
15 Nov
Covalense Digital
Bengaluru
## Job highlights
- Experience with LLM inference frameworks like vLLM, model quantization, LoRA/QLoRA fine-tuning, and benchmarking
- Set up and optimize LLM inference infrastructure, develop fine-tuning pipelines, benchmark and optimize models, and build evaluation frameworks
Key Responsibilities
- Set up and optimize LLM inference infrastructure, including migrating from Ollama to vLLM
- Benchmark and optimize open-weight models such as Gemma, Qwen, Llama, Mistral, etc.
- Optimize GPU utilization, latency, throughput, memory usage, and inference cost
- Develop and experiment with model quantization and calibration techniques to minimize quality degradation
- Build fine-tuning pipelines using techniques such as LoRA/QLoRA for specific use cases
- Establish automated LLM and RAG evaluation frameworks
- Benchmark models across quality, latency, memory, and throughput
- Profile and optimize workloads on NVIDIA RTX 4090 / RTX 5090 and future GPU infrastructure
- Evaluate emerging open-source models, inference frameworks,
and optimization techniques
- Build reusable foundations for model serving, evaluation, experimentation, and deployment What We're Looking For
- Strong Python and Linux engineering skills
- Hands-on experience with LLMs and Hugging Face Transformers
- Experience with vLLM or other LLM inference frameworks
- Solid understanding of LLM inference, GPU memory, attention, KV cache, batching, etc.
- Experience with quantization, model optimization, or compression
- Experience with LoRA/QLoRA/fine-tuning
- Experience designing LLM evaluation and benchmarking systems
- Strong debugging, profiling, and performance optimization mindset Good to Have
- CUDA / Triton / FlashAttention
- GPTQ / AWQ / FP8 / INT8 / INT4
- SGLang / TensorRT-LLM / llama.cpp
- Distributed or multi-GPU inference
- RAG evaluation and retrieval optimization
- Experience working with production LLM infrastructure
📌 AI / LLM Systems Engineer (Consultant/Freelance) (Bengaluru)
🏢 Covalense Digital
📍 Bengaluru