Location: Noida or Bengaluru

About the Role

Serving models efficiently is its own discipline, distinct from training them. As an Inference Runtime Engineer, you'll build the serving infrastructure that keeps GPU utilization high and cost per token low across everything we run, from rollout generation to production model serving.

What You'll Do

- Build and optimize model-serving infrastructure: KV-cache management, continuous batching, and speculative decoding
- Implement quantization (FP8/INT4) and intelligent routing to reduce serving cost per token
- Support low-latency rollout serving for both internal model training and customer-facing workloads
- Diagnose and resolve GPU utilization bottlenecks in production serving
- Work with product engineering to expose serving capabilities through customer-facing APIs

What You'll Bring





- 5+ years of experience in systems or ML infrastructure engineering
- Robust experience with model-serving systems (vLLM, SGLang, TensorRT-LLM, or comparable)
- Hands-on CUDA or Triton experience
- Solid understanding of quantization and batching strategies for inference
- Proficiency in C++ and Python

Education

B.Tech in Computer Science or a related field; M.Tech a plus for senior scope.

Nice to Have

- Experience at an inference-focused AI infrastructure company or a hyperscaler's model-serving team
- Exposure to speculative decoding or other latency-reduction techniques in production

📌 Performance Engineer (Inference) (Bengaluru)
🏢 HCLTech
📍 Bengaluru