21 Nov
|
Spydra
|
Secunderabad
21 Nov
Spydra
Secunderabad
CPositions: 1 | Experience: 5–7 years
About the role You will design, develop and optimise distributed LLM inference systems that deliver high throughput, low latency and productive GPU utilisation.
Key responsibilities
Deploy and optimise LLM serving using vLLM, SGLang, TensorRT-LLM and NVIDIA Dynamo.
Implement appropriate tensor, pipeline and expert parallelism.
Design prefill/decode disaggregation and optimise continuous batching, scheduling, speculative decoding and prefix caching.
Develop cache-aware routing and load balancing across inference replicas.
Integrate and evaluate LMCache, Mooncake and other distributed KV-cache technologies.
Optimise long-context and high-concurrency workloads; diagnose compute, memory and scheduling bottlenecks.
Required technical skills Python, C++, PyTorch, vLLM, SGLang, NVIDIA Dynamo, Kubernetes, distributed systems, GPU memory management, model-serving APIs.
Expected deliverables Scalable serving deployments, optimised configurations, measured performance gains and reproducible benchmarks.
📌 Senior Distributed Inference Engineer Secunderabad
🏢 Spydra
📍 Secunderabad