Our training infrastructure runs distributed jobs across many GPUs at once, and it has to just work no lost checkpoints, no silent slowdowns, no surprises at 2am. As a Training Runtime Engineer, you'll build and harden the distributed training infrastructure our model development depends on, working on some of the hardest reliability and performance problems in the stack.
What You'll Do
Build and maintain distributed training infrastructure, including tensor, pipeline, data, and expert parallelism and FSDP
Implement and harden distributed checkpointing and fault-tolerant recovery for long-running training jobs
Optimize NCCL communication patterns and diagnose distributed-training performance issues
Support multi-tenant job scheduling and isolation for concurrent training runs
Partner with our research team to translate recent training recipes into runtime requirements
What You'll Bring
5+ years of experience in distributed systems or ML infrastructure engineering
Solid experience with distributed training frameworks (PyTorch, Megatron-LM, DeepSpeed, or similar)
Hands-on experience with CUDA, NCCL, and RDMA-based communication
Solid grasp of distributed checkpointing and fault-tolerant systems design
Proficiency in C++ and Python
Education B.Tech in Computer Science or a related field; M.Tech a plus for senior scope.
Nice to Have
Experience at a distributed-database, HPC, or cloud infrastructure company — the reliability and consistency challenges transfer directly
Prior work on a multi-tenant training or scheduling system
📌 Distributed Training Engineer Bengaluru
🏢 HCLTech
📍 Bengaluru
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.