28 Oct
|
Larsen & Toubro Vyoma
|
Chennai
28 Oct
Larsen & Toubro Vyoma
Chennai
Job Purpose Deliver and operate ultra‑low‑latency, high‑bisection network fabrics for GPU training at ~10K GPU scale with deterministic performance. Roles & Responsibilities
• Implementation Build IB (HDR/NDR) east‑west fabrics and Ethernet north‑south connectivity; leaf‑spine with ECMP. Deploy and tune RoCE v2, PFC/ECN, MTU, buffer profiles; IB routing/credit/timers; SHARP/UCX offloads where applicable.
• Operations Day‑2 fabric health (congestion hotspots, slow‑drain, link flaps); automate config with Ansible/Terraform. Lifecycle of switch OS/firmware; link‑level optics and cable qualification; inventory and spares.
• Performance & Optimization Validate NCCL collectives performance across pods; troubleshoot all‑reduce bottlenecks and imbalance. Implement QoS classes for training vs inference vs control‑plane.
• Reliability & Incident Lead P0/P1 critical incident response for NVIDIA GPU infrastructure, including InfiniBand, RoCE, Ethernet fabric and AI/ML cluster environments. Troubleshot performance bottlenecks involving InfiniBand fabrics, RoCE networks, congestion management,
NCCL communications, GPUDirect RDMA, and distributed AI training workloads. Led post-incident reviews and implemented automation, monitoring enhancements, and architectural improvements to increase overall cluster availability, stability, and performance. Create runbooks for path failures, fabric partitions, and credit starvation; design graceful degradation.
• Security & Compliance BGP/EVPN multi‑tenancy, ACLs, role‑based access to fabric managers; config signing/change control. Experience & Educational Requirement BE/B-Tech or equivalent with Computer Science or Electronics & Communication Certifications:
- NVIDIA Networking; CCNP/CCIE DC; JNCIP‑DC / Arista ACE. RELEVANT EXPERIENCE
• 7–15 years DC networking; deep InfiniBand (HDR/NDR) and RoCE tuning at scale; GPU cluster comms patterns. Tools / Tech NVIDIA UFM/NetQ (or equivalent), Spectrum/Quantum families, Cumulus/Onyx/SONiC; iperf/NCCL‑tests; sFlow/telemetry; Ansible/Terraform.
📌 Network Engineer (Chennai)
🏢 Larsen & Toubro Vyoma
📍 Chennai