Job Purpose

Deliver and operate ultralowlatency, highbisection network fabrics for GPU training at ~10K GPU scale with deterministic performance.

Role Description

Key Responsibilities

- Implementation
- Build IB (HDR/NDR) eastwest fabrics and Ethernet northsouth connectivity; leafspine with ECMP.
- Deploy and tune RoCE v2, PFC/ECN, MTU, buffer profiles; IB routing/credit/timers; SHARP/UCX offloads where applicable.
- Operations
- Day2 fabric health (congestion hotspots, slowdrain, link flaps); automate config with Ansible/Terraform.
- Lifecycle of switch OS/firmware; linklevel optics and cable qualification; inventory and spares.
- Performance & Optimization
- Validate NCCL collectives performance across pods; troubleshoot allreduce bottlenecks and imbalance.
- Implement QoS classes for training vs inference vs controlplane.
- Reliability & Incident
- Lead P0/P1 critical incident response for NVIDIA GPU infrastructure, including InfiniBand, RoCE, Ethernet fabric and AI/ML cluster environments.
- Troubleshot performance bottlenecks involving InfiniBand fabrics, RoCE networks, congestion management, NCCL communications, GPUDirect RDMA, and distributed AI training workloads.
- Led post-incident reviews and implemented automation, monitoring enhancements,



and architectural improvements to increase overall cluster availability, stability, and performance.
- Create runbooks for path failures, fabric partitions, and credit starvation; design graceful degradation.
- Security & Compliance
- BGP/EVPN multitenancy, ACLs, rolebased access to fabric managers; config signing/change control.

Experience & Educational Requirements

Qualifications and Experience

EDUCATIONAL QUALIFICATIONS: (degree, training, or certification required)

BE/B-Tech or equivalent with Computer Science or Electronics & Communication

RELEVANT EXPERIENCE: (no. of years of technical, functional, and/or leadership experience or specific exposure required)

- 715 years DC networking; deep InfiniBand (HDR/NDR) and RoCE tuning at scale; GPU cluster comms patterns.

Tools / Tech
NVIDIA UFM/NetQ (or equivalent), Spectrum/Quantum families, Cumulus/Onyx/SONiC; iperf/NCCLtests; sFlow/telemetry; Ansible/Terraform.

Certifications
NVIDIA Networking; CCNP/CCIE DC; JNCIPDC / Arista ACE.

KPIs
Fabric throughput/latency SLOs, congestion events, NCCL job throughput variance, change success rate.

Work Mode
Onsite for build/turnups; oncall for P1 network incidents.

📌 Nvidia Network Engineer (InfiniBand / Ethernet) (Tamil Nadu)
🏢 Larsen and Toubro (L&T)
📍 Tamil Nadu