05 Nov
|
Ishan Technologies
|
Ahmedabad
05 Nov
Ishan Technologies
Ahmedabad
About the Role
Ishan Technologies is looking for an experienced Network Performance Engineer to build, operate, optimise and maintain the high-performance network infrastructure supporting Saksham AI Services' 1000+ GPU cluster. The role will focus on backend GPU networking, high-speed Ethernet/RoCE and InfiniBand fabrics, front-end connectivity, storage and management networks. The ideal candidate should have strong hands-on experience in data centre networking, network performance optimisation, RDMA technologies, congestion control and collective communication benchmarking.
The selected candidate will also act as an L2/L3 escalation point for network and performance-related issues raised by Ishan Technologies' 24x7 NOC team.
Key Responsibilities
1. GPU Network & Data Centre Infrastructure
- Install, configure and operate high-speed backend GPU networks using RoCEv2/Ethernet or InfiniBand.
- Manage front-end, storage and out-of-band management networks.
- Configure and maintain switches, network interface cards (NICs), transceivers and fibre connectivity.
- Work on rail-optimised leaf-spine network architecture.
- Maintain network topology, cable mapping and infrastructure documentation.
2. Network Performance & Optimisation
- Configure and optimise BGP, EVPN-VXLAN and leaf-spine network architecture.
- Implement congestion control mechanisms, including PFC, ECN/DCQCN, adaptive routing and QoS.
- Monitor network bandwidth, latency, packet loss, link flaps and congestion.
- Troubleshoot GPU-to-GPU communication and collective performance issues.
- Ensure full-bandwidth network performance across different workload and cluster sizes.
3. RoCE, InfiniBand & RDMA Networking
- Manage RoCEv2 and InfiniBand fabrics for GPU/HPC environments.
- Work with NVIDIA UFM and InfiniBand management tools.
- Configure and troubleshoot RDMA drivers, SR-IOV and GPUDirect RDMA.
- Work with NVIDIA ConnectX and AMD Pensando Pollara networking technologies.
- Maintain VLAN/VRF segmentation, PKeys and fabric-level tenant isolation.
4. AI Workload & Collective Communication
- Perform collective communication benchmarking using RCCL, NCCL, MPI and related testing tools.
- Conduct RDMA performance testing using perftest,
ib_write_bw and ib_read_bw.
- Work with nccl-tests and rccl-tests to validate network performance.
- Understand how data, tensor and expert parallelism impact network utilisation.
- Investigate slow-running AI jobs, communication bottlenecks and straggler GPUs.
- Collaborate with compute and MLOps teams to optimise AI workload performance.
5. Network Monitoring & Automation
- Develop network monitoring dashboards using Prometheus and Grafana.
- Monitor switch health, link errors, optics performance and network congestion.
- Automate network configurations, health checks and compliance activities using Python and Ansible.
- Maintain network performance baselines and conduct acceptance testing after infrastructure changes.
- Support Kubernetes RDMA networking using Multus, SR-IOV device plugins and network operators.
6. Incident Management & Operations
- Handle L2/L3 network and performance escalations from the 24x7 NOC team.
- Lead network incident investigations and root cause analysis.
- Coordinate with customers, solution architects, vendors and internal engineering teams.
- Plan and execute switch, NIC and optics firmware upgrades under change management.
- Maintain network runbooks, technical documentation and operational standards.
- Support internet edge connectivity, peering and private customer network links.
Required Technical Skills
- Robust hands-on experience in Data Centre Networking.
- Experience with RoCEv2 or InfiniBand networking.
- Strong understanding of BGP, EVPN-VXLAN and leaf-spine architecture.
- Knowledge of PFC, ECN/DCQCN, QoS and adaptive routing.
- Experience with RDMA, GPUDirect RDMA and SR-IOV.
- Hands-on experience with NVIDIA ConnectX, Spectrum-X or AMD Pensando Pollara.
- Knowledge of RCCL, NCCL, MPI and collective communication benchmarking.
- Experience with NVIDIA UFM and InfiniBand management.
- Understanding of VLAN, VRF, PKeys and network tenant isolation.
- Experience with 400G/800G networking, optics and fibre connectivity.
- Working knowledge of Python, Ansible and network automation.
- Experience with Prometheus, Grafana, SNMP and streaming telemetry.
Educational Qualifications
- Bachelor's degree in Engineering or Technology.
- Preferred specialisations: Computer Science, Information Technology, Electronics, Telecommunications or related disciplines.
- Relevant data centre networking certifications will be an added advantage.
Preferred Certifications:
- Cisco CCNP/CCIE Data Center.
- Arista ACE.
- Juniper JNCIP-DC.
- NVIDIA Networking (InfiniBand/Spectrum-X) certifications.
Experience Requirements
- 6–10 years of overall skilled experience.
- Minimum 4+ years of relevant experience in Data Centre Networking for GPU, HPC or hyperscale environments.
- Hands-on experience with RoCEv2 or InfiniBand fabrics is essential.
- Experience with large-scale leaf-spine networks, cabling and optics is essential.
- Experience in RCCL/NCCL or MPI performance optimisation is preferred.
- Exposure to AMD Pensando, NVIDIA Spectrum-X or ConnectX networking is preferred.
Key Competencies
- Strong troubleshooting and analytical skills.
- Hands-on approach to network engineering and performance optimisation.
- Ability to handle critical production incidents under pressure.
- Strong ownership and root cause analysis capabilities.
- Effective communication with customers, NOC teams and technical stakeholders.
- Willingness to participate in a 24x7 on-call support rotation.
- Ability to learn and adapt to evolving AI and GPU networking technologies.
Key Success Outcomes
- Reliable, high-performance GPU networking infrastructure.
- Full-bandwidth collective communication across cluster workloads.
- Verified network isolation between multiple customers and tenants.
- Improved network performance and reduced communication bottlenecks.
- Faster resolution of network incidents and performance escalations.
- Standardised, automated and well-documented network operations.
📌 Network Performance Engineer – GPU/HPC Infrastructure (Ahmedabad)
🏢 Ishan Technologies
📍 Ahmedabad