20 Nov
|
Stratsol Software Systems
|
Chennai
20 Nov
Stratsol Software Systems
Chennai
Hiring: NVIDIA GPU Infrastructure Engineer / HPC Engineer
Location:
Chennai, India
Experience:
7–12 Years Work Mode: Onsite / Hybrid
About the Role
We are looking for an experienced NVIDIA GPU Infrastructure Engineer / HPC Engineer to build, manage, and optimize large-scale AI/HPC GPU clusters supporting high-performance LLM training and inference workloads. The ideal candidate will have strong hands-on expertise in NVIDIA H100/B-series GPUs, CUDA, Slurm, Kubernetes, NCCL, and GPU performance optimization, with experience supporting enterprise-scale GPU infrastructure. Must-Have Technical Skills
• GPU Infrastructure: NVIDIA H100/B-series GPUs, large-scale GPU clusters (up to 10K GPUs), multi-pod architecture, GPU/CPU/Memory/NIC capacity planning, and heterogeneous GPU SKU bin-packing.
• GPU Virtualization: NVIDIA MIG, vGPU profiles, multi-tenant GPU partitioning, quota management, and resource allocation.
• GPU Computing: CUDA, cuDNN, TensorRT, NCCL, UCX, GPU clocks/persistence, and CUDA runtime tuning.
• Cluster Orchestration: Slurm, Kubernetes (K8s), GPU Device Plugins, DaemonSets, Node Feature Discovery (NFD), scheduling, accounting, and quotas.
• GPU Monitoring: NVIDIA DCGM, NVML, nvidia-smi, Prometheus, Grafana, ELK, and Splunk.
• Automation & Deployment: Helm, ArgoCD, golden images, node remediation (cordon/drain/reimage), and auto-healing workflows.
• High-Performance Networking: InfiniBand (IB), Ethernet, RDMA, TOR/leaf connectivity, NUMA/locality optimization, and GPUDirect Storage.
• Performance Benchmarking: HPL, HPL-AI, MLPerf-like benchmarking, performance baselines, regression analysis, and SLO monitoring.
• Infrastructure Operations:
Rack/power/cooling layouts, firmware and driver lifecycle management, change management, and zero/low-downtime maintenance.
• Security: MIG isolation, cgroups/device cgroups, mpsd, secure GPU drivers/containers, SBOM, and image scanning.
Key Responsibilities
• Deploy, configure, and operate enterprise-scale NVIDIA GPU clusters supporting AI/ML training and inference.
• Integrate Slurm/Kubernetes scheduling with GPU resource management and multi-tenant workloads.
• Optimize CUDA, NCCL, UCX, RDMA, GPU performance, and distributed training efficiency.
• Manage GPU firmware, drivers, monitoring, cluster health, and capacity planning.
• Lead P0/P1 production incidents involving GPU, CUDA, networking, NCCL, RDMA, and scheduling failures.
• Coordinate with Networking, GPU Operations, Platform Engineering, Storage, and Application teams to restore services within SLA targets.
• Perform root cause analysis (RCA), implement corrective actions, and improve infrastructure reliability.
• Maintain runbooks, performance baselines, and application tuning documentation.
• Track GPU utilization, queue wait-time SLOs, failed-job rates, performance baseline adherence, and incident MTTD/MTTR. Qualifications
• BE/B-Tech or equivalent in Computer Science or Electronics & Communication.
• 7–12 years of hands-on experience building and operating HPC/AI GPU clusters at scale.
• Deep expertise in CUDA, MIG, and vGPU.
• Proven experience with NVIDIA H100/B-series GPUs, Slurm and/or Kubernetes device scheduling, and NCCL/UCX performance tuning. Preferred Certifications
• NVIDIA Certified Skilled – AI Infrastructure
• RHCE / LPIC (Linux).
📌 NVIDIA GPU Infrastructure Engineer (Chennai)
🏢 Stratsol Software Systems
📍 Chennai