Title Lead GPU Engineer

Product Server Fleet Management Platform: management of data-centre server fleets, CPU and GPU, through their BMCs (licensed in tiers)

Team You lead the NVIDIA and AMD GPU engineers, inside a platform team of backend, frontend and test engineers

Tech stack Go (including CGo), C, Linux, NVIDIA NVML/DCGM, AMD SMI/ROCm, PostgreSQL, VictoriaMetrics, NATS JetStream, Temporal, Redfish/IPMI, PLDM/MCTP

Location Bangalore preferred; other cities in India considered

Experience 10+ years in systems or infrastructure software, including 5+ years building software for GPU servers and 2+ years leading engineers on it. Experience with the specific models in our fleet is not required.

Roles & responsibilities

1. Lead the GPU team. Lead the NVIDIA and AMD engineers, set the technical direction, review all GPU code, and help hire and onboard the team.

2. Own characterisation. Run the tests on every model at the start, turn the results into the capability map, and use them to refine the plan and dates.

3. Design the cross-vendor model. Data model, identities, units, capability detection and the common severity scale that both vendor collectors feed.

4. Design fleet-scale telemetry. Metric and label design, collection cadence, retention and storage for 25,000 GPUs on VictoriaMetrics and PostgreSQL.

5. Design control safety. The guardrails every GPU write goes through: model limits, idle checks, read-back, re-apply after reboot, scoped resets, drains, BMC recovery and audit.

6. Sequence the work. Monitoring first,



then diagnostics and control, so every write can be confirmed by reading back.

7. Work across the platform. Agree interfaces with the backend, frontend and test teams, and bring GPU data into the platform’s out-of-band (BMC) path.

8. Own GPU field escalations. Lead root-cause analysis of customer GPU issues and keep detection rules, thresholds and documentation current.

Must-have skills

- Led engineers building GPU or hardware fleet software in production: monitoring or management software that ran across thousands of servers or GPUs. You can walk through its architecture and what you would change.

- Deep API-level work with NVML/DCGM or AMD SMI/ROCm, and working knowledge of the other vendor.

- Go and C in production: calling C libraries from Go, concurrency, profiling, long-running agents.

- Linux on bare metal and GPU faults: PCIe, drivers, kernel logs, XID and RAS errors, ECC, GPU loss and recovery.

- Metrics at scale: cardinality, retention and alerting design (VictoriaMetrics, Prometheus or similar).

- Technical leadership: you have hired or grown engineers, written design documents and run code review.

Pay: ₹3,500,000.00 - ₹5,000,000.00 per year

Advantages:

- Paid time off
- Work from home

Application Question(s):

- What is your current CTC, Expected CTC & Notice Period ?

Experience:

- infrastructure software: 10 years (Required)
- software for GPU servers: 5 years (Required)
- leading engineers: 2 years (Required)

Work Location: Remote

📌 Lead GPU Engineer (India)
🏢 SecHorizon Technologies
📍 India