Home/Jobs/System Software Engineer - Local AI

System Software Engineer - Local AI

Nvidia

Pune

2+ years

Today

$36.1K–56.6K/yr

Full-time

Onsite

Skills Required

LLM

Gen AI vLLM

Llama.cpp

PyTorch

TensorRT

WinML

DXCGC

CUDA

Vulkan

DirectX

C++

GPU programming quantization pruning

Description

NVIDIA is a leading AI computing company focusing on delivering AI models locally for improved latency, real-time processing, and privacy. The LocalAI team seeks a Systems Software Engineer to build efficient on-device AI software for RTX and DGX-class systems.

Company: NVIDIA

Role: Systems Software Engineer

Experience

- 2+ years of experience with Bachelor's, Master's, or PhD in Computer Science, Software Engineering, Mathematics, or related field, or equivalent experience

Qualification

- Bachelor's degree in Computer Science, Software Engineering, Mathematics, or related field
- Master's degree in Computer Science, Software Engineering, Mathematics, or related field
- PhD in Computer Science, Software Engineering, Mathematics, or related field

Responsibilities

- Partner with software, research, architecture, and product teams to align strategies and technical needs
- Build and optimize local AI inference stack for RTX, RTX Pro and DGX GPUs focusing on performance, stability, and scalability
- Architect and develop up-to-date inference runtimes and execution stacks covering frameworks like Llama.cpp, vLLM, PyTorch, WinML, DXCGC, TensorRT-RTX
- Perform end-to-end optimization of AI models, data pipelines, and inference runtimes
- Apply model optimization techniques such as quantization, pruning, sparsity, and distillation
- Perform system-level debugging, performance optimization, and performance–accuracy trade-off analysis
- Develop infrastructure for performance and accuracy sweeps and analyze results to identify gaps and drive fixes




- Establish engineering guidelines to accelerate bring-up and ensure production readiness of new models and inference backends

Additional Responsibilities

- Facilitate effective collaboration with management and engineering teams through outstanding written and oral communication skills
- Multitask effectively in a dynamic environment

Nice To Have

- Understanding of up-to-date techniques in Machine Learning, Deep Neural Networks, and Generative AI
- Relevant contributions to major open-source projects
- Consistent track record of delivering end-to-end products with geographically distributed teams in multinational product companies
- Proficiency in lower-level system/GPU programming and CUDA
- Experience developing high-performance systems
- Contributions to open-source inference runtimes, model tooling, or performance infrastructure
- Hands-on experience building applications with frameworks and APIs like Llama.cpp, PyTorch, TensorRT, Vulkan, DirectX, vLLM

More Skills debugging, data structures, algorithms, machine learning, AI inferencing pipelines, ML/DL frameworks, inference backends, runtime internals, scheduling, memory management, KV-cache behavior, graph execution, hardware-aware optimization

Prepare for this role

Recommended resources to build the skills for this position. Sponsored.

Top 20 LLM System Design Questions (Quick)

Zenaique A focused LLM system design interview question set.

Top 50 LLM System Design Interview Questions

Zenaique

Curated LLM system design questions from Zenaique.

Top 15 Together AI Interview Questions

Zenaique A Together AI-focused interview question set from Zenaique.

More LLM jobs

AI/ML Engineer

Calsoft

Pune

Today

Senior AI Engineer

Siemens Technology

Bengaluru

Today

AI Tech Lead

Valtech

Bengaluru

Today

Staff GenAI Engineer

EY

Pune

Today

Agentic Engineer Level II

Netomi

India

Today

Senior AI Engineer

Siemens

Bengaluru

Today

📌 System Software Engineer - Local AI (Pune)
🏢 Top Gen AI Jobs
📍 Pune