Home/Jobs/System Software Engineer - Local AI

System Software Engineer - Local AI

Nvidia

Pune

2+ years

Today

$36.1K–56.6K/yr

Full time

Onsite

Skills Required LLM

Gen AI vLLM

Llama.cpp

PyTorch

TensorRT

WinML

DXCGC

CUDA

Vulkan

DirectX

C++

GPU programming quantization pruning

Description NVIDIA is a leading AI computing company focusing on delivering AI models locally for improved latency, real-time processing, and privacy. The LocalAI team seeks a Systems Software Engineer to build efficient on-device AI software for RTX and DGX-class systems.

Company: NVIDIA

Role: Systems Software Engineer

Experience

- 2+ years of experience with Bachelor's, Master's, or PhD in Computer Science, Software Engineering, Mathematics, or related field, or equivalent experience

Qualification

- Bachelor's degree in Computer Science, Software Engineering, Mathematics, or related field
- Master's degree in Computer Science, Software Engineering, Mathematics, or related field
- PhD in Computer Science, Software Engineering, Mathematics, or related field

Responsibilities

- Partner with software, research, architecture, and product teams to align strategies and technical needs
- Build and optimize local AI inference stack for RTX, RTX Pro and DGX GPUs focusing on performance, stability, and scalability
- Architect and develop modern inference runtimes and execution stacks covering frameworks like Llama.cpp, vLLM, PyTorch, WinML, DXCGC, TensorRT-RTX
- Perform end-to-end optimization of AI models, data pipelines, and inference runtimes
- Apply model optimization techniques such as quantization, pruning, sparsity, and distillation
- Perform system-level debugging, performance optimization, and performance–accuracy trade-off analysis
- Develop infrastructure for performance and accuracy sweeps and analyze results to identify gaps and drive fixes




- Establish engineering guidelines to accelerate bring-up and ensure production readiness of new models and inference backends

Additional Responsibilities

- Facilitate effective collaboration with management and engineering teams through outstanding written and oral communication skills
- Multitask effectively in a dynamic environment

Nice To Have

- Understanding of modern techniques in Machine Learning, Deep Neural Networks, and Generative AI
- Relevant contributions to major open-source projects
- Consistent track record of delivering end-to-end products with geographically distributed teams in multinational product companies
- Proficiency in lower-level system/GPU programming and CUDA
- Experience developing high-performance systems
- Contributions to open-source inference runtimes, model tooling, or performance infrastructure
- Hands-on experience building applications with frameworks and APIs like Llama.cpp, PyTorch, TensorRT, Vulkan, DirectX, vLLM

More Skills debugging, data structures, algorithms, machine learning, AI inferencing pipelines, ML/DL frameworks, inference backends, runtime internals, scheduling, memory management, KV-cache behavior, graph execution, hardware-aware optimization

Prepare for this role

Recommended resources to build the skills for this position. Sponsored.

Top 20 LLM System Design Questions (Quick)

Zenaique A focused LLM system design interview question set.

Top 50 LLM System Design Interview Questions

Zenaique

Curated LLM system design questions from Zenaique.

Top 15 Together AI Interview Questions

Zenaique A Together AI-focused interview question set from Zenaique.

More LLM jobs

Staff AI Engineer (Remote Eligible)

Capital One

San Francisco

Today

Senior GenAI / Machine Learning Engineer

Tential

United States

Today

Senior Gen AI Developer

Diverselynx

India

Today

Conversational AI Engineer

Bright Vision Technologies

United States

Today

AI Engineer – Generative AI

AppZime Technologies

Hyderabad

Today

AI & Data Engineering

VG AI Solutions

Hyderabad

Today

📌 System Software Engineer - Local AI (Pune)
🏢 Top Gen AI Jobs
📍 Pune