Senior Software Engineer - AI Inference Performance - CA, Santa Clara, United States
$184,000 - $356,500 USD yearly
Originally posted 26 August 2026 by the employer.
About the role
What you'll do
- Lead end-to-end analysis of LLM/VLM inference processes and define representative prefill and decode workloads.
- Build speed-of-light and roofline models to quantify performance headroom.
- Profile workloads using NVIDIA Nsight Systems, Nsight Compute, PyTorch Profiler, and custom instrumentation.
- Tune serving hyperparameters and techniques such as batching, KV-cache management, quantization, speculative decoding, and CUDA Graphs.
- Build and optimize performance-critical kernels, including attention, matrix multiplication, mixture-of-experts routing, quantization, and data movement.
- Establish repeatable benchmarks, canonical run records, and performance regression gates.
What you'll need
- More than 6 years of experience in full-stack LLM/VLM inference performance involving models, serving, distributed runtimes, kernels, and hardware.
- Strong programming skills in Python, Rust, and/or C++, plus hands-on experience with CUDA or another GPU programming environment.
- Demonstrated expertise in speed-of-light analysis, roofline models, microbenchmarks, and tools including NVIDIA Nsight Systems and Nsight Compute.
- Deep understanding of GPU architecture, including Tensor Cores, memory hierarchy, caches, occupancy, synchronization, and numerical formats.
- Practical experience optimizing inference servers and model execution, choosing techniques for the workload including batching, scheduling, KV-cache management, quantization, speculative decoding, and various parallelism strategies.
- BS or MS in Computer Science, Computer Engineering, or a related field, or equivalent experience.
Skills: LLM/VLM inference, GPU-accelerated systems, CUDA kernels, performance optimization, Nsight Systems
This role has been open 49 days — below the 56-day median for Infrastructure/Platform Software roles.
Infrastructure/Platform Software · GPU Software Stack
|
Open roles in category
231
|
Median days open
56 d
|
Median salary
$236k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | NVIDIA | All employers we track in this specialty (231 roles · 20 employers) |
|---|---|---|
| Open roles in this specialty | 114 | 231 |
| Open roles in the wider Software, Firmware & Systems family | 993 | 6127 · 145 employers |
| Median days open | 55 d | 56 d (−1 d vs this employer) |
| Median salary (USD postings) | — | $236k |
Skills observed across this category: LLM/VLM inference, GPU-accelerated systems, CUDA kernels, performance optimization, Nsight Systems
Who's hiring in this category
- NVIDIA (this employer) · 117 open roles · median 55 d
- AMD · 53 open roles · median 65 d
- Qualcomm · 19 open roles · median 106 d
- Intel Corporation · 17 open roles · median 6 d
- Arm Holdings · 5 open roles · median 112 d
- d-Matrix · 2 open roles · median 120 d
How we counted: 231 open Infrastructure/Platform Software (GPU Software Stack) roles from 20 employers tracked in the SemiconductorJobs index, counted 15 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.