Software Engineer, Inference Systems - Palo Alto, CA, United States
$200,000 - $420,000 USD yearly
Originally posted 12 September 2026 by the employer.
About the role
As a Software Engineer, Inference Systems, you will build the engines that serve large models through the River API, focusing on delivering fast, reliable inference while efficiently utilizing GPU compute and memory.
What you'll do
- Optimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters.
- Improve batching, caching, and admission control to balance throughput, latency, memory use, and fairness.
- Improve RL sampling throughput while keeping samples and log probabilities tied to the correct model version.
- Build reliable streaming, cancellation, and recovery under failures and overload.
- Profile bottlenecks and validate improvements through reproducible performance and correctness tests.
What you'll need
- Bachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience.
- Experience building inference engines or performance-sensitive distributed services.
- Strong understanding of transformer inference, GPU memory, concurrency, and networking.
- Proficiency in Python and C++ or Rust.
- Strong debugging and profiling skills across models, runtimes, and services.
Nice to have
- Experience extending SGLang, vLLM, TensorRT-LLM, or similar frameworks.
- Work on advanced serving techniques, such as speculative decoding or disaggregated prefill and decode.
- Familiarity with expert parallelism, GPU collectives, and quantized inference.
- Experience with multi-adapter serving, dynamic checkpoint loading, or RL sampling.
- Familiarity with CUDA graphs, custom kernels, and NVIDIA profiling tools.
- Experience operating model-serving systems under production traffic.
Skills: inference engines, GPU compute, distributed model execution, serving runtime, transformer inference
This role has been open 45 days — below the 54-day median for Infrastructure/Platform Software roles.
Infrastructure/Platform Software · Infrastructure Platform
|
Open roles in category
1009
|
Median days open
54 d
|
Median salary
$223k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | River | All employers we track in this specialty (1009 roles · 91 employers) |
|---|---|---|
| Open roles in this specialty | 3 | 1009 |
| Open roles in the wider Software, Firmware & Systems family | 7 | 6292 · 146 employers |
| Median days open | 45 d | 54 d (−9 d vs this employer) |
| Median salary (USD postings) | — | $223k |
Skills observed across this category: inference engines, GPU compute, distributed model execution, serving runtime, transformer inference
Who's hiring in this category
- NVIDIA · 282 open roles · median 53 d
- Qualcomm · 65 open roles · median 54 d
- AMD · 57 open roles · median 28 d
- Intel Corporation · 49 open roles · median 13 d
- Graphcore · 48 open roles · median 102 d
- Cerebras · 35 open roles · median 92 d
How we counted: 1009 open Infrastructure/Platform Software (Infrastructure Platform) roles from 91 employers tracked in the SemiconductorJobs index, counted 27 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.