Software Engineer, Inference Systems - Palo Alto, CA, United States
$200,000 - $420,000 USD yearly
Originally posted 12 September 2026 by the employer.
About the role
As a Software Engineer, Inference Systems, you will build the engines that serve large models through the River API, focusing on delivering fast, reliable inference while efficiently utilizing GPU compute and memory.
What you'll do
- Optimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters.
- Improve batching, caching, and admission control to balance throughput, latency, memory use, and fairness.
- Improve RL sampling throughput while keeping samples and log probabilities tied to the correct model version.
- Build reliable streaming, cancellation, and recovery under failures and overload.
- Profile bottlenecks and validate improvements through reproducible performance and correctness tests.
What you'll need
- Bachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience.
- Experience building inference engines or performance-sensitive distributed services.
- Strong understanding of transformer inference, GPU memory, concurrency, and networking.
- Proficiency in Python and C++ or Rust.
- Strong debugging and profiling skills across models, runtimes, and services.
Nice to have
- Experience extending SGLang, vLLM, TensorRT-LLM, or similar frameworks.
- Work on advanced serving techniques, such as speculative decoding or disaggregated prefill and decode.
- Familiarity with expert parallelism, GPU collectives, and quantized inference.
- Experience with multi-adapter serving, dynamic checkpoint loading, or RL sampling.
- Familiarity with CUDA graphs, custom kernels, and NVIDIA profiling tools.
- Experience operating model-serving systems under production traffic.
Skills: inference engines, GPU compute, distributed model execution, serving runtime, transformer inference
This role has been open 33 days — well below the 50-day median for Infrastructure/Platform Software roles.
Infrastructure/Platform Software · Infrastructure Platform
|
Open roles in category
973
|
Median days open
50 d
|
Median salary
$235k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | River | All employers we track in this specialty (973 roles · 85 employers) |
|---|---|---|
| Open roles in this specialty | 3 | 973 |
| Open roles in the wider Software, Firmware & Systems family | 7 | 6145 · 145 employers |
| Median days open | 33 d | 50 d (−17 d vs this employer) |
| Median salary (USD postings) | — | $235k |
Skills observed across this category: inference engines, GPU compute, distributed model execution, serving runtime, transformer inference
Who's hiring in this category
- NVIDIA · 289 open roles · median 44 d
- Qualcomm · 61 open roles · median 57 d
- AMD · 57 open roles · median 21 d
- Graphcore · 44 open roles · median 90 d
- Intel Corporation · 40 open roles · median 9 d
- Cerebras · 39 open roles · median 80 d
How we counted: 973 open Infrastructure/Platform Software (Infrastructure Platform) roles from 85 employers tracked in the SemiconductorJobs index, counted 15 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.