Software Engineer, Distributed Training - Palo Alto, CA, United States
$200,000 - $420,000 USD yearly
Originally posted 12 September 2026 by the employer.
Build the distributed training engines behind the River API.
About the role
This role involves building distributed training engines for the River API, focusing on making fine-tuning and reinforcement learning fast, numerically correct, and reliable across large GPU clusters.
What you'll do
- Build and optimize distributed training for large dense and mixture-of-experts models, including low-rank adapter training.
- Improve reinforcement-learning pipelines by coordinating sampling, reward computation, training updates, and weight transfer.
- Optimize GPU memory use, parallelism, and communication to increase training throughput.
- Implement reliable checkpointing, resumption, and worker recovery while preserving consistent training state.
- Validate losses, gradients, and optimizer behavior, and diagnose numerical or distributed execution failures.
- Partner with researchers to implement new algorithms and make them accessible through the River API.
What you'll need
- Hands-on experience building or substantially improving distributed model-training systems.
- Strong proficiency in Python and a modern deep-learning framework, such as PyTorch or JAX.
- Solid understanding of backpropagation, optimizers, mixed-precision training, and GPU memory management.
- Strong debugging skills across concurrent execution, collective communication, and distributed failure recovery.
Nice to have
- Experience with reinforcement-learning infrastructure, rollout generation, or asynchronous training.
- Familiarity with tensor, pipeline, expert, or data parallelism and their performance tradeoffs.
- Work on LoRA, mixture-of-experts training, distributed optimizers, or activation checkpointing.
- Experience with NCCL, communication profiling, and overlapping computation with data transfers.
- Proficiency in C++, Rust, or CUDA, with experience investigating performance below the framework layer.
- Contributions to training frameworks or a track record of operating large training runs.
Skills: distributed training engines, GPU clusters, training workloads, optimizer updates, mixed-precision training
This role has been open 33 days — well below the 50-day median for Infrastructure/Platform Software roles.
Infrastructure/Platform Software · Infrastructure Platform
|
Open roles in category
973
|
Median days open
50 d
|
Median salary
$235k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | River | All employers we track in this specialty (973 roles · 85 employers) |
|---|---|---|
| Open roles in this specialty | 3 | 973 |
| Open roles in the wider Software, Firmware & Systems family | 7 | 6145 · 145 employers |
| Median days open | 33 d | 50 d (−17 d vs this employer) |
| Median salary (USD postings) | — | $235k |
Skills observed across this category: distributed training engines, GPU clusters, training workloads, optimizer updates, mixed-precision training
Who's hiring in this category
- NVIDIA · 289 open roles · median 44 d
- Qualcomm · 61 open roles · median 57 d
- AMD · 57 open roles · median 21 d
- Graphcore · 44 open roles · median 90 d
- Intel Corporation · 40 open roles · median 9 d
- Cerebras · 39 open roles · median 80 d
How we counted: 973 open Infrastructure/Platform Software (Infrastructure Platform) roles from 85 employers tracked in the SemiconductorJobs index, counted 15 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.