Inference Systems Performance Architect - San Jose, California, United States
$245,000 - $325,000 USD yearly
Originally posted 16 August 2026 by the employer. Relisted by the employer on 21 October 2026.
Architect end-to-end LLM inference performance across Reconfigurable Dataflow Unit (RDU) and GPU disaggregated architectures.
About the role
You will be the architect for end-to-end inference performance, covering how requests move through tokenization, prefill, decode, and the interconnecting fabric. The focus centers on heterogeneous, disaggregated inference across GPU prefill and Reconfigurable Dataflow Unit (RDU) decode architectures.
What you'll do
- Define and drive technical strategy for inference systems performance, spanning workload capture, benchmarking, modeling, and simulation. - Build workload-capture and agentic-benchmarking capabilities using representative production traffic to evaluate real-world contention and cache-hit behaviors. - Own performance modeling and simulation practices to predict configuration impacts and inform capacity planning against customer SLOs. - Drive end-to-end profiling tooling to pinpoint bottlenecks across distributed pipelines spanning host, accelerator, and fabric. - Guide cross-functional technical decisions across model optimization, systems, hardware, and product teams. - Mentor principal and senior engineers while establishing systems, tools, and methodologies that enhance engineering productivity.
What you'll need
- B.S. in Computer Science, Computer Engineering, or a related field. - 12+ years of experience in performance engineering with demonstrated technical leadership on large-scale, complex distributed systems. - Deep expertise in end-to-end performance analysis of distributed systems to localize bottlenecks across complex pipelines. - Proven command of realistic workload generation, simulation, and performance modeling calibrated against variable production workloads. - Demonstrated ability to independently scope and deliver high-ambiguity performance initiatives impacting hardware or product roadmaps.
Nice to have
- M.S. or Ph.D. in Computer Science, Computer Engineering, or a related field. - Direct experience with LLM inference serving, including continuous batching, prompt/KV caching, prefill/decode disaggregation, and tail-latency SLOs. - Familiarity with inference simulation frameworks or agentic benchmarking efforts. - A public technical voice through talks, writing, or community contributions on systems performance.
Skills: Inference Systems Performance, LLM inference, performance modeling and simulation, heterogeneous, disaggregated inference, AI Network Infrastructure
This role has been open 66 days — well below the 83-day median for AI/ML Hardware Engineering roles.
AI/ML Hardware Engineering · AI ML Hardware
|
Open roles in category
511
|
Median days open
83 d
|
Median salary
$236k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | SambaNova | All employers we track in this specialty (511 roles · 76 employers) |
|---|---|---|
| Open roles in this specialty | 6 | 511 |
| Open roles in the wider Software, Firmware & Systems family | 26 | 6296 · 145 employers |
| Median days open | 83 d | 83 d (same as this employer) |
| Median salary (USD postings) | — | $236k |
Skills observed across this category: Inference Systems Performance, LLM inference, performance modeling and simulation, heterogeneous, disaggregated inference, AI Network Infrastructure
Who's hiring in this category
- Qualcomm · 103 open roles · median 112 d
- NVIDIA · 91 open roles · median 56 d
- AMD · 40 open roles · median 85 d
- Micron Technology · 27 open roles · median 56 d
- Analog Devices · 18 open roles · median 39 d
- Mobileye · 18 open roles · median 84 d
How we counted: 511 open AI/ML Hardware Engineering (AI ML Hardware) roles from 76 employers tracked in the SemiconductorJobs index, counted 21 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.