ML Infrastructure Engineer - New Grad - Austin, Texas, United States
$131,200 - $196,800 USD yearly
Originally posted 27 October 2026 by the employer.
Develop inference serving features on the Reconfigurable Dataflow Unit (RDU) architecture.
About the role
As an ML Infrastructure Engineer - New Grad, you will write production Python for SambaNova's inference serving stack, including the vLLM integration and public APIs. You will bring features such as speculative decoding, prompt caching, and tool calling into production on the Reconfigurable Dataflow Unit (RDU).
What you'll do
- Implement inference features on the RDU such as speculative decoding, constrained decoding, tool calling, prompt caching, and long-context inference. - Adapt vLLM and related serving frameworks to the RDU architecture. - Build and maintain endpoints in the public inference API. - Write accuracy verification and regression tests for inference features. - Debug latency, throughput, and reliability issues in production services.
What you'll need
- B.S. in Computer Science, Electrical Engineering, or related field. - Strong programming skills in Python. - Understanding of algorithms, data structures, concurrency, and systems design. - Coursework or project experience in distributed systems or machine learning. - Experience with LLM serving frameworks such as vLLM, SGLang, or TensorRT-LLM. - Experience with inference techniques such as speculative decoding, KV caching, or continuous batching. - Experience with PyTorch.
Nice to have
- M.S. or Ph.D. in Computer Science, Electrical Engineering, or related field. - Experience building or maintaining APIs or backend services. - Internship or research experience in ML systems or backend infrastructure. - Experience with disaggregated systems - prefill/decode. - Open source project ownership or contributions.
Skills: vLLM, inference, speculative decoding, Python, RDU, PyTorch
This role has been open 0 days — well below the 89-day median for AI/ML Hardware Engineering roles.
AI/ML Hardware Engineering · AI ML Hardware
|
Open roles in category
507
|
Median days open
89 d
|
Median salary
$236k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | SambaNova | All employers we track in this specialty (507 roles · 76 employers) |
|---|---|---|
| Open roles in this specialty | 7 | 507 |
| Open roles in the wider Software, Firmware & Systems family | 31 | 6292 · 146 employers |
| Median days open | 72 d | 89 d (−17 d vs this employer) |
| Median salary (USD postings) | — | $236k |
Skills observed across this category: vLLM, inference, speculative decoding, Python, RDU, PyTorch
Who's hiring in this category
- Qualcomm · 104 open roles · median 118 d
- NVIDIA · 88 open roles · median 60 d
- AMD · 32 open roles · median 110 d
- Micron Technology · 27 open roles · median 62 d
- Mobileye · 20 open roles · median 88 d
- Analog Devices · 18 open roles · median 45 d
How we counted: 507 open AI/ML Hardware Engineering (AI ML Hardware) roles from 76 employers tracked in the SemiconductorJobs index, counted 27 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.