Senior Product Manager – AI Inference Performance - CA, Santa Clara, United States
$208,000 - $327,750 USD yearly
Originally posted 13 August 2026 by the employer.
This role defines the product roadmap for AI inference performance across TensorRT-LLM, vLLM, and SGLang.
About the role
This role involves owning the products that enable customers to extract optimal performance from AI models and applications on NVIDIA hardware, focusing on latency, efficiency, and cost per token.
What you'll do
- Own the inference performance roadmap, setting direction for model representation, memory management, request scheduling, and token generation.
- Build platforms that generalize across model families, deployment topologies, and customer sizes for easy adoption and extensibility.
- Define the performance strategy for agentic applications, addressing cross-turn cache reuse, request prioritization, and efficient idle time handling.
- Define how optimizations land across TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo.
- Own how AI inference performance is measured, published, and reproduced.
- Run the product day to day, covering release readiness, quality bars, regression tracking, and customer feedback.
What you'll need
- 12+ years in product management at a technology company, or comparable experience as a founder, engineering lead, or technical product owner.
- Depth in AI inference optimization, including KV caching and reuse, quantization, speculative decoding, and disaggregated serving.
- Familiarity with inference and orchestration frameworks such as TensorRT-LLM, vLLM, SGLang, NVIDIA Dynamo, and the surrounding serving and scheduling ecosystem.
- Proven track record of working independently to define strategy and drive outcomes.
- Operational experience running a live product, including release management, quality rigor, and customer support processes.
- Skill at translating low-level capability into business value for engineers and executives.
- BS, MS, or PhD in Computer Science, Computer Engineering, or another relevant area of study (or equivalent experience).
Skills: AI inference performance, TensorRT-LLM, vLLM, SGLang, product roadmap, benchmarking
This role has been open 64 days — well above the 43-day median for Product Management roles.
Product Management · AI ML Hardware
|
Open roles in category
70
|
Median days open
43 d
|
Median salary
$248k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | NVIDIA | All employers we track in this specialty (70 roles · 22 employers) |
|---|---|---|
| Open roles in this specialty | 22 | 70 |
| Median days open | 32 d | 43 d (−11 d vs this employer) |
| Median salary (USD postings) | — | $248k |
Skills observed across this category: AI inference performance, TensorRT-LLM, vLLM, SGLang, product roadmap, benchmarking
Who's hiring in this category
- NVIDIA (this employer) · 22 open roles · median 31 d
- Qualcomm · 10 open roles · median 65 d
- AMD · 8 open roles · median 50 d
- NXP Semiconductors · 4 open roles · median 16 d
- Analog Devices · 3 open roles · median 35 d
- SambaNova · 3 open roles · median 37 d
How we counted: 70 open Product Management (AI ML Hardware) roles from 22 employers tracked in the SemiconductorJobs index, counted 17 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there. Figures refresh nightly.