Director, Product Management - San Jose, California, United States
$260,000 - $351,800 USD yearly
About the role
This role leads the strategy and execution of an AI inference platform across distributed compute environments. It sits at the intersection of hardware and software, defining how modern AI models are executed, served, and optimized at scale.
What you'll do
- Define and drive the product vision, strategy, and roadmap for large-scale AI inference systems. - Own product direction across compute execution, inference serving, and control plane systems. - Drive inference architecture and orchestration strategy, including distributed serving, routing, batching, and scheduling. - Define capabilities for KV cache management, memory optimization, and token lifecycle efficiency (prefill vs decode). - Partner with engineering to enable hardware–software co-design, improving performance across accelerators and interconnects. - Define and track key performance metrics including P99 latency, TTFT, throughput, cost per inference/token, and availability.
What you'll need
- Proven experience in product management within AI/ML infrastructure, cloud platforms, or distributed systems. - Deep understanding of AI inference systems and large-scale serving architectures. - Solid understanding of LLM inference concepts including prefill vs decode, KV cache, and token streaming. - Demonstrated ability to deliver scalable, high-performance platform products. - Strong technical expertise and ability to collaborate directly with engineering teams.
Nice to have
- Experience with accelerator-based systems and performance optimization. - Familiarity with modern inference frameworks such as TensorRT-LLM and vLLM. - Experience working on production-scale AI systems.
Skills: product vision, AI inference platform, LLMs, accelerators, roadmap, market analysis
This role has been open 125 days — well above the 48-day median for Product Management roles.
Product Management · AI ML Hardware
|
Open roles in category
67
|
Median days open
48 d
|
Median salary
$248k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | Arm Holdings | All employers we track in this specialty (67 roles · 22 employers) |
|---|---|---|
| Open roles in this specialty | 2 | 67 |
| Median days open | 119 d | 48 d (+71 d vs this employer) |
| Median salary (USD postings) | — | $248k |
Skills observed across this category: product vision, AI inference platform, LLMs, accelerators, roadmap, market analysis
Who's hiring in this category
- NVIDIA · 22 open roles · median 39 d
- Qualcomm · 10 open roles · median 76 d
- AMD · 5 open roles · median 55 d
- NXP Semiconductors · 4 open roles · median 27 d
- Analog Devices · 3 open roles · median 46 d
- Micron Technology · 3 open roles · median 48 d
How we counted: 67 open Product Management (AI ML Hardware) roles from 22 employers tracked in the SemiconductorJobs index, counted 28 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there. Figures refresh nightly.