Skip to main content
Arm Holdings

Staff Software Engineer, AI Inference Runtime

Arm Holdings Seattle, Washington, United States 2 months ago
Embedded & Systems Software

$209,100 - $282,900 USD yearly

Set technical direction for distributed inference runtimes spanning scheduling, KV-cache management, and kernel optimization.

Originally posted September 9, 2026 by the employer.

About the role

As a Staff Software Engineer on the AI Inference Runtime team, you will set technical direction for critical components of distributed inference runtime for running AI models. You will lead work across scheduling, batching, KV-cache management, memory allocation, distributed workload execution, and kernel development.

What you'll do

  • Define the architecture, interfaces, and roadmap for AI inference runtime capabilities and abstractions supporting evolving models, workloads, and compute platforms.
  • Enable new model architectures end to end through operator support, production validation, and optimization of scheduling, batching, model execution, memory management, and KV-cache efficiency.
  • Profile system bottlenecks and develop optimized kernels and data-movement paths across compute, memory, networking, and framework integration.
  • Evaluate new inference techniques and build benchmarking, regression, validation, and safe-rollout systems to improve latency, throughput, reliability, and resource efficiency.
  • Partner with cloud, framework, compiler, hardware, and research teams while mentoring engineers and leading technical reviews.

What you'll need

  • 8+ years of experience, or equivalent demonstrated impact, in ML systems, high-performance systems, compilers, kernel development, or production AI inference.
  • Deep understanding of modern AI inference, including model execution, Attention, MoE, batching, prioritisation, and KV-cache behavior.
  • Strong programming skills in C++, Rust, Python, or a comparable language, with knowledge of concurrency, parallel programming, handling of memory resources, and data movement.
  • Proven ability to profile, debug, and optimize performance across kernels, runtimes, frameworks, operating systems, and hardware.

Nice to have

  • Experience developing or modifying inference schedulers, cache managers, batching systems, disaggregated or distributed execution paths.
  • Experience optimizing kernels using accelerator programming tools, assembly, or intrinsics, including attention, matrix multiplication, operator fusion, and low-precision execution.
  • Familiarity with model parallelism, collective communication, high-performance networking, compilers, or graph optimization.
  • Contributions to open-source ML runtimes, frameworks, compilers, or kernel libraries.

More semiconductor jobs: Arm Holdings jobs · Embedded software jobs

Market context

Tracked in our index since 29 Oct 2026. This is our observation date, not the employer posting date. Specialty: Systems Software Engineering (Systems Software). 722 active roles in this specialty at 78 employers in our index, as of 21 Nov 2026.

How we counted: postings read from employer career sites. How we count

Apply now
Seattle, Washington, United States
On-site
$209,100 - $282,900 USD yearly
2 months ago

Share this job