Engineering Manager, Inference Router

The Inference Router team builds global request routing, prefill/decode (P/D) disaggregation, and prefix-aware routing for KV-cache reuse across data centers.

Originally posted 13 October 2026 by the employer.

About the role

As an Engineering Manager, Inference Router, you will lead the team responsible for the intelligent control plane of the generative AI inference platform, owning global request routing, prefill/decode (P/D) disaggregation, and prefix-aware routing. The team builds systems that balance heterogeneous compute, maximize cache utilization, and deliver low time-to-first-token across data centers and third-party integrations.

What you'll do

• Own the architecture and evolution of the inference routing layer across data centers and third-party inference integrations.
• Guide the design of multi-region request routing, load balancing, heterogeneous compute routing, prefix-aware routing for KV-cache reuse, and low-TTFT request scheduling.
• Build, manage, and grow an engineering team focused on distributed systems, infrastructure, and ML systems.
• Build high-fidelity fleet simulation models to validate routing policies, failover behavior, and scaling decisions without consuming accelerator capacity.
• Drive execution of complex initiatives spanning prefix- and cache-aware routing policies, autoscaling, high availability, observability, and SLOs.
• Partner with inference runtime, cloud/platform, hardware, and ML teams to optimize end-to-end performance and latency.

What you'll need

• 8+ years of experience in large-scale software engineering, distributed systems, or infrastructure.
• 2+ years of engineering management experience.
• Strong programming skills in C++ and/or Go for performance-critical, concurrent production systems.
• Deep experience operating production services on Kubernetes, including deployments, autoscaling, rollout strategies, and multi-cluster or multi-region topologies.
• Proven track record building and operating highly available systems, covering load balancing, failover, scale-out, and reliability engineering.
• Experience designing or operating request routing, load balancing, scheduling, or traffic-management systems at scale, along with cloud infrastructure and microservices.

Nice to have

• Experience with large-scale LLM/inference serving systems such as vLLM, TensorRT-LLM, SGLang, or comparable tools.
• Familiarity with prefill/decode (P/D) disaggregation, KV-cache management, and prefix/cache-aware routing.
• Machine learning background with an understanding of inference workloads, batching, and performance characteristics of serving LLMs and multimodal models.
• Experience with autoscaling and capacity management across heterogeneous compute.
• Experience building simulation, modeling, or load-testing frameworks for distributed systems.
• Strong grounding in observability (metrics, tracing, SLOs) and incident response for high-traffic production systems.

Skills: Kubernetes, C++, Go, Inference Router, Distributed Systems

More semiconductor jobs: Cerebras jobs · Sunnyvale jobs