Staff Software Engineer, Inference API - Sunnyvale, CA, United States

Originally posted 23 September 2026 by the employer.

About the role

This role involves building and evolving the ML API layer for a new generation of disaggregated AI inference systems.

What you'll do

• Design, implement, and maintain APIs for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, and multimodal inputs.
• Create consistent request and response semantics across GPU prefill and Cerebras decode.
• Integrate emerging foundation models, tokenizers, prompt formats, sampling methods, attention variants, and multimodal inputs into the serving platform.
• Maintain compatibility with widely adopted inference interfaces while designing Cerebras-specific extensions, including versioning and deprecation.
• Extend and integrate custom inference services with vLLM, PyTorch, Hugging Face libraries, the AMD ROCm stack, and Cerebras runtime components.
• Build the control and data paths required to coordinate GPU prefill with Cerebras decode, including request routing, state transfer, error handling, retries, and lifecycle management.

What you'll need

• 5+ years of software engineering experience, with substantial individual-contributor ownership of production software or distributed systems.
• Strong programming ability in Python and Go, plus experience developing performance-sensitive or highly concurrent services in C++, Rust, or a similar systems language.
• Experience building stable APIs with clear validation, error handling, observability, compatibility, and versioning practices.
• Experience integrating software across service, framework, runtime, and infrastructure boundaries.
• Experience designing or maintaining OpenAI-compatible, gRPC, REST, or streaming inference APIs.
• Experience with Linux, containers, Kubernetes or comparable orchestration systems, CI/CD, and operating latency-sensitive services in production.
• Ability to diagnose correctness, reliability, and performance issues across multiple components of a distributed serving system.
• Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice to have

• Experience modifying or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another open-source ML systems project.
• Experience creating API conformance, model-quality, numerical-comparison, determinism, or performance-regression test systems.
• Experience building SDKs, developer tools, model registries, configuration systems, or self-service ML platforms.
• Experience with multi-model or multi-tenant inference platforms, including routing, admission control, fairness, quotas, rate limiting, and capacity-aware scheduling.
• Understanding of model-specific tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, and constrained decoding.
• Experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, memory-aware admission control, or request scheduling.
• Experience designing, building, or operating production APIs and services for machine learning, large language models, or other data-intensive applications.
• Familiarity with reduced-precision inference and quantization formats such as BF16, FP8, FP4, INT8, or INT4.

Skills: ML API layer, heterogeneous serving system, inference APIs, model integration, distributed systems