Senior Software Engineer, ML API – AI Inference - Sunnyvale, CA, United States

Originally posted 27 October 2026 by the employer.

Cerebras Wafer-Scale Engine heterogeneous inference platform.

About the role

Cerebras is hiring a Senior Software Engineer to build and evolve the ML API layer across inference APIs, model integration, and request-routing services.

What you'll do

- Design, implement, test, and operate features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, and multimodal inputs. - Implement clear request and response semantics across GPU prefill, Cerebras decode, and other inference backends, maintaining compatibility with widely adopted inference interfaces. - Integrate foundation models, tokenizers, chat templates, generation configuration, and sampling methods into the serving platform. - Extend inference services and their integration with frameworks and runtimes such as vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components. - Implement routing, state-transfer integration, error handling, retries, cancellation, and request lifecycle management to coordinate GPU prefill with Cerebras decode. - Diagnose bottlenecks to improve time to first token, streaming latency, throughput, and resource efficiency via tokenization, serialization, batching, scheduling, and communication.

What you'll need

- 5+ years of software engineering experience, including ownership of production services, distributed systems, or substantial software components. - Strong programming ability in Python and proficiency in Go, C++, Rust, or another language used for production services. - Experience building and evolving production APIs with REST, gRPC, or streaming protocols, including validation, error handling, compatibility, and testing practices. - Experience operating services on Linux using containers, Kubernetes or comparable orchestration systems, and CI/CD pipelines. - Experience integrating software across service, framework, runtime, or infrastructure boundaries. - Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice to have

- Experience building or operating production ML or LLM inference APIs, including OpenAI-compatible interfaces. - Experience extending or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another ML systems project. - Understanding of tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, or constrained decoding. - Experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, or inference request scheduling. - Experience developing performance-sensitive or highly concurrent services in Go, C++, Rust, or a similar language. - Familiarity with GPU inference, AMD Instinct accelerators, ROCm, CUDA, collective communication, or reduced-precision quantization formats (BF16, FP8, FP4, INT8, INT4).

Skills: Python, distributed systems, inference API, model serving, vLLM