Senior Staff LLM Inference Engineer - Santa Clara, United States

Originally posted 30 June 2026 by the employer.

About the role

As a Senior Staff LLM Inference Engineer at d-Matrix, you will work across every layer of the inference stack from kernel-level optimization to distributed orchestration and high-level serving APIs. The scope includes pathfinding novel deployment patterns, developing custom kernels, and building systems that integrate d-Matrix silicon with CPUs, GPUs, and custom accelerators.

What you'll do

- Identify and prototype LLM inference use cases suited to heterogeneous hardware deployments. - Build proof-of-concept systems demonstrating d-Matrix capabilities to customers, partners, and internal stakeholders. - Develop and tune custom kernels and operator-level optimizations to maximize throughput and minimize latency. - Drive quantization, sparsity, and batching strategies tailored to the d-Matrix computational model. - Build and maintain inference runtimes, serving frameworks, and evaluation tooling. - Contribute to distributed inference systems covering tensor and pipeline parallelism, disaggregated prefill/decode, and KV-cache management.

What you'll need

- Bachelor’s degree in Computer Science, Electrical Engineering, or a related field with 10+ years of relevant engineering experience (or Master’s/PhD with 6+ years of relevant industry experience). - Strong proficiency in Python and C/C++. - Hands-on experience optimizing LLM inference, including attention kernels, KV cache, batching strategies, and quantization (INT8/FP8/INT4). - Contributor-level experience with at least one major inference framework such as vLLM, SGLang, TensorRT-LLM, or ONNX Runtime. - Familiarity with GPU kernel programming (CUDA/Triton) and performance profiling tools.

Nice to have

- Experience with heterogeneous compute deployments scheduling inference workloads across dissimilar hardware (accelerators, CPUs, GPUs). - Familiarity with custom silicon or ASIC-based inference beyond GPU-only environments. - Experience with distributed inference covering tensor parallelism, pipeline parallelism, and disaggregated serving. - Contributions to open-source inference or ML systems projects. - Experience with production inference serving at scale, including latency SLOs, continuous batching, and multi-model serving. - Familiarity with speculative decoding, mixture-of-experts routing, or long-context-serving techniques. - Systems-level understanding of modern LLM training and inference equivalent to the material in the JAX Scaling Book.

Skills: LLM inference, GPU kernel programming, CUDA, Triton, heterogeneous compute