Technical Director, Large-Scale AI Model Inferencing - San Jose, California, United States

Originally posted 10 September 2026 by the employer.

This role focuses on memory as the core product of AI inference, covering GPU HBM, host DRAM, CXL-attached memory pools, and NVMe/SSD tiers.

About the role

This role involves owning the requirement for full-stack AI memory solutions at scale, connecting model behavior to memory-system design.

What you'll do

• Serve as expert on how different model families consume and move memory, translating into memory-product requirements for Dense Transformers, Mixture-of-Experts, and State Space Models.
• Model the memory footprint, bandwidth demand, and access patterns of frontier open-weight models like Llama/Qwen, DeepSeek, and Jamba-class hybrids.
• Own deep expertise in production inference stacks such as SGLang, vLLM, NVIDIA Dynamo, and TensorRT-LLM, including their memory-management internals.
• Drive inference performance engineering, including continuous batching, chunked prefill, disaggregated prefill/decode, and speculative decoding.
• Define engineering requirements for tiered memory systems for inference at fleet scale, utilizing HBM, host DRAM, CXL-attached memory pools, and NVMe/SSD.
• Develop and deliver POCs demonstrating inference TCO improvement from the memory stack, focusing on context capacity multiplication or cost-per-token reduction.

What you'll need

• 12+ years in systems engineering, with 4+ years hands-on in large-scale LLM inference or GPU systems performance.
• First-principles understanding of transformer-class model internals, including KV-cache size formulas, MQA/GQA/MLA tradeoffs, and activation-memory peaks.
• Working expertise with MoE model behavior, including routing, expert parallelism, load skew, and weight-memory economics.
• Direct experience with memory-management internals of inference stacks like vLLM PagedAttention/block manager, SGLang HiCache, TensorRT-LLM KV manager, or llama.cpp compute buffers.
• Strong performance-engineering skills: bandwidth-bound vs. compute-bound analysis, NUMA and PCIe topology reasoning, RDMA basics, and fluency with GPU/CPU profilers.
• Track record of building systems software at the memory/storage/IO layer—caches, tiering, paging, or storage engines—with production deployments.

Nice to have

• Experience with State Space Model or hybrid SSM-attention serving (Mamba-class state management, cache swapping for recurrent models).
• Contributions to open-source inference/caching projects (vLLM, SGLang, LMCache, HiCache, Mooncake, KTransformers, llama.cpp).
• Experience with CXL memory pooling, CXL-attached tiering, or near-memory processing in real deployments or serious prototypes.
• Experience with SSD/NVMe as a KV or expert cache tier, including QoS engineering for inference-grade latency.
• Background in memory/storage product companies bringing hardware-software co-designed solutions to market.

Skills: AI model inferencing, memory-bandwidth, GPU HBM, CXL-attached memory, production inference stacks, memory tiering