Sr. Principal Machine Learning Software Engineer – Inference Serving Engine

Build and optimize AI inference-serving software across compilers, runtimes, and kernels for Ampere deep learning accelerators.

About the role

As a Sr. Principal Machine Learning Software Engineer, you will lead the architecture and development of high-performance AI inference-serving software for Ampere accelerators. You will work across model serving, frameworks, compilers, runtimes, and optimized kernels to enable deployment of modern AI workloads, including generative AI and large language models.

What you'll do

• Drive end-to-end serving performance, addressing throughput, latency, tail latency, memory efficiency, scalability, and cost per inference.
• Develop serving capabilities including batching and scheduling, KV-cache management, model loading, memory planning, quantization, and multi-device or distributed inference.
• Optimize model execution through graph compilation, operator fusion, precision optimization, efficient attention, and hardware-specific execution strategies.
• Collaborate across model-serving, framework, compiler, runtime, kernel, and hardware teams, influencing accelerator architecture through hardware/software co-design.
• Establish technical direction, design standards, benchmarking practices, and performance-validation processes while evaluating and integrating open-source AI serving technologies.

What you'll need

• Bachelor’s degree in Computer Science, Computer Engineering, Mathematics, Electrical Engineering, or a related field with 12+ years of experience, Master’s with 8+ years, or Ph.D. with 5+ years.
• Experience developing or optimizing production inference runtimes, serving engines, deployment toolkits, or high-performance AI systems.
• Experience with technologies such as vLLM, llama.cpp, TensorRT/TensorRT-LLM, ONNX Runtime, OpenVINO, Triton Inference Server, TVM, XLA, MLIR, or comparable systems.
• Expert proficiency in modern C++ and strong experience in C, including concurrent programming, memory management, and systems-level performance optimization.
• Understanding of AI inference performance factors such as batching, scheduling, latency/throughput tradeoffs, memory bandwidth, KV-cache behavior, quantization, and accelerator utilization.
• Experience with profiling, benchmarking, performance analysis, and low-level systems debugging.

Nice to have

• Experience with LLM, transformer, multimodal, or generative-AI inference.
• Experience with AI frameworks and heterogeneous computing technologies such as PyTorch, oneDNN, SYCL, OpenCL, OpenMP, MPI, CUDA, or ROCm.
• Experience with ML compilers, graph optimization, code generation, kernel development, or device runtimes.
• Familiarity with distributed inference, cloud-native serving, containers, Kubernetes, orchestration, observability, or multi-tenant systems.
• Experience with model optimization techniques including mixed precision, INT8/INT4/FP8 quantization, sparsity, or efficient-attention methods.
• Contributions to open-source AI, compiler, runtime, or HPC projects, or experience with hardware/software co-design.

Skills: C++, inference engine, vLLM, compiler, runtime

More semiconductor jobs: Santa Clara jobs