Sr. Staff Software Engineer, Inference Platform - Sunnyvale, CA, United States

Originally posted 19 June 2026 by the employer.

This role on the Inference Platform team primarily owns the orchestration layer that runs inference on datacenter clusters.

About the role

This role is for a Sr. Staff Software Engineer on the Inference Platform team, primarily owning the orchestration layer connecting cloud components with machine learning services.

What you'll do

• Design, develop, test, and maintain production software across testing, continuous development, observability, security, networking, debugging, and productionization.
• Help shape the technical direction for the Inference Platform, including Kubernetes custom resource definitions, failure domains, service boundaries, and system evolution, and own the roadmap for major technical areas.
• Architect active-active systems with rapid failover, graceful degradation, and clear SLOs, driving system-level improvements in latency, throughput, capacity efficiency, and resilience.
• Write and review production code in important parts of the platform and make high-consequence architectural decisions.
• Lead on production issues and cross-system bottlenecks, driving observability, incident response, capacity planning, and post-incident improvement.
• Partner with ML, Product, Infrastructure, and Cloud teams to translate product and business requirements into scalable system designs.

What you'll need

• 8+ years of experience in software engineering, with substantial individual contributor experience building and operating large-scale distributed systems or cloud infrastructure.
• Deep expertise in distributed systems architecture, ideally with Kubernetes.
• Strong track record of making sound architectural decisions for highly available, latency-sensitive systems at scale.
• Experience with security (certificates, TLS, mTLS).
• Experience optimizing latency, throughput, and efficiency in high-QPS systems.
• Strong proficiency in backend or systems languages such as Go or C++, with the expectation to contribute production code directly.
• Experience designing observability and reliability practices, including metrics, logging, tracing, alerting, incident response, and SLO-driven operations.
• Ability to influence senior engineers and cross-functional partners through technical credibility, communication, and judgment.

Nice to have

• Experience with TTFT and tail-latency reduction.

Skills: Inference Platform, orchestration layer, Kubernetes, distributed systems, production software