Build the runtime architecture, hardware abstraction layer, and device management interfaces for in-memory LLM inference hardware.
Originally posted October 20, 2026 by the employer.
About the role
As a Runtime Engineer at Normal Computing, you will own the technical direction, design, and implementation of the runtime layer that loads, schedules, and executes compiled programs on custom hardware. You will build this layer against simulators and FPGA prototypes to create a stable hardware abstraction for compiler and serving layers before silicon arrives.
What you'll do
- Own the runtime architecture, including device management, memory allocation, host-device transfers, executable loading, and command submission.
- Build a hardware abstraction interface that conceals the differences between simulators, FPGA prototypes, and future silicon generations.
- Manage persistent device state across requests, including resident model weights, to support serving layers.
- Minimize runtime overhead on the critical path for LLM inference to optimize end-to-end latency and throughput.
- Define interface boundaries with the compiler layer above and the driver and firmware layers below.
What you'll need
- Experience building runtimes, drivers, or hardware abstraction layers for accelerators or non-standard hardware.
- Strong systems programming experience in C++ or Rust covering memory management, concurrency, and low-level performance.
- Track record of owning a complex systems component end-to-end from initial design through production.
- Working understanding of LLM inference workloads and their runtime requirements.
- Experience developing software for pre-silicon targets across simulation and bring-up environments.
Nice to have
- Experience with open runtime architectures such as IREE's HAL, or internal familiarity with CUDA or ROCm runtimes.
- Experience with in-memory compute or other non-von-Neumann hardware architectures.
- Experience integrating a custom runtime with inference serving frameworks such as vLLM.
- Contributions to open-source runtime or systems infrastructure projects.
More semiconductor jobs: Normal Computing jobs · Embedded software jobs · Mountain View jobs
Tracked in our index since 20 Oct 2026. This is our observation date, not the employer posting date. Specialty: Systems Software Engineering (Systems Software). 720 active roles in this specialty at 79 employers in our index, as of 20 Nov 2026.
How we counted: postings read from employer career sites. How we count