Principal Software Engineer, At-Scale Reliability and Fleet Intelligence — CSP Engagements
$272,000 - $431,250 USD yearly
Originally posted 20 July 2026 by the employer.
About the role
As a Principal Software Engineer on the CSP Engagements team, you will serve as the technical focal point for fleet-scale reliability, working directly with engineering teams at CSP and hyperscale customers to ensure NVIDIA platforms achieve target MTBI in production.
What you'll do
- Drive reliability work streams with CSP engineering teams, ensuring shared understanding of MTBI measurement methodology, failure classification, and health monitoring architecture. - Gather and synthesize CSP fleet reliability data to identify cross-customer failure patterns and champion improvements into firmware, driver, and hardware teams. - Conduct fleet-scale failure pattern analysis using statistical methods such as Pareto analysis, survival analysis, and Weibull distributions. - Drive fleet health monitoring integration architecture to align health agents, telemetry, and reporting with CSP operational workflows and automation. - Define burn-in reliability test environments and cluster certification criteria in collaboration with quality teams, validating criteria directly with customers. - Develop predictive failure models using fleet telemetry and validate their effectiveness in customer production environments.
What you'll need
- 15+ years of experience in systems software at datacenter scale, or reliability engineering with a focus on at-scale challenges. - BS or MS in Computer Science, Electrical Engineering, Statistics, or related field (or equivalent experience). - Deep expertise in multi-NUMA, rack-scale system software, firmware, and statistical failure analysis methods including MTBF/MTBI calculation, Pareto analysis, and root cause classification. - Experience with fleet-level telemetry and observability systems, including time-series databases, anomaly detection, health scoring, and event correlation. - Understanding of hardware failure modes in large-scale GPU/accelerator deployments across compute, interconnect, memory, power, and thermal domains. - Experience defining or operating burn-in, stress testing, or certification frameworks for complex hardware systems, alongside predictive maintenance or anomaly detection techniques.
Skills: fleet telemetry, reliability software/firmware, GPU/accelerator deployments, statistical failure analysis, hyperscaler
This role has been open 116 days — well above the 50-day median for Infrastructure/Platform Software roles.
Infrastructure/Platform Software · Infrastructure Platform
|
Open roles in category
1020
|
Median days open
50 d
|
Median salary
$230k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | NVIDIA | All employers we track in this specialty (1020 roles · 89 employers) |
|---|---|---|
| Open roles in this specialty | 293 | 1020 |
| Open roles in the wider Software, Firmware & Systems family | 1004 | 6296 · 145 employers |
| Median days open | 47 d | 50 d (−3 d vs this employer) |
| Median salary (USD postings) | — | $230k |
Skills observed across this category: fleet telemetry, reliability software/firmware, GPU/accelerator deployments, statistical failure analysis, hyperscaler
Who's hiring in this category
- NVIDIA (this employer) · 293 open roles · median 48 d
- Qualcomm · 65 open roles · median 49 d
- AMD · 58 open roles · median 23 d
- Graphcore · 50 open roles · median 97 d
- Intel Corporation · 48 open roles · median 13 d
- Cerebras · 36 open roles · median 87 d
How we counted: 1020 open Infrastructure/Platform Software (Infrastructure Platform) roles from 89 employers tracked in the SemiconductorJobs index, counted 22 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.