Agent Evaluation and Self-Improvement Engineer - CA, Santa Clara, United States
$224,000 - $431,250 USD yearly
Originally posted 16 October 2026 by the employer.
About the role
This role involves setting evaluation strategy for autonomous agents across NVIDIA, from individual skills to entire products, and designing how agents use these results to improve themselves.
What you'll do
- Build evaluation that works consistently across harnesses, with automated checks for regressions.
- Mine production traces for failures, usage patterns, missing skills, cost/latency issues, and successful runs to create evaluation cases and improvement signals.
- Evaluate evaluation datasets for real usage coverage, duplicate cases, and production match.
- Create a self-improvement loop for evolving harnesses, where agents propose and test changes to their skills, plugins, and harness configuration in sandboxed environments.
- Measure multi-agent swarms, including work distribution, handoff success, and permission adherence.
- Calibrate model and agent judges against human labels and detect reward hacking or eval gaming.
What you'll need
- BS or MS in Computer Science, Engineering, or related field (or equivalent experience).
- 12+ years building large-scale systems, including evaluation systems for ML or agents and their data pipelines.
- Hands-on experience building and evaluating agents on an open or closed agent harness.
- Solid grounding in experiment design: statistical significance, variance across repeated runs, and A/B testing.
- Experience setting technical direction across teams as a senior individual contributor.
- Proficiency in Python, Go, Rust, or similar.
Skills: autonomous agents, evaluation systems, data pipelines, experiment design, Python
This role has been open 1 day — well below the 51-day median for Infrastructure/Platform Software roles.
Infrastructure/Platform Software · Infrastructure Platform
|
Open roles in category
962
|
Median days open
51 d
|
Median salary
$236k
|
See the full market breakdown ▾Category comparison, skills in demand, and who else is hiring
| Metric | NVIDIA | All employers we track in this specialty (962 roles · 86 employers) |
|---|---|---|
| Open roles in this specialty | 278 | 962 |
| Open roles in the wider Software, Firmware & Systems family | 971 | 6118 · 145 employers |
| Median days open | 46 d | 51 d (−5 d vs this employer) |
| Median salary (USD postings) | — | $236k |
Skills observed across this category: autonomous agents, evaluation systems, data pipelines, experiment design, Python
Who's hiring in this category
- NVIDIA (this employer) · 274 open roles · median 47 d
- Qualcomm · 62 open roles · median 53 d
- AMD · 54 open roles · median 19 d
- Graphcore · 44 open roles · median 93 d
- Intel Corporation · 43 open roles · median 12 d
- Cerebras · 36 open roles · median 83 d
How we counted: 962 open Infrastructure/Platform Software (Infrastructure Platform) roles from 86 employers tracked in the SemiconductorJobs index, counted 18 Oct 2026. Specialty figures count only roles carrying this exact specialty label, so an employer's related work in neighbouring specialties is not included there — it is counted in the wider Software, Firmware & Systems family row. Figures refresh nightly.