Build Failure Analyzer
- Role
- Built it, now lead
- Since
- 2024
- Status
- In production
- Used by
- Teams across NI
When a CI build fails, someone has to read the logs, decide whether the cause is the infrastructure or a code change, and work out which pull request broke it. That happens dozens of times a week. I built a system that does the first pass.
How it works
- A failed build lands in the review queue.
- Deterministic checks run first: known infrastructure patterns, duplicate detection, failure clustering and dependency tracing. about 65% of failures end here
- What’s left goes to a root-cause agent. It reads the logs and PR history through tools and names the pull request that most likely caused the break.
- A second agent, running on a different model, checks that call before anything is posted.
- The result is commented on the work item. A triager confirms it, and only then does the system file or link a bug.
Models are slow, cost money and are sometimes confidently wrong. So the cheap checks go first, a second model reviews the first, and a person signs off before anything is filed. Different jobs use different models: Claude Sonnet for root cause, GPT as the validator, Claude Haiku for clustering.
Results
- 1,350+ failed builds analyzed across 600+ work items.
- Saves roughly 10-15 minutes of manual triage per failure.
- Scores 217 of 250 on a labeled eval set, which I use to check every change to the prompts and rules.
- Won an R&D Excellence Award in Emerson’s 2026 R&D Global Awards.
Where it falls short
It still gets 33 of my 250 eval cases wrong, and a model can be wrong with total confidence. That’s why a triager’s sign-off is a hard gate before any bug is filed, not an optional step.
The code is internal to NI. A small public version shows the first step: finding the real error in a build log and labelling it infrastructure or code. build-failure-triage on GitHub (opens in new tab)
Stack Python · Azure DevOps REST API · GitHub Copilot SDK · SQLite · pytest · Poetry