Submission type
Generic use case
Name
JonathanSolvesProblems
Industry category in which use case would best fit in (Select up to 2 industries)
Banking and financial services
Compliance
Complexity level
Advanced
Summary (abstract)
FlakeWarden is an agent that triages failing tests on UiPath Test Cloud. When a build goes red, it decides whether each failure is a real defect, a flaky test, or an environment issue, then routes it to the right action with a human approving anything that changes. Clear-cut cases are settled by exact, auditable math; only the genuinely ambiguous failures reach a grounded Agent Builder classifier. On a labeled set of 150 failures it reached 90.7% accuracy with a measured 0% rate of ever hiding a real regression. It won 1st place in the Test Cloud track at UiPath AgentHack 2026.
Detailed problem statement
Flaky tests are the most corrosive failure mode in continuous integration, because they quietly erode trust in the entire test suite. Google published that about 16% of their tests showed some flakiness and that flaky failures were behind roughly 84% of their red builds. The practical result is that when a build goes red, most of the time it is noise, and the one signal that matters, a genuine regression, is buried underneath it.
Teams respond in one of two damaging ways. Either an engineer spends real time digging through logs to work out whether each failure is real or flaky, multiplied across dozens of failures a day, or the team stops trusting red builds entirely and just reruns until green, at which point a real regression eventually slips straight through because everyone assumed it was noise again.
The core problem is triage: reliably separating real defects from flaky noise and environment issues, at scale, without a human having to read every stack trace, and without ever silently hiding a real bug.
Detailed solution
FlakeWarden expresses the whole triage flow as one governed UiPath Maestro process.
Before anything touches AI, a deterministic scorer looks at each test’s run history and computes five statistics: how often it flips between pass and fail, what fraction of its failures recover on retry, whether it fails in isolation while everything else stays green, how varied its error signatures are, and how unstable its runtime is. Each is normalized to a 0-1 value, weighted (retry-recovery and flip-rate dominate, since those are the classic flaky tells), and summed into a single flake score. Simple thresholds then settle the confident cases: above 0.62 is confidently flaky, below 0.38 is confidently not, and only the ambiguous middle band is escalated. There is also a hard rule: if a selector changed at the moment the test broke, the score is capped, so a real UI regression can never be waved off as flaky.
Only the genuinely ambiguous failures, about two-thirds of the set, reach a grounded classifier built in UiPath Agent Builder. It reasons over the actual evidence (stack trace, selector diff, commit, logs), must cite that evidence in its rationale, and escalates toward a real defect when the evidence is split rather than guessing.
Whatever the path, nothing is quarantined, promoted, or changed autonomously. Every verdict becomes a proposal that lands in UiPath Action Center for a human to approve, and only then does UiPath Orchestrator write back the action. A negative-control gate fails the build if a real defect is ever auto-healed, which is what enforces the safety guarantee.
The entire solution, including the full Maestro BPMN flow, was built with a coding agent (Claude Code) driving the uip CLI in plain English from the terminal, which is the UiPath for Coding Agents path. It runs offline on stdlib Python with an optional Anthropic Claude backend through the UiPath AI Trust Layer.
Narrated video link (sample: https://bit.ly/4pvuNEL)
Expected impact of this automation
- Reclaims triage time: engineers no longer read stack traces to sort real failures from noise. Since flaky failures drive the large majority of red builds, most of that manual triage is eliminated, and the confident cases (about a third of failures) are resolved by exact math with no model call at all.
- Protects release integrity: a measured 0% safety-direction false-positive rate on 150 labeled failures means a real regression was never once hidden as flaky. This is the benefit that matters most in regulated environments where shipping a hidden defect is unacceptable.
- Restores trust in the suite: because red builds are triaged instead of ignored, teams stop the “rerun until green” habit that lets regressions slip through.
- Cost-efficient by design: only ambiguous failures reach the LLM, so model spend scales with genuine ambiguity, not with total failure volume.
- Governed and auditable: every change is a human-approved proposal with a cited rationale and a reproducible deterministic score.
Measured on a labeled corpus of 150 failures: 90.7% overall accuracy, 0% safety-direction false-positive rate, roughly one-third of cases resolved deterministically without any AI. Reproducible from a git clone in three commands.
UiPath products used (select up to 4 items)
UiPath Action Center
UiPath Agent Builder
UiPath Maestro
UiPath Test Manager
UiPath Test Suite
Automation Applications
N/A - operates on CI/CD test-suite results, not a line-of-business app
Integration with external technologies
Anthropic Claude (LLM backend for the Agent Builder classifier, accessed through the UiPath AI Trust Layer). Runs offline on the Python standard library when no model backend is configured.
