Submission type
UiPath for Coding Agents
Name
Anthony Yanza
Industry category in which use case would best fit in (Select up to 2 industries)
Information technology and services
Operations
Complexity level
Advanced
Summary (abstract)
Brittle UI tests are the number one reason teams abandon test automation, and self-healing was supposed to fix that. It does, right up until it creates a worse problem: a test that blindly heals itself can heal straight past a real bug, turn green, and quietly ship the regression.
SelfHeal QA is a UiPath Coded Agent that heals brittle locators and knows when not to. Give it a code change and a plain-English spec. It picks the affected suites, writes the test, runs it in a real browser, and on failure it triages the root cause: brittle selector or real bug. Brittle, it rewrites the locator from the live DOM and re-runs. Real bug, it refuses to heal and files a defect in UiPath Test Manager instead of masking it. The restraint is enforced in deterministic code, not just asked of the model.
Built solo as a first-time UiPath builder for UiPath AgentHack, where it won Best Cross-Platform Integration.
Detailed problem statement
Test automation programs do not usually die because the tests were wrong. They die because the tests became expensive to keep alive.
Someone renames a button, changes an id to a data-testid, or reorders a form. The next morning the suite is red, and an engineer loses hours working out whether anything is actually broken. Do that often enough and teams stop trusting the suite, then stop running it, then stop writing it.
Self-healing is the standard answer. Tools detect the drifted locator, patch it, and move on. That solves the maintenance cost, but it introduces a failure mode that is far more expensive and almost invisible: over-healing.
A blind self-healer cannot distinguish between “the locator moved” and “the product is broken.” It sees a failing step, patches around it, and turns the build green; the regression ships. You find out when a customer does. Every silent false heal converts a caught bug into a released bug, and it does it while making the dashboard look healthier.
So the real bar for autonomous QA is not “tests that fix themselves.” It is tests with judgment: something that heals when healing is safe, and pulls the alarm when it is not. That means the agent needs a defensible answer to one question, on every single failure, without a human in the loop:
Is this a flaky locator, or is this a real bug?
And critically, it needs to be wrong in the safe direction. Escalating a brittle selector to a human costs a few minutes. Healing away a real regression costs a production incident.
Detailed solution
SelfHeal QA is a UiPath Coded Agent (Python, scaffolded, run, and published with the UiPath CLI) that owns the full loop: select, generate, execute, triage, heal or file, report. It calls the UiPath Identity and Test Manager v2 REST APIs directly, and the published package runs on Orchestrator.
THE LOOP
- Select. It maps changed files to the suites they actually touch and runs only that slice, rather than re-running everything on every change.
- Generate. A plain-English spec becomes executable test steps.
- Execute. The test runs in a real browser (Chromium via a Playwright executor), so the agent is reasoning about a live DOM, not a static fixture.
- Triage. On failure, it stops and asks the real question. Claude reads the live page and returns a verdict, BRITTLE_SELECTOR or REAL_BUG, with a confidence score and written reasoning.
- Heal gate. This is the part that matters. The verdict is model judgment, but the gate that acts on it is deterministic code. A heal is applied only if confidence is at or above 0.7 AND the page snapshot shows no error or alert state. Anything else is withheld and escalated, because a real bug very often masquerades as a drifted selector.
- Heal or file. Passes the gate; it rewrites the locator from the live DOM and re-runs until green, with no human involved. Fails the gate, it refuses to heal, logs the failure, and files a defect in UiPath Test Manager with the verdict, confidence, and reasoning attached, so a person sees the regression instead of it getting buried.
WHY THE GATE IS CODE AND NOT A PROMPT
Asking a model to “be careful” is not a safety property. The confidence floor and the error-state check live in src/heal.ts and are mirrored in the agent’s main.py, with a unit-test suite pinning the invariants: a confident heal on a clean page is applied, a low-confidence heal is withheld and the locator is never rewritten, a confident heal on an error-state page is still withheld, and a real bug is filed. There is also a cross-language parity test so the TypeScript and Python gates use the same floor and the same signals.
HOW I KNOW IT DOES NOT HEAL PAST REAL BUGS
That question kills every self-healing demo, so I answered it with an adversarial benchmark: 16 labeled failures including look-alike cases built specifically to fool the triage. One is a real bug where the button still exists but the payment was declined. Another is ordinary locator drift dressed up with a scary-looking error banner.
Results: 100% triage accuracy (16/16), and a 0% false-negative rate across the 8 real regressions. It never healed a real bug away.
Being straight about the limits, because that matters more than the number: the app states are seeded fixtures, the set is small and author-curated and deliberately weighted toward adversarial cases, and it is not a statistical guarantee. The verdicts are live model calls, not canned, and the harness refuses to record a run as valid if any case fell back to the deterministic path, so a fallback cannot fake a clean score. The full per-case transcript (verdict, confidence, model, timestamp) is committed in the repo and regenerated on every benchmark run.
VERIFIED ON PLATFORM
The published package was executed as a serverless Orchestrator job and completed successfully in 10.2 seconds. The job record is committed in the repo alongside the Test Manager run history and the auto-filed defect.
WHY THIS IS NOT JUST AUTOPILOT AGAIN
UiPath already self-heals tests in Autopilot for Testers, and Testim, mabl, and Healenium all optimize for heal coverage. I did not try to out-heal any of them. I went after the blind spot they share. SelfHeal QA inverts the objective function: the novelty is knowing when not to heal, and turning that restraint into an auto-filed defect.
The pattern generalizes beyond locators. Any autonomous agent that mutates state on failure needs the same rule: act when confident and safe, escalate otherwise.
WHAT GOT IN THE WAY
Honest friction from a first-time UiPath build, in case it saves someone else the hours:
- invalid_scope is all or nothing. Request a token with several scopes, and it fails if one name is wrong, with no indication which. I had to probe scopes one at a time.
- Scope names are hard to discover. TM.TestExecutions (plural) is required for executions and easy to miss, and the valid Test Manager scopes are not surfaced where you create the external application.
- A test execution requires a test set. Posting testCaseIds alone to /testexecutions returns a 500 rather than a 400 telling you what is missing.
- Cloudflare blocks the default urllib User-Agent with a 403 on the token endpoint, which any coded agent using stdlib HTTP hits immediately. Setting a UA header fixes it.
- UiPath publish is interactive only, which is awkward inside a coding-agent workflow.
None of these are dealbreakers, and the coded agents plus Test Manager combination is genuinely the right direction for agentic testing. I sent the full list to the team as product feedback.
BUILT WITH
Solo, end-to-end, with Claude Code through UiPath for Coding Agents, as my first project on the UiPath platform.
Narrated video link (sample: https://bit.ly/4pvuNEL)
Expected impact of this automation
MEASURED IN THE BUILD
- Triage accuracy 16/16 (100%) on an adversarial benchmark, with a 0% false-negative rate across 8 real regressions. Live model verdicts, per-case transcript committed to the repo.
- The published Coded Agent ran as a serverless Orchestrator job and completed successfully in 10.2 seconds.
- Test selection runs only the suites the changed files touch, instead of the whole suite on every change.
WHERE THE TIME GOES
Every red build caused by locator drift currently costs an engineer a full triage cycle: open the run, read the trace, find the renamed element, patch the locator, re-run, confirm. SelfHeal QA removes that cycle completely for the drift cases and escalates the ones it cannot classify safely, so the saved hours scale with (drifted locators per week) x (minutes per manual triage). I am deliberately not quoting an industry ROI figure here, because the only honest measurement is the one you take against your own suite and your own flake rate.
The second-order effect is the bigger one. Suites get abandoned because maintenance cost outruns perceived value. Remove the routine maintenance and the suite stays trusted, which means it keeps getting run.
THE FAILURE MODE IT PREVENTS, WHICH IS HARDER TO PRICE
A blind self-healer that patches around a real regression turns a red build green and ships the bug, while making the dashboard look healthier. One silent false heal can cost more than a year of saved locator maintenance. The design target for that number is zero, which is exactly why the heal gate is deterministic code with a confidence floor and an error-state check, and not an instruction to a model.
FIT WITH EXISTING PROCESS
Defects land in UiPath Test Manager as the system of record, with the verdict, confidence, and reasoning attached, so there is no new tool for the team to adopt and no new place to look. Humans review real bugs only.
UiPath products used (select up to 4 items)
UiPath Apps
UiPath Assistant
UiPath Coded Agents
UiPath Orchestrator
UiPath Studio
UiPath Test Cloud
UiPath Test Manager
Automation Applications
Web and browser-based applications under test (any UI reachable by URL). The public demo runs against a seeded login and checkout web app in Chromium. GitHub is the source of the code changes that trigger suite selection.
Integration with external technologies
Anthropic Claude (claude-sonnet-4-6) via API for failure triage and locator rewriting, with a deterministic fallback path. Claude Code, via UiPath for Coding Agents, for the build itself. Playwright with Chromium for real-browser execution. Python 3.11 for the Coded Agent, Node 20 and TypeScript for the companion executor and the guardrail test suite. GitHub for source control and the published evidence artifacts.
TO-BE workflow/architecture diagram (file size up to 4 MB)
Other resources
Source code, including the Coded Agent, the heal gate, and the guardrail tests: GitHub - yanzaaa/selfheal: The testing agent that knows when not to heal. A UiPath coded agent for Test Cloud that self-heals brittle locators but refuses to heal over real regressions. · GitHub
Demo video: https://www.youtube.com/watch?v=GWa3EnbFI5M
DevPost submission (UiPath AgentHack, Best Cross-Platform Integration winner): SelfHeal QA | Devpost
Evidence committed in the repo:
- submission/benchmark-results.json: full per-case adversarial benchmark transcript (verdict, confidence, model, timestamp for all 16 cases)
- submission/orchestrator-job.json: the serverless Orchestrator job record, Successful in 10.2s
- submission/screenshots/: self-heal pass, refuse-to-heal with filed defect, Test Manager run history, benchmark output, Orchestrator job
- tests/heal.test.ts: the unit tests pinning the confidence and error-state gate
- submission/PRODUCT_FEEDBACK.md: the full platform friction list summarized above
Run it yourself with no API keys in about 10 seconds: git clone, npm install, npm run demo