This topic goes in-depth about the improvements in UIAutomation, Computer Vision, and SAP Automation. To read about other products, please navigate to the main topic here.
Previously we had a static timeout of 60 seconds to start the browser. This proved to be insufficient in unattended cases, on low spec machines, when the workflow is started from the Orchestrator and a new session is created.
Since users may need to set a longer timeout for the browser startup, we made it configurable from the Use Application/Browser and Open Browser activities. The following project settings have been added:
UI Automation Classic → Browser → Open browser timeout (seconds)
UI Automation Modern → Application/Browser → Open browser timeout (seconds)
A new workaround is available for the known issue Click activity with Simulate fails on specific web page elements.
The workaround is available through a new project setting UI Automation Classic → Browser > Enable workaround for simulate click with MV3 extension (default: false).
The purpose of this setting is to allow users that have many workflows affected by this issue to apply this workaround through a simple switch - instead of manually checking each Click activity from their workflows and then change the change the input method from Simulate to any other input method (Chromium API, Hardware Events, Window Messages).
For UiPath.UIAutomation.Activities 22.10 and older and Adobe Reader DC version 22.003.20258, PDF automation may be affected. It was not possible to identify individual elements in PDF documents, instead the entire document was selected. The issue was caused by a security option available in Adobe Reader DC, Enable Protected Mode at startup, which is enabled by default.
We improved the support for Chromium-based embedded browsers (Electron, CEF 103+) and added support for orphaned WebView2 windows.
We also added support for the latest WebView2 runtime 106.0.1370.34.
We extended the support for automating Chromium Widgets/Electron applications with HTML selectors, introduced with version 22.8 of the UI Automation package.
Now, this support is available for Chromium versions 108, 109, and 110.
Electron applications can create multiple BrowserViews inside a single BrowserWindow. Each view translates into a new and separate WebContents instance added to the Aura window widget. In the previous versions, our Chromium support used only the first WebContents it could find.
Now we support to:
- iterate over all WebContents
- get WebContent’s position
- check if a WebContent is active
Also, we have added support for CefSharp applications with multiple UI threads.
If the Group Policies in an organization are defined at the user or cloud level, problems may appear since the UiPath browser extensions installed via Group Policy update the policies at the machine level, overwriting the user/cloud level policies (see Group Policy conflict).
For this case, the recommendation is to only install the Native Messaging Host component (ChromeNativeMessaging.exe). With this version it is easier to use the Self-hosted extension (for the Group Policy Offline installation method) by just adding the following value to the ExtensionInstallForcelist policy:
We have added the WebRequest permission to the Chrome/Edge extensions to allow using the web request API.
As a result, the Chrome extension v23.4 has been published on the Web Store with new Extension IDs:
kmejkhcmhgilmppjodlgaklnmdmmhhcl– on Google Web Store
dknkgjgkdpkmddgdjlgdhfojlaehikmk– on Microsoft Store
The Self-hosted extension, used for Group Policy Offline installations, has also been updated with this permission, but kept the same Extension ID:
We improved the Select Item activity to support complex HTML dropdowns, like non-native and non-SAP Fiori/Salesforce dropdowns.
We have added check support for non-standard checkboxes. This includes labels that have a
for attribute, which indicates the hidden checkbox, labels that have a single (hidden) checkbox descendant, and other case that are highly reliable.
Previously, for event data that returned modifiers, the
Modifiers field consisted of an unreadable bitmask, having the
int data type. Therefore, without having some additional skills to search for information, users might have been confused.
Modifiers field for event data is now mapped to the
NKeyModifier enum and returns a string (such as Shift, Alt, or Ctrl) instead of an integer when
ToString() method is called. For example, before this change the returned value for
Modifiers was 23, while now it is Shift, Alt, Ctrl.
In the context of the Trigger-based Attended Automation initiative, where execution performance is critical, we improved the event monitoring usability and performance, by implementing new sets of events supported by different types of application technologies:
The events are available through the Application Event Trigger activity, which is meant to help set up a trigger based on a native event for any type of UI element. The user needs to indicate the target element on the screen and then choose the native event from the list of events supported by that element.
Main native events exposed for each supported technology (Web with
<webctrl> tags, Active Accessibility with
<ctrl> tags, Java with
webctrl: Appeared, Disappeared, Text changed, Key pressed, Click, Double click, Right click, Mouse move, Mouse entered, Mouse exited, Mouse pressed, Mouse released, Focus gained, Focus lost
java: Appeared, Disappeared, Text changed, Key pressed, Click, Mouse move, Focus gained, Focus lost, Enabled, Disabled, Selection changed, Mouse entered, Mouse exited, Cell selected, Cell value changed, Mouse pressed, Mouse released, Menu selected, Menu deselected, Menu canceled, Popup menu visible, Popup menu invisible, Popup menu canceled
wndtop-level windows: Appeared, Disappeared, Title changed, State changed, Location changed, Foreground gained, Foreground lost, Minimized, Restored
ctrl: Appeared, Disappeared, Text changed, Focus gained, Focus lost, State changed, Selection changed
For example, the editable text of the Notepad application is a
ctrl element, and the Event type list displays the available events for this technology:
The Application Event Trigger activity exposes these properties:
- Target – selected UI element. Only strict selectors are supported, without anchors.
Scheduling mode – It specifies how to execute the actions when a trigger is fired. Available values:
Sequential(actions are executed one after another),
Concurrent(actions execution can overlap),
OneTime(executes one action and exits monitoring).
Include children – when selected, the children of the specified UI element are also monitored. It is supported by all event types except
Disappeared, and for any UI element except top-level windows.
We have migrated the most used Classic triggers to the Modern framework. The new Click Event Trigger and Keypress Event Trigger modern activities are designed to identify the elements via a unified target concept, use an Object Repository element and offer a friendlier user interface.
One other important reason why we want them into the CC solution is that they can block the original event and replay it later on, or ignore it. This specific capability allows implementation scenarios like a form submission is stopped until all the fields are correctly filled in.
Set Text activity just got a modern version. It can identify targets with the unified target concept, and it can use an Object Repository stored element. Moreover, it can cache the target element to increase the execution speed when executing repeatedly e.g., when synchronizing text between multiple UI elements.
Modern activities require an application card/scope, to minimize the area of searching the target element, thus, increasing the automation reliability. However, when the activities receive an UIElement as input, they do not require the scope. So, all the modern activities have been updated accordingly.
Page option allows the script to interact with the page as well.
Building and running reliable automations is the key to success for any automation program. The Unified Target concept is UiPath’s solution for ensuring the UI Automation based robots are reliable and trustworthy.
Computer Vision is an important AI powered targeting image-based technology that in some situations is the only one that can be used to reliably automate an application. Use-cases span from CV being used atomically to automate certain areas/elements that the driver cannot drill into, to VDI automation where CV is the only viable technology, and everything in between.
Computer Vision is now part of the Unified Target solution to help customers:
- reduce the complexity of building UI-based automation: customers should not know nor care about technical underlying concepts like selectors, AI-based screen searches, or OCR technology; they should simply point and click the screen elements: the rest is UT’s job.
- to increase the reliability and resilience to changes, in UI-based automation: often, customers do not control application UI changes; the UiPath robots must deal with these situations out of the box, and be smart enough not to break, remaining uninfluenced by the UI deterministic changes.
Computer Vision is enabled by default in Project Settings > UI Automation Modern for Desktop, Java and Web and disabled for SAP (as the happy path for SAP is using the strict selectors):
Computer Vision can also be enabled from the Helper:
The UIA Helper now has a streamlined new design focusing on optimizing vertical space, using a horizontal icon bar on top for the most common functions used:
- Pause with a dropdown for changing the number of seconds of pausing the configuration
- Hoverable elements & Image region selection buttons
- Enable/disable Native Text & Computer Vision buttons
- [F8] can now be used to enable/disable Computer Vision
- Changing the UI Framework is now a dropdown
- Default has been renamed to Auto
- Active Accessibility and UI Automation Frameworks now show their own icons
After indicating the target, the Configuration section is expanded, now holding an Options, Windows Selector and Target/Anchor sections (also expanded by default):
Current scope & limitations:
Starting from UiPath.UIAutomation.Activities package v23.4 we are introducing a new way of identification and automation of SAP elements of SAP WebGUI. This allows us stable and reliable identification independent from the dynamical nature of the interface.
The entire range of activities from the UIAutomation Activities Pack can be used for SAP WebGUI automation. For some SAP specific controls, we are providing an additional extended support. See the full documentation.
SAP WebGUI uses the proprietary SAP HTML framework to display its elements on the page. Selectors generated for these elements are easy to recognize by the
sapweb- attributes in selectors.
A set of
sapweb- attributes allows stable and reliable identification of display elements similar to SAP WinGUI technical selectors.