This topic goes in-depth about the improvements in UI Automation, Computer Vision, SAP automation. To read about other products, please navigate to the main topic here.
UIAutomation now uses the latest .NET release, .NET 6. Windows and Cross-platform projects now also target .NET 6.
We often see the need to automate native Windows applications embedding web applications with the help of Electron/Chromium Widgets technology. From now, the web module rendered inside a Chromium widget can be automated with standard web selectors (
webctrl), instead of IA2-based selectors (
ctrl). This brings all the web automation power at hand when automating desktop applications embedding web applications through Chromium widgets.
A few examples of applications with Chromium (mostly Electron) are UiPath Assistant, Slack or Microsoft Teams.
Please note that the support only works with compatible Chromium versions v80 to v102.
Using the RDP ActiveX control, one can build custom remote desktop applications, besides custom remote apps. One such example is the Picture in Picture functionality. We’ve added support for automation such applications.
Select Item activity can be now used with VB6 drowpdown controls. Also, we have added data extraction support for VB6 grid components:
ctrlid attribute is extracted and added to the selector for the following controls with Default framework: TextBox (input field), Button, RadioButton, CheckBox, PictureBox, ListBox (select item), ComboBox. The
ctrlid attribute is unique for each window of the application. It is also stable between application runs, making it a great option for control identification.
Similarly, for UIA framework the
automationid attribute is added to the selector for the same type of controls.
We have improved the Kinaxis Java support, with proper grid table cell identification and data extraction.
foregroundColor properties of Java elements are now exposed as strings containing RGB triplets and can be used in selectors.
The default selectors for toolbar buttons in Java applications have the
name attribute now added by default to their selectors.
Support for applications based on the Qt Framework has been developed through the new type of selector tag: qt.
Selectors would look like this (with
<wnd app='widgetstestapplication.exe' cls='Qt*QWindowIcon' title='Qt widgets test application' />
<qt name='QPushButtonObjectName' rawtext='Add' role='QPushButton' />
A well-known implementation built on Qt 5 is Tableau.
We currently support Qt 5 Widgets (no Qt Quick/Qt QML), and Qt 6 is on our roadmap (currently falling back on the AA framework).
Several improvements have been implemented for the web selectors.
The default selectors have new attributes, such as the HTML
class attribute or the
aria-labelledby attributes, as well as the improvement of priorities of the attributes from the default selectors.
title attribute has been removed from the list of available attributes for the
webctrl tag. It also added the reliability verification for the
id attribute from the
Starting with version 22.4 the attribute
visibleInnerText, which reflects the actual
innerText of the underlying HTML element, was added to UI Explorer → Property Explorer. From now, the
visibleInnerText attribute has been exposed as a selector attribute in UI Explorer.
When using Native Remote Automation for a Remote Desktop (Citrix or RDP) it is possible now to launch a browser inside the remote session, and then automate it, by using the modern “Use Application/Browser” activity.
Previously, the classic “Open Browser” activity did not allow this functionality. As a workaround, an “Open Application” activity should have been used, followed by an “Attach Browser” activity.
The different installation methods for browser extensions (per user, per machine, Group Policy Online, Group Policy Offline) were available only from the command line.
In Studio → Tools, for each browser extension, a combo box has been added to allow the user to select the installation method:
- Group Policy Offline
- Group Policy Online
- All Users
- Current User
- Native Host Only
For each of these methods, tooltips are provided for explaining their purpose.
Once the extension is installed, the installation method remains displayed in the disabled combo box, so that the extension may be uninstalled with the same method used for installation.
In the extension status info (
/status-admin), extra fields have been added for description, documentation info and warnings. These fields contain localized text displayed in Studio->Tools or
New Design and continuous improvements:
- New Menu at selection stage for extracting text/URL on web scraping scenarios
- Added Preview option at the element level
- New pagination link toggle with clearer messaging
- New On/Off highlighting option of confirmed series of elements
Several improvements have been made:
- Create screens automatically when adding a new element to empty Object Repository
- Edit Screen for Browsers - Arguments are now being exposed as editable text
- For Each UI Element & Extract Data - Improved Indicate & Editing experience when linked to Object Repository
- Pre-populated modern Select Item items list when an Object Repository target is paired with it
- Object Repository Screen edit with expression editors for FilePath, Arguments, URL
Allow passing variables/arguments in the Shortcuts fields.
The Keyboard Shortcuts activity now offers a
DelayBetweenKeys property that can be used to set a delay between keystrokes, similar to the Classic behavior. A corresponding project setting, Delay between keys, has been added in the Keyboard Events section.
It is now possible to configure automations to run using the default system browser, not only the specific browser set at design time. This is allowed through the
DefaultSystemBrowser class which has been made available for usage in the Default value for variables.
DefaultSystemBrowser.Name is the executable name (
firefox.exe) and it can be used in selector definition. Also,
DefaultSystemBrowser.BrowserType can be used for getting the browser type string (“Chrome”, “Edge”, “Firefox”).
Previously the “detecting auto-anchor” step took approximately 4 seconds, but now there is a big improvement as this step takes approximately 1 second.
By default, we were allowed to add a single anchor, and we needed to explicitly click the “add anchor” button to add more. But sometimes there are duplicates right after indicating something, which makes things hard to correct.
With this improvement, if after indicating an element (or an anchor) and validating the current selection as “duplicates found (yellow)”, we automagically allow adding another anchor (until the maximum of 3 anchors is reached).
As the anchors are the healthy way of automating and we want to promote them, we removed the “Enable anchors” flag from project settings, for all targeting methods (the default value is true). For the projects that have this project setting on false, we still show the option, so they aren’t stuck with no anchors.
UI-PST-001 is the Workflow Analyzer static rule applicable to UIAutomation Modern, that in its first version was for Robot Logging properties only. The description has been changed now to “Audit log level in project settings”, with the scope Project and the Default Action Warning.
Audit log level in project settings (UI Automation Modern → Robot logging) can be restricted via this rule. Possible values for AuditLogLevel are Off/Info/Trace (default Off).
The Take Screenshot activity now supports region selection via the F3 hotkey when indicating on screen.
For consistency, we have enabled the modern selection screen for the Classic activities and Recorders.
When migrating workflows from one browser to another, we are migrating the Object Repository targets as well.
When migrating a workflow to Chrome/Edge/Firefox, if the attributes
innertext or aaname` are part of the IE selector, they won’t work, because the attribute values look different for Chrome/Edge/Firefox browsers.
This issue is caused by the fact that for IE the
innertext selector attribute maps to the HTML
innerText property, while for Chrome/Edge/Firefox the
visibleInnerText selector attribute uses the HTML
For the migration from IE to Chrome/Edge/Firefox, a new option is presented to the user:
IE - Replace innertext/aaname attribute with visibleInnerText (requires at least v22.4).
If this option is selected, for all
webctrl selector tags from IE:
innerTextis replaced with
aanameis replaced with
- trim begin-end whitespace from the value of
- if both
aanameare present in the same
webctrltag from the IE selector only the first
visibleInnerTextis kept in the migrated selector
This option should not be used when migrating from IE to Edge IE Mode. For this type of migration, is enough to select IE as the Source and Edge as the Destination browser.
We’ve deployed a new iteration of our CV AI Model for Cloud & On-Prem, significantly better performing when working with tables and OCR data due to an improvement in the internal neural net architecture.
The Localserver is the compressed version of the CV Model running on the Cloud and On-Prem CV servers. It has a lower number of neurons (so slightly lower UI elements detection accuracy) and is running on the local machine’s CPU – no GPU required. It is enabled via a property of the CV Screen Scope:
For the first time since we’ve launched this version of the CV AI model, we’re very happy to announce it is now on-par feature-wise with the Cloud and On-Prem model, as it can also be used to extract tables and is outputting auto-anchors.
Detection accuracy has also improved, while the AI model is now 1.5x faster!
Our On-prem customers can now have their design-time data learned by the UiPath’s CV AI Model by enabling the “Include in AI Learning” toggle in the CV Screen Scope. Any new iteration of the AI Model will learn, thus detecting all the data the previous iteration was, making all the On-prem CV automations with this feature enabled as resilient to model updates as any Cloud CV automation.
Since the AI Model only learns from the shapes and colors of the UI elements and not the real data content, we highly recommend using mock data when building any CV workflow to avoid sending any sensitive information, such as personal data. Only valid screen UI elements will be saved, everything else will be discarded.
This feature is turned off by default and can be enabled on a granular level, per each CV Screen Scope – so our On-prem customers still have full data residency control. After enabling it, all screenshots from the specific Scope will be sent to our cloud. When disabled, automatic deletion requests are created, thus all screenshots will be discarded and excluded from the training set.
Automating scrollable content in webpages or apps using CV activities is now much easier and can be done using the usual delayed screen refresh UX [F2]: scrolling actions performed at design-time are replicated at run-time so that the application will reach the same state needed to find the target element.
The 3 seconds counter in the delayed screen refresh UX [F2] will reset with each mouse scroll while indicating and a Refresh will be triggered at the end of this timer, generating a new screenshot and capturing the scroll direction value in a new property called ScrollDirection (from None to Up or Down). When ScrollDirection is None, no scrolling will be performed at run-time.
At run-time, the ScrollDirection’s value is used to search for the target element; if it is not found during this first attempt, the robot will do a top-most (when ScrollDirection is Down) or bottom-most (when ScrollDirection is Up) scroll and then try again.
The NumberOfScrolls property sets the scrolling speed at run-time (default value is 2), while DelayScreenshotAfterScroll is useful when working with dynamic content such as icons that load when scrolling.
When extracting tables in scrollable pages or apps, we recommend setting the ScrollableTable property (“Scroll” property in previous releases) to True; otherwise, an incomplete table might be extracted.
Since this feature is a big change for any current CV user, we’ve updated the [F2] “Delayed Screen Refresh” tooltip and added a “What’s New?” message when first entering the delayed screen refresh UX (which can be hidden when next indicating using the “Don’t show again” checkbox) for a better visibility.
Also, the “Refresh Scope” button in the CV Helper has been relabeled “Screen Refresh” and the “Refresh with delay” button has been relabeled “Delayed Screen Refresh”.
A quick demo of CV Run-time Auto-scroll can be seen here.
The OCR-CJK engine is now available by using a CV API Key as well:
Currently available endpoints per region:
More info on CJK-OCR can be found here.
When using UiPath Screen OCR as a standalone activity, there is no need for inputting the CV API Key anymore if you are logged to your UiPath Cloud account.
The AI model is now outputting automatic anchors for text targets as well when indicating.
The activity card design has been slightly updated to match the corresponding UIA activities.
“Click type” and “Mouse button” have been added to CV Click.
“Save to” has been added for CV Extract Table and CV Get Text.
We have deployed a new CV service endpoint in the USA to better serve the customers closer to that region. We estimate that customers that are closer to the US than to the current endpoint deployment in West Europe can see more than 100ms decreases in latency by simply switching to using the new service URL.
- Current default endpoint, https://cv.uipath.com will still point to our cloud server in West-Europe
- To use the new US endpoint, you will need to manually edit the values in either:
- Project Settings > Computer Vision > Server to https://cv-us.uipath.com
- CV Screen Scope Properties > Server > URL value to https://cv-us.uipath.com
Design-time data will be centralized in our West Europe cloud storage, no matter the region of origin. UiPath’s Data Residency page has been updated.
When both OCR and CV are requested for the same screenshot, we are now merging the two requests into a single server call. Since the requested UI elements and text are found in the same image, we are thus optimizing for both latency and rate-limiting.
The single call endpoint will be set to the value of the CV endpoint, as follows:
|CV endpoint||OCR endpoint||Single Call endpoint|
|West Europe||West Europe||West Europe|
|West Europe||United States||West Europe|
|United States||United States||United States|
|United States||West Europe||United States|
|Localserver/On-Prem||Any Cloud Endpoint (WE/US)||N/A|
|Any Cloud Endpoint (WE/US)||Localserver/On-Prem||N/A|
The single-call logic will only be applied if both defined CV and OCR endpoints are either:
- UiPath cloud servers
SAP Fiori Tree Lists are very complex elements used to display hierarchically structured items, which UiPath now is able to automate in a stable and reliable way. All variations of trees are supported for all browsers.
Searchable Combo Boxes are now easy to automate and allow the selection in Studio as usual with help of Select Item Activity.
Extended support for the most required customer features, such as Global Busy Indicator. Now automation is even more reliable.
Which features excite you the most and you are going to try right away? We want to hear what you think! Please use the button below
SAP Fiori UI / SAP SuccessFactors Searchable Combo Boxes
Searchable Combo Boxes are now easy to automate and allow the selection in Studio as usual with help of Select Item Activity.