UiPath Community 2023.10 Release - UIAutomation, Computer Vision & SAP Automation

[Luiza_Surdu-Bob (Luiza Bob)]

UIAutomation, Computer Vision & SAP Automation

This topic goes in-depth about the improvements in UIAutomation, Computer Vision & SAP Automation. To read about other products, please navigate to the main topic here.

Our 23.10 release focused on improving core components in Unified Target and Windows attended automations by supporting use-cases involving Microsoft UI Automation / Embedded browsers applications and by providing new tools for helping our clients identify the events triggered by certain UI elements actions.

Improving Unified Target

In 23.10, we continued our story around seamlessly integrating Computer Vision in Unified Target by addressing the main limitations in 23.4:

We also optimized core issues related to Unified Target itself by accounting for each targeting method’s relevancy in terms of targeting power and resilience to UI changes.

In short, we have continued to pursue our north-star vision of a unified UIA UX where both pure CV and combined CV & driver powered automations can seamlessly be created without the need for the user to be aware of the intricacies of each targeting method while also providing flexibility in targeting customization for power users.

Table Extraction

Table extraction is now CV-boosted for selector-based extraction

You can now just indicate full tables, no need to indicate a relevant table cell first. Since CV “sees” entire tables, it is used at design-time to correct the sometimes-faulty selector-based extraction when indicating a full table. Note: in case the selectors have any issues extracting the table at run-time, we don’t have a CV fallback mechanism in place yet.

While also allowing pure CV table extraction

No need to switch to a CV Screen Scope & use a CV Extract Table; you can use the existing UIA Table Extraction wizard to extract pure CV tables as well for scenarios where selectors are not available at all, such as remote desktops without the remote run-time installed. You can now easily extract tables spanning across multiple pages in pure CV scenarios as well by indicating the Next button in the wizard.

Extracting scrollable tables is also possible, using the same logic as in the CV Extract Table activity.

Run-time targeting methods are now optimized

We improved overall targeting accuracy and automation resilience, while also providing the user with better transparency in terms of selector health.

Using a new hierarchical concept

Ranking now decides which targeting method will be used at run-time, accounting for each targeting method’s relevancy in terms of targeting power and resilience to UI changes. Selectors are now considered the primary targeting method, Computer Vision is a secondary targeting method, while Image is a tertiary targeting method (and is also now disabled by default).

There is now a new setting in Project Settings > UI Automation Modern, which is set to True by default for new projects:

This setting allows the primary targeting methods to consume their full timeout until any fallback targeting method can be used, aiming to bring more predictability and visibility into selector related issues so that the RPA developers will have better control over their UI elements targeting, while also having the solid fallback CV provides.

Failing strict selectors at run-time are highlighted in logs

Failing strict selectors will show in the logs as warnings, accompanied by suggestions of the top 10 closest matches that can be used to replace the faulty selectors.

This feature is enabled via a setting in Project Settings > UI Automation Modern, which is True by default:

One example of an output:

Using a different OCR engine

In addition to the default CV embedded OCR engine, UiPath Screen OCR, you can now use any other custom OCR engine to power the text extraction in CV when used in Unified Target. The OCR engine can be changed on the fly in each scope, from the hamburger menu and it will only affect the CV targets; the rest of the OCR activities will still use the default OCR set in Project Settings > OCR section.

Streamlined indicate experience when CV is enabled

The CV screenshot no longer needs an initial delay, so now the start of the indicate experience is faster. Also, the enabled/disabled state of the CV toggle in the Helper will persist throughout all the indicate sessions inside a scope.

Auto-enable CV when indicating in scope-less activities

Given the targeting and fallback mechanism in Unified Target, having fully loaded (both selectors and CV descriptors) targets is the happy path for all users, so auto-enabling CV is needed when indicating. Scope-less activities (activities that can be used outside an Use Application/Browser UIA scope) can now also benefit from CV being enabled by default (when also enabled in Project Settings).

Optimize UI elements indicate experience to favor fully loaded targets

Since the CV and driver UI element bounding boxes can differ a lot, reconciling the two is needed. We are now avoiding ending up with CV-only targets by reconciling such elements with their matching driver element.

Extended Trigger-based Attended Automation

With enterprise version 23.4 we released the Trigger-based framework, a new way to build Attended Automations, with event-driven architecture and support for native events monitoring through the Application Event Trigger activity.

With version 23.10 we continued the investment in Windows attended automation to support use-cases involving Microsoft UI Automation / Embedded browsers applications and provided tools to improve user experience for our clients by helping them to identify the events triggered by certain UI elements actions.

Support for UI Automation Apps Native Events Triggers

After releasing the Trigger-based framework in version 23.4, with native triggers support for certain technologies, we have seen users who would need to monitor events from applications built on Microsoft UI Automation framework (such as Office applications – Excel, Word, Access).

Based on this we introduced this feature that extends the native events support through the Application Event Trigger activity for applications built on UI Automation framework, in addition to the native events triggers provided with version 23.4 for Web, Java, Active Accessibility.

With the new feature we offer the possibility of handling complex use cases based on different applications, built on different technologies.

Support for Native Embedded Browsers Web Triggers

In the past years, we have seen an increased demand for automation standalone applications embedding browsers via technologies like WebView2, CefSharp, Electron.

Given that these apps are becoming popular very quickly, it is imperative that they support attended automation and native triggers generated by these types of apps.

For version 23.10 we provide support for webctrl based native events in native embedded browsers (Slack, Discord, Electron test app) and in the next release we will also add support for managed embedded browsers (CefSharp).

Event Inspection Tool

In the same context of Trigger-based Attended Automation, users may need help to identify the events triggered by certain UI elements actions.

For such cases, we introduced a new tool for developers, integrated with UI Explorer, that will help them understand the type of native events triggered/exposed by various UI elements when the user interacts with them.

The users can benefit from Event Inspection Tool for scenarios where they need to identify what events to monitor on which UI elements.

Dynamic Application Event Trigger (Monitor Multiple Targets)

For the Augment functionality to allow centralized configuration of URL/UI element mapping to processes, we needed to create a way to dynamically observe each URL/UI element from this centralized repository.

With version 23.4 a triggered workflow with an Application Event Trigger only monitors a single target element, requiring developers to create a triggered workflow for each target element they want their automation to monitor. We introduced a way to monitor multiple target elements with a single Application Event Trigger and allow the number of monitored targets to be dynamic (the monitored targets may change after design time). In addition, the Application Event Trigger activity has been updated so that it can be used with coded workflows.

Improved Browser Support

Support for Multiple Browser Profiles

Until version 23.10 we had a known limitation: we only supported automation for one browser profile at a time. UiPath browser extensions did not support automation of multiple browser instances running with multiple user profiles at the same time. As such, browser automation required you to close all active browser instances and restart the browser using a single user profile.

This new feature allows user design automation with multiple browser instances opened under different profiles for Chrome/Edge/Firefox. We now support multiple native messaging application instances running at the same time to accommodate scenarios of multiple browser processes running for different browser profiles, such as the case when a specific application must run on a dedicated profile to isolate sensitive data from other applications.

Opening the browser with a specific profile is now possible when using the Use Application/Browser activity. To specify a profile at browser startup, we need to append an argument to the browser command line. This can be specified in the Arguments property in Unified Application Target section:

• for Chrome/Edge: --profile-directory="profile_path". The profile path is the relative path to the User data folder path specified for that Use Application/Browser activity. If the User data folder path is not specified, then the profile path is relative to the default user data directory %LOCALAPPDATA%. The profile path can be found by navigating chrome://profile-internals/ and retrieving it from the Profile path.

• For Firefox: -profile "full_profile_path". The full profile path can be found by navigating to about:profiles and getting the Root Directory for that profile.

Support for installing RDP, VMware and Firefox extensions from UiPathRemoteRuntime.msi

For versions lower than 23.10 it was possible to install some of the UiPath extensions on the remote machine only from command line, using the SetupExtension.exe tool after UiPath Remote Runtime was installed on that remote machine. This was the case for RDP and VMware extensions and Firefox extension.

With version 23.10 you can choose to deploy any extension you need directly from the UiPath Remote Runtime installer, by selecting them from the features list.

If you choose to install UiPath Remote Runtime from the command line, in addition to the other extensions you want to deploy, you may also specify these extensions as arguments:

UiPathRemoteRuntime.msi ADDLOCAL=RemoteRuntime,RemoteRuntimeTask,JavaBridge,ChromeExtension,EdgeChromium, FirefoxExtension,CitrixClient,WindowsRdpExtension,VMwareExtension

The above command installs the Remote Runtime component, the Task Scheduler entry, the extension for Java, the extension for Chrome, the extension for Microsoft Edge, the extension for Firefox, the extension for Citrix, the extension for Microsoft Remote Desktop and Apps, and the extension for VMware Horizon.

New Project Settings for Open Browser

We addressed some extension initialization issues affecting the Open Browser and Use Browser activities by adding six UI Automation Classic – Browser project settings: Extra arguments for OpenBrowser Chrome/Edge, Custom executable path for OpenBrowser Chrome/Edge, and Enable retry for OpenBrowser Chrome/Edge.

Workaround for Edge v115 issue with InPrivate option

The Microsoft Edge browser update to version 115.0.1901.183 introduced a bug regarding the initialization of MV3 extensions for Edge started “InPrivate” mode. More details are available in the forum post about this issue.

Since the bug is not yet fixed in Edge v116, v117 or v118, we implemented a workaround for this issue, that is available starting with UIAutomation v23.8.0-preview.

Workaround for Chrome/Edge v117 issue with Active Accessibility support for IFRAMEs and PDFs

Starting with Chrome and Edge v117 we are seeing a new bug that affects Active Accessibility support for web pages that contain IFRAMEs or PDF documents. More details and available workaround can be found in our forum post about this issue.

The code fix included in UiPath.UIAutomation.Activities v23.10 removes the need to add the
--force-renderer-accessibility=complete flag when starting the browser.

No “Restore pages” popup when opening Chromium browsers

Sometimes, when the browser closes unexpectedly or when the browser was closed during Windows shut down, the “Restore pages” popup is displayed. This can break the automation if no specific logic is implemented on browser opening, to check if the popup appears and close it with a Click activity.

We have implemented a change that prevents the “Restore pages” popup window from appearing when opening the browser, so that no additional logic is needed to close it.

Copy the error message to clipboard

In all dialog boxes that show errors coming from UIAutomation, we’ve added an option “Copy to Clipboard” for the Runtime execution error messages. This makes it easier for users to get the error message when reporting issues.

SAP Automation

SAP Table Cell Scope improvements

We have heard the feedback from the customer and improved and made it more flexible to perform automation with help of Table Cell Scope activity.

Now, we are allowing the filtering on level of Columns and Rows. The identification of columns in SAP table is done per default based on the Tooltip, which provides the readable and, in most case, reliable access to the column. But sometimes columns within one table has a similar ToolTip. Now we are able to recognize such scenarios automatically and select ColumnName as an alternative solution for identification. Additionally, the user has an ability to use any other property to identify the element.

Example of SAP Table with 2 and more columns with the same Tooltip

New Table Cell Scope Activity offering the right identification.

Another example, where we are introducing the ability to provide a context for the row.
Use Case: I want to double-click the Rachel McMiller in Column Name1, but I do not know the exact Row Number. I know that the customer number is 1000273.

This is how it works now with Table Cell Scope Activity:

SAP WinGUI 8.0 - support multi-monitor scaling awareness

We are introducing support and improving the multi-monitor scaling issue of the latest version of SAP WinGUI Client.

The settings in SAP Logon is:

If you are using multiple monitors with different scaling percentages, go into UiPath Project settings and activate a new setting, which enables an additional scaling to support SAP native multi-monitor scaling awareness. This will ensure proper identification of elements and you would be able to move the SAP window freely across different monitors.

UiPath Studio → Project Settings → UIAutomation Classic → SAP → Enable additional scaling.

We are happy to provide additional support for the following SAP FIORI and SAP WebGUI Elements.

SAP Trees

We have significantly improved the automation of all available SAP Trees in SAP Fiori and SAP WebGUI. It includes stable identification of the Trees, reliable automation across all SAP FIORI versions and all browsers.

See the screenshot below to learn more about SAP Trees and possible operations.

SAP Menu Item

SAP Menu automation in SAP WebGUI has become easier and more convenient, just use Select Menu Item activity and get the list of all available SAP Menu Items on the screen.

SAP Click Toolbar Button

We extend the support of this activity to cover SAP WebGUI. Now you can get all elements displayed and available in the dropdown menu direct in Studio. See example here:

Expand Tree Activity

Now you can expand the SAP Tree element in SAP WebGUI and start working with them as usually. We introduced a set of new SAP WebGUI specific attributes such as sapweb-id, sapweb-path, etc. At design time you need to indicate the node that you need to expand. During the run time this node will be expanded.