UiPath Community Stable Release 2023.4 - UI Automation, Computer Vision, SAP automation

This topic goes in-depth about the improvements in UIAutomation, Computer Vision, and SAP Automation. To read about other products, please navigate to the main topic here.

UIAutomation - General

Browser startup timeout configurable from Use Application/Browser and Open Browser activities

Previously we had a static timeout of 60 seconds to start the browser. This proved to be insufficient in unattended cases, on low spec machines, when the workflow is started from the Orchestrator and a new session is created.

Since users may need to set a longer timeout for the browser startup, we made it configurable from the Use Application/Browser and Open Browser activities. The following project settings have been added:

UI Automation Classic → Browser → Open browser timeout (seconds)

UI Automation Modern → Application/Browser → Open browser timeout (seconds)

Workaround for Simulate Click with MV3 Extension

A new workaround is available for the known issue Click activity with Simulate fails on specific web page elements.

The workaround is available through a new project setting UI Automation Classic → Browser > Enable workaround for simulate click with MV3 extension (default: false).

The purpose of this setting is to allow users that have many workflows affected by this issue to apply this workaround through a simple switch - instead of manually checking each Click activity from their workflows and then change the change the input method from Simulate to any other input method (Chromium API, Hardware Events, Window Messages).

PDF automation with Protected Mode enabled

For UiPath.UIAutomation.Activities 22.10 and older and Adobe Reader DC version 22.003.20258, PDF automation may be affected. It was not possible to identify individual elements in PDF documents, instead the entire document was selected. The issue was caused by a security option available in Adobe Reader DC, Enable Protected Mode at startup, which is enabled by default.

Starting with UiPath.UIAutomation.Activities 22.12, you can automate Acrobat Reader even if you have the Enable Protected Mode at startup option enabled.

Improved support for Chromium-based embedded browsers

We improved the support for Chromium-based embedded browsers (Electron, CEF 103+) and added support for orphaned WebView2 windows.

We also added support for the latest WebView2 runtime 106.0.1370.34.

We extended the support for automating Chromium Widgets/Electron applications with HTML selectors, introduced with version 22.8 of the UI Automation package.

Now, this support is available for Chromium versions 108, 109, and 110.

Electron applications can create multiple BrowserViews inside a single BrowserWindow. Each view translates into a new and separate WebContents instance added to the Aura window widget. In the previous versions, our Chromium support used only the first WebContents it could find.

Now we support to:

  • iterate over all WebContents
  • get WebContent’s position
  • check if a WebContent is active

Also, we have added support for CefSharp applications with multiple UI threads.

Easier use of Self-hosted extension with Native Host Only installation

If the Group Policies in an organization are defined at the user or cloud level, problems may appear since the UiPath browser extensions installed via Group Policy update the policies at the machine level, overwriting the user/cloud level policies (see Group Policy conflict).

For this case, the recommendation is to only install the Native Messaging Host component (ChromeNativeMessaging.exe). With this version it is easier to use the Self-hosted extension (for the Group Policy Offline installation method) by just adding the following value to the ExtensionInstallForcelist policy:

ndmegdjihnhfmljjoaiimbipfhodnbgf;file:///C:/ProgramData/UiPath/UiPath.Common/ChromeExtension_SelfHosted/extension_manifest_chrome.xml

Added the WebRequest permission to the Chrome/Edge extensions

We have added the WebRequest permission to the Chrome/Edge extensions to allow using the web request API.

As a result, the Chrome extension v23.4 has been published on the Web Store with new Extension IDs:

  • kmejkhcmhgilmppjodlgaklnmdmmhhcl – on Google Web Store
  • dknkgjgkdpkmddgdjlgdhfojlaehikmk – on Microsoft Store

The Self-hosted extension, used for Group Policy Offline installations, has also been updated with this permission, but kept the same Extension ID: ndmegdjihnhfmljjoaiimbipfhodnbgf.

Select Item activity improvements

We improved the Select Item activity to support complex HTML dropdowns, like non-native and non-SAP Fiori/Salesforce dropdowns.

Check support for non-standard checkbox controls

We have added check support for non-standard checkboxes. This includes labels that have a for attribute, which indicates the hidden checkbox, labels that have a single (hidden) checkbox descendant, and other case that are highly reliable.

Improved Modifiers field for event data

Previously, for event data that returned modifiers, the Modifiers field consisted of an unreadable bitmask, having the int data type. Therefore, without having some additional skills to search for information, users might have been confused.

The Modifiers field for event data is now mapped to the NKeyModifier enum and returns a string (such as Shift, Alt, or Ctrl) instead of an integer when ToString() method is called. For example, before this change the returned value for Modifiers was 23, while now it is Shift, Alt, Ctrl.

UIAutomation – Modern

New native events monitoring framework

In the context of the Trigger-based Attended Automation initiative, where execution performance is critical, we improved the event monitoring usability and performance, by implementing new sets of events supported by different types of application technologies: wnd, ctrl, java, webctrl.

The events are available through the Application Event Trigger activity, which is meant to help set up a trigger based on a native event for any type of UI element. The user needs to indicate the target element on the screen and then choose the native event from the list of events supported by that element.

image017

Main native events exposed for each supported technology (Web with <webctrl> tags, Active Accessibility with <wnd>, <ctrl> tags, Java with <java> tags):

  • webctrl: Appeared, Disappeared, Text changed, Key pressed, Click, Double click, Right click, Mouse move, Mouse entered, Mouse exited, Mouse pressed, Mouse released, Focus gained, Focus lost
  • java: Appeared, Disappeared, Text changed, Key pressed, Click, Mouse move, Focus gained, Focus lost, Enabled, Disabled, Selection changed, Mouse entered, Mouse exited, Cell selected, Cell value changed, Mouse pressed, Mouse released, Menu selected, Menu deselected, Menu canceled, Popup menu visible, Popup menu invisible, Popup menu canceled
  • wnd top-level windows: Appeared, Disappeared, Title changed, State changed, Location changed, Foreground gained, Foreground lost, Minimized, Restored
  • wnd, ctrl: Appeared, Disappeared, Text changed, Focus gained, Focus lost, State changed, Selection changed

For example, the editable text of the Notepad application is a ctrl element, and the Event type list displays the available events for this technology:

application-event-trigger

The Application Event Trigger activity exposes these properties:

  • Target – selected UI element. Only strict selectors are supported, without anchors.
  • Scheduling mode – It specifies how to execute the actions when a trigger is fired. Available values: Sequential (actions are executed one after another), Concurrent (actions execution can overlap), OneTime (executes one action and exits monitoring).
  • Include children – when selected, the children of the specified UI element are also monitored. It is supported by all event types except Appeared and Disappeared, and for any UI element except top-level windows.

Modern Click & Keypress Event Trigger activities

We have migrated the most used Classic triggers to the Modern framework. The new Click Event Trigger and Keypress Event Trigger modern activities are designed to identify the elements via a unified target concept, use an Object Repository element and offer a friendlier user interface.

One other important reason why we want them into the CC solution is that they can block the original event and replay it later on, or ignore it. This specific capability allows implementation scenarios like a form submission is stopped until all the fields are correctly filled in.

click-event-trigger

keypress-event-trigger

Modern Set Text activity

Set Text activity just got a modern version. It can identify targets with the unified target concept, and it can use an Object Repository stored element. Moreover, it can cache the target element to increase the execution speed when executing repeatedly e.g., when synchronizing text between multiple UI elements.

No scope for activities working with UIElement as an input

Modern activities require an application card/scope, to minimize the area of searching the target element, thus, increasing the automation reliability. However, when the activities receive an UIElement as input, they do not require the scope. So, all the modern activities have been updated accordingly.

Inject Js Script Modern/Classic activities got a new property: ExecutionWorld

The Execution world property for the Inject Js Script activity sets the JavaScript world for a script to execute within. The Isolated option allows the content script to make changes to its JavaScript environment without conflicting with the page. The Page option allows the script to interact with the page as well.

Computer Vision integration in Unified Target

Building and running reliable automations is the key to success for any automation program. The Unified Target concept is UiPath’s solution for ensuring the UI Automation based robots are reliable and trustworthy.

Computer Vision is an important AI powered targeting image-based technology that in some situations is the only one that can be used to reliably automate an application. Use-cases span from CV being used atomically to automate certain areas/elements that the driver cannot drill into, to VDI automation where CV is the only viable technology, and everything in between.

Computer Vision is now part of the Unified Target solution to help customers:

  • reduce the complexity of building UI-based automation: customers should not know nor care about technical underlying concepts like selectors, AI-based screen searches, or OCR technology; they should simply point and click the screen elements: the rest is UT’s job.
  • to increase the reliability and resilience to changes, in UI-based automation: often, customers do not control application UI changes; the UiPath robots must deal with these situations out of the box, and be smart enough not to break, remaining uninfluenced by the UI deterministic changes.

Computer Vision is enabled by default in Project Settings > UI Automation Modern for Desktop, Java and Web and disabled for SAP (as the happy path for SAP is using the strict selectors):

Computer Vision can also be enabled from the Helper:

The UIA Helper now has a streamlined new design focusing on optimizing vertical space, using a horizontal icon bar on top for the most common functions used:

  • Pause with a dropdown for changing the number of seconds of pausing the configuration
  • Hoverable elements & Image region selection buttons
  • Enable/disable Native Text & Computer Vision buttons
    • [F8] can now be used to enable/disable Computer Vision
  • Changing the UI Framework is now a dropdown
    • Default has been renamed to Auto
    • Active Accessibility and UI Automation Frameworks now show their own icons

After indicating the target, the Configuration section is expanded, now holding an Options, Windows Selector and Target/Anchor sections (also expanded by default):

Current scope & limitations:

SAP WebGUI Automation

Starting from UiPath.UIAutomation.Activities package v23.4 we are introducing a new way of identification and automation of SAP elements of SAP WebGUI. This allows us stable and reliable identification independent from the dynamical nature of the interface.

The entire range of activities from the UIAutomation Activities Pack can be used for SAP WebGUI automation. For some SAP specific controls, we are providing an additional extended support. See the full documentation.

SAP WebGUI uses the proprietary SAP HTML framework to display its elements on the page. Selectors generated for these elements are easy to recognize by the sapweb- attributes in selectors.

A set of sapweb- attributes allows stable and reliable identification of display elements similar to SAP WinGUI technical selectors.

4 Likes

Cool to see that Computer Vision is now integrated directly into the Unified Target! That should make for even more reliable automations.