UiPath Community Stable Release 2022.10 - UI Automation, Computer Vision, SAP automation

[gheorghestan]

UiPath Community Stable Release 2022.10 - UI Automation, Computer Vision, SAP automation

This topic goes in-depth about the improvements in UI Automation, Computer Vision, SAP automation. To read about other products, please navigate to the main topic here.

Generic support

Upgrade UIAutomation to .NET 6

UIAutomation now uses the latest .NET release, .NET 6. Windows and Cross-platform projects now also target .NET 6.

Support to automate native Chromium Widgets with on-screen rendering

We often see the need to automate native Windows applications embedding web applications with the help of Electron/Chromium Widgets technology. From now, the web module rendered inside a Chromium widget can be automated with standard web selectors (html, webctrl), instead of IA2-based selectors (wnd, ctrl). This brings all the web automation power at hand when automating desktop applications embedding web applications through Chromium widgets.

A few examples of applications with Chromium (mostly Electron) are UiPath Assistant, Slack or Microsoft Teams.

Please note that the support only works with compatible Chromium versions v80 to v102.

Support for automation of custom Remote Desktops

Using the RDP ActiveX control, one can build custom remote desktop applications, besides custom remote apps. One such example is the Picture in Picture functionality. We’ve added support for automation such applications.

VB6/Epic software UI Automation improvements

VB6 improved support

Select Item activity can be now used with VB6 drowpdown controls. Also, we have added data extraction support for VB6 grid components: MSFlexGridWndClass, ListView20WndClass and MSHFlexGridWndClass.

Better VB6/Epic selector attributes

The ctrlid attribute is extracted and added to the selector for the following controls with Default framework: TextBox (input field), Button, RadioButton, CheckBox, PictureBox, ListBox (select item), ComboBox. The ctrlid attribute is unique for each window of the application. It is also stable between application runs, making it a great option for control identification.

Similarly, for UIA framework the automationid attribute is added to the selector for the same type of controls.

Java Improvements

Kinaxis Java support

We have improved the Kinaxis Java support, with proper grid table cell identification and data extraction.

Add backgroundColor and foregroundColor properties for Java elements

The backgroundColor and foregroundColor properties of Java elements are now exposed as strings containing RGB triplets and can be used in selectors.

Add the ‘name’ attribute by default in the selector for toolbar buttons

The default selectors for toolbar buttons in Java applications have the name attribute now added by default to their selectors.

Qt UI Support

Support for applications based on the Qt Framework has been developed through the new type of selector tag: qt.

Selectors would look like this (with qt tag):

<wnd app='widgetstestapplication.exe' cls='Qt*QWindowIcon' title='Qt widgets test application' />
<qt name='QPushButtonObjectName' rawtext='Add' role='QPushButton' />

A well-known implementation built on Qt 5 is Tableau.

We currently support Qt 5 Widgets (no Qt Quick/Qt QML), and Qt 6 is on our roadmap (currently falling back on the AA framework).

Web Selectors Improvements

Several improvements have been implemented for the web selectors.

The default selectors have new attributes, such as the HTML class attribute or the aria-label and aria-labelledby attributes, as well as the improvement of priorities of the attributes from the default selectors.

The title attribute has been removed from the list of available attributes for the webctrl tag. It also added the reliability verification for the id attribute from the webctrl tag.

Starting with version 22.4 the attribute visibleInnerText, which reflects the actual innerText of the underlying HTML element, was added to UI Explorer → Property Explorer. From now, the visibleInnerText attribute has been exposed as a selector attribute in UI Explorer.

Open Remote Browser

When using Native Remote Automation for a Remote Desktop (Citrix or RDP) it is possible now to launch a browser inside the remote session, and then automate it, by using the modern “Use Application/Browser” activity.

Previously, the classic “Open Browser” activity did not allow this functionality. As a workaround, an “Open Application” activity should have been used, followed by an “Attach Browser” activity.

Setup Extension Improvements

Expose all the methods for installing the browser extensions in Studio → Tools

The different installation methods for browser extensions (per user, per machine, Group Policy Online, Group Policy Offline) were available only from the command line.

In Studio → Tools, for each browser extension, a combo box has been added to allow the user to select the installation method:

• Group Policy Offline
• Group Policy Online
• All Users
• Current User
• Native Host Only

For each of these methods, tooltips are provided for explaining their purpose.

Once the extension is installed, the installation method remains displayed in the disabled combo box, so that the extension may be uninstalled with the same method used for installation.

Provide description for extensions in the extension status

In the extension status info (/status-admin), extra fields have been added for description, documentation info and warnings. These fields contain localized text displayed in Studio->Tools or UiPathStudio.msi.

UIAutomation Modern

Table Scraping in Studio Desktop

New Design and continuous improvements:

• New Menu at selection stage for extracting text/URL on web scraping scenarios
• Added Preview option at the element level
• New pagination link toggle with clearer messaging
• New On/Off highlighting option of confirmed series of elements

Object Repository Support

Several improvements have been made:

• Create screens automatically when adding a new element to empty Object Repository
• Edit Screen for Browsers - Arguments are now being exposed as editable text
• For Each UI Element & Extract Data - Improved Indicate & Editing experience when linked to Object Repository
• Pre-populated modern Select Item items list when an Object Repository target is paired with it
• Object Repository Screen edit with expression editors for FilePath, Arguments, URL

Keyboard Shortcuts Improvements

Allow passing variables/arguments in the Shortcuts fields.

The Keyboard Shortcuts activity now offers a DelayBetweenKeys property that can be used to set a delay between keystrokes, similar to the Classic behavior. A corresponding project setting, Delay between keys, has been added in the Keyboard Events section.

Add support for DefaultSystemBrowser to be used in variable’s default value

It is now possible to configure automations to run using the default system browser, not only the specific browser set at design time. This is allowed through the DefaultSystemBrowser class which has been made available for usage in the Default value for variables.

For example, DefaultSystemBrowser.Name is the executable name (chrome.exe, msedge.exe, firefox.exe) and it can be used in selector definition. Also, DefaultSystemBrowser.BrowserType can be used for getting the browser type string (“Chrome”, “Edge”, “Firefox”).

Anchors Improvements

Improve AutoAnchor detection on Salesforce fields that use “for” attribute

Previously the “detecting auto-anchor” step took approximately 4 seconds, but now there is a big improvement as this step takes approximately 1 second.

Automatically enter add-anchor mode when duplicates are found

By default, we were allowed to add a single anchor, and we needed to explicitly click the “add anchor” button to add more. But sometimes there are duplicates right after indicating something, which makes things hard to correct.

With this improvement, if after indicating an element (or an anchor) and validating the current selection as “duplicates found (yellow)”, we automagically allow adding another anchor (until the maximum of 3 anchors is reached).

Remove “Enable anchors” from project settings

As the anchors are the healthy way of automating and we want to promote them, we removed the “Enable anchors” flag from project settings, for all targeting methods (the default value is true). For the projects that have this project setting on false, we still show the option, so they aren’t stuck with no anchors.

UI-PST-001 Project Settings rule

UI-PST-001 is the Workflow Analyzer static rule applicable to UIAutomation Modern, that in its first version was for Robot Logging properties only. The description has been changed now to “Audit log level in project settings”, with the scope Project and the Default Action Warning.

Audit log level in project settings (UI Automation Modern → Robot logging) can be restricted via this rule. Possible values for AuditLogLevel are Off/Info/Trace (default Off).

Region Selection for TakeScreenshot activity

The Take Screenshot activity now supports region selection via the F3 hotkey when indicating on screen.

UIAutomation Classic

Modern Selection Screen for Classic Activities and Recorders

For consistency, we have enabled the modern selection screen for the Classic activities and Recorders.

Browser Migration Tool

Migrate the Object Repository targets

When migrating workflows from one browser to another, we are migrating the Object Repository targets as well.

New option for migration from IE to Chrome/Edge/Firefox

When migrating a workflow to Chrome/Edge/Firefox, if the attributes innertext or aaname` are part of the IE selector, they won’t work, because the attribute values look different for Chrome/Edge/Firefox browsers.

This issue is caused by the fact that for IE the innertext selector attribute maps to the HTML innerText property, while for Chrome/Edge/Firefox the visibleInnerText selector attribute uses the HTML innerText property.

For the migration from IE to Chrome/Edge/Firefox, a new option is presented to the user:

IE - Replace innertext/aaname attribute with visibleInnerText (requires at least v22.4).

If this option is selected, for all webctrl selector tags from IE:

• innerText is replaced with visibleInnerText
• aaname is replaced with visibleInnerText
• trim begin-end whitespace from the value of visibleInnerText
• if both innerText and aaname are present in the same webctrl tag from the IE selector only the first visibleInnerText is kept in the migrated selector

This option should not be used when migrating from IE to Edge IE Mode. For this type of migration, is enough to select IE as the Source and Edge as the Destination browser.

Computer Vision

Smarter Cloud & On-Prem CV AI Model

We’ve deployed a new iteration of our CV AI Model for Cloud & On-Prem, significantly better performing when working with tables and OCR data due to an improvement in the internal neural net architecture.

Smarter & Faster, Full Feature Set Localserver AI Model

The Localserver is the compressed version of the CV Model running on the Cloud and On-Prem CV servers. It has a lower number of neurons (so slightly lower UI elements detection accuracy) and is running on the local machine’s CPU – no GPU required. It is enabled via a property of the CV Screen Scope:

For the first time since we’ve launched this version of the CV AI model, we’re very happy to announce it is now on-par feature-wise with the Cloud and On-Prem model, as it can also be used to extract tables and is outputting auto-anchors.

Detection accuracy has also improved, while the AI model is now 1.5x faster!

On-Prem AI Model update resilience on par with Cloud

Our On-prem customers can now have their design-time data learned by the UiPath’s CV AI Model by enabling the “Include in AI Learning” toggle in the CV Screen Scope. Any new iteration of the AI Model will learn, thus detecting all the data the previous iteration was, making all the On-prem CV automations with this feature enabled as resilient to model updates as any Cloud CV automation.

Since the AI Model only learns from the shapes and colors of the UI elements and not the real data content, we highly recommend using mock data when building any CV workflow to avoid sending any sensitive information, such as personal data. Only valid screen UI elements will be saved, everything else will be discarded.

This feature is turned off by default and can be enabled on a granular level, per each CV Screen Scope – so our On-prem customers still have full data residency control. After enabling it, all screenshots from the specific Scope will be sent to our cloud. When disabled, automatic deletion requests are created, thus all screenshots will be discarded and excluded from the training set.

Run-time Auto-scroll Support

Automating scrollable content in webpages or apps using CV activities is now much easier and can be done using the usual delayed screen refresh UX [F2]: scrolling actions performed at design-time are replicated at run-time so that the application will reach the same state needed to find the target element.

The 3 seconds counter in the delayed screen refresh UX [F2] will reset with each mouse scroll while indicating and a Refresh will be triggered at the end of this timer, generating a new screenshot and capturing the scroll direction value in a new property called ScrollDirection (from None to Up or Down). When ScrollDirection is None, no scrolling will be performed at run-time.

At run-time, the ScrollDirection’s value is used to search for the target element; if it is not found during this first attempt, the robot will do a top-most (when ScrollDirection is Down) or bottom-most (when ScrollDirection is Up) scroll and then try again.

The NumberOfScrolls property sets the scrolling speed at run-time (default value is 2), while DelayScreenshotAfterScroll is useful when working with dynamic content such as icons that load when scrolling.

When extracting tables in scrollable pages or apps, we recommend setting the ScrollableTable property (“Scroll” property in previous releases) to True; otherwise, an incomplete table might be extracted.

Since this feature is a big change for any current CV user, we’ve updated the [F2] “Delayed Screen Refresh” tooltip and added a “What’s New?” message when first entering the delayed screen refresh UX (which can be hidden when next indicating using the “Don’t show again” checkbox) for a better visibility.

Also, the “Refresh Scope” button in the CV Helper has been relabeled “Screen Refresh” and the “Refresh with delay” button has been relabeled “Delayed Screen Refresh”.

A quick demo of CV Run-time Auto-scroll can be seen here.

Using OCR – Chinese, Japanese, Korean with a CV API Key

The OCR-CJK engine is now available by using a CV API Key as well:

Currently available endpoints per region:

West Europe: https://du.uipath.com/cjk-ocr
Japan: https://du-jp.uipath.com/cjk-ocr
Singapore: https://du-sg.uipath.com/cjk-ocr

More info on CJK-OCR can be found here.

Token Authentication for UiPath Screen OCR

When using UiPath Screen OCR as a standalone activity, there is no need for inputting the CV API Key anymore if you are logged to your UiPath Cloud account.

Auto-anchors for Text Targets

The AI model is now outputting automatic anchors for text targets as well when indicating.

Updated Design for CV Click, CV Get Text, and CV Extract Table Activities

The activity card design has been slightly updated to match the corresponding UIA activities.

“Click type” and “Mouse button” have been added to CV Click.

“Save to” has been added for CV Extract Table and CV Get Text.

New US Endpoint

We have deployed a new CV service endpoint in the USA to better serve the customers closer to that region. We estimate that customers that are closer to the US than to the current endpoint deployment in West Europe can see more than 100ms decreases in latency by simply switching to using the new service URL.

• Current default endpoint, https://cv.uipath.com will still point to our cloud server in West-Europe
• To use the new US endpoint, you will need to manually edit the values in either:
  • Project Settings > Computer Vision > Server to https://cv-us.uipath.com

• CV Screen Scope Properties > Server > URL value to https://cv-us.uipath.com
  

Design-time data will be centralized in our West Europe cloud storage, no matter the region of origin. UiPath’s Data Residency page has been updated.

Single call for CV + OCR requests

When both OCR and CV are requested for the same screenshot, we are now merging the two requests into a single server call. Since the requested UI elements and text are found in the same image, we are thus optimizing for both latency and rate-limiting.

The single call endpoint will be set to the value of the CV endpoint, as follows:

CV endpoint	OCR endpoint	Single Call endpoint
West Europe	West Europe	West Europe
West Europe	United States	West Europe
United States	United States	United States
United States	West Europe	United States
Localserver	Localserver	Localserver
On-Prem	On-Prem	On-Prem
Localserver/On-Prem	Any Cloud Endpoint (WE/US)	N/A
Any Cloud Endpoint (WE/US)	Localserver/On-Prem	N/A
Localserver	On-Prem	N/A
On-Prem	Localserver	N/A

The single-call logic will only be applied if both defined CV and OCR endpoints are either:

• UiPath cloud servers
• Localserver
• On-Premise

SAP Automation

SAP Fiori UI Tree Lists

SAP Fiori Tree Lists are very complex elements used to display hierarchically structured items, which UiPath now is able to automate in a stable and reliable way. All variations of trees are supported for all browsers.

SAP Fiori UI / SAP SuccessFactors Searchable Combo Boxes

Searchable Combo Boxes are now easy to automate and allow the selection in Studio as usual with help of Select Item Activity.

SAP Fiori UI Global Busy Indicator

Extended support for the most required customer features, such as Global Busy Indicator. Now automation is even more reliable.

Please tell us what you think

Which features excite you the most and you are going to try right away? We want to hear what you think! Please use the button below
SAP Fiori UI / SAP SuccessFactors Searchable Combo Boxes
Searchable Combo Boxes are now easy to automate and allow the selection in Studio as usual with help of Select Item Activity.

[StefanSchnell]

@gheorghestan

Hello Gheorge,

great, thank you very much. I like the dotNET 6 support, the improvements of the CV and the SAP Fiori best, here especially the busy indicator.

Best regards
Stefan

[oscar]

Great improvements all around. Looking forward to using it!