UiPath Community 2021.10 Stable Release - UIAutomation

[gheorghestan]

UiPath Community 2021.10 Stable Release - UIAutomation

This topic goes in-depth about the improvements to UIAutomation. To read about other products, please navigate to the main topic here.

Modern UIA as default for new Studio projects

With Studio 21.6-preview, for new projects, we are enabling the UIAutomation Modern design experience by default. At any moment, one can either use the Classic activities or switch to Classical design experience entirely.

Nested Use-app cards

Until today, automating multiple apps simultaneously can sometimes be cumbersome, as it requires careful nesting of Use application/browser cards, and placing UI activities in the correct card. Starting today, no more! You can just nest Use application/browser cards however you want, and inside them, configure each UI activity to reference whichever card you need.

Background automation

Finally, this often-requested feature is here! It allows you to run UI automation in the background, while you are working on something else. It’s not a replacement for Picture-in-Picture, as PiP fully guarantees that the automation will not interrupt you in any way, while this new Background mode still has some limitations that might briefly show the running automation for some specific actions that are not background-compatible, like keyboard shortcuts and screenshots. But it’s just the first version, we have quite a few improvements for it in the pipeline, so stay tuned.
In order to enable it, set the Input mode of any Use app/browser card to Background .

Capture Elements – Capture all elements on a screen

This one is pretty exciting. Apart from ComputerVision, it’s the first AI-powered feature in UI automation and Studio.
The Capture all elements tool lives inside Capture Elements from Object Repository, and it extracts all the relevant elements from a screen (along with their detected anchors) and adds them to your Object Repository library. This frees you from manually adding them one by one.
We believe this will be a great timesaver for companies that embrace the “Object Repository way”, so give it a try and let us know what you think.

Capture Elements – Highlight an element, or all of them

In addition to the feature above, we also added the ability to highlight elements in Capture Elements. Either one by one, or a whole screen of them. If you’re an Object Repository user, you’ll find this ability pretty useful.

Object Repository support in UIA Modern Recorder

The Recorder has been integrated with Object Repository, meaning that now, when you are recording a process, the recorder will automatically reuse existing objects from your library, or create new ones on the fly. Essentially, if you have a UI library, it will be automatically used, and if you don’t have an existing library, it will be created as you record your process. Win-win!

Preview and advanced configuration for Get Text

The modern Get Text activity was always pretty smart in the way it handled text extraction: it tries multiple extraction methods and returns only the best result. But in some situations, the various methods return different results, which sometimes might result in not getting exactly what you need. With this new feature, we exposed more advanced configurations of the extraction methods and added the ability to preview all of them, so you can pick the one that gives the best result for your use case.

The Modern GetAttribute

The GetAttribute activity has been upgraded to the modern experience, receiving a few enhancements along the way: the list of attributes now shows the current/live value of all the attributes, and descriptions for most used ones, so you are more efficient when working with them.

Excel & PowerPoint scopes now support UI automation

In previous versions, if you wanted to supplement your excel automations with a few actions that are only possible through Excel’s UI, you had to add a new Use-app activity, and add all the clicks/types inside it. That was a bit cumbersome and harder to read.
But starting with 21.10, you can just add UI activities inside the Excel or PowerPoint scopes, and they work just like they do when placed inside an Use-app scope.

Nicer variable support

Adding or editing selectors with variables is now easier. All variables generated from the selection screen are now better configured, with default values so they whole descriptor can be evaluated in real time, and there is an automatic find & replace functionality that ensures your changes are propagated correctly.

Mouse Scroll

A highly requested feature, this scroll activity is smart! It allows the basic row-based scrolling, as well as To-element scrolling, for those situations when you need to scroll to a specific object on the page. On top of that, it supports “endless” pages, as a lot of modern websites are built this way.

Verify execution

This capability introduced in 21.4 quickly became a favorite of many, and for good reason. It makes the automations much safer, by ensuring that the actions of the Click, TypeInto, Hover and KeyboardShortcuts actually have the desired outcome. Pretty powerful stuff.

With 21.10, activities can check not only if the target element appeared or disappeared, but also if its text has been modified, or if its visual appearance has somehow changed.

Excel cards as App cards

Starting with 21.8, StudioX’s Use Excel File is promoted to act as an Use app card. Meaning, now you can add a single Use Excel card and inside it, place both specific Excel activities, as well as any UI automation activities, like Click, Type, and so on, and they will work just like inside an Use app/browser . This will greatly increase the flexibility of automating Excel with a mix of dedicated Excel activities + UI automation activities, to achieve any and all Excel functionality.

Type Into - Automatically set the correct input method

When building UI automation when building UI automation, setting the correct input method is paramount, as it can greatly increase reliability and speed. Unfortunately, picking the correct one is a bit of a black art, as there is no hard rule to follow, so some trial & error is involved. This new improvement automates this process, performing all the trial & error for you, with the click of a button, located right next to the input mode. Enjoy!

Browser migration tool

A simple standalone tool that converts the XAML files targeting a (IE, Chrome, Edge, Firefox) into XAML files targeting a (Chrome, Edge, Firefox).

It can be used with XAML files generated with any Studio or UIAutomation. version (e.g., 18.4, 19.10, 20.10, 21.4).

The tool does:

• Migrate the <html …> selectors by updating their ‘app’ property.
• Change the BrowserType property of the ‘Open browser’/‘Attach browser’ activities.
• Change the default value of the variables from “sourceBrowser” to “targetBrowser”.
• Updates the KillProcess activity is it targets the “sourceBrowser” process.

The tool does not:

• Modify the selectors starting with <wnd …> (window specific selectors). This type of selectors, identify UI elements rendered differently by IE in comparison with Edge/Chrome/FF. Thus, they must be regenerated manually (reindicate the target element in the new browser).
• Ensure 100% success rate. This can only be achieved through thorough testing

Because it has no dependencies on Studio or UIAutomation.Activities packages, the tool can be launched in command line or using its basic UI:

When run in command line, the minimal configuration is required:

The tool will be available for download from the ‘Resources Center’ of the cloud.uipath.com, under the ‘Other products’ section.

Browser runtime switch

If you build up your Classic automation for a specific browser type but want to run it on a different browser type, now you can. Simply use the ‘Runtime Browser’ option from the UI Automation Classic project settings, to switch to the target browser. All the specific selectors & activities will be switched automatically at runtime.

Note: the automatic conversion will not work for selectors starting with <wnd …> (window specific selectors). This type of selectors, identify UI elements rendered differently by the IE/Edge/Chrome/FF. Thus, they must be regenerated manually (indicate the target element in the new browser).

Microsoft RemoteApp UIAutomation support

RemoteApp is a virtual application solution that allows users to run virtualized Windows-based applications. It allows users to launch virtual applications from a server that appear on their computer as if it is installed locally, but in reality, are running on a remote server.

The UIAutomation support is now available for automating this application types: both designtime and runtime (via de UiPathRemoteRuntime).

RuntimeGovernance & RT-UIA-001 allowed/prohibited apps/URLs runtime rule

With the help of AutomationOps policies, we are launching the UIAutomation runtime governance. The RT-UIA-001 runtime rule, allows setting the allowed and/or prohibited applications/URLs for being used when running UI automation. The verification is done at runtime and the execution it’s permitted or stopped, based on the rule’s configuration.

Project Setting - Show closest matches in the “Element not found” error message

Sometimes, you don’t want the ElementNotFound error message to log the closest matches, as they might contain confidential data. Thus, there is a new Project Settings option available now to control this behavior:

New input method: Chromium API

A new input method is now available for the Modern Click, TypeInto and Hover activities: Chromium API. This comes in addition to the existing methods Simulate, HardwareEvents and SendWindowMessage and adds new capabilities, one of which being better support for automating browser tabs that run in the background.

Get/SetBrowserData activities

Imagine a scenario where you and the robot work together to complete a task. On the same machine and the same browser application. And without stepping on each other’s toes.

You login into the business browser applications via your company authentication/authorization mechanism (e.g., SmartCard PKI, other 3rd-factor auth), the robot takes over your work, inherits your browser session data (like authentication cookies - once the automation moves into PIP, you will not need to re-authenticate) and finish the work without bothering you. And you can monitor its progress in a small PIP window.

The Get/SetBrowserData data activities come to help you implement this experience.

The GetBrowserData activity exports the browser session data from a specified browser instance, while the SetBrowserData sets the session data into another browser instance. They are available in both Classic and Modern experience and for now, they only work for the Edge browser. Chrome support to come soon.

OpenBrowser: ensure WebDriver Compatibility option

So far, running automation via WebDriver required keeping manually in sync, the browser, and the WebDriver versions. No longer the case. We do it automatically now, via the OpenBrowser.AutomaticallyDownloadWebDriver new option:

Salesforce Lightning improved UI automation

Salesforce Lightning is a UI interface of the SFDC, built with Lightning Component UI Framework. Creating UI automation over these components is specifically challenging because various UI attributes are changing on the fly, at every page refresh. But, the UiPath UIAutomation is now much stronger:

• we have implemented proper UI controls identification and easy usage of the dedicated activities

  

• we generate selectors more reliable and resilient to changes, based on the attribute at the very core of the Lightning framework

  Example:

  <html app='chrome.exe' title='Users | Salesforce' />
<webctrl aaname='Object Manager' sfl-path='oneOne/lafAppLayoutHost:host/navexDesktopLayoutContainer/navexContentRegionNavContainer:contentRegionNavContainer/navexWorkspaceManager/navexConsoleTabset:parentTabset/navexConsoleTabContainer:tabBar/navexConsoleTabItem/auraHtml' tag='SPAN' />

Edit Table Extraction

The table extraction wizard can now be restarted to resume work from where it was left. The already added data is highlighted so you can remove it, add new data or configure the columns properties.

ComputerVision

Automatic authentication – no more API key

Starting with 21.10, the authentication to the ComputerVision Cloud service can be done automatically, if you are logged into Automation Cloud in Studio. The API key authentication will still work and can be used in hybrid scenarios, when you use an on-prem Orchestrator, but need to access the ComputerVision Cloud service.

Table cell targeting

Targeting cells was always possible using the native anchors available in ComputerVision, but in cases where tables had a lot of empty cells, there weren’t enough usable anchors, and it was hard to target specific cells. With this new feature, we introduced a dedicated cell targeting method, that allows a more natural definition of a target cell, based on row and column. Each row/column can be defined either by row/column number, or by name, or by the content of the row.

[Arpit_Kesharwani]

Since now I am not a big fan of the modern design experience but the new feature looks great mainly switching process to run on another browser