A new UI Automation input method: the ChromiumAPI - highlights

As you might have read here, we have added a new browser automation input method along to the existing ones (Hardware Events, Simulate and SendWindowMessage), named ChromiumAPI .

What is it?

Based on Devtools protocol, this feature comes with many benefits when automating a Chromium-based browser.

Which browsers is working with?

Edge and Chrome. Although, for 21.6-preview we have released only Edge support. Chrome support will soon come.

Which UIAutomation activities can be used with?

The Modern UseApplication/Browser, Click, TypeInto and Hover activities.
image

How does it compare with Simulate & HardwareEvents?

Here are some insights you might want to consider when design browser automation with the ChromiumAPI input method:

ChromiumAPI vs HardwareEvents

Reliability

Sending keys/mouse input via hardware events means sending them to the operating system’s mouse/keyboard input stream. From there, the events are forwarded to the target application, the browser in our case, which parses them and sends them further to the web pages.
Sending keys/mouse input via ChromiumAPI means sending them directly to the browser. Therefore, this input method eliminates a communication channel that can fail (as any other communication channel).
In conclusion, directly communicating with the browser via this new method means fewer communication channels, thus improved automation reliability.

Works in the background

Hardware events need the elements to be on the screen, focused, in the foreground. The Chromium API does not have this requirement. It works while the browser is minimized and another tab is focused (mind that the modern activities tend to activate the working tab).

ChromiumAPI vs Simulate

Improved efficiency

The ChromiumAPI sends trusted events triggers, which are the exact chain of events triggered when a human does the same action. This means that, besides the automation that worked with Simulate, a lot more is available now through the ChromiumAPI.

Other benefits

It allows sending hotkeys directly to the browser.
The ‘Clear before typing’ is natural with this method, since does not empty the target element field by default.
The ‘Click with modifiers’ actions are more efficient and reliable.

What do I need to try it out?

You need the latest UiPath Edge extension and the 21.6-preview UIAutomation.Activities package.

The latest UiPath Edge extension can be installed from the Studio 21.4.4 Stable or 21.6-preview.

Note: When the browser is started with the ChromiumAPI, a ribbon shows up stating that the browser started in debug mode(highlighted below in blue). This message does not show up if the extension is installed via policy.

Enjoy! And do not forget that feedback is always welcomed!

Looking to hear from you.

Gheorghe.

28 Likes

Looks cool. Thanks for the clear steps !

1 Like

Thanks for this update.

1 Like

Good Update, Waiting for chrome browser support.

1 Like

@gheorghestan

Sounds excellent Gheorghe.
:+1:

3 Likes

Sweet! I like Background processing enabled automations! :slight_smile:

What about Firefox? I’m asking because if we build Browser-neutral automations and all of the top 3 Browsers support Chromium API communication channel, then it’s easy. If not we have to make an exception with Firefox ( or any other Browser that isn’t supported) .

Is the Input mode list an Enumeration? I was thinking if there is a way to pass a parameter into it so that I can set it back to the regular method of communication if a Browser doesn’t support Chromium API.

thanks!

Hardware event will work in the background. Just wow :smiley:

@AndyMenon Firefox itself does not support ChromiumAPI. But if they’ll add support in the future, we will enhance the UI automation input for FF as well.

1 Like

looking good @gheorghestan, thank you for this!

Thanks @gheorghestan .:slightly_smiling_face:

@gheorghestan While using Chromium API Input option with Edge version 92 I am getting error ‘Edge Chromium API’ should only be used with Edge Chromium browsers.
Edge version 92 is chromium based. Does anyone know why it could be coming?
Thanks in advance!

Hi @ritikanand ,

That is a little bit strange. Can you share a sample of your workflow file where you meet this error? Also, it would be useful to know if the Edge extension is enabled while you get the error and which version of UiAutomation activities package you are using.

Thanks