A new UI Automation input method: the ChromiumAPI - highlights

News Product News
,
1

As you might have read here, we have added a new browser automation input method along to the existing ones (Hardware Events, Simulate and SendWindowMessage), named ChromiumAPI .

What is it?

Based on Devtools protocol, this feature comes with many benefits when automating a Chromium-based browser.

Which browsers is working with?

Edge and Chrome. Although, for 21.6-preview we have released only Edge support. Chrome support will soon come.

Which UIAutomation activities can be used with?

The Modern UseApplication/Browser, Click, TypeInto and Hover activities.
image

How does it compare with Simulate & HardwareEvents?

Here are some insights you might want to consider when design browser automation with the ChromiumAPI input method:

ChromiumAPI vs HardwareEvents

Reliability

Sending keys/mouse input via hardware events means sending them to the operating system’s mouse/keyboard input stream. From there, the events are forwarded to the target application, the browser in our case, which parses them and sends them further to the web pages.
Sending keys/mouse input via ChromiumAPI means sending them directly to the browser. Therefore, this input method eliminates a communication channel that can fail (as any other communication channel).
In conclusion, directly communicating with the browser via this new method means fewer communication channels, thus improved automation reliability.

Works in the background

Hardware events need the elements to be on the screen, focused, in the foreground. The Chromium API does not have this requirement. It works while the browser is minimized and another tab is focused (mind that the modern activities tend to activate the working tab).

ChromiumAPI vs Simulate

Improved efficiency

The ChromiumAPI sends trusted events triggers, which are the exact chain of events triggered when a human does the same action. This means that, besides the automation that worked with Simulate, a lot more is available now through the ChromiumAPI.

Other benefits

It allows sending hotkeys directly to the browser.
The ‘Clear before typing’ is natural with this method, since does not empty the target element field by default.
The ‘Click with modifiers’ actions are more efficient and reliable.

What do I need to try it out?

You need the latest UiPath Edge extension and the 21.6-preview UIAutomation.Activities package.

The latest UiPath Edge extension can be installed from the Studio 21.4.4 Stable or 21.6-preview.

image
image889×191 20.9 KB

Note: When the browser is started with the ChromiumAPI, a ribbon shows up stating that the browser started in debug mode(highlighted below in blue). This message does not show up if the extension is installed via policy.

image
image1112×574 190 KB

Enjoy! And do not forget that feedback is always welcomed!

Looking to hear from you.

Gheorghe.

37 Likes
2
3

Looks cool. Thanks for the clear steps !

1 Like
7

Thanks for this update.

1 Like
8

Good Update, Waiting for chrome browser support.

1 Like
9

@gheorghestan

Sounds excellent Gheorghe.
:+1:

3 Likes
10

Sweet! I like Background processing enabled automations! :slight_smile:

What about Firefox? I’m asking because if we build Browser-neutral automations and all of the top 3 Browsers support Chromium API communication channel, then it’s easy. If not we have to make an exception with Firefox ( or any other Browser that isn’t supported) .

Is the Input mode list an Enumeration? I was thinking if there is a way to pass a parameter into it so that I can set it back to the regular method of communication if a Browser doesn’t support Chromium API.

thanks!

1 Like
11

Hardware event will work in the background. Just wow :smiley:

12

@AndyMenon Firefox itself does not support ChromiumAPI. But if they’ll add support in the future, we will enhance the UI automation input for FF as well.

1 Like
13

looking good @gheorghestan, thank you for this!

14

Thanks @gheorghestan .:slightly_smiling_face:

15

@gheorghestan While using Chromium API Input option with Edge version 92 I am getting error ‘Edge Chromium API’ should only be used with Edge Chromium browsers.
Edge version 92 is chromium based. Does anyone know why it could be coming?
Thanks in advance!

16

Hi @ritikanand ,

That is a little bit strange. Can you share a sample of your workflow file where you meet this error? Also, it would be useful to know if the Edge extension is enabled while you get the error and which version of UiAutomation activities package you are using.

Thanks

17

Did the use of ChromiumAPI input mode break with one of the latest releases of Chrome (haven’t tested Edge)? It works fine for me on Chrome Version 100.0.4896.127 (Official Build) (64-bit) which is what is installed on my dev machines but I get the error “Type Into ‘Welcome to’: Click with “Chromium API” in Version 101.0.4951.54 (Official Build) (64-bit) and similar of text input mode.

18

Hi @Justin_Lynch , just tested on my machine with the same Chrome version and it works. Can you provide more details on your error? Some screenshots would help.

19

Hi @vlad.coteanu,

I have a few in this thread. Thanks.

20

Ran into a problem where ChromiumAPI wasn’t actually performing a click, while Simulate worked like a charm. looks like depending on your selectors etc. there might still be issues with this mode.

Any ideas how one can easily change input modes in all activities from one value to the other? I would hate to manually switch hundreds activities from Chromium to Simulate mode.

1 Like
21

I am constantly changing my Use App/Browser and Click activities because they default to the Chromium API - which never works for me. It always tells me “click for this object not supported by Chromium API” or whatever.

Hasn’t been of any use to me. Simulate and Hardware Events work fine.

22

The Click, Type Into, etc activities inherit from the scope (ie Use App/Browser, etc)

23

Not really an answer. Question was on how to change individual click, type into, etc from Chromium to anything else quickly for 100+ activities.

Hint for those in need: you can open xaml file in notepad and do search and replace. First make sure at least one element is with “Simulate” mode. Then search for it in notepad, you will see attribute that you will need to change. Deleting this will default activity to use browser/app.

Funny thing is that when activity has mode set to “Chromium” in notepad it says “Debug*”