Studio CE 2022.4.3 - Type Into Activity - SimulateType and SendWindowMessages are buggy

Hello,

The Type Into activity is buggy in the latest stable version of UiPath Studio. This is kind of awkward to say, but it does not work correctly with the UiPath UiBank application. The problem is largely with automating the input text field as shown on this page:

Issues with Simulate Click:
When enabled, the automation types in wrong values into the Amount column
Example(s):

  • If the input value is 22.45 it types in only 45
  • If the input value is 10.11, it types in 11
  • If the input value in 10, it types 10

Issues with SendWindowMessages

  • When enabled the automation detects the Amount field and types in the correct values
  • In other words, it works correctly when the screen in unlocked
  • However, when the screen is locked, the automation fails to detect the very same Amount field and stops with error

What solutions have been tried thus far:

  • System has been restarted
  • Browser Extension has been disabled/enabled
  • The “Amount” Label field as the Anchor tag was put in place in order to detect the Amount input field
  • Likewise, Context Aware Anchor tag was used to no effect
  • All packages updated to the latest version to no consequence

Using Inject JS:

  • Inject JS was used to insert the value into the Amount field
  • It works behind locked screen as well
  • However, that value is not picked up by the next step
  • Therefore, this solution cannot be used even if it works

Version Information

Studio CE Version:

UiPath Browser Extension for Chrome:

Version
22.4.8

Google Chrome Browser Version:
Version 102.0.5005.63 (Official Build) (64-bit)

Package Stack:
“UiPath.Credentials.Activities”: “2.0.0”,
“UiPath.Database.Activities”: “1.6.1”,
“UiPath.Excel.Activities”: “[2.12.3]”,
“UiPath.Mail.Activities”: “[1.15.1]”,
“UiPath.System.Activities”: “[22.4.1]”,
“UiPath.UIAutomation.Activities”: “[22.4.5]”

cc: @loginerror

Is that a service mode installation with an Un-Attended bot license

No, this is a normal installation.
I cannot use Robot Accounts as this automation uses Integration Service and cannot work with Robot accounts.

The automation is deployed to a modern folder with an automation cloud user account assigned to it.

This account is assigned to a client machine via machine credentials and 1 Unattended robot + runtime to the Directly Assigned User Template.

Thanks

1 Like

Hey @AndyMenon

Yes thats fine as you said you have an Un-Attended configured for the cloud user account.

Now I just want you to make sure, the robot installed in the server machine is of service mode ?

Thanks
#nK

No, this is still working on a Dev machine. It is not installed in service mode.

1 Like

Hey @AndyMenon

Are you triggering that in Un-Attended mode ?

Thanks
#nK

The job is started by an Integration Service Trigger.

Just to clarify:

The entire system works well and as expected. The Integration Service triggers the job flawlessly each time. As long as the screen is open, the execution is accurate. I’ve conducted more than 28 tests, and it has worked in every instance in which the screen is unlocked.

And to be specific, all other controls work flawlessly behind the locked screen.

For instance , the automation opens the UiBank site, logs in, navigates to the money transfer page, and selects the from/to accounts accurately.

It's just the amount field that does not work behind the locked screen.

I don’t know, but I’m guessing that the complicated nature of that text box may be causing this.

It has a couple of up and down arrows and when those arrows are clicked, the numbers increase or decrease in whole increments/decrements. If the number has decimal points, we have to manually type it in.

`Whilst I understand that this input field may have been created for learning experience, I am not sure which bank in the real world has this kind of amount input field.

Therefore in my honest opinion, this text box many not be in the right place. If there is a UI application that must run behind locked screens, it’s a banking application. I’ll be hard pressed to run an application such as this one in full view of an open screen!`

Thanks

1 Like

Okay clear :+1::slightly_smiling_face:

But still on the suspects of the mode.

Sorry, suspicion of which mode?

Hi,

In my environment (Studio 21.10.6 + UiAutomation 21.10.5 + Chrome browser), this matter can be reproduced with SimulateType. (SendWindowMessages seems good)

I guess it causes by SimulateType improperly handles type attribute which value is “number” in Input element.

If value of the type attribute is modified to “text”, it works.

Unfortunately,even if we add step attribute (ex. step='any'), it doesn’t resolve it.
I also confirm same issue occurs in other web site (however it seems all the site which has number attribute does not occur this matter)

As an aside, if we use firefox, the input value became “2245” (lack of decimal point).

I think this matter should be raised as a bug.

Hope this helps you.

Regards,

1 Like

Thank you @Yoichi . Please do vote this post up as it is a Bug report.

-Andy

Strange bug. I pointed it out to our team.

In the meantime, maybe that property could help?
image

Or, depending on your locale, maybe a comma instead of a dot for the decimal (a shot in the dark here)?

I will try this option out as I’m working on a more advanced scenario with Integration Service with Outlook 365.

To your point, a comma is typically used to separate large amounts . Example: $1,235,456.77

If the locale used commas in place of a decimal, I’m wondering how would they differentiate between a decimal and a comma in the above example.

I’m still curious on why that Amount field has been set that way? Is it a convenience of the style sheets used to build the app? Or is there a specific learning goal? I’m asking because I see a similar text box in the Loan application section of the UiBank website.

Even then, it still does not compute why a Banking Application would have such a field unless that page runs from a kiosk or an ATM, where it is convenient to click up and down arrows to increment the deposit/withdrawal amount in whole numbers. If ATMs restrict withdrawals to whole number amounts, they must still allow decimal deposits.

Thanks for moving this along @loginerror ! :slight_smile: :handshake:

1 Like

@AndyMenon Thank you for reporting this issue.

Indeed, Type Into using the Simulate input method does not work for that particular input field.
It looks like the input field does some custom validation logic and it reports these errors when the Simulate input method is used:
image
This is not something that we can fix - it’s just the way this input field behaves.

Luckily, all the other input methods work correctly for this field:
ChromiumAPI, Window Messages or Hardware Events.

But only the ChromiumAPI input method works if Windows session is locked:
This is also the recommend input method for browsers.

Thanks @Marius_Stan.
Will give this a try and update.

Andy

Hi @Marius_Stan ,

This does not work consistently. I’ve performed at least a dozen tests in Studio after having switched to Modern design. It works only the first time behind locked screens, but fails to do so when run repeatedly.

This is the error I keep getting frequently:

Error reported was: Could not find the user-interface (UI) element for this action. Possible solutions:
 •  Ensure application is opened and the UI element is visible on the screen at execution time
 •  Edit the Target of the UI activity and use Validation to debug the issue. 
 •  If needed, re-indicate the element as its properties might have changed
 •  Use "Check state" activity to check the application state before executing the action
 •  Increase the "Delay before" value to allow time to the application to render entirely and become responsive

I’ve tried all options of testing and revalidated the element selections several times, in addition to using up to 2 seconds as the “Delay Before” time.

Regardless, it does not work with repeat testing behind locked screens.

FYI,
Andy

Hello @AndyMenon !

I apologize for replying so late.
They key thing that I forgot to mention is that you must also start Chrome using the “Use Browser” activity.
This is required to add extra command line arguments to Chrome that make it run in background.

Please try the following:

  1. Close all your running Chrome instances.
  2. Run your workflow with UiAutomation v22.4 that
    2.1 Uses the “Use Browser” activity to start Chrome
    2.2 Uses the ChromiumAPI input method
  3. Lock your screen
  4. The Robot should now be able to automate Chrome behind the lock screen.

Hi @Marius_Stan ,

I will look into it later today. I have performed all actions inside the Use Application/Browser Modern activity.

Are we talking about the same thing? Or am I missing the “Use Browser” activity specifically?

Thanks for your persisted attention to this issue.
Andy

Hey @AndyMenon !

Yes, we are talking about the same thing: the “Use Application/Browser” Modern activity.
Please let me know if you get this working or not.

Regards,
Marius

Right. All Browser actions are within this scope.

I still cannot get it working behind locked screen.

Thanks.