Cannot Click Element Found in Java JViewport

[grosner (Gary)]

With a Java SunAWTFrame app, the Click activity fails when clicking on element that is a node in a JTree. The JTree is contained in a JViewPort. The Click fails while attempting to click on a selector (element), if a portion (where selector can be seen) of the JViewPort is off-screen.

If the JViewPort is entirely on-screen, then the Click activity works as expected

Failure Scenario:

• Find Element successfully finds the Jtree node’s target and saves it to a variable (foundNode).
• Click (target = foundNode) activity FAILS with message:

Click: Cannot send input to UI element because it is outside of screen bounds.

Failure Reason(s):

• A portion of JViewPort and target element are located outside screen bounds.
• Or the window was resized and obscures the UI Element from the human eye.
• When the JViewPort and target element are both located inside screen (and window) bounds, then the click will work.

Notes:

• The application’s window was positioned so half of the window (and JViewPort) is offscreen (see image).

• In this Scenario, the UI element is always visible when the window is maximized.

• Using an Anchor does not help.

• Simulate Click does not work with this Java application

• The Select Item activity does successfully select an item in the JTree, even when not visible to the human eye.

• Using UiPath.UIAutomation.Activities 21.10.3

Workaround:
I can workaround the issue using either of these techniques:

1. Maximize the window:

   • Maximizing the window and ensure the entire JViewPort and target item is visible on screen.

2. Use the Select Item and Type Into Activities

   • Select Item target: <java cls=‘JTree’ role=‘tree’ />
   • Select Item To Select: String variable or the exact text in the JTree to select.

Requested Fix:

• Fix the Click activity so that If an element (i.e. selector) can be found (using Find Element), then the same selector it should be clickable, EVEN WHEN offscreen.

Summary:

If Find Element can successfully locate a selector, then Click should also work.
Alternatively, perhaps the UiPath.UIAutomationNext.Activities.NSelectItem (that, unlike click, does work to select an offscreen element) can include an option to automatically make uielement visible onscreen (either before or after the select activity.

[grosner (Gary)]

There are other Java situations where the above workarounds are not possible because the Java control does not permit any keyboard (type into) entry and the simulate click is unsupported.

In this case, the only thing the user can do is to mouse scroll until the Found element becomes visible, then click the the element.

But UiPath cannot easily do this like a human. This is because the Find Element looks past what is visible on screen and so there is no way to know how when the mouse scroll activity brings the element into the viewable area (where it can be clicked).

So, I reiterate that if a Find Element is successful, then so should the mouse click, even when the element is hidden down below, contained in the control (e.g. list) but outside the control’s visible area.

Note: The Click activity does try to click where it “thinks” the element was found, but because it clicks outside of the border where the control is found, the click occurs on the application window rather than inside the control.

If an element is found inside a control, and the mouse is instructed to click that element, the mouse should NEVER click outside of the control, else unexpected things could occur!!

[vlad.coteanu]

Hi @grosner ,

FindElement activity is designed to find both elements that are visible and those that are not visible. This is because some input methods should work in the background (For example, click with Simulate, where it is possible, will work if the element is not in view ).
In your case, scrolling is the only way to get the element in the view and you can try to rely on the “javastate”, “visibility” or “visible” properties to determine whether an element is in the viewable area or not.

[grosner (Gary)]

Perhaps UiPath’s Find Element should have a check box to only find visible elements. That would save a lot of extra programming to get the attributes, etc.

Scrolling, while possible is a slow and tedious approach. Adding the ability for the Click activity to bring the element into the viewport (automatically scroll) would save programming time.

Lastly, perhaps someone can shed some light on why some java elements can accept a simulate click and others cannot.

[vlad.coteanu]

@gheorghestan , FYI:
Perhaps UiPath’s Find Element should have a check box to only find visible elements. That would save a lot of extra programming to get the attributes, etc.

Modern activities contain a Mouse Scroll activity, that is not currently working for Java elements. However, we will research and proceed in enabling this activity also on scrollable Java elements.

To understand better why some controls can receive input with Simulate and why some controls can’t do that, we need to take a look at some Java interfaces. We use the Accessibility features to enable that behavior. Let’s take a look at this JButtonSource code: openjdk-jdk8u/JButton.java at master · AdoptOpenJDK/openjdk-jdk8u · GitHub.

public class JButton extends AbstractButton implements Accessible

So the JButton implements that Accessible interface, whose main role is to return an AccessibleContext. The accessible context it returns is an AccessibleJButton, which extends the AccessibleAbstractButton class.

public AccessibleContext getAccessibleContext() {
        if (accessibleContext == null) {
            accessibleContext = new AccessibleJButton();
        }
        return accessibleContext;
    }

Now, to support Click with Simulate, an element on the screen must expose an accessible context class that implements the AccessibleAction interface, which, in our case, is implemented in the AccessibleAbstractButton class.

protected abstract class AccessibleAbstractButton
        extends AccessibleJComponent implements AccessibleAction,
        AccessibleValue, AccessibleText, AccessibleExtendedComponent

The implication is not necessarily the other way around since not every class that supports AccessibleAction is exposing a way of Clicking through that interface. I am saying this because AccessibleJTreeNode class implements the AccessibleAction interface but is exposing something else through that, which is not the similar behavior as clicking.

[gheorghestan]

@grosner

Hi Gary,

Perhaps UiPath’s Find Element should have a check box to only find visible elements. That would save a lot of extra programming to get the attributes, etc.

The option in already available in the Modern selection screen:

`

Scrolling, while possible is a slow and tedious approach. Adding the ability for the Click activity to bring the element into the viewport (automatically scroll) would save programming time.

The MouseScroll activity released in 20.10 will be enhanced in the near future to work for Java apps as well.

Lastly, perhaps someone can shed some light on why some java elements can accept a simulate click and others cannot.

Explained by @vlad.conteanu above.

Best regards,
Gheorghe

`

[grosner (Gary)]

@gheorghestan

The option in already available in the Modern selection screen:

I believe the image in your post is from to the modern UI Click activity. The option to Enforce element visibility does not appear to make any change to the selector. The UiPath documentation does not (IMHO) fully explain the purpose for this option (Enforce element visibility),

As previously mentioned, my target selector is a variable (foundElement) which is the output from the UiPath.Core.Activities.WaitUiElementAppear activity. This activity has no apparent way to restrict the find to “visible” elements. By “visible” I mean something I can see on the screen.

the WaitUiElementAppear activity successfully finds the element when the element is contained anywhere in the “tree” (see yellow outline on image). What I require is restrict the UiPath.Core.Activities.WaitUiElementAppear to look only inside the ‘viewport’, rather than the entire tree. Perhaps someone can let me know how to accomplish this.

In short:
How can I restrict the WaitUiElementAppear to look only inside the ‘viewport’ instead of in the entire ‘tree’

[Cosin (Cosmin Voicu)]

@grosner

The option to Enforce element visibility does not appear to make any change to the selector.

Yes that’s correct, the visibility checks are enforced separately, and in 22.4 we’ll expose visibility and “existance” separately. But could you try it out anyway to see if it works?
TBH I kinda expect it not to, because we perform a similar check, but is worth a try.