Using Get OCR Text to Obtain Text Coordinates for Use in Click Activity

Help Studio
, ,
1

Yeah so I know this is long but…

I’ve built a bot that automates a desktop app that is very unpredictable for certain elements. No matter how dynamic I make them (wildcards, anchors, regex, fuzzy match, passing variables, etc) I cannot get the bot to work reliably. The annoying part is that it will work in one instance then the next minute it won’t. Click image is not an option because the image is never the same twice.

It seems the bot would have to switch between SendWindowsMessages, SimulateClick and neither to work. Rather than having to venture down the nested try catch block rabbit hole I figured a better alternative would be to use the “Get OCR Text” Activity to get the coordinates (aka the wordinfo) of the text I want clicked on screen. The only OCR method that seems to work is OmniPageOCR.

I can obtain the Left, Top, Right, and Bottom coordinates in the Get OCR Text activity, which is exactly pointless because you can’t use variables inside Click Activty’s ClippingRegion field. When I create a variable in the Result field under output (OmniPageOCR) it creates a variable of type:

System.Collections.Generic.IEnumerable<System.Collections.Generic.KeyvaluePair<System.Drawing.Rectangle, System.String>>

I can iterate over this variable to locate the string in question but THEN I have to use regex to get the X and Y coordinates (I don’t retrieve height or width because neither can be used inside click activity). When I put these inside the CursorPosition section of Click Activity, the bot clicks very close to where I want it to… but I have to go a step FURTHER and add static numbers to the X and Y values to get it to click the right spot:

OffsetX: cint(locX.ToString) + 15
OffsetY: cint(locY.ToString) + 15

There has got to be a less insane way to accomplish this without having to add static numbers to the Offsets no?

2

Did you try the Click OCR text activity?

3

Hi

Did we try with COMPUTER vision activities

https://docs.uipath.com/activities/docs/computer-vision-cv

Cheers @tSingularity20XX

4

Hi Charbel. I tried using Click OCR Text first. What is strange is that the rectangle/cell around the element highlights, so it is clear that the activity is identifying the right element. Unfortunately it times out after the default 30 seconds and I’ve tried this with and without SendWindowsMessages to no avail. Is an API key/Endpoint needed for this activity to work? Not sure what the issue is.

5

Hi Palaniyappan. Unfortunately this is one of the few features we don’t have enabled yet as it is not yet setup on our server.

6

Are you sure that the windows you want to click is ready? Maybe it is clicking before the button is ready.

Try adding a 3-5 seconds delay if this is the case.

If the click activity is working, this one should work as well…