"Find OCR Text " output element is same as selector

I have a page where I need to search for some text and grab the uielement. When I use the uiexplorer and click on the text, I get this:

<html title='PD4 - Indirect Purchasing' />
<webctrl id='normalPriorityGrid' tag='TABLE' />
<webctrl idx='1' isleaf='1' tableCol='9' tablerow=’5’ tag='TD' />

Since the text I am searching for can be anywhere in the table, I tried the “Find OCR Text” activity. I used

<html title='PD4 - Indirect Purchasing' />
<webctrl id='normalPriorityGrid' tag='TABLE' />

for the selector attribute. When I run the robot, two strange things happen. The first is that it won’t find the text unless it is visible on the screen (if you have to scroll down for the text, it can’t find it). The second thing is that it returns the selector above for the output UIElement. This seems useless, so I must be missing something. I get similar results when I try “Find Text Position.” I want the tablerow of the element because I actually need to select an icon in that row.

What am I missing? How can I get the element returned by UIExplorer?

Thanks!

You’re using OCR so the robot has to be able to “See” what it is scraping. That’s how OCR works - it’s Optical Character Recognition.

You’re telling the Get OCR Text activity to look at the full table so it’s returning the full table as the output UiElement. It’s not returning the table row/column because it doesn’t know the table row/column.

You could use the Find Text Position or the WordsInfo parameter of the Get OCR Text activity to get the coordinates of your word and scrape or click relative from there.

Otherwise you could manually search each row in the table. Let’s say your data is in column 3 and you’ve got 100 rows of information. You could do a For Each with Enumerable.Range(0,100). Then use a try/catch to scrape item

<html title='PD4 - Indirect Purchasing' />
<webctrl id='normalPriorityGrid' tag='TABLE' />
<webctrl idx='1' isleaf='1' tableCol='3' tablerow=’" + item + "’ tag='TD' />

The item variable will increment every time so you can check each row one by one.

2 Likes

Hi Daniel. Thanks for the quick response.

You’re telling the Get OCR Text activity to look at the full table so it’s returning the full table as the output UiElement. It’s not returning the table row/column because it doesn’t know the table row/column.

Yes, that was what I assumed, but a part of me was hoping UIPath did something automatic in the background because I couldn’t imagine everybody using a scroll down activity and rescanning each time until the text was found. Some web pages with data could be dozens of clicks to scroll down. But it sounds like I would have to do that in order to scan the entire page.

You could use the Find Text Position or the WordsInfo parameter of the Get OCR Text activity to get the coordinates of your word and scrape or click relative from there.

I tried the Find Text Position and it did the exact same thing. It just returns the selector without the tablerow node. I assumed these activities would return what the UIExplorer returned when I clicked on the text. When I tried the WordsInfo it returned a huge string that oddly enough, did not contain the text I was searching for (despite it being visible on the screen). It has other numbers in there, but not the one I was looking for. Could there be a size limit on WordsInfo?

I may have to resort to your last suggestion and enumerate through the rows, although I am not sure how to get the total rows at this point, other than pulling it in from the DB.

Thanks again!

UiExplorer gives a selector. WordsInfo gives a region defined by an x-coordinate, a y-coordinate, a height, and a width. This region is defined per word. You should be able to do the OCR w/ WordsInfo, loop through WordsInfo, and if item.Text matches your search string then you can set the clipping region to the text location (plus or minus whatever offset you need to get to the icon).

If the Get OCR Text activity isn’t returning your word then I’d try different engines/scaling/etc to try and help it read the table a little bit better than it currently is. If you run the screen scraping wizard on your table does it show you the word that you’re looking for?

I ran the screen scraper wizard and selected the WordsInfo option and it did return the text I wanted followed by some coordinates:

0001111122 (821, 3264, 891, 3274)

So I guess I could use these to find the icon that I want to click? The icon is a few columns over. If there is an easy way to do this, I think I would still need to run that scraper in a loop as I scroll down the page, right?

Thanks!

So after you do your scraping you can use a For Each with the TypeArgument parameter set to UiPath.Core.TextInfo. Use your wordsInfo variable as the collection to iterate through.

Then, inside the loop you can do an If with item.Text = "0001111122" (or whatever your variable is). Use a Set Clipping Region activity. Under the Size parameter you can specify the left, right, top, and bottom of the region. Transform the scraped region values to where they need to be (this will require some trial and error). The word region data can be found with:

item.Region.Rectangle.Value.Left
item.Region.Rectangle.Value.Right
item.Region.Rectangle.Value.Top
item.Region.Rectangle.Value.Bottom

For example if the region you need to go to is 30 pixels left of the word and is only 15 pixels wide then you could do:

item.Region.Rectangle.Value.Left-30  //for the left bound
item.Region.Rectangle.Value.Left-15 //for the right bound
item.Region.Rectangle.Value.Top //keep top and bottom the same
item.Region.Rectangle.Value.Bottom

I’ve never used the Set Clipping Region activity before. Are you saying that I should plug different numbers into the size parameters until I find the region with the icon? Then I could see what the difference is in coordinates and set the region by subtracting the differences? This sounds like a good idea, I’m just a little fuzzy on the size parameters and the region parameter.

Thanks!

Yes, you’ll need to figure out how many pixels are in between your word and the icon you need. Then you can set the clipping region to be the same region as your word, but offset by the x and y values needed to get to the icon.

Ok, let me play around with that and see what happens. I’m getting ready to leave for the day, but hopefully I can make some progress and post an update here tomorrow. I appreciate your help on this.

Thanks!

2 Likes

One more quick question. When I plug in some size coordinates, say (700, 3264, 891, 3274), what activity do I need to find out what is in that region?

It depends what you want to do in that region. If you want to click use a Click activity and use the ClippingRegion parameter to tell the bot where to click. If you want to get text you can do Get OCR Text and do the same thing.

After you do that you should use a Set Clipping Region activity with (0,0,0,0) to reset the clipping region for next time.

2 Likes

Hi Daniel. I’m feeling a bit obtuse on this clipping region activity. I set up my workflow to scrape the screen and then I loop through the WordsInfo that is returned to find the coordinates of the text I am looking for. It returns what I expected (821, 3264, 891, 3274). I then used a click activity and set the clipping region to (700, 3264, 891, 3274). That didn’t work so I thought that maybe I should try to set up a loop that alters the numbers until it clicks on what I want. But I couldn’t put in a variable for the clipping region in the click activity parameters. Is there a way to do this? It seems to just accept numbers. Then I decided to find the coordinates using the click activity “Indicate on screen” so that I could at least get an idea of the offset. When I did this, the coordinates in the preview were -888x-874 and -693x-681. This isn’t even close to the region numbers returned in my loop of the WordsInfo. If I put those numbers into the clipping region of the click activity it does what I want. So I am a bit confused on 1. How to use variables in the clipping region parameters and 2. Why the numbers in the WordsInfo are so different than those from the “Indicate on screen” values.

I appreciate your input. I’m sure it’s something I’ve overlooked. In searching for info on the clipping region, I didn’t see anybody using variables for the parameters, so perhaps I am completely going down the wrong path here.

Thanks!