ClippingRegion - browser vs window

Help
, ,
1

Hi,

In one of the projects I’ve needed to do OCR inside browser (it’s an embedded RDP inside of it, I know that OCR is not recommended for native apps).
While pulling my hair out why the clipping regions were acting like they weren’t being taken into account at all, found out that when inside AttachBrowser, SetClippingRegion doesn’t do anything (at least that’s what my tests have shown).

After changing just AttachBrowser to AttachWindow, everything else being an exact copy-paste, now it works.

It would be nice to have this either confirmed as working as intended, or investigated as a bug.

Regards.

2 Likes
2

I encountered the same issue when I did a scrape relative using attach browser. Only Attach window seems to support clipping region.

2 Likes
3

Thank you for confirming.

Is this documented somewhere (couldn’t find anything on clipping region TBH) or would it be possible to know where the clipping region context does work?
I was thinking of doing a workaround with ElementScope if needed, but time is always short and blind checking options is not the most fun thing to do.

Regards.

1 Like
4

Looks like a bug, @adrian will confirm.

5

Hi andrzej.kniola

I will check this issue. I think clipping region was not added because one doesn’t usually need clipping region inside a browser. Could you please provide more info about your OCR scenario; I want to better understand your particular case.

If you go over the RDP area when using the recorder / UiExplorer what selectors are generated?
Did you use the scraping wizard to generate the OCR scraping workflow or create it manually?

It would be great if you could share the OCR part of your workflow (attach window and scrape should be enough).

Thanks,
Adrian.

2 Likes
6

Hi @adrian,

Unfortunately I can’t share that xaml, but I’ll try to provide as much details as possible here. Since this is an IE hosted RDP, attach window works correctly even on browser level. I’d guess it’s not the case with Firefox/Chrome, but I don’t have that kind of app to test.

  1. UiExplorer generated selector for the RDP subwindow:
    <html htmlwindowname='{name}' title='{title}' /> <wnd cls='ICOWndClass' title='{title} /> <wnd cls='{appSpecificClassName}' title='{title} /> <wnd cls='CtxICADisp' />
  2. Application is hosted as an IE window.
  3. When checked with recorder for a simple ClickImage, it generated with AttachWindow.
    Attach Window selector: <html htmlwindowname='{name}' title='{title}' />
    ClickImage activity name: Click image 'ICOWndClass'
  4. When trying to manually use AttachWindow (with IndicateOnScreen), it only goes to html level. Using a selector generated with UiExplorer (as above) works though.
  5. Workflow was made manually.

Closest thing to describe the process would be an OCR-based VLookup on a table. It would probably be possible to build a repro using OCR and Excel, but to be honest it’s tiresome enough as it is :slight_smile:
Input: ValueToFindInColumn1 Output: ValueFromColumn4 _ Find top left AnchorPoint of table for row offset calculations; Find reference header positions for column offset calculations; Scrape Column7 until found text indicating correct section, scroll if needed; When in correct section, scrape Column1 for searched value (with vertical offset based on which row it begins with); Scrape and scroll until value is found or end of list; If value is found, scrape Column4 and return found value;
There are some additional parts for multiscale and common OCR mistakes tolerance, but that’s not relevant I think.

Offsets for column regions are similar to:
New UiPath.Core.Region(New Rectangle( _ tableAnchor.GetAbsolutePosition.Left - windowXOffset, _ tableAnchor.GetAbsolutePosition.Top - windowYOffset + {rowHeight}*{rowOffset}, _ {width}, {height} - {rowHeight}*{rowOffset} ))
So that it’s possible to read the whole column at once and then calculate which row was it.

I’ve tried with GetPosition at first, but couldn’t get consistent results when some windows were moved (especially when dev is on multimonitor setup, but prod will be single).

So to summarize:

  1. There was a need to read a Table from an RDP inside IE.
  2. Tried with AttachBrowser at first (out of habit mostly - rest of this application is a webapp), ClippingRegion didn’t work (it was scraping the whole IE window every time).
  3. Tried with AttachWindow - works for IE hosted RDP.

If you’ll need additional info for investigation/use case analysis, let me know, but like said unfortunately I can’t share this workflow.
I also agree that it’s not common to need clipping region with browsers, although with more and more SaaS applications with web interfaces it might be in the future (f.e. for reading scanned files opened in non-IE browser).

Regards.

3 Likes
7

Thanks for detailed information!

I have one more question, if you could test it: if you use the scraping OCR wizard on the RDP window what selector is generated? Is it or ?

I think it’s a good idea to add clipping region for browsers too; there are also windowless plugins where the clipping region could be useful.

Adrian.

1 Like
8

Whn using ScreenScraping wizard, it generates just the <html htmlwindowname='{name}' title='{title}' />, but the application itself is split to 9 different HTML containers, so I wouldn’t take this as a definitive test.

9

Hi,

I am trying to do a dynamic region of screen scrapping.

How can I parameterize the data of the region?

The Set clipping region no allow use variable.

I should put a code in my flowchart?

Thanks

10

Hi,

I’ve found it easiest to just make regions as variables and pass them as Region argument to SetClippingRegion activity (see my previous post here with New UiPath.Core.Region(New Rectangle( _ ).
That way you can control it with coordinates, either absolute or relative based on positions of other elements.

Regards.

4 Likes
11

Hi.,
My requirement is to set dyamic clippling region:

As per instruction i have added Set clipping Region activity and assign Uipath.Core.Region(New UiPath.Core.Region(New Rectangle(335, 246, 473, 283))).
But i am not sure how to map the new region to element,please check and give some suggestion to complete this task.

Thanks.,

12

Hi,
I have a question, anybody know how i could convert the UiPath.Core.UIElement into string?