Set Clipping Region; Input.Direction (Rectangle vs Translate vs Left, Right, Top, Bottom)

Regarding the UiPath.Core.Activities.SetClippingRegion:

I’ve been having a difficult time understanding exactly how the Set Clipping Region [Input.Direction] calculations are performed.

First question: is there a way to shade-in or highlight a Clipping Region on-screen so you can see a selected region for testing purposes?

I know I can use Recording>Screen Scraping>Scrape Relative to automatically populate the Find Image (Activities.WaitImageAppear), Translate Clipping Region (Activities.SetClippingRegion), GetOCRText, and Reset Clipping Region steps. But from what I’ve seen this always sets the [Input.Direction] to “Translate”. Yet there are also options for “Left”, “Top”, “Right”, “Bottom” and “Rectangle”?

I plotted a quick example image and I’m hoping someone can help identify how to fill in the missing coordinates for these other types of [Input.Directions] so this Rookie can better understand!

The attached image represents an area of 20 x 20 pixels. Please correct my logic if anything that follows is not accurate: coordinates should be plotted relative to the greenDot in the top-left; i.e. the starting point (x, y) = (0, 0) or (left, top, right, bottom) = (0, 0, 0, 0).

The blueRect consists of all positive integers (LEFT = 1 from the left, TOP = 1 from the top, RIGHT = 19 from the left, BOTTOM = 18 from the top) … (1, 1, 19, 18). Likewise: wholeScreen would be written as (0, 0, 20 20).

My goal is to capture the bottom right corner point of the blueRect (x, y) = (19, 18) which I believe could also be written as a rectangle, example: redPt = (19, 18, 19, 18). This would become my anchor point from which I could set relative coordinates for new rectangles, such as: yellowRect.

To help illustrate I have 2 columns: first is “Transform from prplRect” and second is “Transform from redPt” — with the intent of creating the yellowRect using Activities.SetClippingRegion.

a) Are my numbers correct for the “Translate” option from both the redPt (-5, -3, -2, -1) and the prplRect (-4, -2, -2, -1)? I believe all would be negative integers as I am going backwards on both the x and y axis in all cases.
b) How would this differ if I were to use the alternate options of “Rectangle”, “Left”, “Top”, “Right”, and “Bottom” for the [Input.Direction]?

The closest topic I could find to this was answered here:


But I posted questions to it because I was confused on the solution which uses [Input.Direction]=”Left”.

Any feedback is greatly appreciated!

6 Likes

I can briefly answer one thing on this.
To highlight your region you can use the Highlight activity.
Steps would look like this:

  1. Find Element to store to an element variable.
  2. Set Clipping Region using the element’s rectangle properties.
  3. Highlight with the element variable using desired color and length of time.
  4. Perform actions using the element variable in the element parameter.

I have a hard time figuring out the rectangle also, but have used trial and error (with the .Top .Bottom .Height and .Width I think) in the past to create my rectangle. Keep in mind that if you load process within an environment having a different resolution setting, your clipping region will change on you; additional handling on that may be necessary or just use the same environment settings all the time.

Regards.

1 Like

Great tip ClaytonM, I have been concerned about the resolution issue but had not delved into it yet. Your reminder led me to this post: Display resolution so I’ll probably play around with the Activities.InvokeWorkflowInteractive a little bit to see if I can account for this potential problem.

Also potentially a good resource on Resolution settings: https://www.uipath.com/kb-articles/resolution-considerents

Awesome about Highlight: that will help me greatly with testing! Also good to know about .Height and .Width

Cheers

@mr_steve did you find any information on this? I just started with the UiPath and stuck with Clipping Region. I am not sure how it works?

Here are my own notes regarding the Input Direction for Clipping Region.Hope it helps for anyone who are still confused like myself before this.

  • Coordinates for Clipping Region relative to current application window: left=x1, top=y1, right=x2, bottom=y2

  • Direction: RECTANGLE
    ** this is used to specify a new ClippingRegion by position relative to the origin(Top=0, Left=0) of current application window
    ** the new coordinates can be specify either with the Region or Size property

  • Direction: TRANSLATE
    ** this is used to update the current ClippingRegion by specifying the delta values and shift the current coordinates accordingly
    ** use Size property = (left, top, right, bottom)
    ** result --> x1=x1-left, y1=y1-top, x2=x2+right, y2=y2+bottom

  • Direction: LEFT|TOP|RIGHT|BOTTOM
    ** Updating current ClippingRegion by specifying the delta value for the specific data element
    ** use Size property = (left, top, right, bottom)
    ** data elements other than the one specified in the context are being ignored
    ** e.g. if Direction=LEFT, only the first value (i.e. left) will take effect, and the result will be x1=x1-left

6 Likes

:+1: Set Clipping Region is one of mysterious activities. I wonder why UiPath folks can’t provide such description in the activity documentation.

J0ska

5 Likes

It took me quite a while to figure this out. Basically, Translation is a really cool method in GUI automation but it’s not straight forward. Lack of documentation made it really a myth. I here by gives the asthmatically demonstration and example equation with which you can solve for any region you need. Enjoy.

Just to write down the derivative equations:
1- From known Origin Rect and Translating Rect’ we can obtain the Translated Rect " .
Rect= ( l,t,w,h) and Rect’=(l’.t’,w’,h’) => Rect" =(l",t",w",h") , where

         l" = l-l'                                          (1.1)
         t" = t-t'                                          (1.2)
        w" =l+w+l'+w'                               (1.3 ) 
         h" =t+h+t'+h'                               (1.4)

2- With given Origin Rect and the final Translated Rect", we can obtain the Translating Rect ’ :
Rect= ( l,t,w,h) and Rect" =(l",t",w",h") => Rect’=(l’.t’,w’,h’) , where

         l' = l-l''                                           (2.1)
         t' = t-t''                                            (2.2)
        w' = w"-w+l"                                    (2.3)
         h' =h"-h+t"                                      (2.4)

3- The classical euclidean translation is defined in Wikipedia as a geometrical transformation that moves every point of a figure or a space by the same distance in a given direction.( https://en.wikipedia.org/wiki/Translation_(geometry))
In this case, we don’t change the size of rectangle but rather move it paralleled: left ,right, up and down. Let’s call them “Movement”. So we got 2 new equations:
w"=w (3.1)
h"=h (3.2)
I noticed the left and right side ( samey for upper and bottom sides ) moves equal distance in the space. Then I’ve found 2 more equations

4- let’s note
horizontal Movement of left side dL
horizontal Movement of right side dR
horizontal Movement between Rect and Rect" dX
dX= dL=dR (4.1)
dL = l"-l (4.2)
dR = l"+w"-(l+w) (4.3)

5- now for the vertical Movement:
Vertical Movement of Upper side dU
Vertical Movement of Lower side dL
Vertical Movement between Rect and Rect" dV
dY= dU=dL (5.1)
dU = t"-t (5.2)
dR = t"+h"-(t+h) (5.3)

6-
Finally , synthesize respectively:
a) (1.1),(1.3) (4.1)(4.2)(4.3):
l’= -dX (6.a.1)
w’= 2 * dX (6.a.2)

b) (2.1),(2.3)(5.1)(5.2)(5.3)
t’=-dY (6.b.1)
h’=2 * dY (6.b.2)

Now it comes the most useful formula that I promise you won’t regret if you have spent so much time reading my article :
To move a UIPATH region ( element, search area …etc ) with horizontal and vertical distance dX ,dY , you only need to Set the Clipping Region as
Setting Clipping Region : ( -dX,-dY,2dX,2dY)

don’t forget to choose translation for your direction

1 Like