Yeah so I know this is long but…
I’ve built a bot that automates a desktop app that is very unpredictable for certain elements. No matter how dynamic I make them (wildcards, anchors, regex, fuzzy match, passing variables, etc) I cannot get the bot to work reliably. The annoying part is that it will work in one instance then the next minute it won’t. Click image is not an option because the image is never the same twice.
It seems the bot would have to switch between SendWindowsMessages, SimulateClick and neither to work. Rather than having to venture down the nested try catch block rabbit hole I figured a better alternative would be to use the “Get OCR Text” Activity to get the coordinates (aka the wordinfo) of the text I want clicked on screen. The only OCR method that seems to work is OmniPageOCR.
I can obtain the Left, Top, Right, and Bottom coordinates in the Get OCR Text activity, which is exactly pointless because you can’t use variables inside Click Activty’s ClippingRegion field. When I create a variable in the Result field under output (OmniPageOCR) it creates a variable of type:
I can iterate over this variable to locate the string in question but THEN I have to use regex to get the X and Y coordinates (I don’t retrieve height or width because neither can be used inside click activity). When I put these inside the CursorPosition section of Click Activity, the bot clicks very close to where I want it to… but I have to go a step FURTHER and add static numbers to the X and Y values to get it to click the right spot:
OffsetX: cint(locX.ToString) + 15
OffsetY: cint(locY.ToString) + 15
There has got to be a less insane way to accomplish this without having to add static numbers to the Offsets no?