Thomson Reuters OneSourceTax automation / Image Based Recognition

Hi everyone, hoping I’m not the only one trying to automate this online tool and someone can offer some help.

We use an older version of the tool, where there is no selectors I’m able to find, everything appears to be set behind a window, so I have sadly opted (after countless house trying everything else) to use image based recognition to hit the Run and Save buttons on the reports. However, I’d say the accuracy on this is around 20% at best, for whatever reason. Has anyone found a better or more efficient way to do this?

Used to just click the button but that wasn’t great as it usually clicked when it was greyed out before it had loaded, so now I get UiPath to hover over it, as it changes colour when you do, and then wait for it to be orange before clicking it, but when you try and do that it often seems to unhover/switch off orange, and therefore not click either. There are no keyboard shortcuts, the section in the middle is in a frame so contains no code I can see to use when opening page in dev tools etc.

TLDR: need to click a button on a website, using image based as it’s not selectable otherwise, button takes a variable amount of time to load, changes colour on hover, still won’t click even when trying hover, wait for colour change, click

Anyone have any ideas? I guess last resort is just wait a minute or two as it should load by then, then click on it?

Many thanks

Have you looked at using Computer Vision?

It is much more intelligent way of interacting with apps using OCR

Thanks, something we might expand to in the future, right now however this isn’t a tool we’re using. Does look like a solid potential for future automation though!

If only image based works, have you tried something along “on image appear” to see when the button turns orange?

So what I’ve tried is:
Find element on white-backed button
When found, hover over
Look for orange-backed button (only orange on hover)
Click green centre of button

I originally had it clicking the orange-backed button, but for whatever reason sometimes it was no longer able to recognise it as when it tried to click the button would go back to being white-backed just before, so then it couldn’t see what it was meant to click.

image
For reference

So it recognises the colour change to orange at least through the image appear.

So maybe you can have click text at that point as a follow up activity rather than the click image?

Oooh, clicking the text for “Run” might be a good idea, thanks, I’ll try that!

1 Like