Seletors in modern activities

Hi everyone,

@postwick is the winner here in the guessing game. Sorry for all this confusion, I’ll work directly with the documentation team to update the erroneous docs page. Paul is 90% right about the 23.4 behavior: we first try to find the target using the strict selector, then we also enable the fuzzy selector (so we now have both strict & fuzzy searching for a target); if they still both fail, we enable both CV & Image at the same time.

The 23.4 approach has a number of issues:

  • it can result in an Image target even if Image sometimes can hallucinate
  • CV is a powerful fallback, but it hides issues with selectors - there’s no feedback something’s wrong with them, which is not ideal; ideally, we recommend Unified Target to be fully loaded

This is why we’ve implemented a number of significant changes in our 23.10 GA package:

  • there is a new clear hierarchical targeting system, based on each targeting method’s relevancy
  • there is a new setting called “Wait for primary targeting method until timeout” (enabled by default for new projects in Studio >=23.10)
    • selectors (both strict and fuzzy) are the primary targeting methods
    • secondary is CV and Image is tertiary (so it’s de-prioritized vs. CV)
  • the output of an inferior targeting method in this hierarchy will only be used in case the upper targeting methods are failing
  • Image is disabled by default in Project Settings > UI Automation Modern, since it’s the weakest targeting method available (uses direct pixel values) + we have CV enabled by default, which is a far superior targeting method in terms of accuracy
  • we now output warnings in case CV or Image are able to target an element, but the selector are not + we output some recommendations for the selectors possible new values (see attached pdf: Unified Target.pdf [191.7 KB])

Please watch this demo video - it should be more clear vs. a wall of text :slight_smile:
And let us know what you think. How do you find these changes?

2 Likes