UIAutomation Semantic Fill Form Activity

Introducing Fill Form, the first addition to our Semantic Activities lineup in the 23.12 Preview release! :rocket:

Fill Form is our first dive into Semantic Activities, where AI meets automation magic to streamline your workflow like never before. Say goodbye to endless Type Into activities and hello to effortless form filling!


With Fill Form, simply indicate your Data Source, and watch the magic happen: the activity automatically detects the destination form from the UIA Scope, making automation a breeze.

Whether you’re dealing with structured data objects: Document Understanding DocumentData, Dictionaries, Data Entity Record, JSON files or Datatables or unstructured text (e.g.: the body of an email or any other string), Fill Form handles it all with ease. Plus, its semantic matching ensures your automation stays resilient to UI changes as long as the semantic meaning of the text labels stay the same – no manual intervention required.

The magic doesn’t stop here: no specific element targeting input is required! No selectors or CV descriptors are needed. Everything is covered by the proprietary DOM analysis engine.

Here’s the kicker: the data schema is automatically extracted at run-time and AI semantically matched with the destination form fields, simplifying your automation setup even further.

Currently available for web applications in cross-platform projects, we’re already eyeing its inclusion in Windows projects as well.

Now it’s your turn! Dive into Fill Form, explore its capabilities, watch some video demos (here and here) and share your feedback. We’re eager to hear your thoughts, use-cases, and any suggestions you might have. Happy automation, let’s embark on this new journey together!

20 Likes

Amazing activity, just a question what if the activity doesn’t identify a property or find more than one match for a particular attribute/property?

2 Likes

Interesting :star_struck: Eager to understand how much time it will take AI to align data against the fields. Also, will there be configuration option to process whatever fields available and skin unavailable one.

Thanks,
Ashok :slight_smile:

1 Like

This looks exciting! Im particularly interested on the DOM analysis engine, without having any selectors and how it works. How much flexibility do we have for any corrections, if required? Does the model/engine get fine tuned as part of the process?

2 Likes

By “properties”/“attribute/property” I’m guessing you’re referring to input/output fields in the source/target form (since selectors are not involved here, there’s no selector attributes either).

If my guess is right, then the answer is this: the activity will just fill-in whatever it is able to extract from the source and whatever it is able to find in the target form. If the source is missing some of the fields in the target form, the activity will just skip them; if the source has some extra fields that are not in the target form as well, it will again ignore them. Feel free to try it out for yourself :wink:

Let me know if I answered your question.

1 Like

Processing time doesn’t seem to be a big issue, from our tests (but please tell us if you encounter such issues).
We wanted to first release the simplest, most magical AI powered version of this activity and then see what advanced options would be needed; so no configuration option to edit source/target fields for now (it’s done automatically - please see my previous reply above + give it a go :wink: )

1 Like

It’s the same engine we developed to support Attended scenarios (such as Clipboard.AI); we do a proprietary AI powered DOM filtering so we can build a reliable schema. It’s also used in Capture All for Object Repository, enabling (finally!) full scrollable content capture for OR - limited to web content for now. It’s a pretty cool technology for getting a complete UI snapshot of the application. Although there are no selectors directly involved, we could also link this new Semantic targeting to selectors & CV descriptors as well in the future - maybe even integrate this in Unified Target; what do you think? :wink:

2 Likes

@Bogdan_Sultana,

Thank you for the inputs and setting fair expectation from the activity. Surly I will be going to try it.

Thanks,
Ashok :slight_smile:

1 Like

We deliberately skipped any corrections for now: we wanted the cleanest, simplest experience for form-filling; flexing our muscles in terms of everything we know about UI element extraction, targeting and execution so that users can just drag&drop such an activity and forget about tedious selector tweaking or configuring activities.

But we do plan to add some relevant advanced editing in the future - we want to see how the simple version is doing first :slight_smile:

One possible idea is to enable you to generate a list of “classic” (well, Modern, not Classic, but compared to Semantic, I need to call them “classic” :stuck_out_tongue: ) UIA activities using AI, reusing our currently in Public Preview Autopilot technology. This would be an alternative way to do form-filling that provides more control.

2 Likes

thanks, this makes sense i.e. consolidation of different targetting methods.

1 Like

Hello community,( @Bogdan_Sultana ), I am going to test the new version… I hope the improvement will be amazing. Regards :star_struck:

Is this activity only for cross-platform projects? I am trying to find it in a Windows project, but no results.

Yes, only for cross-platform for now :point_up::wink:

Really enjoying the learning experience

1 Like