UIAutomation Semantic Fill Form Activity

News Product News
,
1

Introducing Fill Form, the first addition to our Semantic Activities lineup in the 23.12 Preview release! :rocket:

Fill Form is our first dive into Semantic Activities, where AI meets automation magic to streamline your workflow like never before. Say goodbye to endless Type Into activities and hello to effortless form filling!

image
image904Ă—458 89 KB

image
image904Ă—256 41.3 KB

With Fill Form, simply indicate your Data Source, and watch the magic happen: the activity automatically detects the destination form from the UIA Scope, making automation a breeze.

Whether you’re dealing with structured data objects: Document Understanding DocumentData, Dictionaries, Data Entity Record, JSON files or Datatables or unstructured text (e.g.: the body of an email or any other string), Fill Form handles it all with ease. Plus, its semantic matching ensures your automation stays resilient to UI changes as long as the semantic meaning of the text labels stay the same – no manual intervention required.

The magic doesn’t stop here: no specific element targeting input is required! No selectors or CV descriptors are needed. Everything is covered by the proprietary DOM analysis engine.

image
image904Ă—442 89.4 KB

Here’s the kicker: the data schema is automatically extracted at run-time and AI semantically matched with the destination form fields, simplifying your automation setup even further.

Currently available for web applications in cross-platform projects, we’re already eyeing its inclusion in Windows projects as well.

Now it’s your turn! Dive into Fill Form, explore its capabilities, watch some video demos (here and here) and share your feedback. We’re eager to hear your thoughts, use-cases, and any suggestions you might have. Happy automation, let’s embark on this new journey together!

21 Likes
3

Amazing activity, just a question what if the activity doesn’t identify a property or find more than one match for a particular attribute/property?

2 Likes
4

Interesting :star_struck: Eager to understand how much time it will take AI to align data against the fields. Also, will there be configuration option to process whatever fields available and skin unavailable one.

Thanks,
Ashok :slight_smile:

1 Like
5

This looks exciting! Im particularly interested on the DOM analysis engine, without having any selectors and how it works. How much flexibility do we have for any corrections, if required? Does the model/engine get fine tuned as part of the process?

2 Likes
6

By “properties”/“attribute/property” I’m guessing you’re referring to input/output fields in the source/target form (since selectors are not involved here, there’s no selector attributes either).

If my guess is right, then the answer is this: the activity will just fill-in whatever it is able to extract from the source and whatever it is able to find in the target form. If the source is missing some of the fields in the target form, the activity will just skip them; if the source has some extra fields that are not in the target form as well, it will again ignore them. Feel free to try it out for yourself :wink:

Let me know if I answered your question.

1 Like
7

Processing time doesn’t seem to be a big issue, from our tests (but please tell us if you encounter such issues).
We wanted to first release the simplest, most magical AI powered version of this activity and then see what advanced options would be needed; so no configuration option to edit source/target fields for now (it’s done automatically - please see my previous reply above + give it a go :wink: )

1 Like
8

It’s the same engine we developed to support Attended scenarios (such as Clipboard.AI); we do a proprietary AI powered DOM filtering so we can build a reliable schema. It’s also used in Capture All for Object Repository, enabling (finally!) full scrollable content capture for OR - limited to web content for now. It’s a pretty cool technology for getting a complete UI snapshot of the application. Although there are no selectors directly involved, we could also link this new Semantic targeting to selectors & CV descriptors as well in the future - maybe even integrate this in Unified Target; what do you think? :wink:

2 Likes
9

@Bogdan_Sultana,

Thank you for the inputs and setting fair expectation from the activity. Surly I will be going to try it.

Thanks,
Ashok :slight_smile:

1 Like
10

We deliberately skipped any corrections for now: we wanted the cleanest, simplest experience for form-filling; flexing our muscles in terms of everything we know about UI element extraction, targeting and execution so that users can just drag&drop such an activity and forget about tedious selector tweaking or configuring activities.

But we do plan to add some relevant advanced editing in the future - we want to see how the simple version is doing first :slight_smile:

One possible idea is to enable you to generate a list of “classic” (well, Modern, not Classic, but compared to Semantic, I need to call them “classic” :stuck_out_tongue: ) UIA activities using AI, reusing our currently in Public Preview Autopilot technology. This would be an alternative way to do form-filling that provides more control.

2 Likes
11

thanks, this makes sense i.e. consolidation of different targetting methods.

1 Like
13

Hello community,( @Bogdan_Sultana ), I am going to test the new version… I hope the improvement will be amazing. Regards :star_struck:

14

Is this activity only for cross-platform projects? I am trying to find it in a Windows project, but no results.

15

Yes, only for cross-platform for now :point_up::wink:

16

Really enjoying the learning experience

1 Like
17

Hello Bogdan

UiPath.Semantic.Activities Package doesn’t come up for me in Manage Packages (even with prerelease enabled). I have a VB Cross-Platform project (I tried C# as well with the same result) open. I have Studio 2024.4.0 Community Edition installed on my machine. Can’t be found either in Official or Marketplace feed

What feed UiPath.Semantic.Activities is available on?

Capture
Capture2148Ă—719 56 KB

1 Like
18

@maxim_pechura1,

No need to install in as a package separately. Search directly for the activities.

image
image1218Ă—746 26.9 KB

Thanks,
Ashok :slight_smile:

2 Likes
19

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.