Forms AI in Public Preview

Hello to all friends of Document Understanding,

The day you have all been waiting for is upon us! Forms AI is in public preview starting today and here is how to find it, use it, and provide feedback so we can improve it.

What is Forms AI?

Forms AI is a way to extract data from documents with a fixed layout, like forms. You label the values you are interested in on at least 1 sample, and Forms AI trains a model in seconds behind the scenes to recognize those values on other samples.

Prerequisites

Document Understanding Cloud

You need to make sure the Document Understanding service is enabled on your tenant.

Once it is enabled, you will see the Document Understanding option appear in your left navigation bar.

|246x237.5172419372268

Automation workflow (Optional)

If you have never used UiPath Document Understanding product, please take a look at the Quick starters documentation page. For more advanced users who are ready to build scalable and robust RPA automations, please use the Document Understanding process template available in Studio 21.4 and later. A full tutorial is also described here: RPA Framework for Document Understanding.

For using Forms AI, populate the endpoint input property in Machine Learning extractor activity with the endpoint generated using Forms AI (described in detail below)

Quickstart Guide

Getting started with Forms AI is very easy. Follow the simple steps below to use Forms AI.

Before you get started, first download the sample documents that will be referenced in this guide:

Forms AI Example Docs.zip (1.0 MB)

Step 1 - Click on the “Document Understanding” icon on the right-hand panel to access the

Document Understanding Cloud. If this is your first time, you will see an empty page like this:

If you have already created projects in the past, you will see a table with past projects (the remainder of this guide will use such a set-up)

Step 2 - Click on “+ New project” to create a Document Understanding project. You need to enter the project name. If using UiPath Document OCR, no need to make any other changes. If you plan to use other OCR engines, you need to provide the URL and the API Key of those OCR engines.

Step 3 - Once the project is created, click on the project. This will open a view like this:

Step 4 - You can now create a Forms AI session by either clicking “+ New” or accessing it from the quick starters panel on the right-hand side.

Step 5 - On the next screen, provide a session name

Step 6 - Once you hit “save”, the next screen will start the core of Forms AI experience. The first step is document upload. You need to upload 2-5 documents of the same layout to create a model. You can upload a maximum of 20 documents. Each document cannot have more than 5 pages.

Step 7 - When the document upload is done, hit “Next” and you will be taken to a screen that will show you the uploaded documents. You will notice that some fields have already been detected and are already labelled for you, like the name and the social security number.

  • If you want to add more fields, simply click on “+ Add field”, provide a unique field name and hit “OK”

  • To assign a value to the newly added field, simply select the value on the screen and hit the shortcut key to assign the value to this new field.

  • If you want to edit a field or delete it, hover over the field and an edit button will appear on the top right side. Click on it and a pop-up will open to modify the field. “Multi line” option is for fields like addresses where the text may be on more than one line of text.
    Also, the column fields have the very important option “Split items”. This option is to determine how the rows (line items) of a table are split up. If you mark a field as “Split items” then every value of that field will be placed on a different row.

  • When you are satisfied with your field selection, go to the next document by clicking the arrow navigator on the top . All the changes are auto-saved as you make them

  • While you navigate to the second document, Forms AI trains a custom ML model for you in the background. When you hit “Predict” on the second document, the predictions are generated using the trained and customized model. If the model misses something, you can make corrections and the model learns from it and customizes itself. You can now go through a few more documents and run predictions on them to test and refine the Forms AI model

Step 8 - Now you are the third and final step for Forms AI

  • When you feel the model is ready for your task, simply hit “Publish” and then provide an extractor name to create a custom endpoint for the model

  • Once you hit “publish”, the Forms AI interface will provide the option to “Copy Endpoint”

  • You can verify the status of created model by simply going to the “Extractors” tab under the Forms AI session you created in your project

Step 9 - The endpoint created can now be used in ML extractor in your automation workflow for use as described earlier. You can also access the model endpoint created earlier by clicking on the three dots and copying the URL

You are done!! :tada: :partying_face:

If you run into difficulties, please respond to this post and let us know. We are working hard to make Forms AI more powerful and more easy to use every day.

Happy automating!

Your friendly Document Understanding team.

21 Likes

Already tried it and it’s absolutely amazing, so easy to setup!

1 Like

@alexcabuz thanks for sharing this update…

Hi @alexcabuz ,

Looking great! Is the intent to also incorporate the user feedback from Action Centre Validation Station for FormsAI model re-train (manual/auto?) in the future?

Thanks alot @alexcabuz !

Quick observation: When selecting multiple lines for a field, it works to use CTRL to select multiple values, other times it only works if I use SHIFT.

Thanks!

Hi @raf667

Would you be able to attach a small gif/recording of this behaviour?

@loginerror I cannot use my current setup since documents contain sensitive information. I will try to find some nonsensitive data and demo it later this week.

1 Like

Hello everyone, I just launched tutorial about Forms AI, where we will learn from scratch how to setup and use it on Document Understanding project

6 Likes

Hey Readers,

I found this new feature interesting and I created one video on it.
Here am posting the link have a look at it and let me know if you found this useful.

3 Likes

Here is the part 2 of the video: How to consume the created extractor.
Hey @alexcabuz Thanks for sharing, It’s really great feature to have across the team by just with a Endpoint and API key mapped to the Tenant. I tried using the extractor created in different tenant. It worked absolutely great.

Amazing post/guide @alexcabuz and great job UiPath team! :clap::clap::clap:

1 Like

Hi @warren_lee yes, that is on the roadmap for 2022.10 release.

Alex.

1 Like

Hi @raf667 this is a known issue, we are working to improve the selection with Ctrl or Shift.

Alex.

Great work. Very easy to use. We are doing some self-study using 4 ROMANIAN ID. On one of our sample, Google OCR did not recognize the “SEX” field value of M. UiPath OCR recognized the CNP field but will not let us tag the value though it can see the value. The value box has a lock icon on it. If we create a new cnp2 field, we can do things just fine so we deleted the one it found. Predict button works too. So cool.

@anthony.garcia just click on the lock and it will go away. You proably manually edited the value. That is when the lock appears, to indicate the value was edited manually, and therefore it will not be repopulated automatically, because presumably the human knows better than the machine what the true value is. :stuck_out_tongue_winking_eye:

Alex.

Hello @alexcabuz,

I have already trained/ developed extractors using Forms AI. And it’s way too easy for beginners. :slight_smile: Superr happy with this new feature :innocent:

However, there seems to be an issue or bug when it comes to extracting table items/ line items. It is considering the entire column as a single field. Hence table data is not extracted correctly.

Not sure if you can help with this. However, I wanted to highlight the issue I was facing and I hope this issue can be fixed in the next update.
Attached is the invoice document and the extracted data for your reference.

Sample Export.xlsx (11.2 KB)
.
AKASH PLASTIC.pdf (130.8 KB)

Happy Automating :fire:

Hello @kiran.suryavanshi ,

When you configure a Column field you can set the flag Split items. You should set this flag on fields which appear only once in a table row, and have a short value, like a line item, or quantity, or unit price on invoices.

split items

Hope this helps,
Alex.

1 Like

Thanks a lot @alexcabuz for the prompt response. You suggestion helped me fix the issue!!.

Thanks again !! :slight_smile: :love_you_gesture:

Cheers :vulcan_salute: