New UiPath Document Understanding features have been released!

[loginerror (Maciej Kuźmicz)]

New UiPath Document Understanding features have been released!

Our new UiPath Document Understanding features have been released to Limited Availability in February 2020!

Note: The Receipt, Invoice, and Purchase Order ML Extractor are a Cloud Platform offering and will be released to general availability on April. Additionally, the on-prem solution will be also released in April.

Key Benefits

Flexible approach to extracting information from documents

You have the option to use templates with the Fixed Form (ex-Position-Based) Extractor, regular expressions with the Regex Extractor, or pre-trained models with the machine learning - ML Extractor. Also, you can bring your own OCR technology like Abbyy or Microsoft Forms to digitize documents before data extraction.

High accuracy for even ‘noisy’ documents

We are continuously improving our AI to handle the complexities found in real world documents that make them hard to read, such as noise, rotated and skewed documents, or low resolution from printing and scanning.

Drag-and-drop AI skills directly into your workflow in Studio

IntelligentOCR package (to be renamed to UiPath Document Understanding) allows you seamlessly drag-and-drop our new ML Extractor into the workflow - similarly like any other extractor. Same infrastructure, new supercharged brain.

How to Install in Studio

Note that you need to create an account on platform.uipath.com and use a License key from the Licenses page for these features to work. The key must be pasted in API-Key property of the document understanding activities.

In order to utilize the Document Understanding framework you must first install the UiPath.IntelligentOCR.Activities activity package:

1. Open the ‘Manage Packages’ window
2. Click ‘All Packages’ in the left navigation, then search for “UiPath.IntelligentOCR.Activities”
3. Select the result, make sure the newest version is selected, then click the ‘Install’ button in the right pane

Next, you need to install the UiPath.DocumentUnderstanding.ML.Activities activity pack:

5. Check ‘Include Prerelease’, then search for “UiPath.DocumentUnderstanding.ML.Activities”
   

6. Select the result and click the ‘Install’ button in the right panel

   Important - We must reiterate, make sure you check the ‘Include Prerelease’ checkbox.

The ML Extractor activity should be visible now in UiPath Studio, as shown below:

How to Use

Here are the demo videos on how to use UiPath Document Understanding:

UiPath Document Understanding Demo 1: Setting up the framework in Studio

UiPath Document Understanding Demo 1: Setting up the framework in Studio

UiPath Document Understanding Demo 2: Data extraction configuration

UiPath Document Understanding Demo 2: Data extraction configuration

Additionally, see these step-by-step instructions to help you get started with the ML models:

1. If you haven’t yet defined taxonomy for the documents you intend to process, you can do so using the Taxonomy Manager

2. Drag in the Load Taxonomy activity and store the taxonomy in a variable
   

   

3. Drag in the Digitize Document activity

   • Drag-and-drop an OCR Engine inside the Digitize Document activity
   • Define variables for both outputs: DocumentObjectModel and DocumentText

4. Drag-and-drop the Classify Document Scope activity and once again drag and drop the keyword-based classifier activity inside of it to add a new classifier. Activate the classifiers to enable classification
   

   

5. Drag in the Data Extraction Scope activity

   • Populate input variables DocumentObjectModel, DocumentText, Taxonomy and DocumentTypeId. The DocumentTypeId is a string you can see in TaxonomyManager if you click on the Document Type you need to extract.
   • Define variable for output: ExtractionResults
   • Drag in Machine Learning Extractor and drop it inside the Data Extraction Scope
   • Drag in Regex Extractor and drop it inside the Data Extraction Scope
   • Drag in Position-Based Extractor and drop it inside the Data Extraction Scope
   • Populate Endpoint input property with the URL of the endpoint you would like to use:
   • https://invoices.uipath.com for Invoice processing
   • https://receipts.uipath.com for Receipt processing
   • https://purchaseorders.uipath.com for Purchase Order processing
   • a local custom endpoint in case you have one of these models hosted on-premises

6. Click the “Configure Extractors” link within the Data Extraction Scope activity

7. Expand the Document Type you are interested in and populate, on the right side column, the names of the fields which correspond to the fields in your taxonomy. See below for the full field names you need to fill in on the right side. You can mix and match between values and what you want to extract.

Pre-trained ML model for Invoices extracts the following name and line item fields:

“name” 
“vendor-addr” 
“billing-name” 
“billing-addr” 
“shipping-addr” 
“invoice-no” 
“payment-terms” 
“due-date” 
“po-no” 
“date” 
“net-amount” 
“tax” 
“total” 
“currency” 
“line-no” 
“description” 
“item-po-no” 
“quantity” 
“unit-price” 
“line-amount” 

Pre-trained ML model for Receipts extracts the following name and line item fields:

“name” 
“total” 
“vendor-addr” 
“date” 
“phone” 
“currency” 
“expense-type” 
“description” 
“line-amount” 
“unit-price” 
“quantity” 

Pre-trained ML model for Purchase Orders extracts the following name and line item fields:

“po-number” 
“date” 
“client-name” 
“client-address” 
“vendor-name” 
“vendor-address” 
“shipping-name” 
“shipping-address” 
“billing-name” 
“billing-address” 
“payment-terms” 
“delivery-by-date” 
“net-amount” 
“tax-amount” 
“tax-rate” 
“total-amount” 
“line-number” 
“description” 
“product-code” 
“delivery-date” 
“unit-measure” 
“unit-price” 
“quantity” 
“line-net-amount” 
“line-tax-rate” 
“line-tax-amount” 
“line-amount” 
“currency”

[MarkusDS (Markus)]

So the ML extractors can be used to do extraction without templates, correct?
this can be done without AI fabrics and doesn’t count towards Machine Learning library use, right?

[Adam_Hanson]

Yes. You can use our models in the cloud without AI Fabric.

AI fabric enables retraining and managing / deploying models generically, so it can be used with DU and other models.

The DU solution comes with everything you need to run DU on it’s own, so it can be purchased separately from AI Fabric.

However, using our models for DU does count towards ML use.

[mz_psl]

Hi - is it possible to re-train the models on-prem now? And hopefully the validation station can be completely skipped? thanks

[irahmat (Indra Rahmat)]

is it support for Thai Character?

[chaithanyapallegar]

Hi,

I would like to know what Extractors can be used On-Premises and is there an alternative for using the Cloud based Extractors on Non Cloud environment.

One of the requirements we have doesnt have active internet connection and unable to connect to the cloud.

[Ioana_Gligan]

Hello @chaithanyapallegar,

Regex based extractor and Keyword based classifier are usable with no internet connection.
Machine learning models can also be hosted on prem (so no external internet connection - except for the ML model infrastructure that needs to be able to do licensing calls only).
The rest are only available with an internet connection on the robot machines, but we have it as a WIP to make them available on prem as well.

Hope this helps,

Ioana

[Ferdinand]

If I understand correctly you should not have a newer Studio version than Orchestrator version.

So do you need/recommend to have Orchestrator 2020.4 to use Document Understanding, or can you do it on 2019.10 LTS?

[Riya_Lanjewar]

Can we use this for Structured as well as for semi structured Invoices?

[Jersey_Practical_Sho (Jersey Practical Shooter “X X Sonic Loss Xx”)]

So Im having a play around and got down to

“Drag in Position-Based Extractor and drop it inside the Data Extraction Scope”

But there isn’t an option for this?

Si i’ve used the Inteligent Form Extractor and Configured it. Question is how many times do I have to run the workflow through before it actually starts to pick up the information on its own, as everytime I do it picks up the document type but nothing else

[tudor.serban]

@Jersey_Practical_Sho: Form Extractor is the new name for Position-Based Extractor.

[Jersey_Practical_Sho (Jersey Practical Shooter “X X Sonic Loss Xx”)]

and which APi key should we be using with it?

and does the endpoint change if using enterprise?

As Im using the same items as ML Api key and standard endpoint and getting Licence couldn’t be validated

SOLVED IT

[AndyMenon]

@loginerror

4. Drag-and-drop the Classify Document Scope activity and once again drag and drop the keyword-based classifier activity inside of it to add a new classifier. Activate the classifiers to enable classification

I have followed the steps and installed all the packages. But I see only one classifier in my configuration dialog. How do I get the other classifier as shown in the instruction #4 above?

[tudor.serban]

Hi @AndyMenon: You can drag and drop any number of classifiers in the Classify Document Scope. They will then appear in the Configure Classifiers wizard as a new column in the table.

[AndyMenon]

Thanks @tudor.serban. I see now how this works.

[chaithanyapallegar]

Hi,

I am receiving the Error : Create Document Validation Action: The remote server returned an error: (500) Internal Server Error, when i am testing a workflow based on Document Understanding Process (orchestrator background process).

When I run the same process via studio, it works fine, but when I run this as a background process (Non production Robot machine), I get this error

Error Below

Process: _AP_Process_Data_Extraction
Environment:
Robot: AP_Process
Machine name: UIROB1
Info: The remote server returned an error: (500) Internal Server Error.

RemoteException wrapping UiPath.DocumentUnderstanding.Persistence.OrchestratorException: The remote server returned an error: (500) Internal Server Error.
at UiPath.IntelligentOCR.Activities.BaseOrchestratorClientAsyncActivity.ThrowIfNeeded(Task task, Boolean suppressThrowException)
at UiPath.IntelligentOCR.Activities.BaseOrchestratorClientAsyncActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)
at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)
Start Time: 07/17/2020 4:15:33 PM
End Time: 07/17/2020 4:16:10 PM
Input Values: null
Output Values: Empty
Has Recording: false

[AndyMenon]

@chaithanyapallegar

I am receiving the Error : Create Document Validation Action:

In your error message there is this part I have highlighted below - is your activity missing something?

[ernesto.limon]

Is there any way to customize the DU extractor for cheques samples currently ? or what options do we have to customize the ML based extractor ?

[GouthamVijay]

I couldn’t able to find UiPath Document Understanding package on 2019.10.4 on-premise Enterprise Version, is it only available for Uipath Cloud Automation licenses ?