How To Implement A Document Understanding Training Loop On Cross Platform Activities or Studio Web

The new Document Understanding Cross Platform created for use in Studio Web and also available in Studio Desktop simplify a huge amount of work we need to do with in a Document Understanding project and also give us easier objects to work with as outputs and are well worth using over the legacy versions, however it is currently missing a key feature, the training loop to retrain your Machine Learning Model.

In collaboration with UiPath we have figured out a workaround for this by leveraging the ‘classic’ Training Scope from the Windows activities, it does require some advanced knowledge for the setup though so I will explain that here.

Firstly lets briefly discuss the new Cross Platform Activities. This assumes you have a project set up in the Document Understanding Center and have deployed a ML model based on it. You can skip this if you are already confident you understand these activities and want to get to the training loop.

To add the new cross platform activities to your project use the dependency ‘UiPath.DocumentUnderstanding.Activities’

The first activity is the ‘Extract Document Data’ activity which will handle the taxonomy and digitization of the document you give it, its based on a document type you have previously defined in the Document Understanding Center.

The output type will change based on the document you select. For example for a Purchase Order it would be an IDocumentData<PurchaseOrder> for a Bill Of Lading it would be IDocumentData<BillOfLading>.
This object contains properties for all the data points in your document, which makes it much easier to use. Make sure you assign your variable for the output using ctl+k AFTER you select your ML Skill, otherwise you get the default, generic type which is much harder to use since all the properties from your document cannot easily be found.
Here you can see an example where all the fields in the document can easily be accessed.

We then provide this input to a ‘Create Validation Task’ which creates the action in the action center. The output of this will also be dynamic based on the IDocumentData<T> that we give it. Pay attention to the folder and the Storage Bucket used in this activity as we need data from it later.

We then need to wait for the human so we use the ‘Wait for validation Task and Resume’. The job will suspend and wait for the human to validate the data.

Leave a task in the action center to be validated and go to the storage bucket associated with the validation task, you should see various documents associated with the extraction there which have been handled for you automatically.
We need the Taxonomy for the ‘classic’ activities so download it.

The Training Loop activities need to be in a different UiPath project than the one containing the Cross Platform activities ones due to compatibility issues so we need to add to transfer data via a queue item, so lets add an item to a queue after we get back the human validated data in the first project.

We need the following Keys on the queue item specific content, this assumes the output from the Wait For Validation Task and Resume’ outputs to a variable called ‘validatedData’.

We must serialize some of the data to a string so it can be added to the queue.

Key Data Type Value
Document Path string
Document Text string validatedData.DocumentMetadata.Text
Document DOM string Newtonsoft.Json.JsonConvert.SerializeObject(validatedData.DocumentMetadata.DocumentObjectModel)
Human Validated Data sting Newtonsoft.Json.JsonConvert.SerializeObject(validatedData.Data.GetProperties().GetExtractionResult())

In another project we can read the queue item and process that into a training dataset so lets start a new project.

We need the dependences ‘UiPath.DocumentUnderstanding.ML.Activities’ and ‘UiPath.IntelligentOCR.Activities’ in this second project to build the training loop.

The classic activities require you to build the taxonomy manually so we need the taxonomy that was generated for us from before that we downloaded (to save us making it manualy), add a folder to the UiPath Project called ‘DocumentProcessing’ and add the Taxonomy you downloaded there validate it works by opening it via the taxonomy manager in the project.

Assuming we set up the queue item as indicated earlier, we want to read these keys from it and assign the values we added to the specific content to variables so we can use them in the training activities.

image

Variable Name Data Type Express to Get From Queue Item
documentPath string queueItem.SpecificContent(“Document Path”).ToString()
documentText string queueItem.SpecificContent(“Document Text”).ToString()
documentDOM UiPath.DocumentProcessing.Contracts.Dom.Document Newtonsoft.Json.JsonConvert.DeserializeObject(of UiPath.DocumentProcessing.Contracts.Dom.Document)(queueItem.SpecificContent(“Document DOM”).ToString())
humanValidatedData UiPath.DocumentProcessing.Contracts.Results.ExtractionResult Newtonsoft.Json.JsonConvert.DeserializeObject(of UiPath.DocumentProcessing.Contracts.Results.ExtractionResult)(queueItem.SpecificContent(“Human Validated Data”).ToString())

Now add a ‘Training Scope’ activity and add the variables above to it.

We lastly need a ‘Machine Learning Extractor’ to add to the scope that we can configure.

Make sure you add all the fields from the Taxonomy to the extractor.

Set the output you want the data to be sent to, either a local folder, data set in the AI center or public endpoint.

Untitled

From there we are done, you can run your two projects and validate they work! Be sure to check you get output as expected, if you do not there could be a mismatch in your taxonomy the document type ID is important as if they dont match they wont generate any training data.

4 Likes

Hi @Jon_Smith ,

Thanks for sharing this proposed solution. The final result is the exported labels that could be later used for retraining purposes. Let’s assume you exported to a local folder, how can the exported labels be used later to retrain an extractor in a modern du project?

Same way as a classic one I believe, its all about retraining the skill in the AI Center.

I’d suggest looking at the existing documentation there or open a new thread.
If I am honest I focused on not losing the validated data and havent done a re-training run on a modern project yet, its on my Todo list.

If you get there before me please share your insights.

When you create a modern du project it doesn’t create an AI Center project, so the integration/link with AI Center is not explicit as it is with classic du projects.
The modern experience continuously trains the models and isn’t clear for me if that training runs over pipelines in an AI Center hidden project or if it is a complete different infrastructure.
Maybe uploading both the file and the annotations (labels) works in modern, if you ever test it please share your findings.
In the mean time, this is an alternative for retraining modern du models:

Incorrect and incorrect. The modern experience does not continuously retrain the models, can you tell me where you are getting that information from?

The link you share explains that its a missing feature, not how to do it?

My understanding is that training for fine-tuning is missing, so the feedback from the action center (human validation) is not yet automatically used to improve the model
image

When I say “continuously trains” I am referring to the active learning experience, when you annotate new documents uploaded in the modern du project the model retraining is triggered (this is not fine-tuning)
image

I see the distinction you are making.
Its certainly easy to get wires crossed here on that, but once a skill is deployed from the DU center I believe its still an AI center skill so stops getting retrained, its only during labelling it does what you say.

I again need to double check that assertion though. I’ve being using the modern activities but not had the opportunity to fully using the modern DU experience that integrates the AI learning.