Should queue items be uploaded manually or using a dispatcher for large volume of data to be fetched from a single CSV file

Help Orchestrator
,
1

Is it preferable to use a dispatcher to upload data to Orchestrator queue or upload the queue items manually when a process has large volume of records around 17000 and needs to run with multiple bots running at the same time?

2

Hi @fardeen.shaikh.91

You need to use add bulk queue Item activity to Upload into Queue!

Regards

3

Thanks for your reply @pravin_calvin. The add bulk queue item activity has a limit of 15000 so will it be preferable in this case?

4

Hi @fardeen.shaikh.91

Check below thread similar to yours!

Regards

5

Hi @fardeen.shaikh.91 and @pravin_calvin,

If you do not want to make any changes to the queue item’s specific data, then uploading a processed CSV is a quicker option. This is assuming that you only need to upload queue items once. If it is a repetative process, then I suggest you to not upload the CSV directly.

If you use a dispatcher process and advantage is that you can add more specific content to the queue item and this may help the performer process. For example, if you wanted a specific content to also contain a table as json string and rest of the specific content as standard strings then it is best to use a dispatcher process. This way your dispatcher process can also run on schedule by using orcehstrator triggers.

Now there are four options to send items to queue if you use a dispatcher process.

  1. Add Queue Item
  2. Bulk Add Queue Item
  3. Using Orchestrator API
  4. Trigger a stored-procedure (which connects to your orchestrator via SQL and uses the Orchestrator API calls) credit to @KanadMehta
    Tutorial: Call UiPath Orchestrator API from SQL Server - Marketplace - UiPath Community Forum

Lets keep it simple. In your case the first 2 options are sufficient. Option 3 is a complete overkill and Option 4 is something you can use if your data is already in a SQL table
In my tests for a dispatcher process (which also had a lot of fields AKA specific content) 5000 rows was a limit and Bulk Add to Queue failed. Why? It is because Bulk Add to Queue recieves a json response back in UiPath. If this json reponse is too large, UiPath fails to return it and the dispatcher process will end as faulted.

What we currently do for large input data (maximum we have tried is 15000 rows, which takes the dispatcher close to 11 minutes)
We check the rows in the datatable which has to be sent via dispatcher and depending on the row count, we can then use a condition to trigger either Add Queue Item or Add Bulk to Queue.

This ensures that when the data is small all our queue items are uploaded in a single call to orchestrator and if the data is large, we take a conservative approach and send items one by one.

If InputDatatable.Rows.Count > 5000 
Then : Add Queue Item
Else : Add Bulk to Queue

Hope this helps you and others.

Update
There is also another alternative of using Add Bulk to Queue and use slices of your input datatable.

Lets assume your Input Data is of 10000 rows, you divide them into 5 (n number) batches. For each batch you use Add Bulk to Queue and this way you can both reduce the time taken to send items to queue and avoid failure of the Bulk Add to Queue activity.

2 Likes
6

@jeevith Thank you for your reply. That pretty much clears my doubt but please let me know what do you think about this solution where I add the queue items from the CSV file manually and do it twice by uploading the data from 2 CSV files, since the data is more(1700) then the limit(1500). Please note that no change is required to be made in the CSV files.

7

@pravin_calvin Thank you. That was informative.

8

Hi @fardeen.shaikh.91,

You could do that, it is a safe method. Split your current CSV into two files each consisting a max of 1499 rows of data. Also, I hope you can one column in your csv which you can use as a Reference in your queue.

To use a reference you need to have an additional column named “Reference” and copy the values of a unique column in your CSV. This way when you upload the csv you will also get a unique Reference per queue item. This is handy when you have thousands of items in a queue. I am here assuming you chose to not have unique reference values when creating your queue. If your queue does not support duplicates of Reference then orchestrator will let you know and will fail to upload items to queue.

Again, if your queue items have to be uploaded only once then this is an OK approach, if you have to do the same frequently then I would suggest you use a dispatcher process.

9

@jeevith Legend, thank you!