Bulk Add Queue Items - Is there a maximum data size? What is a suggested work around

Help Studio
1

I have a project which requires me to send the rows of a spreadsheet to a queue as transaction items for processing. There are almost 52,000 records in the spreadsheet. If I send them all to the Orchestrator queue in a single Bulk Add Queue Items activity, I get a “BadRequest” error.

So, I’m guessing there’s a maximum data size allowed… but I can’t find documentation for it.

So, if there IS a maximum data size and I’ve run into it, is there a suggested work around? Sending the transactions to the queue one at a time is not an option because that would take longer than the SLA allows.

2

There is a limit, though I’m unsure of what that limit is. You can insert each item indvididually using Add Transaction Item with a loop over each row of the data.

3

As mentioned, adding each item individually, in my timing tests, would take way too long. It took 6 seconds in my test to do that for 10 records… for 51,000+ records, that’s 85 hours… and that’s too long for our SLA.

Thanks, though.

4

Orchestrator queues aren’t built well for large data loads like this unless you preload them in a separate overnight process. If you need the data loaded more quickly, you can read from the source directly and process each record, but then you will need to create a mechanism for handling retries, logging, etc.

5

Unfortunately, that won’t work either as the processing of each record also has a short SLA. One record at a time will take in excess of 16 days to process for a single bot so we are loading to orchestrator queue to have a “farm” of bots processing in parallel.

6

If you follow the Dispatcher-Performer model, you can have a bot load the queue transaction by transaction, and the queue can trigger the other bots to act on the queue. In the meantime, the dispatcher is still loading accounts to the queue. If you do it this way, you won’t have to wait for all of the accounts to be loaded, as they will likely take less time to load than it takes to process each transaction. Also, even if the bots finish reading all of the queue items faster than they are loaded in, the minimum time to complete all transactions will be the length of time it takes the dispatcher to load them all.

6 Likes
7

Duh… why didn’t I think of that. Processing a transaction will certainly take less time than uploading so an item being added to the queue will trigger available bots. Thanks!

1 Like
8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.