Queue setup for inter-dependent queue items

Help Orchestrator
1

Hey guys, I’m looking for some advice on how to set up queues when individual items depend on one another. The scenario is as follows:

  • The input consists of mutiple excel files
  • Each of the files contains a list of items to be processed
  • The processing is a straightforward data update in a web app
  • However, the items need to be completed in the exact order as specified in the file. If any of the items fail, the following items from that file should not be processed.
  • The process needs to be scalable, i.e. we need to be able to add additional bots as item volumes increase
  • After processing each file there needs to be a summary of the items processed

My initial thought was to split the bot into 2 queues and 3 robots:

1. Dispatcher - looks for new excel files and adds them to Queue 1. Each excel file is added as a work item, and within that work item there is information about all the items that need to be processed as part of the file

2. Dispatcher/Performer - consumes Queue 1 - reads the next work item and for each item that needs to be updated, creates a queue item in Queue 2 and waits for that item to be completed. If the item is successful, it creates the next item in Queue 2. If failed - it doesn’t process any more items. The run successfully ends when all the items added to Queue 2 are processed - then the robot creates a summary and picks up the next item from Queue 1.

3. Performer - consumes Queue 2. This robot just picks up the next item and performs the update in the web app. Since new items are added one-by-one by robot 2, a standard queue retry can be used.

Queue 1 has got retries turned off
Queue 2 uses a standard queue auto-retry logic

Theoretically, this set-up should allow to add additional bots for any of the processes. The only problem is that if there are any new items left in Queue 2 (like if someone manually retries a failed item), they may be processed outside of Robot 2 run, which violates the requirement that items need to be processed in a specific order.

What do you guys think? Is this the way to go or should I rather stick to 2 bots and 1 queue where an excel file is a work item, and robots 2 and 3 are merged into 1 robot?

Thanks for any comments.

1 Like
2

Hey @totoro84

  • Do you need to track status of each item from the excel input file

Or

  • It is fine to just track the overall excel file status as either Success or Failure

Thanks
#nK

3

Hi Nithin, thanks for your response.

I need to track the status of an excel file as a whole, and the status of each item in the file. And I need to do it via queues for reporting purposes and due to using a queue-based framework. And I need to be able to add machines, so they dont run into one anothers jobs messing up the order of processing.

I may end up tracking the progress of an individual excel row locally in a datatable, and use add transaction item activity to populate the queue one item at a time once it is being processed, but this means handling retries locally and changing the framework logic significantly