Hey guys, I’m looking for some advice on how to set up queues when individual items depend on one another. The scenario is as follows:
- The input consists of mutiple excel files
- Each of the files contains a list of items to be processed
- The processing is a straightforward data update in a web app
- However, the items need to be completed in the exact order as specified in the file. If any of the items fail, the following items from that file should not be processed.
- The process needs to be scalable, i.e. we need to be able to add additional bots as item volumes increase
- After processing each file there needs to be a summary of the items processed
My initial thought was to split the bot into 2 queues and 3 robots:
1. Dispatcher - looks for new excel files and adds them to Queue 1. Each excel file is added as a work item, and within that work item there is information about all the items that need to be processed as part of the file
2. Dispatcher/Performer - consumes Queue 1 - reads the next work item and for each item that needs to be updated, creates a queue item in Queue 2 and waits for that item to be completed. If the item is successful, it creates the next item in Queue 2. If failed - it doesn’t process any more items. The run successfully ends when all the items added to Queue 2 are processed - then the robot creates a summary and picks up the next item from Queue 1.
3. Performer - consumes Queue 2. This robot just picks up the next item and performs the update in the web app. Since new items are added one-by-one by robot 2, a standard queue retry can be used.
Queue 1 has got retries turned off
Queue 2 uses a standard queue auto-retry logic
Theoretically, this set-up should allow to add additional bots for any of the processes. The only problem is that if there are any new items left in Queue 2 (like if someone manually retries a failed item), they may be processed outside of Robot 2 run, which violates the requirement that items need to be processed in a specific order.
What do you guys think? Is this the way to go or should I rather stick to 2 bots and 1 queue where an excel file is a work item, and robots 2 and 3 are merged into 1 robot?
Thanks for any comments.