Pauls setup using single jobs is the correct one, however I personally differ on some architect and triggering mechanism as I dont think queue triggers and using queues for them are the correct way.
The way I have solved this for years and advocated for is via ‘process orchestration’.
This use to be harder, but we have alot of options now, we have either the ‘Long Running Automation’, or now Maestro.
In the long run I think Maestro will become the better method to do this, but its currently quite buggy and not feature complete as its still in public preview.
The long running workflow would actually orchestrate your process end to end.
I’ll take Paul’s example as a comparison and suggest the Architecture I’d do instead.
Please note, the architecture style I propose has been possible for about 5 years now, but the tooling for it etc is only now becoming widely available and popular.
So, if we had this
Dispatcher - loops through items on a web page and creates items in the “performer” queue
Performer - on a queue trigger, loops through items, looks up data in mainframe, makes comparisons and if comparison fail it creates a queue item in the “actioncenter” queue
ActionCenter - processes one item and stops, because in the middle it suspends waiting for the Action Center Task. This process is on a queue trigger that starts multiple jobs, one per queue item
I would instead do this.
Dispatcher - Loops through items on a web page and starts an Orchestration job per item, passing details of the item directly to the job it starts (you can optionally track these jobs in the dispatcher for reporting).
Orchestration - This is a long running process and will suspend several times during the execution, which can take several days, it takes the details from the item and creates a queue item for Performer 1 to start and then suspends, waiting for that item to be done.
Performer 1 - This functions as a normal performer, triggered by a queue trigger and will process queue items made from any Orchestration job.
Orchestration (resumed) - Once the queue item from performer 1 is compelted the Orchestration job is triggered automatically to wake up, it can read data from the queue item as necessary and do the next step. In this context that is the human in the loop, the orchestration process already supports persistence as we just had a ‘robot in the loop’ so we can do this directly with no issues.
After this we can continue processing if neccessary, maybe there is a second performer we need, and that can be used in the same way as the first one.
This is a strong design pattern and one that doesn’t rely on one performer being hard coded to trigger another in a ‘daisy chain’, which can help significantly reduce complexity in your performers, it also centralises alot of the major business logic in one layer making it easier to understand than ‘chaining’.
This will be the design pattern Maestro enforces and due to the way we can also resume maestro workflows from an error state it will also become very strong, but for now, you can try this using the template I propose.
