I built a dispatcher that is crawling a web page for specific data.
In case the dispatcher runs again before the performer has finished running on all of the transactions then the same queue item might be added again.
I thought of adding a unique reference to the transaction, I even surrounded that with try-catch block (because there is an execption if I add the same reference) but then even if I have a transaction with “Removed” status it is still not added to the queue.
Why is that? And would like your suggestion on what is the best way to the handle this issue?