If you are using the REFramework, I suggest you throw the error as system exception. Every REFramework has a try catch at the process transaction state.
In the catch flow, you have both business and application exception error handling.
You can set a counter here for consecutive application error which keeps incrementing when there is an application exception, but returns to zero if a business exception occurs between items. This counter will be a global variable.
In the main.Xaml you can then use the state machine condition to not retry an item after a given number failures say 5 application failures. Instead send it gracefully to the end state of the state machine.
Hope this gives you some food for thought
We use a similar logic to avoid robot setting application error for all the queue items. This is one of the most important features which is absent in the REFramework. It’s upto us as developers to ensure that an application failure does not result in 3 retires for every item in the queue.