That’s true your in progress item will just be ignored and will not affect the performer. However, I understand your concern with keeping the queue clean.
Your issue or concern is something similar to this thread
I have a suggested workflow in the link above to delete your in progress items. The solutions does not require you to delete the queue.
Hope it helps!