I've often thought there is an analogy with broken "real-world" processes where the same thing happens.
I've often thought there is an analogy with broken "real-world" processes where the same thing happens.
And generally this system should not use the message queue to just get things resent - because most likely in this situation you still need to be mostly handling new messages rather then old ones when you're stuck trying to catch up.
All of which tends to be a product of just utterly magical thinking from managers and solutions architects as to how anything works, and how likely an outage is (basically guaranteed no matter what anyone says about their product).
I can't find the reference, but consistent with that, I remember reading someplace that the number of items in your queue should typically be zero, since queues can't actually make the rest of the system any faster. All they can really do is buffer temporary bursts, or buffer while the rest of the system recovers. And for that you'll definitely need your recovery system working as a prerequisite.
Depending on your requirements you either stop the state of the whole system or use something like dead-lettering and/or delay-queues + retry count headers to make sure that you can reprocess failed messages without letting them accumulate until you're wasting CPU processing messages that will never work.
The other problems with message queues is that you need to make sure you have alerting on all your queues otherwise you're just waiting for trouble.
There's sender channels, receiver channels, xmit queues, local queues, remote queues, dead letter queues, topics, subscriptions and all kinds of other things to learn about. Add that to the crappy IBM tools and their documentation that is full of dead links and you're in for a lot of learning through trial-and-error.
2nd'd & 3rd'd ...
You hired a 50 man lumberjack crew to cut down a sapling.
I've been spending a some time to remove IBM MQ out and going to "simpler" queuing solutions. First of all, cost. Guess what, we're not getting anything out of licensing per year to justify the cost.
With improving the end points, we don't need all that complexity. With 5 9's uptime on the network and smarter end points, retrying messages isn't expensive.
As always, YMMV.