The issue was processing messages for live pro esports tournaments. Each game had to be processed and the biggest issue was that messages from the queue were not guaranteed to ordered and because of processing issues upstream you could get a large amount of messages for a specific match in sequence. Thus we decided to run just a light weight worker on the original external queue copying to our datastore which is very fast and can be run massively parallel.

Then run a unique job per running match that does the ordering and processing. This worked well enough but you need to now make sure that each worker gets called again after it finished and originally we had a recursion going on but the best place to do it was eventually deprecated in a new sidekick or unique jobs version update. We eventually settled on a supervisor that would monitor and restart workers. Which is just a bad version of elixir in the end