There are many things here that concern me from a system view.

1. They need guaranteed delivery, but chose to use UDP 2. They jacked up the default rmem buffer to ~2GB which is insane. Also, applies to all sockets not just UDP, so I wouldn't be surprised if they where also running into issues with memory pressure especially under load 3. Support didn't seem to let them know that's a pretty unconventional configuration

That was an interesting debugging story, and catching a bug like this is always good IMO. But, there is just so much WTF in this setup.