What made it so painful?
But as soon as you start touching HA, you have to unbuckle all of it's components, and that's where it becomes a real, real pain. Because gitlab does not assume you will do this and starts _freaking_ out when it tries to start something which doesn't exist on the local system.
It takes a long time to find all the dials and triggers to turn local starting off, and even then you basically have a full omnibus installation sitting on each of your nodes, because even if features are turned off they sometimes need some part of the code for the feature.
At this point upgrading anything is not just a tedious process, it's also error-prone and time consuming. Take every web-node down, upgrade the local packages on the web node you use to upgrade the database, then do the db migrations, then you go around upgrading everything else to the latest version, then you can slowly bring everything online and hope that it works.
Don't event get me started on extending it's authentication system.
You can open issues about specific complaints in [1] and [2].
Funny enough I believe I finally figured it out, after years! It was hanging on mail notification jobs because the MTA wasn't properly setup. Once the MTA was fixed sidekiq was working better and I was also receiving mail notifications.
Even so, syncing with AD I get internal names like shortname@addomain.local and most users don't update their profile. Thankfully the MTA just queues them and discards them without blocking sidekiq.
I've worked a lot with async jobs in Python using both RMQ and rq, I don't see why sidekiq should be such an issue. I've never had issues like that myself in my own projects.
Also felt that it was hard to manage compared to my own experiences with job management in Python. Adding to the frustration was having to understand a (to me) foreign language like Ruby. Something I never wanted to learn and avoided successfully. I ended up having to write a little bit of code that could kill sidekiq jobs because I couldn't figure out how to do it through any interface provided by Gitlab.