I think the biggest problem was that it broke a stablished UI for no apparent reason, but that was 10 years ago. And people still hold a grudge to it, maybe?
Still, I've seen more than a few workplaces that still use pre-ribbon era Office [0], and haven't upgraded because the employees hate the ribbon...
[0] ...on Windows XP, with 12+ year old hardware... the horror!