The short answer: the choice to use Ubuntu in particular was an incredibly bad decision, and we're ripping it out, but it's going to take forever.
A lot of the anti-ubuntu bias you hear (and my own strong anti-ubuntu feelings) are developed from trying to make these things work at scale and in an automated fashion. I'll be the first to admit that you can find lots of smart people (particularly here) who have made this work, but at quite a bit of effort. Woe be unto you if you try to do anything they didn't foresee.
* Their packaging guidelines and standards are quite poor--some packages start the daemons on installation (bad), and some don't, but you'll never know which do and which don't until you install them or inspect the debs, and the universe/multiverse repos are even worse. Additionally, some packages require interactive use on installation (NIS, I'm looking in your direction). Yes, you can preseed these questions away, but I find it astoundingly irritating.
* Their work on user experience is to the detriment of everything else you'll need to get a stable, predictable system, and they admit this. They half-finish major system changes, and don't give a shit if it works or is documented. Here's a rich one: http://manpages.ubuntu.com/manpages/precise/en/man8/mountall.... That's one of the most critical pieces of software on the system, and that man page has been unchanged for over 2 years. It does different things depending on what signals it traps, and the only way to find that out was to look through the source. I'd say I generally find something stupid like that every month or so.
* Canonical support is a complete and utter joke. The support contract we signed would have been more useful as toilet paper. My company also has quite a few kernel hackers on staff, and we damn well needed them while we waited for canonical to get off their asses and patch serious issues we had (that were, by the way, fixed in the upstream kernel). On two such occasions, I sent in bug reports pointing to lkml posts that pointed out the issues, I backported the patch directly to the Ubuntu release kernel, and still waited months for any movement on the issues.
* This is not Canonical/Ubuntu/Debian's fault, but it's still an issue. Any third party software that is sufficiently complicated is rarely tested on debian-based distros before it is released. In some cases, this is fine; in other cases, it's a disaster. There are subtle differences in kernel versions, libraries, etc. that will make some software just plain not work. Other software (Infiniband in particular) is a huge hassle to get working just because nobody who packaged it ever cared to make it work on ubuntu. This is less of an issue in the startup/web app world since a lot of that software really is developed and tested on multiple operating systems. My background is in HPC and research/academic computing, and quite a bit of that software is only used on Scientific Linux or some other RHEL derivative. This is changing a bit with the OpenNebula stuff, but not fast enough for my taste.