That may be a reasonable thing to ponder until you try to run Ubuntu on a device which formerly ran Android. I thought the same as you and installed Ubuntu on my Asus Transformer TF101.
While the original Transformer sports reasonably dated specifications, it's provides a very neat, fast and responsive Android-experience.
I expected Ubuntu to offer the same, but with a wider-selection of (non touch-optimized) software. I was thoroughly disappointed. Just opening the Terminal Emulator took 5-10 seconds. Trying to open Firefox or any web-browser was a joke.
What most people fails to realize (I guess because Android has adopted the Java-is-slow-meme, either it wants to or not) is that Android is heavily optimized because it has to be responsive on ultra-light hardware. Given its feature-set, it's a very fast OS compared to most other out there.
Ubuntu on the other hand, is heavy enough to sometimes make my Core i7 wish it had more CPU-cycles. Putting Ubuntu on underpowered machines is something I wouldn't recommend to anyone.
I'm not disagreeing with your general point, but I wonder if a lighter DE would make a big difference. (Unless you were already using Xfce, and got the results you describe... that would be bad news.)
Android obviously handles low memory situations better. Ubuntu can't match its ability to free up the ram used by inactive apps.
Don't forget who this is aimed for. Alas, Ubuntu is still mostly for folks above some level of computer proficiency. Android, on the other hand, is probably one of the most "familiar" experiences for people using smartphones, and the (huge) amount of applications is easily accessible through the Play store with lots of reviews and hence high quality. You just don't get that for Ubuntu.
There are lighter Window Managers, but most of these don't give quite such a modern , easy and complete user experience.
Even with that though, the kernel not getting updates and continued patches makes it harder to keep things built for an end user, but I'm not sure if that is so much of an issue past the Linux 2.x to 3.0 jump with Android devices.
Updates for anything Android outside of Nexus devices that Google maintains are generally far from automatic or dependable.