Like, the iPhone? Or maybe a Lumia 800? They both fit comfortably in all mens' pockets (and most women's, assuming they have pockets).
The problem is Android is such a power hungry OS that you need to be constantly updating the specs as hardware and app demands grow higher, and high-end specs require space to fit everything in (including ever-growing battery demands), so small Android handsets have to make compromises on specs. It feels sort of like PCs did 10 years ago - 3 years of use and your top-of-the-line PC was near unusable. I find Windows Phone and iOS don't suffer from this nearly as much.