Sony is making a smaller Android device, the "Z3 Compact," exactly for the reasons you list.
Although, with the recent announcement of Android Lollipop, the inevitable next question will be "when is the update going to be rolled out to phone [X]".
[1] http://www.sonymobile.com/us/products/phones/xperia-z3-compa...
What we should strive for is choice.
I personally find the 4.3-4.7" range compact enough, especially if the bezel is minimal.