I have the same issue. I’m on a Nexus S as well and it’s starting to show its age with the latest operating system and apps. But the form factor is great, and all these newer flagship phones are huge.
I'll be getting the Nexus 4 once it's out, but like both of you, the screen size is the biggest drawback for me.
I really hope Google realizes that bigger is not always better!