I had the 4.65" Galaxy Nexus; it was still usable but just barely. I'd have a hard time being convinced to go any bigger (I'm now on a 4.3" Razr M; no fault of the SGN, I switched networks and can't stand TouchWiz)
Totally, I went from the Droid OG to the Galaxy Nexus and swore I would never go any larger, although it feel's kind of small now after using a Galaxy Note on and off!
The Razr M has a nice form factor, hopefully we can look forward to a Motorola Nexus in the future.