I liked the Nexus 4 form factor as well. I've bounced around to larger phones, including the Nexus 6, Pixel and Pixel 2... but I recently went back to a Sony device (XZ1 compact) because it:

(1) Has a headphone jack

(2) Has the same processor as my Pixel 2

(3) Is actually a usable size (roughly the same as the Nexus 4)

I have giant hands and large pockets available but will always prefer a smaller phone.

Nexus 4 was my favorite phone - beautiful form factor, nice screen, worked well.

The best phone form factor ever made. I loved my N4.

The best designed Android phone was the Moto X 2013. Perfect design, but a terrible camera.

I mean, the Nexus 5 and 5X weren't /that/ much larger than the 4.