When using the phone as, well, a phone, I find smaller phones annoying. I’m never quite sure if the mic can pick up my voice if the earpiece is near my ear. I’m also much more likely to drop the smaller phone.
I’ve been this way for a long time. An early phone of mine was the LG Env which opened up and had a large-for-the-time QWERTY keyboard. It required two hands even way back in 200x.
I doubt that I’m representative of the average person in this regard. But I doubt you are either. I suspect for most people one-handedness is a desirable trait in a phone, but not as strongly desirable in the average person as it appears to be for you, so it just gets eaten by trade offs.