There isn't a huge market for small portable phones because most people shelling out $500+ for a smart phone want to actually use it as a smart phone. They play games, watch shows and movies, read full novels, and browse the web. These are all experiences that are enhanced by having a larger screen. My wife upgraded to the iPhone 6s from her old phone (5s maybe?) and one of the first things she said about it was that the screen was small, despite it being considerably larger than her old phone.
I cannot understand how the "I want a small phone" crowd don't realize that they are in the minority.