The iPhone SE's popularity is driven by price, not the demand for a smaller phone. The iPhone SE is 5% taller and wider than an iPhone mini. People didn't buy the iPhone mini at $100 cheaper than an iPhone. If they were going to settle for a smaller phone, they wanted half-off.
It allows Apple to be in the $400 market by putting out a device that won't compete with the regular iPhone. If Apple offers an iPhone SE sized device at $700, people don't want it. iPhone SE buyers mostly want a larger phone - they just don't want to pay for it.
I love the iPhone mini, but people didn't want a small phone. They wanted a cheaper phone. I'd love it if this weren't the case. I do want a 5" phone, but when Apple put out a 5" phone, people wouldn't buy it. They'll only buy it if you price it at half-off.