1) hardware engineering: there is no dead space in a phone, every single square inch is used by something. The PCB is as compact as possible. If you design a modular phone (like project Ara), you easily double the space required. Instant losing proposition: less efficient, more bulky.
2) market: even if you offered a wonderful selection of decent modules, market reality means that 90% of your customers would use no more than 2 or 3 different configurations, at most. If that's the case, forget about modularity, just build those 3 configs and optimize them to death.
There's also a lot of possibilities with unusual add-on for scientific measurements, add-ons for employees in service industries, add-ons for hacking(think of repurposing your phone as a microcontroller similar to RPi), etc. There's a very wide and diverse market waiting to explode from the possibilities.
So, in conclusion, completely unrealistic in the near term, or our lifetime.
In practice, it's easy to buy a new phone every 1-2 years, and give the old one to a relative.