USB by itself is a terrible answer. The connector is small and weak so you can't hang a peripheral device from it without risk of breakage. The only way this would work is if phone manufacturers agreed to add mounting rails or magnets at a standardized position near the USB port in order to provide a secure and robust mounting point. But realistically I doubt that the major manufacturers would ever cooperate on that type of standardization.

Dongles slapped on after the fact are a poor design. They ruin the ability to hold the device one handed, the ability to pocket the phone with the module attached, or add substantial bulk to cases that must be redesigned for every variant of phone.

USB is a decent electrical interconnect and API standard... But not a good cellphone mechanical modularity standard.

Possibly due to the placement of USB on most phones - at the bottom, where a drive would stick out awkwardly.

Also, MicroUSB on a lot of devices doesn't seem like the most robust port ever - I think it would eventually wear out with more than just the nightly plug/unplug for charging.