That depends on the modules you use.
For example, many people keep each function in a separate file, instead of merging them into psm1 during build. That will speed it up A LOT.
You can have different profiles and load them when you need them, even when you enter specific folder.
There are ways around it. You complain that you want everything everything everything loaded in advance for your wizardry but are u really a wizard if you can't solve this? So if you buy a porche and then put a house on top of it, shop in cart in the back of it, tires for every season, and so on, and then complain its slow, then guess what...