Because the batteries wouldn't have to be tested while you wait. Like laptop batteries, they could have their own embedded controllers that would keep track of each individual unit's charge state and condition. (Actually it's almost certain that they already do.)
The analogy I like to use is swapping out propane cylinders for a gas grill. It should take less time to refuel an EV than a gasoline-powered vehicle, not more.
This is also important for power management, if Musk's prediction comes true and half of all cars sold by 2025 are indeed electric. Swappable batteries can be charged when/where the power is available, using industrial rates and infrastructure. Instead of seeing gasoline tankers driving around, you would see trucks delivering and picking up batteries from service stations.
People will understand why hardwired batteries are a bad idea as soon as half their neighborhood tries to charge their EVs every night the way we currently charge our smartphones.