Powering those ion thrusters is a bit of a problem at the mass ranges you need for peopled interplanetary flight. Either you have an absolutely enormous solar array, or an absolutely enormous radiator array for your nuclear reactor.
Either way, any vessel travelling beyond Mars or Venus in a time short enough to be safe for humans is going to require in orbit assembly, so size in terms of volume becomes less of a constraint. But as long as we're boosting all of our mass from Earth, that's what costs the money.