Interstellar travel doesn't have to be slow for the travellers. You could get to Alpha Centauri and back to Earth in a day (at least as far as relativity is concerned), but for those who stayed on Earth, a decade would have passed.
That's the twin paradox btw.