Instead, I think need to find some way of generating a lot of power over extended periods of time if we want to crack space travel beyond lobbing items into Earth orbit. Particularly for human space travel.
We need significantly more energy density than what chemical motors can provide, and nuclear is the easiest and most "within-reach" that we know of. Things like anti-matter and even more exotic forms of propulsion are pure sci-fi for the time being (in terms of practicality). I suppose fission could work in a very primitive way of basically detonating small nukes behind a pusher plate, but fusion feels so much more elegant (at least to a layman). Mostly in the sense of not having to use effectively the same nukes we stick in weapons. Something tells me that this will never fly (excuse the pun), for obvious reasons.