And yet we have no way to know that for sure. We have no idea what the average lifespan of an intelligent species is. It may just be that humans are particularly short-lived, among all the species in the galaxy, or the entire universe. If a species has a lifetime of a couple thousand years then interstellar travel, even at sub-light speeds, would be a lot more manageable than it is for us.
Which to me, means we can relax a little about the risk of being destroyed by hostile aliens. We don't know what we don't know. Chances are, if they were going to destroy us, they would have already done so in the last million years or so. We're probably a lot less appealing, as a world, than we think we are. Perhaps the universe really hates salt water or oxygen atmospheres? Who's to say?