My go-to counterargument for this is that galaxies are really far apart (from us to Andromeda is 25x the diameter of the Milky Way). If you haven't developed FTL travel, it's a long haul with very little benefit on the way.