Even at the speed of Voyager 2, (15km/s), in a billion years it would cover about 4 * 10^17 km or about 40 thousand light years. I am pretty sure people who have been developing for hundreds of millions of years can do better than that, by orders of magnitude.
Also you put a totally strange assumption that all civilisations will only restrict themselves to visiting planets. If there is a lot of civilisation it is more likely they will have different motivations.
For all we know, there could be a gigantic tungsten rod slowly and purposefully accelerating toward us that would destroy all life on earth in ten thousand years, and we'd have no way of knowing.
Although I guess they could also just paint their weapon black.
A different matter how they hide the engine's infrared signature. Perhaps by just not running it when approaching, except for the final course corrections.
Imagine you are in impactor frame of reference -- any particle of dust it meets will cause tremendous damage to it regardless of what the impactor is built of (assuming our existing technology).
It is in my opinion very likely that over long distances the impactor will just get converted to a shower of high energy particles over very large area.
https://en.wikipedia.org/wiki/Stellar_kinematics
Gliese 710 is probably going to cross Oort cloud in 1.29M years. Perhaps we should wait and hop to this star?
https://en.wikipedia.org/wiki/Gliese_710#Computing_and_detai...