They wouldn't need generation ships. You "just" need sufficient tech to put engines on a suitable astroid and accelerate it to a decent fraction of c; it only needs enough logic to fine tune the path. An invasion force is horribly inefficient vs. just slamming a lot of mass at someone at high enough speed.

And it only takes one massively paranoid, xenophobic species lobbing big rocks at people to ruin the entire neighbourhood.

I don't think camouflaging necessarily means not leaving your gravity well, but if there's someone lobbing big rocks at potential threats, then the only ones exploring will be the ones powerful enough or good enough at hiding for us to be unlikely to spot them unless they want us to. Everyone else will be dead.

It's one of the more compelling answers to the Fermi paradox to me, while at the same time being profoundly depressing. But it's less depressing than the chance that there might not be any other civilizations.