Even if it took a few million years to do, that's still far less time then the Earth has been habitable and had life. So the question of course is still where are they?
Thus, each expansionist civilization will spread out to encompass an exponentially increasing number of advanced, stable civilizations, and antagonize an increasing number of them. Soon, they will encounter one that chooses to put them in their place, and maybe just wipe them out.
So we may think of the galaxy as an archipelago of stable civilizations that interact little, with splashes of expansionism in between that encounter and annoy them, and are swiftly contained. Such expansions are exceedingly unlikely to get to us before such an event curtails them.
I.e. how do you destroy an expansionist fleet which has scaled up to the point that it's furthest point from you is increasing close to the speed of light?
But they are, remember, incomprehensibly more advanced than you or me. And they know a bunch of other incomprehensibly more advanced civilizations all the way on the other side of the expansionist noobs. And, just because they don't usually see much point in going out there doesn't mean they can't, or won't, when the time seems right for it.
I can come up with one respons within the believed laws of physics at how a hypothetical stable civilization could suppress exponential ones even though exponential ones would quickly be expanding their radius at close to C: the stable civilization would need to pre-seed the visible universe with idle hidden outposts of a sufficient density to detect and suppress exponential expanders.
We are not predicting the future, but trying to account for an observation. For the scenario to remain on the list of reasons why we haven't been visited, it suffices that it is possible. There are of course lots of other possible reasons, among them that there is nobody to visit.
That come around us since we're trying to explain why we haven't seen an expansionists. So this line of thinking predicts that we're in the shadow of a non-expansionists, I guess?
This is not necessarily true. A roiling cluster of ephemeral, competing life forms and civilizations may very well develop warfare and defence technology better and faster than stable civilizations. This is especially true if you consider the cost of stability: all new technology is a potential threat to stability that has to be weighed carefully, so I expect stable civilizations to evolve much slower than unstable ones. In the aggregate, instability is not really a problem: a civilization's progress will be scavenged or stolen by its successors, retaining the advantage, and for that reason I would argue the unstable regime will almost systemically overwhelm the stable one.
You can kind of see this with humanity: a lot of the big problems we have to face as a civilization (but not all of them) result from the careless adoption of technology. Climate change would not be a problem if we had industrialized slowly enough. The Internet spread too fast for us to understand its effects on the propagation of (dis)information. A truly stable civilization would have taken a thousand years to do what we did in a hundred. Perhaps it would be happier for it, but if we met them, we would eat them alive.
Puny humans are confined to one solar system, and even just the one planet, but are pretty dynamic. Population in wealthy countries is in decline.
Remember we are not just talking about humans with bumpy heads, here. Social universals on Earth do not apply. Expansion might be limited just by a physical inability to reproduce beyond their home planet, without giving up any dynamism.
You described barbarians. But barbarians don't need Von Neumann probes, they can do everything they want with a stone hammer.