We can’t hope to colonize even a fraction of 1% of our galaxy, even with wildly futuristic technology.
There is one option, though: Become our own aliens. If we set up colonies separated by time and distance, we will culturally diverge within a century, and genetically within a few millennia-ish.
If you can do any colonization at all it shouldn't be that much harder to colonize the whole galaxy. Once you've got the ability to colonize exponential growth will take over and pretty soon* you'll be everywhere.
*In the context of the Fermi paradox, we're wondering why no aliens have already colonized the galaxy (including our solar system). So by "pretty soon" I mean relative to the time interval between when such aliens might have appeared and now.
If you look at the statement you yourself make, it requires a lot of assumptions:
* Interstellar colonization is possible (this is explicitly taken as a hypothesis, though)
* The ability to colonize another star system is the ability to colonize a large number of star systems. This is to say, the social consequences of colonization are not going to grow larger with size. A possible counterfactual would be that large civilizations might put more energy into securing control of known systems rather than embarking on further colonization--and the anecdotal evidence from the history of Earth is that this is indeed the case.
* Colonization is a logistic process (which is probably what you really meant when you said exponential, but the difference is a technicality for this discussion). If interstellar travel is really as difficult as our knowledge of physics suggests it to be, then interstellar colonies are likely to develop as completely independent civilizations without maintaining strong ties to their parent motherlands, and subsequent colonization efforts are going to include attempts to "colonize" already-colonized systems and consequently logistic growth wouldn't be achieved.
ADDED: There are also questions of resources, motivations, and e.g. whether we'd even want to build self-replicating probes.
Life on Earth consists of Von Neumann machines. That's exactly what life is. Yet our life has not converted the entire mass of Earth into itself, and there's still plenty of places where you can't easily find signs of life. And our life has had billions of years. The assumption that Von Neumanns will very cleanly and exponentially increase in number until they are all over the galaxy is too ideal.
And that most if not all probes in every generation will even find something other than empty space because the universe is very very very big and the likelihood of traveling in any direction and ever hitting anything is very very very small.
Although if you assume the universe is simple enough to describe on the back of a napkin then it probably is straightforward.
But also experts claim life only arose once on Earth. So, in a place with perfect conditions, the perfect distance from the sun, the perfect atmosphere, perfect trigger, perfect environmental conditions, life only started once! Doesn't that make it so infinitesimally unlikely that even with millions, billions, of planets with the right conditions we still shouldn't expect abiogenesis to occur.
The same arguments that make for extraterrestrial life surely stand against the uniqueness of anything. We know things that are unique. Seems like it's mostly faith based assertion.
Maybe it started twice or 100M times but 100% of those times the newly started life was "consumed" in some way by already existing life etc.
The especially cool aspect of searching for life elsewhere in the solar system is the possibility that we'll find other variants that dominated.
If we are just one of billions of places that life is going strong then we’ll do the best we can but if we don’t survive the eons who cares.
If Earth is the one and only seed of life, then don’t we have an incredible moral imperative not to wreck things here, and if possible to spread life into the darkness?
Being truly alone in the vast dark is a much scarier idea to live with.