However, the going assumption of the Fermi thesis is intelligent life is different. Intelligent life forms civilizations that are expected to naturally expands forever, eating planets, stars and so-forth.
So far, human behavior looks much like the other phenomena that expand to the limit of their fuel and then die.
And, if somehow against all odds, we wise up and stop exhausting the resources needed for our existence, well, then that infinite expansion tendency quite likely won't be there.
"If we humans are now on the cusp of colonizing our solar system, and we are not much faster than other civilizations, those civilizations should have completed this colonization long ago and spread to other parts of the galaxy."
But what if we aren't on the cusp of this colonization, if one can argue such colonization would happen by quite a few technological advances that are purely hypothetical, that "colonizing the solar system" only seems plausible by incorrect analogy with the colonization of various areas of earth by various local societies.
Space seem wholly hostile to human from it's lack of gravity to its lack of all other components needed by us. Our era has gone from believing "space the final frontier" to seeing just how fragile our condition on earth is.
Are you sure about that? It seems to me that humanity has often made a "leap" in the past... Romans "fed" on slaves, then collapsed, and eventually we found coal. The industrial revolution "fed" on coal, then we found "oil", there's probably the next thing!
The only thing not justified is the expand forever thing.
It just takes one civilization with a runaway von Neumann probe to drop huge monoliths on every single rocky body in the Galaxy over a couple millennia, and since evidently nobody did that in our solar system, the question is why not. If there's a rule that forbids it, what is the rule and does it apply to us? That's the question posed by the Fermi paradox.