What I pointed out is not some iron-clad argument that it's impossible for there to be galactic empires or that people with reasonable assumptions can't be surprised that the universe isn't teaming with life.
My point was that it's also not unreasonable to not be surprised. After all, we know for sure that, even if the universe is eventually full of life, there must still exist at least one species with our experience - looking out and seeing no one else. That one species guaranteed to have this experience is of course the first one that starts colonizing.
To be clear again, I'm not claiming that we are for sure the first ones - there are many other reasonable explanations, but I believe this one is too.