I'm also reminded of Iain M. Banks' Outside Context Problem [2] in a number of different ways.
The first is that space seems, from the human perspective, to be impossible big with even the nearest things being almost impossibly distant. The optimists argue that technology will solve that problem (probably whilst imagining a Star Trek like future) but the laws of physics paint a far bleaker picture if you look at just the energy cost to get to our nearest neighbour even assuming you solve the reaction mass problem and have perfect mass to energy conversion, the problem that even the smallest piece of matter (and eventually even hydrogen atoms) become deadly obstacles at even a modest %c and so on.
The second is that given the abundance of planetary systems we've already detected, the size of the galaxy (and its age) and the very real possibility of constructing self-replicating machines with something not that much beyond our tech, it seems strange that we haven't seen evidence of this.
Anyway, back to interplanetary colonization... given the relative distance to Mercury (6-7 years at current tech for an orbital intercept) such a colony would of course be essentially cut off from the Earth so would need to be self-sufficient (saying nothing of the problem of building a colony ship that could even get people there and keep them alive for such a long period).
I think about it this way: what is the "footprint" of a single person in the developed world? By this I mean we all need food, power, material things and the like. For each of those things, add in all the people required to produce, deliver and service those and keep adding those people until you have a group that is independent and self-sustaining.
Primitive people have a relatively small footprint, requiring a relatively small group but a large amount of area per person.
In the developed world, to maintain anything like our current existence seemingly requires a good portion of the planet. That's a problem for any kind of colonization effort.
But at the same time that interdependence reduces (IMHO) conflict. Imagine a world where 100,000 people could be self-sufficient and effectively cut themselves off from the rest of the world? It seems like a recipe for disaster. It seems like a recipe for creating a technocratic elite and the kind of social divergence that would ultimately create a new species (at first culturally).
So for a Mercury colonization effort you'd need to take enough to establish heavy industry in a hostile environment (assuming you'd mine what you need rather than carry it there), build habitats, food production and so on. It quickly spirals into an impossibly large effort.
Colonization in human history to date has happened at far lower technological levels where transportation and communication were (compared to space travel) ridiculously cheap.
What we probably need is automated, self-replicating heavy industry. This way we send an initial package of robots to Mercury. They build energy sources, habitats, mines, etc without the huge cost of keeping humans alive. Need more robots? They build those too.
Sound familiar? You're only one step away from the self-replicating robots that can (and apparently haven't) colonized the galaxy.
So yes investing in impact defense seems prudent. I don't know what we could really do against something that's 20km across though. That's an awful lot of mass to move out of our way.
What we really need is something that is a large part of artificial intelligence and self-replicating machines to do our work for us. This seems to me like the key to our long term survival and something we'll need to spend significant effort into developing.
As an aside, I tend to agree that the desire to colonize Mars is somewhat misguided but, mistakenly or not, Mars shares a lot more in common with us than Mercury does. It has an atmosphere (although not a terribly useful one). The cold is something that we, as humans, can and do deal with. It's also closer to Earth (~8 months at the right time).
I personally find the rover effort to be useful as the basis for building machiens that can survive in hostile environments for long periods of time, if nothing else.
[1]: http://en.wikipedia.org/wiki/Fermi_paradox
[2]: http://en.wikipedia.org/wiki/Excession#Outside_Context_Probl...