But now there are services to lookup the owner mapped to the house. It's not apples to apples here. A house is stationary, there is a 1-to-1 mapping: house number to name. But say you have a wifi laptop and it is established that a MAC maps to you with reasonable certainty, then it's certainly possible to track your movements using that information.
IMO, I find that Buzz messages on Google Maps to be extremely creepy and am willing to bet a good portion of people doing so have no idea that they can be visualized that way.