Actually, there are third-party services that have made a business of collecting locations of Wi-Fi routers and I believe Google subscribes to them.
http://gigaom.com/2010/09/15/skyhook-sues-google-in-a-locati...
I speculate that further routes for collection may be available. E.g. a user uses a wi-fi connection with a device that has geo-location turned on, and "Bob's your uncle".
I'm sure pioneer map-makers got in trouble for plotting perfectly visible stuff that was located on private property.