Say the TV finds such an access point and uses it. It phones home and so now the manufacturer now knows that such and such serial # TV is located in that neighborhood.

They still need to tie it to you. If you've not given it a shred of personal information, it has to be fingerprinting via surreptitious audio/video recording, or else by obtaining the purchase info from the retailer (like the name and billing address from your credit card) and matching that by serial number (which would have to be recorded during purchase).