I just measured our living room. We have a 43" (4k) TV at about 16' viewing distance. Just about every chart I can find on TV resolution vs viewing distance (assuming 20/20 vision) indicates that I'm in the "720p is fine" region, on the border of "You really won't notice a difference from 480p." Which is consistent with my experience. I can see the difference between 720/1080/4k if I'm right on top of the TV, but once I'm back where I actually watch from, anything I notice is more a factor of bitrate than resolution - I can certainly tell the difference between a low bitrate MPEG2 source and a high bitrate h.264 source, but that has nothing to do with the resolution.
I don't really have the motivation tonight to pull out my trig tables (I'm about to watch a movie instead, of what resolution I honestly don't know), but... at least for my case, yes, I'm saying I can't tell the difference. And it's entirely possible my glasses need updating (quite likely, honestly), but the "TV size vs distance" charts I've found say nothing of what you're claiming either. And they're by people trying to sell TVs!