There is some level of data exposure even if you can't tell which site, specifically, a user has visited. You can tell that "this user has been to at least one page that used X".

Depending on what the common resource is, most people wouldn't necessarily care but there may be some cases where they would.

If I can tell that you used facebook login at one point, even though it is used by a good chunk of the web, I could tailor a phish targeted at FB credentials for example.

No need. Just make a custom asset referenced only by those sites, and get the browser to cache it.