> How would they do this if you never visit the site or download the app?

Facebook SDK is used in more than 40% of mobile apps. Devs do it for analytics and ads. The feature is called Facebook Audience Network.