As it stands, if any app wants access to your contacts, your camera, your microphone, your photos, etc, it must ask first. Allowing sideloading removes this protection, and apps installed that way could simply siphon all your data silently.
You even describe how that can be the case, but you couch it as being with third-party app stores. While what you say is not false, it is also not limited to that case: the removal of both privacy and security protections happens as soon as you stop having the App Store be the sole source for iOS software.
Yes, of course, a hypothetical "Facebook App Store" with all Facebook apps being exclusive to it would have a higher chance of getting nefarious data-siphoning apps onto users' iPhones than any old random sideloaded app, but it's hardly a necessary part of the threat to privacy. It's just a way of guaranteeing much more widespread compromises of privacy.