It's not that bad. The only thing Google/FB/Twitter gets to know is that a user Y is using app X. Nothing more, and not detailed usage stats, just the basic fact.

For that they handle the complete user registration, recovery & auth process for you, with all the work and pain attached to it.

Granted if your OAuth provider were really evil, they could log into the users App X account and access whatever data he has inside the app, so you have decide if that a concern or not.