You are also trusting the Oauth provider to never login as you for their own inscrutable purposes.
Or as the case of twitter demonstrates, you're also trusting all future owners of the Oauth provider, whoever they may be. If an erratic billionaire with a penchant for breaking the rules whenever it suits him buys your Oauth provider, who's to say what he'll do with his new access? He could treat your accounts as his personal toys. Better hope you don't earn his personal ire when he's on another wine and ambien bender.