Let's not forget that a huge part of the pitch was to be part of a developer community that sets out into an as of yet unknown realm.
I don't think people are worried about what Facebook might do to Oculus, the product. They have a hard time seeing Facebook the social engine and Oculus, the developer community coexisting peacefully. In a way, they're already making a rather concrete case against it.