The only pain point in switching I can think of is issues/wiki. Are they exportable from GitHub and importable into something like BitBucket?
For issues GitHub has great API coverage for fetching them [0] so I'd imagine it's not terribly difficult to write a migration script. There are already ones out there for going from BitBucket to GitHub [1] so the reverse should be possible.