Constant hassle with payment processors deciding to ban smut once you get too big, though.
Not doing something because you might get too big is not sound reasoning.
When you get too big, that's the time to worry about it.
If that was even occasionally true, the tons of paid porn sites wouldn't exist. The fact that they both exist and take popular cards tells me that that cannot possibly be true.
Edit: BTW, a large portion of seemingly independent porn sites are actually run by only few companies. Especially the Xvideos corporation.