Unlike Tesla, Spacex is not a threat to any other publicly traded company AFAIK. So the barrage of negativity will be minimal.
For every incompetent investor who comes on board, you'll have a great investor who'll trust you. This is the case in Tesla too. I mean, the odds that most of your investors will be incompetent is fairly low.
It's not that much of a time sink, if you are running just one public company. Running two is a miracle. Jack Dorsey is doing it, so I wouldn't put it past Elon.
All said and done, given a choice, Musk will keep Spacex private forever.