If a cost is evenly moderately costly to create, and abusive comments quickly result in bans, then the ratio of abusers quickly goes down to a very low level.
The problem is that making it costly to sign up goes against the business model and growth model of sites like Twitter, Reddit, or Instagram.
The problem is that it makes it hard to get enough of a critical mass to convince people that it's worth the ten bucks.
That being said, something awful did it well over a decade ago [1] and has a relatively low amount of assholes/harassment, but it's not zero.
"A peanut gallery was, in the days of vaudeville, a nickname for the cheapest and ostensibly rowdiest seats in the theater, the occupants of which were often known to heckle the performers." (Wikipedia CC-SA, via Google)
That's a new one on me.
Maybe the cost is paid in non-monetary ways even (can't message much when anonymous) and maybe the cost is along a continuum depending on how much anonymity is requested (email, but no real name, or location and age but no personally identifying info).
I suppose I should point out explicitly that my goal is not to create a 100% comfortable world in which you never encounter an asshole, but to create something a bit more like the real world where by the time you're an adult and you've found a place for yourself in the world, by and large you probably don't have a problem dealing with lots of people-you-consider-assholes personally. (You may professionally, but that's a different sort of problem.) Even if you maybe have a problem with one or two people in particular, you almost certainly do not set a foot outside your bedroom door and immediately get several thousand people screaming at you at the top of your lungs. But you're going to encounter a few, sure.
More on the Reddit model, I outlined something here: https://news.ycombinator.com/item?id=16570017 The observation that it is computationally expensive is true, but I'm not convinced it's intractably so. There the idea is that if some anklebiter starts replying to your every comment, by default you at most see something greyed out or collapsed if you did not have any previous connection to them, and they can't bother you. I think to jumpstart the system you'd always have to have a couple of random comments show up, and some of them may be assholes, but it wouldn't be the case I was describing where you can get this unrelenting torrent of abuse.
(I think such a system would have more manipulation resistance, too. $MEDDLER could go to some political forum that works that way, and maybe they could convince a few people or a subcommunity to upvote them, but they wouldn't be able to completely take over an entire topic by sheer mass the way they can on Reddit. You can't get a world where $MEDDLERs don't exist, but you don't have to have a world in which only $MEDDLERs and $MEDDLER-approved accounts exist on your forum.)
This change of communications with assholes rate brings asymmetry that is a cause of the problem: you still have the same response capacity and emotional response decay time as in the offline world, but the incoming rate of assholes is so huge, so you can't cope with it anymore.
So perhaps any solution should be seen from this perspective: decreasing the upper boundary rate to some reasonably acceptable value.
But it also may lead to the opposite conclusion – we now in an online world, with different speeds and rates of everything. Maybe solutions "like in the real world" simply won't work anymore here. Perhaps we can use some new tools to solve these sorts of problems, that we never had in the offline world?
For example, there was a brilliant idea to set a price tag for your attention in the online world. Imagine an email system where the sender should pay some price for you to receive it and read. This price can be dynamic and vary according to various parameters – from sender's identity to my current mood. If someone sends me a public speaking request or is trying to hire me, they'll pay like 5$ per mail, but for my friends or coworkers asking for help it could be 0.03$. And if my inbox is empty (ha-ha) and I'm in the mood to explore who's writing me, I can temporarily lower the price for all. That's of course, just a rough sketch of the idea, but it might work in the future if cryptocurrencies become ubiquitous for example.
Making this a market seems like an interesting idea.