Dota 2 addresses this specifically with a time-sink. A minimum number of games is required to play competitively, which increases the cost related to "new accounts at will".

In addition, even for non-competitive games, players who obviously troll and are flagged by the community are pushed to a separate pool with other trolls.

The system isn't perfect but appears to work fairly well for new players.