Sure trolls will support other trolls but I'm pretty convinced most people are good people and should easily outnumber and be able to take down gangs of trolls online.
This way the community self moderates and is able to weed out the undesirables and assholes on the platform all by themself, no intervention required.
The people you rated highly could also float to the top of your stream so if you enjoy the trash talking of milo or similar characters you can still see them without them poisoning the well for everyone else.