Your last line pretty much answered your own question.

Most small-government guys (myself included) distrust anything controlled by flawed humans with the power to target and totally ruin an individual at a whim. I personally hate the idea that I could build myself up from nothing and someone could conjure up a story to take my work away from me.

We are just pessimistic when it comes to institutions... yes it can bring great benefit, but you can also lose a great amount. History has shown us many examples of people striving towards utopia and ending up causing unspeakable loss.