They cause price distortions for consumers by creating state-sanctioned monopolies. Monopolies are a type of market failure and do a great deal of harm. The commonly cited goal of giving monopolies to little inventors and helping the underdog is not a good justification; it doesn't create any value for the user/consumer and ensures they will have fewer choices in the marketplace.
Don't fool yourself, it's not patent trolls that are the problem, it's the whole system. The only innovation I want to see in patents is a huge worldwide lobbying operation to abolish them and make sure everyone knows how detrimental they are to consumer choice.