My thoughts are "abuse is bad" and also "there hasn't been enough history of abuse to really worry". There have been lots of allegations, but nothing documented since Nixon. At least nothing documented according to that wikipedia article.
That said, I'm fine with punishing people who abuse the IRS.