I feel like neutering the IRS is going to have long-term negative impacts on the country. I hate taxes as much as anyone, but I do think the IRS has been a force for good in our country. Not just by ensuring that wealthy people make investment in our society without the expectation of direct financial return, but also by virtue of being a powerful enough to tackle corruption.
The anti-tax rich are totally fine with continuing to shovel government money into their pockets, via contracts, grants, etc. Which suggests that a greater share of the cost burden of enriching these people will fall to regular Americans.