It was never that bad in my case, but the local IRS-equivalent set up two special audits of my personal business within a few months of each other, both times finding that I did everything 100% correct and everything was perfectly in order (no surprise there, I'm paying a tax consultant to handle everything, and they're very conservative).
Still, each time it created a lot of stress because you're essentially in the hand of some random bureaucrat who can make your life hell if they choose to with no oversight and no recourse -- they can single you out for audits whenever they like, and as often as they like, you only get your day in court when they claim to have found something. I did the same thing Louis did: I moved to a different state. And it worked in that regard: not a single audit in the past few years, they're happy taking my money.
That stuff is demoralizing, especially when it's combined with open corruption and support for organized crime in the same institution that's going after you.