There is independent oversight through the tax courts but your typical person off the street can't afford to defend themselves from the united states government.

You have an idealistic assumption that people who are targeted for audits or fines are all guilty of tax fraud. This isn't true. Sometimes is just a matter of them losing supporting documentation.