Think of how horrible it would be if he did something wrong. The average American knows that if they mess up 1 box, as long as all the final numbers come out to about what they should (which is easy to check), it's likely no one will notice.
But someone will notice if the IRS commissioner messes up his taxes, even if it's just forgetting to initial a box, and it would look horrible. Can you imagine how much fun John Stewart would have with it? He'd probably be able to avoid resigning, but his chance of being nominated after Obama's reelection would be basically 0.