Also, it sounds like you're referring to the penalty for underpayment, which I believe is currently 3.6% of the total underpayment for the year [0]. This only kicks in under certain circumstances.
Because this year I got hit by some underpayment penalties (with interest added on top) but at the same time I also got a refund which had no interest applied to it.
That is being self employed and paying estimated taxes every quarter.
I realized that several years back, I had worked at a temp job for a very brief time and I probably never got a W-2 and therefore never paid taxes on it. Thus, I called up the IRS, got the W-2 information which they had, and duly filed an amended return with a check for something like $20.
However, in a rather short amount of time, I got a letter from the IRS stating that they could not accept a payment for a return more than a couple years old and here was a check for the amount, plus interest.
So I can tell you they not only pay interest on your money, but they pay it on money they are returning even though you actually owed it.