2005: Company X sells product Y to person Z for $1000. Company's books show $1000 in revenue. 2006: Company X gives Z a free significant upgrade to product Y and records no revenue.
The argument could be made that what really happened was that the company actually made two sales to Z; one in 2005 and one in 2006 and that the revenue of $1000 should properly be split across the two years.
Not having such a rule apparently allows various accounting shenanigans in which totally unrelated things are treated as "free upgrades" to something else and revenue can be booked in whatever year you want.
So there are two important things here for all this to be an issue: 1) The initial purchase must have cost money. If you get something for free and then get a free upgrade to it, there's no problem. 2) The magnitude of the upgrade must be such that one could be accused of slipping an unrelated product in as a "free upgrade".
So for example security updates to safari would fail test #2 above. Major updates to iTunes may fail test #1, since iTunes is generally available for free to everyone (even if you don't buy a Mac or MacOS).
It's possible that something makes Apple's accountants feel that XCode is part of the OS in a way that means that when you pay for the OS you're also paying for XCode. And that XCode 4 is enough of a change from XCode 3 that if it were free it would fall afoul of the accounting stuff above.
Again, I'm not an accountant, so I could be getting all this totally wrong...
We're not sure what the litmus test is for their accounting practices. We're not even sure if Xcode falls into that category (e.g. maybe they're charging $5 for it now because they want to). It would be nice if Apple shared their accounting algorithm as that would clarify things.
Can you seriously imagine Steve Jobs being told they he has to charge for something he doesn't want to because of accounting practices?!
In its defense it was initially put forward as speculation ("what about this theory"), but now is being taken as fact.
As an example from a few years ago, when I hired a contract developer, the company required a "Contractor Request Form". Fine. There's payroll to setup and so on. But then one day the form was renamed to be the "Contractor SOX Compliance Request Form".
I suspect this exercise was only so that some bean counter could show the outrageous costs of "SOX compliance".
Not sure how they deal with people who never purchase anything, but perhaps they don't have to be 100% thorough about it.
Why don't they give it away for free, say with Software Update? Because of the Sarbanes-Oxley Act (which was passed in the wake of the Enron scandal). iLounge quotes an Apple representative: "It's about accounting. Because of the Act, the company believes that if it sells a product, then later adds a feature to that product, it can be held liable for improper accounting if it recognizes revenue from the product at the time of sale, given that it hasn't finished delivering the product at that point."