Building control require that an architect sign off on things like the size of the beam holding the roof up. If you change that, the architect will have to sign off on the building as constructed before building control will approve it.
Planning permission is given on the basis of external appearance, so you can't change too much in that area - but go look at some planning applications, they're very light on details.
But other than that? If it's your project and you tell the electrician putting in a socket to put it somewhere else, he'll tell you if it's legal under the wiring regulations, and if it is, he'll move it.
The main reason that in large projects builders are required to build exactly to the plan is to stop them substituting cheaper materials. They're under great pressure to do that, because if you allow your builders to replace expensive zinc cladding with cheap ACM cladding, and you ask several builders to tender and choose the cheapest builder, you'll always choose the builder who quoted on the basis of the cheap ACM cladding.
You're right that making arbitrary ad-hoc changes during construction can be expensive, of course. Especially if it's wrong and has to be torn out and redone.
But it's not illegal - if you're in charge of a project and an installer tells you the specified vents are going to be a problem and they recommend a bigger one, nothing in law prevents you from telling them to go ahead and make the change, right there on the spot.
Of course, how often the person with in charge of a large project will be talking to the vent installers while they're working is a different matter...
It of course isn't illegal to make changes - but it is expensive and thus not something you should do. (though some things are much easier than others to retrofit half way in.
I'm talking with some level of experience here. I did a residential project that was obviously much smaller than this but still very expensive (£100+k). Even for this architects recommended a detailed design and a contract that enforced this.
Also, it's possible to request a variation, especially if something wasn't going to work. There is a category called 'non-material variation' which the planning officers are lenient with.
If you got rid of planning (and building control) I guarantee that the quality of buildings would go down. Before building control, many unhealthy buildings were built due to bad drainage, etc.
Also it's because of the financial structure. People invest a lot of money in something with a delayed, sometimes for years. Delay increases everyone's risk, increases interest expense, and defers revenue.