I concur with your technical argument, except for the trust of what's being built. I have repeatedly found the shared state cache to be the achilles heel of my builds. You never want to have a false-positive hit in your cache, yet yocto has repeatedly bit me with that. I have had instances where bitbake claimed it was done, just to find wrong or missing artifacts in my image. Cleaning and rebuilding from scratch is a continuous exercise that's IME sadly necessary. ...let alone letting others use my build directories!
I accept yocto's right to exist, but I'm far from being happy for using it.