Sadly, it's not always that simple. If one of your well-pinned packages has a build dependency which is not correctly pinned, then you're subject to the problem.
You raise a great point. I believe cached builds, say with Nix, are the way to go. What do you do?