Sure, you may lose out of the "faster" part if you have to pull from a remote cache (both inputs and outputs) instead of your local one. And maybe just blindly relying on the community maintained caches isn't right for you, so now you're still in a position maintain a cache and back it up through traditional means. It's not a magic bullet, probably doesn't pay off for a lot of use cases.

But what it buys you is a sort of verifiability that you can't really get any other way. Anyone can build any part of the system from its declared inputs and say "hmm, I got a different hash than you, maybe something is up." If you're restoring from an image which somebody has tampered with, then the image becomes the source of truth, rather than the source code, and it gets a lot more difficult to scrutinize its validity because the pool of available scrutinizers is just the people who care about your particular image--that's likely to be a lot smaller than the set of people who care about whatever sources you're relying on.