How so? I just tested a build, and it used the cache for every layer including the builder layers.
It is of course possible to do for a single project with a bit of effort: build each stage with a remote OCI cache source, push the cash there after. But... that sucks.
What you want is the `max` cache type in buildkit[1]. Except... not much supports that yet. The native S3 cache would also be good once it stabalizes.
It worked wonders for us, on a cache hit the build time is reduced from 10 to 1,5 minutes.