I have this set up in our pipeline, we also build the image early and use assets to move it between jobs. We've also just switched to self-hosted runners, so might look into shared disk.

But in the long run, as annoying as it is out build pipelines reduced but quite a few minutes per build.