What you are looking for is a local S3 cache, which buildx supports as a backend. Just make sure you have an S3 gateway connected to your VPC (and that your S3 bucket is in the same region than your runners!) and enjoy free bandwidth, unlimited cache size, and crazy fast network throughput.
https://runs-on.com/reference/caching/ https://runs-on.com/features/s3-cache-for-github-actions/