I got it working, with intermediate layers, too. All to find that I didn’t see that material a performance benefit after taking into account how long it takes to pull from and push to the cache.
There's also an action out there "GitHub cache dance" that will stick your whole buildkit state dir into the gha cache.