> Using sub-agents means that they can't use that cache

They can, when they're forked off the main session instead of spawned from scratch.