Currently, SRI is not enough for browsers to implement content-addressable storage as you describe here, because it is subject to cache poisoning. See https://news.ycombinator.com/item?id=10311020 - basically, the browser can't know if a script can actually be loaded from the claimed domain without requesting it. This can be used to violate CSP.
Though it would be nice for the browser to cache it for domains that have delivered the script previously. It wouldn't be that different from a normal cache except the timestamp doesn't matter.