it's a security feature. otherwise my malicious site could check for cdn.sensitivephotoswebsite.com and blackmail you if it was cached already
There's no such thing as non-sensitive content from a CDN though. Scripts are obviously sensitive, styles can be used to exfiltrate data through background-url directives, and anything like images has no benefit being cached across sites.
Fonts might be one exception, but I bet those are exploitable somehow.