Cache across domains with a file hash | Hacker News Reader

Cache across domains with a file hash

alexchamberlain.co.uk

62 pointswgx36 comments

I've coded something similar up in WebKit and considered landing it in Chrome, although I was more concerned with being able to use HTTP resources securely in HTTPS pages.

Firstly you need a way to incrementally verify a file: you don't want to have to download and buffer the whole file before figuring out whether it's good. Thankfully you can do this with Merkle trees: just make them `degenerate' (every left child is a leaf node).

If you make the browser cache content addressable (as the post suggests) then you need to do more than look at the caching header. Consider a site with Content Security Policy enabled: If an attacker found an XSS, they could inject <script href="http://correct.origin.com/foo.js digest="degen-hash:sha256:123abc...">. That would match the CSP origin restrictions without the server ever having to serve foo.js. (Thanks to abarth for pointing that out.)

So I believe content addressable caches would need a separate hash:// URL scheme for this. But I didn't attempt to make the cache content addressable in my code.

I don't know what to do about img srcsets[1]. There's no obviously good place to put the hashes in the HTML.

And lastly, and possibly fatally for the "secure HTTP resources in HTTPS pages" use case, many networks will now transcode images on the fly (and possibly other files too). So matching a known hash will result in broken pages for users on those networks.

[1] http://www.whatwg.org/specs/web-apps/current-work/multipage/...

I deliberately avoided images in my proposition, because they are rarely shared across domains and are frequently targeted by dodgy proxies.

I'm not certain that Javascript isn't similarly `optimised' by such networks, but images are significant for the mixed content case.

I believe that we collected some data in Chrome about the efficacy of a content addressable cache in improving density that was underwhelming, despite expectations. I don't have it to hand, but I suspect that it'll appear in a paper at some point.

I've not seen data about how well it reduces network loads yet, although I believe the same colleagues were intending to collect that too.

The fact that JavaScript would break on networks that transparently rewrite JavaScript is a feature, not a bug. If hash codes were implemented by all major browsers, those networks which tamper with JavaScript would have to stop doing so pretty damn quickly, or face a lot of angry customers.

As it is, I think customers would already be angry if they knew that their network provider was rewriting code that will be executed on the customer's computer; anything which makes this more obvious can only be good.

I'm afraid users would be angry, but they would be angry at me. Blame is assigned to the last thing to move.

I'm not claiming that this is good, but it is the reality of writing browsers.

> Firstly you need a way to incrementally verify a file

Following the proposal of the article, you don't really need that. If you don't have the hash in your local cache, you should assume that the link provided by the site is correct and resume as you normally would without the hash present. When it's downloaded, you verify the downloaded contents with the hash, and add the file to the cache if the verification succeeds.

If you already had the hash in the local cache, then you already have the entire file and there's no need for incremental verification.

As I understand it, the point of the hash is purely for caching purposes, not strictly for validating that the downloaded file matches it.

> As I understand it, the point of the hash is purely for caching purposes, not strictly for validating that the downloaded file matches it.

That is a very useful consequence that prevents malicious content.

Anything like this is going to need a mandatory Cache-control header that indicates whether the content can be shared cross-domain. Otherwise you can just probe the cache for arbitrary resources, similar to stuff like the a:visited browser history attack.

Or have we just given up trying to deal with all the privacy leaking side-channels in browsers these days?

Another way to do this would be to allow multiple alternative src urls for a resource. Then the browser could pick the best (for as yet undetermined values of best) source the resource.

This has the extra bonus of advantage of looking mostly like a normal <link> or <script> tag and doesn't introduce yet another way to probe across domains.

I'm sure you're right, but I'm not sure I understand.

It would only apply to scripts you add the `sha256` attribute to and you would probably only do this for public libraries.

I can add whatever SHA I want to the tag though. This means I can tell if you've viewed specific images or other resources on 3rd party domains that wouldn't otherwise by in my browser history.

Unless the original content source has opted into this kind of behaviour this seems like a privacy leak to me.

Only links with a hash initially get put in the addressable cache. So if I download foo.js without a hash in the script tag, it is effectively a different document than foo.js with the script tag. The cache address hash would include the file hash, if you will.

This would effectively mean the person serving the files gets to determine if you are succeptible to this. If I serve you foo.js and don't want others to know you've seen it, I just make sure not to put a hash.

Although, overloading the hash this way sounds wrong. I'd rather see a "shared_cache='public|domain|none' property. I still like the consuming HTML being responsible for deciding that, though.

I think it would actually make more sense for the sha256 to be in the headers for setting it. It seems a bit weird for a property on a link tag pointing at a file to determine what we think the hash of that file is (not to mention quite un-fun to maintain).

Of course, that means without duplicating it you would still need a request similar to If-Modified-Since - something like If-Hash-Differs: <sha256 hash>, but still better than downloading the whole file.

I've thought about this exact problem before and one nice way to solve it would be through some kind of signing process - so you instead tell the browser "I need com.jquery.1.7.2" and there would be some way of verifying that a given instance of that file was legitimate, and could thus be cached across multiple URIs.

I agree the thing serving foo.js should be setting its sha hash in the headers. The client can validate that the header hash and the actual contents match, if it so desires.

On the web application side, by putting a "sha" tag in the script tag, the app is saying, "Fail if foo.js isn't the one I expect, regardless of where it comes from." By putting a "shared_cached" in the script tag, the app is telling the client where it is allowed to look for and to put foo.js.

Finally, a client should be able to ignore any shared cache requests and make everything private at the user's request.

I'm apprehensive to the idea of a cached version of X for Y based on the hash value (could be secure but assuming eventual collisions in SHA-256 it wouldn't be hard to setup a poisoned copy of a library that would live on for a long time).

An alternative use of this would be to validate the script that is being downloaded from the remote source. If you're delegating to Google or any other CDN for a javascript library you really expect it to be the one you requested. Assuming you're not pointing at the HEAD/latest version versus a specific (which would already be pretty amateur for a real site) then you expect it to always be exactly the same (ex: jquery-1.8.1.min.js should always be the same file ... if a bug is fixed it's a new version). It'd be a security improvement to have a way to have the browser verify that.

If the downloaded file's hash does not match the expected one then the browser would flag a security warning or simply not load the file.

Great idea! Could the browser include the libraries and authenticate them in some way?

That's kind of the idea. The hash would be there so the browser could verify it is what is expected. Assuming the source page is delivered via HTTPS (w/ compression disabled :D) then the client could verify the hash against what it received to confirm it's valid.

This trivially exposes browser history to any site on the Internet. That's probably a non-starter. (Edit: I see alex beat me to this, below. He's detailed the attack, too.)

Would this allow some kind of mine-by-SHA cross-site attacks? (sidenote: I've not thought this through, this is just an off-the-cuff thought.)

If you could find a malicious file that had the same hash as the current jQuery before the next one was released, then yes. I don't think this is possible, but I'm no expert!

I guess I was thinking it'd be used for much more than just jQuery, and fast. Then for anything that any site wanted cached, some other site could essentially grab it if all they had was the hash.

Not sure I understand. How would another site grab it? And if they imported it into their page, why would that matter?

If I'm reading it correctly, I think the idea is, imagine your bank uses a hash-cache of an AJAX request containing your bank statements (stupid, yeah, but everyone is stupid sometimes).

A malicious website could then include a request to download a bogus URL from their site with a hash which matched that of the bank statement AJAX response. The browser would then return the cached bank statement, without ever noticing the URLs were bogus or didn't even point to a file with the desired hash. The malicious website could then upload the bank statements back to itself.

This seems like technically an attack path, but it relies on being able to guess a 256-bit hash in a relatively small number of guesses; if you can do that, you might as well just start guessing password hashes and log in to a target website directly.

"If you fear SHA-256 collisions but do not keep with you a loaded shotgun at all times, then you are getting your priorities wrong." http://stackoverflow.com/a/4681221

I think that quote is appropriate if you fear collisions with your own data set. The worry with a cache scheme like this is people would be actively seeking to find cache collisions.

That answer clearly explains why you can't, using current techniques and hardware.

> current techniques

We're talking about a long-term standard.

Dropbox used to be vulnerable to this because of their back-end deduplication. Users could know if a file had been uploaded already by seeing if the dropbox client had to upload it, or if the service "recognized" the hash. http://paranoia.dubfire.net/2011/04/how-dropbox-sacrifices-u... Actually an FTC complaint was filed and Dropbox had to give a up a few of the benefits of deduplication.

Having URI types that are content-centric rather than location-centric has been typical of P2P networks for over a decade... but web browser support has been spotty. (It's sometimes been faked via a localhost HTTP proxy.)

A new chance for this to change soon may be come alongside the forthcoming browser-to-browser connectivity standards under the heading "WebRTC". Then you won't just have to hope a matching file might be in your local cache... you can cooperate with other nearby nodes to find reliable replicas (like the P2P networks do), triggered by a content-hash-specifying URI scheme.

Should be used for static assets over HTTPS. Where the main page could specify the sha for the various resources and avoid bothering with encryption and letting http proxies cache them.

If you want to use content-addressable resources, why not use a content-addressable network, like Kademlia?

This is just another addressing system for the Internet. At some level this has to be about one URL for one resource. The new indexing system just tells you which URL to use. At that point why not just use normal URLs, and implement everything else in Javascript and ETags?

Assuming this is safe and works well, an interesting side-effect is that the need/recommendation of aggregating CSS files would be at least partially obviated.

Are there any security experts that could confirm whether, given current estimates and knowledge, whether a collision attack by a botnet would likely succeed?

Why all the worry about collisions? They will be rare enough that they could be worked around with a hard-coded blacklist in the browser.

Something like a lightweight low-latency bittorrent for browsers would be the ultimate caching solution.

I've proposed that to W3C HTML WG few months ago:

http://lists.w3.org/Archives/Public/public-html/2012Jun/0094...

Reply on news.ycombinator.com