A System to Transparently Compress Hundreds of Petabytes of Image | Hacker News Reader

A System to Transparently Compress Hundreds of Petabytes of Image

usenix.org

69 pointstodsacerdoti18 comments

“As of February 2017, it had compressed more than 203 PiB of user JPEG files, saving more than 46 PiB.”

Not too shabby.

I've tried lepton and packJPG (on which lepton is based) for my personal collection and found packJPG to be faster, give better compression and a lot simpler to compile and install. Only problem was that it did occasionally fail to compress an image.

It compresses them one image at a time. You'd think that if you had millions of images there would be opportunities to index and cluster similar images to obtain further compression.

Then the images are no longer individual. That would be fine for read-only, but I would guess a fair amount of the data is hot.

Fascinating, but wasn’t... Dropbox supposed to have client-side encryption? If the backend is recompressing JPG images, that means it has full access to the file’s plaintext. That’s a bit uncomfortable.

Presumably shared images?

Dropbox does not have client-side encryption; its servers have full access to the content of synced files. You can verify this yourself by uploading a common image filetype like jpeg and seeing that the website shows you compressed file thumbnails.

(I'm a former Dropbox employee who worked on related things, though not specifically the linked paper)

Yep, that makes sense. IIRC, they also have some sort of global deduplication system, don't they? That also implies they have access to your unencrypted files.

Edit: That's what I get for commenting before reading the paper. They literally mention deduplication in there.

Not necessarily. You could use homomorphic encryption and, for example, encode the file with a hash of the file as the key material for encryption. All such files will encode identically for each user, but it will be opaque to the server what the contents are unless the server already has the plain-text of the original file. This was used by other cloud storage companies (that are now defunct).

https://techcrunch.com/2011/06/20/dropbox-security-bug-made-...

This makes me believe there is no problem for Dropbox to read content in clear.

> Not necessarily. You could use homomorphic encryption

I think that's still far from being possible in practice. IIRC, the best homomorphic encryption can do now is to run simple queries on very small datasets, on a supercomputer, with performance of a 386?

Convergent encryption, not homomorphic, is what you use to deduplicate encrypted files.

Mega does client side e2e encryption (or at least they claim to), yet their web interface still shows thumbnails...

I'd guess they implement it by having the client upload an encrypted version of both the original image and a thumbnail, and then decrypting the thumbnails on the client at display time. Javascript has reasonably performant crypto primitives now, so it's very doable to download data from a server, decrypt it with a client side key, and display it.

If mega can do this, it's a shame other services don't.

Dropbox has "client-side" weasel encryption: they use TLS. It is obvious that they do not encrypt the files in any way that would prevent access to Dropbox itself, and probably not to any Dropbox employee.

Dropbox never claimed end-to-end client side encryption. That said, neither do any of the popular cloud drive options - for example Google drive or One drive.

End to end client side encryption is a tough problem from usability. Most people don’t like being told they can’t recover their “cloud backup” if they lose their laptop or phone or forget their password.

I think even iCloud only does end-to-end for specific sensitive data.

There are special providers like SpiderOak who do this, but it is non trivial to use safely.

I've always found Cryptomator or encfs to be great solutions for this.

The only safe assumptions about encryption of your data in the cloud is that 1) its not encrypted, or 2) it is encrypted but at some unknown time in the future there will be a security compromising bug, and the data will be in the wild. But you won't know its in the wild, because the service provider doesn't know about the bug, or it isn't in their interest to tell you.

It seems you're right that Dropbox has the plaintext server side from elsewhere in the thread.

It would also have been possible that they did the recompession client side though, and send enough metadata to calculate metrics. I had thought that's what they were doing, to save bandwidth/reduce upload times as well as save space on the servers.

Reply on news.ycombinator.com