Hacker News Reader

Comment by _flux | Hacker News Reader

> Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong.

Well the images need to be in some order, and if you are fine tuning the very initial parts of the image, this can happen.

But it seems a better transfer algorithm could be used for those individual layers. It rather seems like if they don't match, everything's transferred, while there could be a lot better options as well. If we consider that they are structured like file systems then it opens even more options.

---

So I took a peek on my local Docker installation. It seems the layer files on the drive are gzipped json.

That's right, .json.gz, _that includes the files BASE64-encoded_.

Yes, I believe there is room for improvement, if this is also the format they are transferred in.

View on HN

base64-encoding data only poses a 1% overhead[1] when the data is subsequently gzipped, so this is hardly an issue in practice.

[1] head -c 100000 /dev/urandom | base64 | gzip | wc -c

I wasn't really consider space overhead, which as you notice is small, the encoding just something I noticed. Just json+bloatified binary+compression seemed the epitome of solving problems with a hammer when you know the hammer really well.

But as commented to me, that was incorrect, the payload in the files I saw was actually binary-encoded metadata. The actual archives are stored only in extracted form in the client and .tar.gz in the registry. Not that putting binary metadata in JSON is really optimal to me either, but maybe there are terrific reasons for doing it. Just check /var/lib/docker/image/overlay2/layerdb/sha256/*/tar-split.json.tar.gz; the fragments seem to be straight from tar, so maybe tar is used to interpret them?

Certainly .tar.gz files are not the most optimal for the purpose of incremental updates either. It's difficult to say how much a more sophisticated transfer protocol/storage format would benefit registry servers, though some clients could benefit a lot from it.

If a space-efficient merkle-tree-hashed archive-format suitable for efficient incremental updates doesn't exist, then someone should make one ;-).

json is used only for manifests; actual layers are .tar.gz.

You are correct; I too hastily took the "payload" in the tar-split.json.gz to be the binary itself, but it's something else.

Nevertheles, an archive format other than .tar.gz could be transferred more efficiently given a previous version of the same image. Or, even .tar.gz could be transferred more efficiently if both ends extract the file first, and if there is a need to preseve sha256, then that needs a bit more work.

Reply on news.ycombinator.com