Hacker News Reader

Comment by theamk | Hacker News Reader

I like simple archives, but can it be not tarballs? For the kinds of application described in this article, tarballs are pretty bad:

Either you extract it from scratch every time you run an app, taking a long time penalty...

... or you extract once to cache, and assume that nothing changes the cache. This is pretty bad from both operational and security perspective:

- backups have to walk through tens of thousands of files, thus becoming much slower

- a damaged disk or a malicious actor can change one file in the cache, making damage which is very hard to detect.

There are plenty of mountable container formats -- ISO, squashfs, even zip files -- which all provide much faster initial access, and much better security/reliability guarantees, especially with things like dm-verity.

View on HN

Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.

"... there are some metadata extensions that allow this)."

Where to find these extensions? Are they portable between Linux and BSD?

The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files.

Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.

> Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Yes. If the tar file is on a seek-able medium, just read the headers only and build an index to the offsets of the file contents from the header data.

Then use the offsets index to seek to just the item of interest and read out only it and nothing more.

Now, does such a utility already exist? The answer seems to be yes: https://github.com/devsnd/tarindexer

A zip file is a concatenation of gzipped files. A .tar.gz is a gzip stream of concatenated files. Anything that could do random access into the contents of a zip file entry could do similar things with a tarball.

Not so simple. A bundle of streams is not the same as a stream of bundles.

What about .gz.tar, in which all the files are gzipped, then tarballed? It seems like it would be a slightly fatter .zip file.

With a transparent random access overlay, the difference mostly disappears, reducing to whether the stream needs to be scanned or whether it's indexed, which is itself orthogonal - zip file directory at the end is redundant.

So you mean at each "random access", you actually have to scan the whole .tar.gz file to find the location? For large tarballs, that will definitely hinder performance a lot. The difference does not disappear at all.

Apparently you have a comprehension problem.

Then enlighten me.

Zip files have a directory which lists the files in the archive and their offsets, etc. No such feature in a tar archive.

If we're talking about something that indexes gzip streams, it's not a leap for it to also index the inner tar.

AFAIK a compressor like zip builds a dynamic running table of frequent byte sequences; the resulting archive is written in such a way that when you decompress it, you re-build the table in the process.

So if you concatenate files A, B, and C and then compress the result, then by the time the compressor starts compressing the data of C, it will have that table built from A and B. To extract C, you'll need to re-build the same table and thus you'll need first to decompress A and B.

In a zip file each entry is compressed individually; this gives random access, but worse compression rate, because the table is not re-used between files.

afaik, only pixz does this; they store the index in xz.

> This makes large tarballs annoying to use on systems with slow disk I/O

Funny how tar was originally developed for tape drives!

Tape drives don't really support random access, though, which is reflected in the design of the tar format and its offspring. That is, in fact, the problem here, and why formats designed for random access instead of sequential access are far better for storing file systems for containers and VMs.

I'm pretty sure the article implies this is for user-facing applications where the user would manually extract it once to a place of their choosing then run it from there. I think you're missing the point of the whole article.

But why would you want to extract if you can mount the file directly? For simple archives, extracting is fine. But for larger archives (like a compiler -- 1000 files or more), loop-mounting is much better than extracting:

- Does not slow down your backup by adding thousands of files

- No need to wait for initial file extraction

- You can quickly and easily verify integrity of the whole archive

And if you are using fuse, it does not require any special privileges either!

Mountable formats have the security issue that the kernel is not that great at protecting against hostile images in mount. On disk format fuzzing has not been common and there are definitely bugs.

This is solved very nicely with fuse mounts (and fuse is surprisingly performant on the modern multicore systems)

Do you not still pay a significant performance penalty by reverifying the container upon each application load? Especially considering that, if the container is signed, you need to verify the signature itself before trusting the container, and full signature verification - including checking whether the signature has been revoked - involves expensive network calls?

If your operational and security model really frowns on trusting your extraction cache, then perhaps a different workflow is more appropriate - download the container, verify the container, extract, bake the OS plus extracted apps into an image, sign the image, verify the image upon each boot and mount apps read-only. Then you don't need to re-verify anything upon each launch, instead trusting that your image creation process is routinely updating and re-verifying the software in your current images.

Verification of a single file is much faster than walking entire tree, especially when there are lots of small files, for example when there is a compiler or large python project inside.

A simple example: my /usr/include is 33037 files, 356M uncompressed. On SSD with cold cache, it takes 6.7 sec to read each file individually, or 0.7 sec to checksum a single 356M archive, a 10x difference.

The difference in the backup time is even more dramatic -- the backup program has to call stat() either 33K times, or just once, a 3,330,000% improvement! The other filesystem tools (What takes all the space? What has changed in the last X hours? Please sync this directory elsewhere.) will have similarly high speed improvements.

So if I had a choice, I would love my dev environment to come in mountable form. Similarly, I don't understand why container runtimes (like docker) don't use loop mounts more -- it seems like many advantages and very few disadvantages.

As for signature verification -- I don't care about 3rd party signature and revocation, I just want to ensure that I am running the same code every time. There are many ways one can damage extraction cache, especially if it is owned by the same user as application (like the topicstarter post described) -- sysadmin errors (`sudo find / -name app-old -delete`), application errors (create cache file in bin dir), disk errors (silent corruption), transfer errors (one file did not get transferred to a new computer). Loop mounting makes disk errors easier to detect, and eliminates other classes of error entirely.

Reply on news.ycombinator.com