Hacker News Reader

Comment by 2ion | Hacker News Reader

Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.

View on HN

"... there are some metadata extensions that allow this)."

Where to find these extensions? Are they portable between Linux and BSD?

The 1998 dict project included a utility called "dictzip" for random access to the contents of gzip compressed files.

Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.

afaik, only pixz does this; they store the index in xz.

A zip file is a concatenation of gzipped files. A .tar.gz is a gzip stream of concatenated files. Anything that could do random access into the contents of a zip file entry could do similar things with a tarball.

Not so simple. A bundle of streams is not the same as a stream of bundles.

What about .gz.tar, in which all the files are gzipped, then tarballed? It seems like it would be a slightly fatter .zip file.

With a transparent random access overlay, the difference mostly disappears, reducing to whether the stream needs to be scanned or whether it's indexed, which is itself orthogonal - zip file directory at the end is redundant.

So you mean at each "random access", you actually have to scan the whole .tar.gz file to find the location? For large tarballs, that will definitely hinder performance a lot. The difference does not disappear at all.

Apparently you have a comprehension problem.

Then enlighten me.

Zip files have a directory which lists the files in the archive and their offsets, etc. No such feature in a tar archive.

If we're talking about something that indexes gzip streams, it's not a leap for it to also index the inner tar.

AFAIK a compressor like zip builds a dynamic running table of frequent byte sequences; the resulting archive is written in such a way that when you decompress it, you re-build the table in the process.

So if you concatenate files A, B, and C and then compress the result, then by the time the compressor starts compressing the data of C, it will have that table built from A and B. To extract C, you'll need to re-build the same table and thus you'll need first to decompress A and B.

In a zip file each entry is compressed individually; this gives random access, but worse compression rate, because the table is not re-used between files.

> Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives?

Yes. If the tar file is on a seek-able medium, just read the headers only and build an index to the offsets of the file contents from the header data.

Then use the offsets index to seek to just the item of interest and read out only it and nothing more.

Now, does such a utility already exist? The answer seems to be yes: https://github.com/devsnd/tarindexer

> This makes large tarballs annoying to use on systems with slow disk I/O

Funny how tar was originally developed for tape drives!

Tape drives don't really support random access, though, which is reflected in the design of the tar format and its offspring. That is, in fact, the problem here, and why formats designed for random access instead of sequential access are far better for storing file systems for containers and VMs.

Reply on news.ycombinator.com