Hop: Faster than unzip and tar at reading individual files
github.com
github.com
https://github.com/electron/asar
It's not hard being faster than zip if you are not compressing/uncompressing.
https://www.gnu.org/software/tar/manual/html_node/Standard.h...
tar the utility is a program that can produce tar files, but it is also able to then compress that file.
When you produce a compressed tar file, the contents are written into a tar file, and this tar file as a whole is compressed.
Sometimes the compressed files are named in full like .tar.gz, .tar.bz2 or .tar.xz but often they are named as .tgz, .tbz or .txz respectively. In a way you could consider those files a format in their own right, but at the same time they really are simply plain tar files with compression applied on top.
You can confirm this by decompressing the file without extracting the inner tar file using gunzip, bunzip2 or unxz respectively. This will give you a plain tar file as a result, which is a file and a format of its own.
You can also see the description of compressed tar files with the “file” command and it will say for example “xz compressed data” for a .txz or .tar.xz file (assuming of course that the actual data in the file is really this kind of data). And a plain uncompressed tar file described by “file” will say something like “POSIX tar archive”.
I don't think that's quite how it works. I'm pretty sure it's more similar to doing something like `tar ... | gzip > output.tgz`. That is it streams the tar output through gzip before writing to the file. In fact with GNU tar at least you can use a `-I` option to use an arbitrary program to compress.
You read from one file to get all the information you need instead of reading from N files and N directories.
Can’t easily mount virtual filesystems outside of Linux. However, Linux supports the copy_file_range syscall which also makes it faster to copy data around than doing it in application memory
If we had a random-access file system loopback-image standard (open standard for both the file system and the loopback image container format), maybe we wouldn’t see so many tarballs. But there is no such format.
As for “why archive things at all, instead of just rsyncing a million little files and directories over to your NAS” — because one takes five minutes, and the other eight hours, due to inode creation and per-file-stream ramp-up time.
> But there is no such format.
What do you think of ISO (ISO9660)? just downloaded a random image to double-check, opens on OSX, Windows and Linux just fine.
It's read-only ofc, but that should not be a problem given we're talking about archives
It doesn't seem very well known, which is unfortunate because it's much better suited for archiving files compared to gzipped tar (which is great for distributing files, but not great for archiving/backup).
It claims it's easy to write a parser but then requires "Pickle" (derived from Python's Pickle?) which is just a link to a chrome header file... I'm not aware of a standard and like that it doesn't seem like it even wants to be standardized. Do they mean easy to write a parser as long as you're using chrome's JS engine?
Actually, since CPUs are so fast and disk IO is so slow, it is essentially impossible to beat a properly tuned program that reads data from disk and decompresses it on the fly using one that doesn't use compression at all.
For random access to gzip'd files, bgzip [1] can be used. bgzip also uses an index file (one time creation) that is used to record key points for random access.
The index is stored at the tail end of a zip file for instance, which is really not cool for something likej tape, doubly not cool when you don’t know in advance how big the data on the tape is.
And since you wouldn’t know where the end is by reading the beginning, you’ll have to keep reading until you hit the right marker - and seek backwards (which you generally can’t read backwards on most tape drives, so you need to jump back and re-read).
Tar is of course all-around great for tape, since every file is essentially put on by itself (with just a little bit of header about it). Again this is great for streaming things both on and off tape. But you can't do any sort of jumping around. You have to start at the beginning and then go from file to file. This gets even worse if you try to compress it (e.g. .tar.gz), as you then have to decompress every byte to get to the next one.
a) zip spec does not require file index to be at the end. Only the pointer to file index
b) You can do streaming decompression of zip as each zip entry has a header + filename inline with data.
c) I came up with this optimization(placing index at front of making file io sequential to make use of OS readahead) for omni.ja files in firefox. It's still a standard zip file, but index lives at the front. Most zip tools can open that file unmodified(tho they sometimes complain).
That is actually applicable to a project I’m working on now, any chance your implementation is in python and open source somewhere?
https://github.com/humphd/mozilla-central-old/blob/9d4d9f265...
Curious, how is it applicable?
Seems convenient to allow optimization for high speed sequential reads and random read/writes at different parts of the life cycle, along with indexing, crcs, signatures, etc.
One big issue with zip storing the index at the end of course is a truncated file basically lost most of it’s context and is generally unrecoverable even in part, which this could also help with from a durability perspective.
Storing it at the beginning (without a end pointer) opens up the possibility you have a valid looking archive you’re touching that is truncated and missing a lot of data, and won’t know until you look past the end (or validate total bytes or whatever, which doesn’t work well when steaming).
Storing the index at the beginning, pointer and file sig at the end, and all the other format extensions does solve for all this. Which is convenient.
But I've started using a tool called "ratarmount" (https://github.com/mxmlnkn/ratarmount) which creates an index once (and something I could automate our upload system to generate in advance, but you can also just process it lcoally) and then lets you fuse mount the file. This works pretty great with the only exception that I can't create scratch files inside the directory layout which in the past I'd wanted to do.
I was surprised how hard a problem to solve it is to get a bundle file format that is indexable and compressed with a good and fast compression algorithm which mostly boils down to zstd at this point.
While it works quite well, especially with gzip and bzip2, sadly the zstd and xz (and some other compression formats) don't allow for decompressing only parts of a file by default, even though it's possible the default tools aren't doing it. The nitty gritty details are summarised here: https://github.com/mxmlnkn/ratarmount#xz-and-zst-files
The other main options I found was squashfs which recently grew zstd support and there is some preliminary zip file support for zstd but there are multiple standards which is not helpful!
"Why? Reading and writing lots of tiny files incurs significant syscall overhead, and (npm) packages often have lots of tiny files. (...)"
It seems the author is working on a fast JS bundler tool (https://bun.sh) and the submission links to an attempt to fix some of the challenges in processing lots of npm files quickly. (But of course could be useful beyond that.)
Ah, those trees of is-character packages depending on is-letter-a packages, themselves depending on is-not-number packages each appearing several times in different versions are probably challenging to bundle and unpack efficiently. We might want to address file systems too, so they too can handle NPM efficiently (actual size on disk and speed).
Or maybe the actual fix is elsewhere.
Runs, fleeing their screen screaming in fear and frustration
(no offense to the author, I actually find the problem interesting and the work useful)
- Faster than ESBuild/SWC
- Fast build-time macros written as JSX (likely friendlier to develop than say a Babel plugin/macro). These open up a lot of possibilities that could benefit end users too, by performing more work on build/server and less client side.
- Targeting ecosystem compatibility (eg will probably support the new JSX transform, which ESBuild does not and may not in the future)
- Support for integration with frameworks, eg Next.js
- Other cool performance-focused tools like Hop and Peechy[1] (though that’s a fork of ESBuild creator’s project Kiwi)
This focus on performance is good for the JS ecosystem and for the web generally.
I know I'm way behind the popularity curve* (they're already in debian-stable, for crying out loud); but for the benefit of anyone even more out of the loop than myself, do check these out. The speed increase is amazing: a significant quality-of-life improvement, essentially for free**. Particularly if you're on a modern NVME SSD and not utilizing it properly. (Incidentally, did you know the "t" in coreutil's "tar" stands for magnetic [t]ape?")
* (The now-top reply to this comment says 'ag' is even superior to 'rg', and they're probably right, but I had no clue about it! I did say I'm ignorant!)
**(Might have some cost if you're heavily invested in power-user syntax of the GNU or BSD versions, in which case incompatibility has a price).
(I've edited my first comment in response to this reply: I originally wrote "fdfind". (For a comment about regexp tools, this is a uniquely, hilariously stupid oversight. Sorry!)).
Thus, Rust tools get a priority, at least for myself. There are probably other memory-safe languages but I haven't had the chance to give them a spin like I did with Rust. If I found out about them then I'll also prefer tools written in them if there's no Rust port and if the alternatives are C/C++ tools.
I don't accept your labeling, especially when it's so egregiously misguided.
Also, you're clearly seeing what you want to see.
Plus a memory error might lead to scary consequences (like a random script getting elevated privileges).
Finally, and last I checked, `rg` is quite featureful and I've never felt constrained by it. So for me it's a win all around.
That being said, there are powerful adversaries of anonymity and the right to personal data out there -- and security in depth is what works best against them. There's no one UltimateSecuritySolution™; there are many small ones that we layer on top of each other so we don't allow even a smidgen of air to pass between the cracks.
But yeah, I am paranoid. I am gradually preparing myself to move from macOS to Linux and even though I am not a criminal and never will be, I'll still make a heroic effort to make the odds of any foul play against me practically zero. (And that's why I will start using the userland Rust tools alternatives as well.)
I'll concede that in my case the biggest impact would probably come from running Chrome in a jail, and not from using `rg` vs. `ag`. That much is true, yep.
Some fun benchmarks, and a fair description of how text search works, can be found in this 2016 post https://blog.burntsushi.net/ripgrep/
Since both are instant on every codebase I care about, not so sure about that. These tools were always chiefly I/O bound , SSDs have torn that down especially now.
But for features: there's no lookaround in the regex.
> Since both are instant on every codebase I care about, not so sure about that.
It is very possible that there is no performance difference between ripgrep and ag for your use cases, but that does not mean there isn't a performance difference between ripgrep and ag. For example, in my checkout of the chromium repository:
$ time rg -c Openbox
testing/xvfb.py:2
tools/metrics/histograms/enums.xml:1
ui/base/x/x11_util.cc:1
real 0.448
user 2.593
sys 2.490
maxmem 77 MB
faults 0
$ time ag -c Openbox
ui/base/x/x11_util.cc:1
tools/metrics/histograms/enums.xml:1
testing/xvfb.py:2
real 2.302
user 2.996
sys 10.462
maxmem 15 MB
faults 0
That's not true and hasn't been true for a long time. ripgrep supports PCRE2 with the -P/--pcre2 flag. You can even put `--engine auto` in an alias or ripgreprc file and have ripgrep automatically select the regex engine based on whether you're using "fancy" features or not.
In general, I also claim that ripgrep has far fewer bugs than ag. ag doesn't really get gitignore support correct, although if you only have simple gitignores, its support might be good enough.
And then there's also the fact that ag isn't that great as a general purpose grep tool. It really falls over in terms of speed when searching, say, bigger files:
$ time rg 'Sherlock Holmes' OpenSubtitles2018.raw.en | wc -l
7673
real 1.475
user 1.115
sys 0.356
maxmem 12511 MB
faults 0
$ time ag 'Sherlock Holmes' OpenSubtitles2018.raw.en | wc -l
7673
real 20.276
user 19.850
sys 0.413
maxmem 12508 MB
faults 0
https://docs.rs/regex/1.5.4/regex/#syntax
> This crate provides a library for parsing, compiling, and executing regular expressions. Its syntax is similar to Perl-style regular expressions, but lacks a few features like look around and backreferences.
Edit: I see. It can use pcre2 but it's a build time option which of course Ubuntu has off.
That's unfortunate. Perhaps a bug report to the packagers is in order? Archlinux enabled it: https://github.com/archlinux/svntogit-community/blob/0dc033f...
time rg Query src > /tmp/junk.rg
rg Query src > /tmp/junk.rg 0.01s user 0.04s system 156% cpu 0.037 total
time pt Query src > /tmp/junk.pt
pt Query src > /tmp/junk.pt 1.44s user 10.80s system 220% cpu 5.545 total
time ag Query src > /tmp/junk.ag
ag Query src > /tmp/junk.ag 4.32s user 9.82s system 204% cpu 6.918 total
ls -l /tmp/junk\*
-rw-r--r-- 1 sramam wheel 16492080 Nov 10 16:05 /tmp/junk.ag
-rw-r--r-- 1 sramam wheel 16492080 Nov 10 16:04 /tmp/junk.pt
-rw-r--r-- 1 sramam wheel 27477 Nov 10 16:04 /tmp/junk.rg
(EDIT: formatting)
My query was for a generic term "src" - so that likely trapped the file-path embedded within the long lines of a source-map. Hence the massive file-size difference.
(I did inspect the files to validate)
Happy to answer any questions or feedback
Just an idea to check the "append" box.
See also B-trees, Shadowing, and Clones https://www.usenix.org/legacy/events/lsf07/tech/rodeh.pdf
Not related to bun or hop is the use of Zig. I am wondering if you will do a write up on it someday.
That’s a good way to bypass that 4GB size limit, and instead of wasting 8 bytes per number this will even save a few bytes.
However, I’m not sure how easy it is to implement in the language you’re using. I only did that in C++ and modern C#. They both have intrinsics to emit BSWAP instruction (or an ARM equivalent) to flip byte order of an integer, helps with performance of that code.
Compressing a 57.5 MB .hop archive with `gzip -c` returns an 11.7 MB file. This is a similar compression ratio as tar -> tar.gz
Compression is great for sending files over the network, but for fast reads, it might be undesirable
If you also used the mt option with 7zip and just stored you probably could have a decent read rate as I think it spins extra threads.
I dislike the command-line format of Hop; it seems to missing many features. Also, 64-bit timestamps and 64-bit data lengths (and offsets) would be helpful, as some other people mention; I agree with them, to improve it in this way. (You could use variable length numbers if you want to save space.)
My own opinion for the general case is that I like to have concatenable format with separate compression (although this is not suitable for all applications). One way to allow additional features might be having an extensible set of fields, so you can include/exclude file modes, modifications times, numeric or named user IDs, IBM code pages, resource forks, cryptographic hashes, multi-volumes, etc. (I also designed a compression format with a optional key frame index; this way the same format supports both solid and non-solid compression, whichever way you want to do, and this can work independently from the archive format being used.)
For making backups I use tar with a specific set of options (do not cross file systems, use numeric user IDs, etc); this is then piped to the program to compress it, stored in a separate partition, and then recorded on DVDs.
For some simple uses I like the Hamster archive format. However, it also limits each individual file inside to 4 GB (although the entire archive is not limited in this way), and no metadata is possible. Still, for many applications, this simplicity is very helpful, and I sometimes use it. I wrote a program to deal with these files, and has a good number of options (which I have found useful) without being too complicated. Options that belong in external programs (such as compression) are not included, since you can use separate programs for that.
I agree 100%. I wrote most of it in like three hours; it's not a polished product.
> My own opinion for the general case is that I like to have concatenable format with separate compression (although this is not suitable for all applications). One way to allow additional features might be having an extensible set of fields, so you can include/exclude file modes, modifications times, numeric or named user IDs, IBM code pages, resource forks, cryptographic hashes, multi-volumes, etc. (I also designed a compression format with a optional key frame index; this way the same format supports both solid and non-solid compression, whichever way you want to do, and this can work independently from the archive format being used.)
I'm wary of slowing it down by adding lots of features. I think that, generally speaking, _more_ purpose-built binary formats should exist.
Engineers do this all the time with YAML files and JSON, but why not binary files?
It is a valid point, yes. However, my mention was meant to mean that unknown fields can be easily skipped.
How does that even happen in 2021?
Skimming the source, there are places where the author explicitly chooses to represent lengths with 32-bit types (e.g., schema.zig/readByteArray()). I bet you could massage the code to work with larger data without many issues.
I'm sure the author would appreciate a pull request if it's that straight forward.
Doing so makes the program slower though which might be a non-starter for a performance focused project. Plus you'd need a little more work to properly handle large archives on 32-bit and smaller systems (at least those which support >4GB files).
2^32 = 4294967296
Not sure if this limitation is being enforced by upstream concerns, but this is why this code in particular is limited to that size.
Maybe that’s overly conservative. Wouldn’t be hard to change
I suppose this could easily be changed, right? I could not compile it though. I got this error:
/usr/bin/ld: errno: TLS definition in /lib/x86_64-linux-gnu/libc.so.6 section .tbss mismatches non-TLS reference in cdb.a(cdb.o)
/usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: error adding symbols: bad value
Nevermind, I fixed it. I added `#include <errno.h>` to `error.h`.
Anyway libarchive can read rar files so just use bsdtar. It can do many other archive files [0] like cpio as well. One standardized interface for everything is nice
[0] https://github.com/libarchive/libarchive#supported-formats
ZIPX archives are simply ZIP archives that have been branded with the X for the benefit of the users who may be trying to open them with something ancient, that doesn't yet support LZMA or the other new features.
[0] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
https://en.wikipedia.org/wiki/ZIP_(file_format)#Version_hist...
According to the above version table, that would conform to PKWARE 2.0, which as with the ODF file specification, limits packing methods to STORE and DEFLATE only; as mentioned in the standardization section and ISO/IEC 21320-1.
When you say 'zip' you should be saying 'zipx' at the very least, but you may as well be comparing a tar file to a tar.zstd file in the same case.
It's not exactly hard to get a new compressor on a *nix system. It's either in the repos, or not so hard to compile. On non-nixes, there's usually a binary. Tarballs have not been limited to only tar.gz for a very long time, though a lot of people do choose to be conservative in how their distributed files are compressed.
> When a 'hacker' (programmer) says 'zip file' they refer to the widely compatible implementation which is likely to work in any common implementation of archive handling software.
You don't speak for every hacker, certainly not for me. If you're implementing ZIP these days and it's not, at the very least, capable of reading ZIP64 archives (forget about newer compression methods), then you're just creating obsolete software.
The format has been perverted, and we can't trust zip as the "just works" option that will open in any platform, with any implementation, anymore.
All because somebody thought it a good idea to try to leverage the zip name's attached popularity to try and make a new format instantly popular.
Great job. This is why we can't have good things.
But then, why wouldn't you just use the 7z format?
The expectation with ZIP is (or was) that it'll unpack fine, even under CP/M.
Moving to '.zipx' extension was the right move, but it was done far too late.
[1]: https://devblogs.microsoft.com/oldnewthing/20180515-00/?p=98...
I hope if/when they upgrade their zip support, they only accept the new format if the filename extension is zipx.