Hacker News Reader

Comment by opencl | Hacker News Reader

And ripgrep is quite a lot faster than GNU grep[1], especially if you need unicode support.

[1] https://blog.burntsushi.net/ripgrep/

It depends, I benchmarked all of the usual suspects (fgrep, GNU grep, BSD grep, ripgrep, sift grep) on the same 30GB line separated JSON file last week and GNU grep was the fastest one.

I didn't have a scientific testing setup though. I just ran them one by one like this:

time cat title_complete_json.txt | grep tt9184994 >> result

The infamous "useless use of cat"

That's relevant in some scripts, where unnecessary forks might actually add up to some meaningful resource usage. Not relevant when typed at a tty.

To clarify, it would have been prettier (conceptually less clunky and shorter) to run this:

time grep tt9184994 < title_complete_json.txt >> result

grep takes a file directly, so this could simply be:

time grep tt9184994 title_complete_json.txt >> result

It annoys me whenever somebody brings this up. People do this because it's easier to modify commands at the end of your prompt than near the beginning. Especially in this case, where the grandparent wants to quickly switch between benchmarking grep, ripgrep etc.

Not useless at all.

Though a neat thing to know is that redirections can appear anywhere on the line in Bourneish shells:

    time < title_complete_json.txt grep tt9184994

As it happens, when benchmarking grep-like tools on large files, this is actually somewhat likely to make a significant difference since the "useless" use of cat actually forces the file to be read sequentially as a stream (because it's piped through), while a process receiving a file on standard input can treat it as a file and e.g. mmap it.

I found a benchmark that suggests the performance gap between a pipe and input redirection is insignificant for most workloads, but there would indeed start to be a difference for the exact case discussed here (grepping a large file): http://oletange.blogspot.com/2013/10/useless-use-of-cat.html

Thanks for your tip, it's clearly the best of both worlds for this purpose.

Incidentally, ripgrep will actually be faster in this case generally (on Linux at least) when given an explicit file path, since that lets it use memory maps. For example:

    $ time rg QZQZQZQZ < subtitles.en.txt

    real    1.842
    user    0.552
    sys     1.287
    maxmem  9 MB
    faults  0

    $ time rg QZQZQZQZ subtitles.en.txt

    real    1.210
    user    0.776
    sys     0.433
    maxmem  9510 MB
    faults  0

(See my other comments in this thread for where to get subtitles.en.txt.)

Surely using cat like this is more in line with the unix philosophy?

Did you try The Silver Searcher https://github.com/ggreer/the_silver_searcher? :)

I didn't, but will try it out of curiosity when I'm back on that machine.

You have to throw parallels in there - https://stackoverflow.com/questions/9066609/fastest-possible...

Yeah, rg is definitely not always the fastest. Test for your own use case and all.

In pretty much any case involving more complicated queries and/or unicode I have found rg to be the fastest by a very wide margin. Also if you're searching multiple files rg is multithreaded by default without having to pass through something like xargs or parallel. If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation.

Thanks for noticing ripgrep's performance on Unicode. :-)

> If you're just searching for an ASCII string literal in a single file rg probably gets slowed down by its UTF-8 validation.

This is most definitely wrong. Firstly, ripgrep doesn't do any UTF-8 validation. That would make it much much slower. Secondly, searching a simple literal is one of the cases where ripgrep should actually do better than GNU grep in many cases. This is because of the slightly smarter way in which ripgrep keeps its vectorized loop active more often than GNU grep does by trying to search rarer bytes. I talk about this quite a bit here, which even includes a benchmark where I measured ripgrep at 2x the speed of GNU grep for a simple literal search: https://blog.burntsushi.net/ripgrep/#subtitles-literal

Anyone can try this for themselves:

    $ cd /tmp
    $ curl 'https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2016/mono/en.txt.gz' | gzip -cd > subtitles.en.txt
    $ pv < subtitles.en.txt > /dev/null
    9.28GiB 0:00:01 [4.89GiB/s] [======================================================================================================================================================>] 100%
    $ time grep ZQZQZQZQ < subtitles.en.txt

    real    8.713
    user    7.495
    sys 1.212
    maxmem  9 MB
    faults  0
    $ time rg ZQZQZQZQ < subtitles.en.txt

    real    1.857
    user    0.573
    sys 1.282
    maxmem  9 MB
    faults  0

On my system, /tmp is a ramdisk, so this doesn't factor in I/O. If you look up this thread, you'll note that someone was trying to search a 30GB file. Unless that person could fit that file into memory, it's very likely that they were just measuring the I/O bandwidth of their machine.

Thanks for the detailed explanation and your work on rg, I use it all the time and strongly appreciate the performance with unicode because I do a lot of searching of large directories of files that are largely in Japanese.

No problem! If it's not too much trouble and you can share your data (and queries), I'm always happy to see more details about use cases that take advantage of ripgrep's Unicode support and its associated performance characteristics. At some point, I'm going to have to redo my previously published benchmarks, and it would be great to get some real use cases like yours in there. :-)

A lot of the data I work on is unfortunately NDAed but I'll try to find something that can be shared, no guarantees though. What's a good way to contact you with this stuff?

Thank you! Just file an issue and that should be good enough!

What’s the rationale for using >> ?

I would try to get as small an output as possible, and not hit the file system at all, for example by using

  time cat title_complete_json.txt | grep tt9184994 | grep -v tt9184994

AFAIK ripgrep always uses regular expressions. So for straight substring matching grep will usually be faster.

It does not. Could you please say where you got this impression so I could fix it? See my other comments in this very thread. ripgrep actually has more sophisticated literal extraction & searching than GNU grep does. Moreover, many of these optimizations are inside the regex engine itself, so even if it looks like you're using a regex, literal optimizations may still be applied.

My mistake. I read the first line of the ripgrep readme and assumed because utils like `find` run slower with regex matching for the same searches that ripgrep would be similar.

> ripgrep is a line-oriented search tool that recursively searches your current directory for a regex pattern

I'm not sure if many other people would jump to similar conclusions. So the readme might not really need a revision.

Reply on news.ycombinator.com