I'll need to test this, just to see how wrong my "feeling" on the matter may be.
% time ag hello4294967296 big_file.txt
134217728:hello4294967296
ag hello4294967296 big_file.txt 7.83s user 0.44s system 99% cpu 8.275 total
% time grep hello4294967296 big_file.txt
hello4294967296
grep hello4294967296 big_file.txt 2.17s user 0.91s system 99% cpu 3.085 total
If you enable line numbers with grep -n, grep still wins, probably because ag's printing code is ridiculously inefficient: % time grep -n hello4294967296 big_file.txt
134217728:hello4294967296
grep -n hello4294967296 big_file.txt 3.37s user 0.97s system 99% cpu 4.338 total
Unfortunately, I didn't write ag's printing code very sanely. It grew organically from users asking for small features here and there, and now it's a giant hairball. I've added some tests to reduce the chance of breaking changes, but it'll probably be a while before I fix this performance issue.Despite these limitations, ag tends to be much faster than grep -r in real-world use. There are several reasons why:
1. Ag has a multi-threaded architecture that can search multiple files at once. Grep is single-threaded.
2. Ag ignores binary and hidden files by default. Grep doesn't.
3. Ag obeys .gitignore/agignore/hgignore. Grep doesn't.
These things really add up. For example, when searching my 20GB ~/code directory:
ggreer@lithium:~/code% time ag instantiationname
...
ag instantiationname 3.82s user 2.16s system 157% cpu 3.797 total
ggreer@lithium:~/code% time grep -n -i -r instantiationname
...
grep -n -i -r instantiationname 42.77s user 2.62s system 99% cpu 45.403 total
Both find the same matches (some tests in the Node.js source), but ag does it 12x faster.Wouldn't it be more accurate to say that ag is slower than grep because it does line-by-line searching where as grep does not? What would happen if ag adopted grep's method of searching where it doesn't do line-by-line? Is that possible with PCRE?
I think it would be very interesting to combine the tricks used by grep with the tricks used by ag. I don't think it has been done yet!
Also, ag only uses PCRE if the query looks like a regex. Otherwise, it uses Boyer-Moore strstr.
Really? Am I misunderstanding this code that very much appears to search line by line? https://github.com/ggreer/the_silver_searcher/blob/master/sr...
GNU grep never does "throw line into buffer and then search it." GNU grep just searches the entire buffer at once.
I guess that explains my experience really well, since I'm always searching from a big amount of small files like huge codebases and config files, and almost never from files bigger than several megabytes.
ag has tremendously improved my quality of life in that regard, so please accept my sincere Thank You for making ag :)
Interesting. I would have guessed you were i/o bound there, and file systems usually give higher throughput on sequential reads. Tried it on my bog standard Linux box but couldn't get it close to grep performance wise. Can you share how you did it?
> Ag ignores binary and hidden files by default
Like grep -I? That could easily cause unexpected results. Many files are treated as binary just because they contain parts of a non native character set (such as latin-1 files on a utf-8 system).
Also, modern SSDs (and even HDDs) have tricks like NCQ. With caches cleared (cold reads), multi-threaded ag is 2x faster on SSDs. Disabling NCQ makes multi-threaded perform the same as single-threaded.[2] (Again, with caches cleared.)
> … That could easily cause unexpected results. …
I've received very few complaints about ag's binary detection. It does a nice job of balancing false positives, false negatives, and performance. See is_binary() in src/utils.c for the exact code[3].
1. http://geoff.greer.fm/ag#related-posts
2. http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-...
3. https://github.com/ggreer/the_silver_searcher/blob/7d52078cb...
"The Silver Searcher is a tool for searching code." (http://geoff.greer.fm/ag/)