App::Rak – 21st century grep / find / ack / ag / rg on steroids
github.com
github.com
besides, ripgrep IS much more state of the art than anything else out there. very fast and can fallback to pcre2 for weird lookahead and utf8 (if i remember correctly, sorry). we have to admit that BurntSushi made incredibly good job at it.
ACTUALLY what I need most of the times is qgrep which can grep regexes in 100k files in less than second. is re2 + index essentially and none of its cousins can do similar searches reasonably fast. the only downside - it indexes per-line, so if you wanna grep some fancy multiline-SQL you have to do second pass over the files that match on more general basis
not sure why codeql could not get up to speed with SQL, but I've several times considered actually using multi-passes with tree-sitter in order to dig some SQL neatly hidden in multiline concatenated Java strings. why? well, the usual "order of things" in many large projects is after all nothing more than respectful mess of entangled enterprise-level orchestration (code) piled years after years. i guess this' why someone at google had to write qgrep to dig the monorepo.
Let me give an allegory - perhaps not exact, but you'll get my point. Much of the mathematics (either discreet, differential, computation geometry, etc) is actually not so difficult to understand when told right... but incidentally, there were some folks back in the day who decided to write it using the whole greek alphabet and then some cryptic symbols in it (like these https://cloford.com/resources/charcodes/utf-8_mathematical.h...). By doing so they effectively formed some weird cryptic cult of the Mathematicians, which eventually repelled many people away from the discipline, and still do.
Later one, some perhaps not so smart people, figured how to express stuff as 'sum', 'product' in for-loops and suddenly the whole world was writing programs while decrying math as complex. even though... programs are logic are math, right?
Very cool! It looks like Rak has a lot of nice features.
I will say that getting this thing built took some digging. Could you please add install/build instructions to the README? As it is, I had to go sleuthing through your CI setup to figure it out. On Archlinux, here's what I did:
$ yay -S rakudo-bin
$ export PATH="/usr/share/perl6/site/bin:$PATH"
$ git clone https://github.com/lizmat/App-Rak
$ cd App-Rak
$ zef install .
$ export PATH="$HOME/.raku/bin:$PATH"
$ git remote -v
origin git@github.com:rust-lang/rust (fetch)
origin git@github.com:rust-lang/rust (push)
$ git rev-parse HEAD
699f4a637ef382ea0716c72f60d0c1c4cd79df41
$ time rg accelerated --files-with-matches
compiler/rustc_target/src/spec/thumbv7em_none_eabi.rs
src/llvm-project/lldb/source/Plugins/LanguageRuntime/ObjC/AppleObjCRuntime/AppleObjCTrampolineHandler.cpp
src/llvm-project/llvm/docs/PDB/HashTable.rst
real 0.126
user 0.544
sys 0.850
maxmem 22 MB
faults 0
$ time rak accelerated --files-with-matches
compiler/rustc_target/src/spec/thumbv7em_none_eabi.rs
src/llvm-project/lldb/source/Plugins/LanguageRuntime/ObjC/AppleObjCRuntime/AppleObjCTrampolineHandler.cpp
src/llvm-project/llvm/docs/PDB/HashTable.rst
real 17.005
user 2:07.88
sys 8.866
maxmem 1565 MB
faults 0
$ cd /dev/shm
$ curl -LO https://burntsushi.net/stuff/opensubtitles/2018/en/sixteenth.txt.gz
$ gunzip sixteenth.txt.gz
$ time rg 'Sherlock Holmes' sixteenth.txt -c
426
real 0.068
user 0.047
sys 0.020
maxmem 778 MB
faults 0
$ time rg 'Sherlock Holmes' sixteenth.txt -c --no-mmap
426
real 0.089
user 0.026
sys 0.063
maxmem 14 MB
faults 0
$ time rak 'Sherlock Holmes' sixteenth.txt --count-only
426 matches in 1 file
real 48.236
user 48.010
sys 0.334
maxmem 215 MB
faults 0
$ time grep 'Sherlock Holmes' sixteenth.txt -c
426
real 0.297
user 0.233
sys 0.063
maxmem 14 MB
faults 0> I'm not totally clear on what kind of filtering `rak` does by default
By default, rak search all of the files that look like text, by reading the first 2K (if I remember correctly) and using some algorithm to determine "binaryness".
> I realize features and an expressive language is your main goal
Indeed.
> I wanted to see what it looked like from a perf perspective
The performance times you've given here, coincide with my findings. Which for the "sixteenth.txt" surprises me: a bare loop in Raku reading that file line-by-line finishes in 8 seconds for me. Which would be one order of magnitude less than the search result. Still not in `rg`'s ball park, but speed wasn't a big concern for me. But 8 -> 48 seconds is a pretty big difference, so I guess it's back to the drawing board for me on that one :-)
(sidebar - Larry took to opportunity with perl6/raku to properly reinvent regexes for unicode - there is quite lot to unpack in raku regexes: graphemes management, unicode props as char classes, named regexes, etc.)
soo - I suggest this proposal to anyone looking for a cool rust/unicode project ... extend ripgrep from PCRE/ECMA262 regex 1.0 to embrace the "21st century" regex syntax of raku
some initial reading is here: - https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b - https://docs.raku.org/language/regexes - https://github.com/edumentab/p6-ecma262regex
ripgrep does have fairly complete UTS#18 Level 1 support already.
Indeed.
Also, it should be noted that the Raku Regex engine is not a state machine as such. The regex "slang" in Raku, is basically just another way to write code. Code that uses some state machine primitives underneath, for sure, but still code.
A grammar in Raku is nothing other than a class in which the methods are codegenned using the regex syntax. But a class nonetheless, which can also have attributes and "proper" methods.
This gives the Raku regex engine the flexibility needed to be able to parse Raku source code, and generate executable bytecode from that.
And yeah, I figured Raku might not follow UTS#18 to the letter of the law with respect to syntax, but likely has all the concepts available. Which makes sense given what Raku is.
According to the guide[2]:
> Raku is a high-level, general-purpose, gradually typed language. Raku is multi-paradigmatic. It supports Procedural, Object Oriented, and Functional programming.
That really doesn't tell me enough. Anyone else have any idea why I might want to use Raku vs one of the many other langs to which those adjectives can be applied?
https://docs.raku.org/language/grammar_tutorial
> Regular expressions (Regexes) work well for finding patterns in strings. However, for some tasks, like finding multiple patterns at once, or combining patterns, or testing for patterns that may surround strings regular expressions, alone, are not enough.
> When working with HTML, you could define a grammar to recognize HTML tags, both the opening and closing elements, and the text in between. You could then organize these elements into data structures, such as arrays or hashes.
Given that, it's 23 years old and done by Larry Wall so it's not exactly yet another fly-by-night hype language, just something that is not in the top 50.
It used to be, but not anymore.
It has grown to be its own thing with some happenstance parenthood through being born from an early attempt at Perl 6, and thus is now only being slightly more related to Perl than e.g Ruby is. Ok I'm pushing it but you get the idea: taking a biological analogy, it has mutated away far enough that it is now considered to be its own strain.
And if that helps clear some doubts, Perl 7 is on the table (if there ever needs to be a Perl 7):
At some point in the future, the PSC may decide that the set of features, taken together, represent a big enough step forward to justify a new baseline for Perl. If that happens, then the version will be bumped to 7.0.
So Raku cannot be Perl 6!
Given that, nobody got the memo, so calling it "perl 6" is still a very fast way to orient people as to what it's a neighbor of.
Here, "some swimming creature" is the totality of the objective, we aren't trying to do a crash course in marine zoology or trying to explain what a Sirenia is or what a fish technically means.
Saying it's perl-6 means that it's not say, lisp-like or some kind of prolog or haskell. It's a relatively swift approximation and that's the totality of the objective.
C of 40 years ago is nothing like modern c.
Same for basic.
int foo(s, f)
char* s;
float f;
{ ... }
(disclaimer: I don't actually know Raku that well, but it is very high on my list of languages to learn)
OOP, functional programming, strong/dynamic typing, declarative /imperative - you’ve got it, we got you covered
Instead of just typing
rg '\bfoo\b'
rak '/ << foo >> /'
rg -B2 -A2 foo
rg -C2 foo
rak foo --before=2 --after=2
rg 'foo.*bar|bar.*foo'
rak '{.contains("foo") && .contains("bar")}'
Okay, I haven't actually tried it (and the README doesn't say how to), and rak does have a bunch of features and capabilities ripgrep doesn't have, but those first examples are not very inspiring after the big initial claim.
So
rak -C2 foo
[Edited to add] Also, the `\b...\b` form is shorter with rak - as the README also states where it documents this pattern form:
> §string
> If the pattern starts with §, then it indicates that the string should occur as a word (with word-boundaris on both ends) in the item. Basically a shortcut to specifying string --type=words. Any --smartcase, --smartmark, --ignorecase or --ignoremark arguments will be honoured.
Typing § is not very easy on most keyboards, and is actually impossible on some keyboard-OS combinations (including mine): https://en.wikipedia.org/wiki/Section_sign#Keyboard_entry
> If the pattern starts with §,
Is it April 1st already?!
I wonder if keyboards in the future will settle on a way to provide access to more characters, than is currently common.
I agree the keyboard thing is really annoying. Maybe the West should investigate the efforts China has put into different typing systems.
U+00A7 (codepoint 167) is also the Unicode codepoint for §. Its use on this very web page is via Unicode and the UTF-8 encoding as the bytes \xC2\xA7.
(This is meant as a narrow correction. It doesn't really change the broader point that § is not exactly easy to type on most keyboards. Whether its ASCII or not.)
https://keyshorts.com/blogs/blog/37615873-how-to-identify-ma...
rg -B2 -A2 foo
rg -C2 foo
grep -R -C2 foo .Do the "contains" example with 5 terms.
rg foo1 | rg foo2 | rg foo3 | rg foo4 | rg foo5
rg foo1 | rg :.\*foo2 | rg :.\*foo3 | rg :.\*foo4 | rg :.\*foo5
rak '{.contains("foo1") && .contains("foo2") && .contains("foo3") && .contains("foo4") && .contains("foo5")}'
This actually shows one of the secret powers of the Raku Programming Language: Junctions https://docs.raku.org/type/Junction
Alternately one could express this as:
$ rak '{.contains: <foo1 foo2 foo3 foo4 foo5>.all}'
Or using whatever currying: https://docs.raku.org/type/Whatever
$ rak '*.contains: <foo1 foo2 foo3 foo4 foo5>.all'
rak '{so one ["foo1", "foo2", "foo3"].map(-> \x{$_.contains(x)})}'
Whait what's that? I'm stuck with ag and {a..z}grep.
Alongside `fzf` and `fd` (a similar project for `find`) `rg` was what finally convinced me to put together a few shell scripts to bootstrap getting all my favorite modern command line tools on a new Ubuntu VM [1]. I'm working on an interactive playground you'll be able to SSH into to try it out yourself without even having to spin up your own, so you can see the difference with your own two eyes, no VM required.
At my previous job I often ended up searching raw log files sized around 5-8GB. Regular grep would grind away for 50-60 seconds, while ripgrep returned the same results in 250ms.
I was shocked the first time I used it and have sung it's praises since.
I have used (at different places) Graylog and Splunk, New Relic is also popular.
Not that ripgrep isn't awesome, but you could have dedicated indexing and a UI that can be shown to non-technical or at least semi-technical people to review logs whereas "Just use ripgrep" really isn't accessible for most users.
TBH more frequently I have "argument list too long" problem, than "one 5GB file" problem.
I'm curious (like others here) how `rak` is better than `rg`.
In contrast "rg" only designed to search text files, but it does that really really fast.
I makes sense to have both.
rak is backed by the entirety of raku language and therefore is much easier for some crafting something that's less trivial to generalize in a one-liner. For most of grep-alike tools, their regex engine would be some sort of ceilings of what can be done but for rak, the ceiling is as tall as the raku programming language. That's rak's niche.
- support for --ignoremark, which means you 'e' will match any accented 'e', such as éëêèęėē.
Have you found folks using these particular Unicode features in practice? I don't think anyone has request it for ripgrep.
Well, but that's only one of the reasons why rak is a lot slower. There's something else going on, but I currently don't have the mindset to investigate this deeply.
(...I am somewhat disappointed that I don't have a ridiculously buff nose by now but at least it matches the rest of my body).
Thanks, I wouldn't have been able to tell otherwise.
I was wondering why, in your coding style, you always use $y.ACCEPTS($x) instead of $x ~~ $y?
$x ~~ $y is basically syntactic sugar for $y.ACCEPTS($x). Using ~~ involves an additional subroutine call, so there's a short-term efficiency issue (as in programs running more than a second or so the extra call will have been inlined).
But personally, I like the verbosity of $y.ACCEPTS($x) as it reads better for me, having been burnt by ~~ semantics in Perl, which was symmetrical.
Even though I'm the author of ripgrep, believe it or not, I still use grep in those 5% of cases without any kind of fuss.
I promise, you can learn both grep and ripgrep at the same time.
Some Postgres servers I work on have massive log files, and due to DDOG indexing retention being short, sometimes it’s required to go sleuthing on the host. While LC_ALL=C speeds up grep quite a bit, it’s still an order of magnitude slower than ripgrep.
Luckily, I can add tooling to these servers if there is a valid need, and “massively speeds up searches when speed is of the essence” is very much an accepted case.
I wish that distributives maintainers would enforce inclusion of the modern tools like Rak in the default install, then they would become the norm across whole industry. Like for example MS includes Powershell regardless of the opinion of people who dislike it. And now you can expect that powershell is available in any modern Win install. In latest Win11 new Terminal is now a permanent feature in every install and you can expect to have it everywhere on Win11. Etc.
Regarding distro maintainers, specifically Debian, ripgrep is in Bookworm [0].