Regex visualized as 3D marble run | Hacker News Reader

Regex visualized as 3D marble run

regexmarblerun.com

My coworker used to teach non-technical people how to code. Students always found regex hard to understand.

Let's be honest, regex can be hard to understand even for technical people. (At least it's hard for me.)

So coworker built this to help anyone understand arbitrary regex commands.

AI disclosure: this was built using Agent Duel, a local coding agent arena system (https://github.com/bottomless/agent-duel). Promotion disclosure: our company is building agent duel. To test it and find bugs we are building passion projects through the system. I nonetheless thought HN would find this interesting.

Really cool. Do you teach people the mechanics, like what the app shows? Because I wonder if that actually makes it harder. I'm wondering about where the hard bits are, given that it all boils down to just:

  "|", which means "or"
  ".", which means "any symbol"
  "?", which means "at most once",
  "*", which means "however many times"
  "+", which means "at least once"

Maybe encountering it next to PEG scrambles your brain.

add to that

- using groups "(" .. ")" in replacement "\"i

- character ranges "[" .. "]"

- character classes "[:" .. ":]"

- multiline match

- ignore case

- beginning / end of line / word

I am mostly an autodidact and learned bits regex on the fly when I needed it and I was always baffled why people have auch a hard time with it.

The problem IMO is that a lot of the traditional teaching material starts with the complexity (and impressive possibilities) of regex, when the best way to start learning it is probably just thinking about it as an extension to simple text matching.

Getting to know regex fully is in fact complicated, getting know enough regex to filter/search/replace 90% of everything you will always need is not too complicated.

Autodidact here. Beyond just the five core quantifiers

  .
  |
  ?
  *
  +

regex only has capture groups

()

and lazy variants of the core quantifiers "?", "*" and "+", which are helpfully called:

  ??
  *?
  +?

The rest is just shorthands.

So while I think it's a worthwhile suggestion not to show people the whole thing the first time round, I doubt that the whole thing is really that complicated.

Regex has many more features than this, e.g. backreferences. I agree that it's easy to learn enough regex to be dangerous with that subset, but that's not everything regex offers.

Backreferencing inside of a regex (which is a way to find repeat occurrences of a substring) is a very controversial feature that some regex engines intentionally omit, while most others support it only grudgingly. Supporting backreferences opens up a big can of worms: Most regex engines will fall back to a slow, poorly optimised algorithm if they ever encounter backreferences.

Also, it's important for people not to confuse backreferences with capture groups. They look a bit similar, but capture groups are actually fine.

In fact, backreferencing is such a footgun that I don't see it as a possible regex feature. It's a PCRE feature - perhaps even the core distinguishing feature. While all regexes are PCREs, not all PCREs are regexes. It's simply not possible to create a consistently performant implementation of PCRE.

There are hundred different flavors of what "regex" actually means. The basics are often the same but the extended features differ.

The foundation is in theoretical CS with deterministic and nondeterministic finite automata (DFAs and NFAs), and I found it easiest to teach regexes using DFAs since you can then just draw circles and arrows. Regex notation is just a different notation. It's a bit likeif you know piano and then learn another instrument, your brain can always refer back to the piano and it's all much easier.

Backreferences are not really "controversial" but just an extension which does not map to regular languages (those recognized by finite automata). That is, they make regexes "non-regular", so they are more a hack. Regex engines that want to support them need a fundamentally different approach, that's why some don't support them and in principle will never be able to.

They are controversial because supporting them raises matching time to O(2^n) in the worst case - which is dangerous - while not supporting them enables matching time to be O(n) in the worst case. And a user might not be aware of that danger.

Also, I think it's misleading to mention NFAs and DFAs alone, like there's no deeper reason. The deeper reason is that:

  Regexes (or NFAs, or DFAs) classify exactly the properties of strings which can be checked in constant space.

It turns out that any such string property can also be checked in O(n) time, and in one single left-to-right pass.

Briefly returning to backreferences, supporting them makes constant space unachievable. They also make linear time unachievable, since matching them becomes NP-COMPLETE.

Also, no regex implementation actually uses DFAs. NFAs yes, but DFAs no. The point of NFAs is that they're easy to execute in constant space and linear time, while also making concatenation and Kleene star easy to implement. It seems much harder to implement concatenation and Kleene star using most other constant-space automata - like DFAs - which is what actually forced people to invent NFAs.

I think we are actually on the same page, but it is funny that we express things in exactly the opposite way as well as reach opposite conclusions. :)

It's not about being pedantic, quite the opposite It's intuitive to use circles and arrows as the mental model, instead of the cryptic regex syntax. Both have the same semantics. The quite visual representaton of the finite automata makes the meaning more approachable, not unlike the marble run illustration.

In contrast, bringing "constant space" and "O(n) matching" and NP-completeness etc into the discussion is all formally correct (and I'm a big fan of formally correct) but that's where things become pedantic from the common folk's POV and is also where you lose people who just want to understand how something works practically.

Therefore I'm saying, start withthe easy circles-and-arrows intuition and only go to the theoretical background (aka be pedantic) once people want to know more.

(Backrefs can't be expressed with circles-and-arrows which illustrates why they are something special.)

> The rest is just shorthands.

If you really want to get technical: ., ?, and + are also shorthands. The formal definition of a regular expression is very minimal on purpose.

`.` is just `(a|b|c|...)` - List of all literal characters

`x?` is just `(x|)` - "x or empty string"

`x+` is just `xx*` - "x followed by zero or more x characters"

Whether this is exactly how they are implemented in any particular regex engine is an implementation detail, but formally there are very few operations which aren't just syntactic sugar.

You are barely scratching the surface! You forgot about positive and negative look ahead and behind, lookaround, backreferences, recursion, inline engine flags settings and probably some other things I forgotten about. PCRE2 compatible regexes are potentially monstrously complex things.

I am a big believer in the idea that within any educational context learning is much easier if you start with the problem instead of describing the atoms that make up the solution. This way students will have a much easier way of understanding why they need to learn what they are learning. And then, once they know enough to be dangerous, you introduce more complicated variations and background ideas especially like yours.

Note that I teach tech topics at the university level, in an art school. That means a mixed audience of people who have a hard time with programming and complete nerds, but didn't come here for semantic definitions, but to get shit done. To get everybody on board with such a topic requires you to start small and on a literal "explain this to a 5-year old"-level.

Of course when teaching this to our sysadmin or someone more knowledgeable I could take your approach since I can assume they know the priors and even have seen a regex or two.

However most people don't care about any topic except when it solves their problems in ways other solutions couldn't. And this needs to be the first thing you show them. Why are regexes really awesome for certain problems that commonly exist. Bonus points if it is a problem they already encountered or are currently having.

Regex is conceptually incredibly easy to understand.

But regex is also syntactically designed by Satan himself.

Most programming languages use commands that are semantically understandable, like "for" and "let" and "return" and mostly symbols we already know like "+" and "=". They use newlines and indentation for organization. And they put multiple commands on multiple lines and you can add comments and everything.

Regex is exclusively punctuation, with its own idiosyncratic meanings, all compressed into a single line, zero organization, zero comments, and impossible to remember what each symbol does if you don't use it daily. Not to mention the nightmare of escaping regexes when expressed as strings, on top of escaping regex symbols within the regex itself.

I swear it's high time we come up with a regex replacement that looks like normal Python code or HTML or something. So instead of:

  ^(\+\d{1,3}-?)?\d{3}-\d{3}-\d{4}$

We would write something like:

  <start>
    <optional>  # Optional country code
      "+"
      <digit>[1-3]
      "-"
    </optional>
    <digit>[3]  # Area code
    "-"
    <digit>[3]
    "-"
    <digit>[4]
  <end>

Same thing. But one is completely inscrutable to someone who hasn't learned regex, while with the other you can guess pretty easily how it works...

If you want readability you can write

    ^(?P<country_code>\+[0-9]{1,3}-?)?[0-9]{3}-[0-9]{3}-[0-9]{4}$

Most regex engines even have a verbose mode (typically the x flag) where this is a valid regex:

  ^
  (   
      \+
      [0-9]{1,3}
      -?
  )?   # optional country code
  [0-9]{3}
  -
  [0-9]{3}
  -
  [0-9]{4}
  $

I am not sure if everyone writes unreadable regex because the reputation of regex makes it seem pointless to try, or if being witty one-liners is a critical part of the value proposition of regex

Most regex dialects allow whitespace and comments, so that already gets you halfway there.

You can enable extended mode for most engines, which allows you to add white space and comments.

Personally, I like regexes, in the same way that I like shell string manipulation: it looks utterly inscrutable, and performs magic. If I write one in code meant for others, I usually use extended mode with each line commented.

Regex in xml? Now you have 2 problems. And the comments are not xml comments even.

Ultimately it is a dsl. Should equations in math also use full words and comments?

No, they should not. Maybe a nice syntax for grammars like raku is a better approach.

I couldn't care less if it looks more like XML or Python or C or whatever. It's just an example.

I just care that things have readable text names rather than inscrutable punctuation and single letters.

Yes, it is a DSL. Just like, you know, Python or C or HTML or CSS. Just because it's a DSL doesn't mean it can't actually use keywords instead of punctuation.

Actually yes? If math equations also wrote a fluent English way of saying the full thing as well as the literal syntax as well every time it would help more people actually understand it.

If you want text _in addition to_ regex you can use comments in regex, like in https://softwareengineering.stackexchange.com/questions/1783...

> Should equations in math also use full words and comments?

My "also" referred not to have multiple representations but to also in math have prose instead of equations. I could have expressed it better. I did not think of that interpretation.

As for math, I really do not think that is true. Look at how this beautiful proof is so long in prose.

https://mathcs.clarku.edu/~djoyce/java/elements/bookI/propI4...

Is that better than a^2+b^2 = c^2 ?

https://en.wikipedia.org/wiki/Pythagorean_theorem

Notation helps to be concise. Imagine more complicated things without notation in maths. Of course if you look at Principia Matematica we see the limits of notation as well.

The point is, if you don't use regex that frequently, you can completely forget that ^ means start and $ means end. And adding those as line comments ("^ # Start of expression") would be dumb.

That's why a well-designed DSL tries to make its syntax as self-explanatory as possible. Using "start" instead of "^", or "digit" instead of "d".

We're not doing equations here, we're doing engineering. Analogies to proofs are not helpful. There is a very good reason why, in coding, we give descriptive variable names such as "accountBalance" rather than "b".

Conciseness is not a virtue in programming. Clarity is. Things should not be longer than they need to be -- but neither should they be shorter. The name of something should not be 100 characters long, but neither should it be 1 character long. A word or two is generally the sweet spot.

> There is a very good reason why, in coding, we give descriptive variable names such as "accountBalance" rather than "b".

In languages with an expressive typesystem you see a lot more single letter variables.

> We're not doing equations here, we're doing engineering

This sentence is nonsense.

> Conciseness is not a virtue in programming. Clarity is

I see what you are saying, but conciseness is also related to clarity, as I showed with the theorem.

I think some languages do have a library-style version of regex. But the function names aren't consistent between the different libraries. Maybe then we need to put on special goggles to make them look the same.

There are certainly alternatives, like https://pomsky-lang.org/

wouldn't it be easier to just add a comment block in the code above the regex?

To be fair, that's just begging for the XML in the comment block to get out of sync with the actual regex literal.

In practice, including a unit test for any nontrivial regex in your application is very helpful. You can demonstrate which sorts of strings are expected to match and, maybe more importantly, which ones are not expected to match. The same goes for testing the capture group contents.

It may feel silly to have a whole unit test for a one-line regex, but that one line can be really dense. If you wrote a function to parse the string without regex, that function would be complex enough to warrant a unit test, after all.

Introducing regressions when tweaking a regex in your application is also a big risk which unit tests can help mitigate.

What you want is something akin to a Raku (perl6) grammar, which was their enhancement to matching they developed to go beyond regular expressions.

    grammar IniFile {
        token key   { \w+ }
        token value { <-[\n;]>+ }
        token pair  { <key> \h* '=' \h* <value> \n+ }
        token TOP   { <pair>+ }
    }

    my $text = "user = alice\nrole = admin\n";
    my $match = IniFile.parse($text);

    if $match {
        say "Parsed successfully!";
    }

There are various libraries for other languages that try to accomplish something similar or Port the concept, so if this interests you, maybe you can find one. Or in this day and age, maybe you can have some VC (and much of society) subsidize the fairly quick creation of a library for you language of choice through AI.

If you're not looking for that, and more specifically what you have shown above, there are often ways to accomplish that in certain languages. In Perl it's easy because you can use regex cars in a regex, so you can just assign `my $digit = qr/(? :0-9)/;` and then use it in the regex as you did (and Perl already supports non matching whitespace with a flag). For that matter, Perl regexes allow you to define chunks purely within the regex to be reused in a DEFINE block, so possibly that is a feature other regex engines have as well.

If you're using Python, you may be in luck: https://andrewshitov.com/2026/10/03/using-raku-in-python/ which specifically shows using Raku grammars / action in Python.

Melody (https://github.com/yoav-lavi/melody) is very similar in syntax to your example.

For example:

  <start>;
  
  option of "v";
  
  capture major {
    some of <digit>;
  }
  
  ".";
  
  capture minor {
    some of <digit>;
  }
  
  ".";
  
  capture patch {
    some of <digit>;
  }

  <end>;

Matches semver tags, e.g. v1.0.0

FWIW, lpeg is nice, of course you'll still have to deal with the terseness, but I don't think there's anything that can be done, since you will have to write an expression.

  patt <- country area "-" phone fin
  country <- ("+" %digit^-3 "-")?
  area <- %digit^3
  phone <- %digit^3 "-" %digit^4
  fin <- ! .

Not that impressive, but at least it doesn't group captures and groups and string escapes \ with a class labeling symbol.

this is basically how raku fixed perl regex

America's Cup and Formula One races are won at the design stages. We praise the drivers, but what succeeds is the engineering.

If billions of dollars of IPO value were at stake, competing to use animation to convey an understanding to test subjects, this design would be toast. The model is cripplingly realistic, wasting most of its cognitive bandwidth on provoking a "Gee willikers, it really looks like someone built that out of wood!" reaction, leaving us to squint to see the balls or the parts of the machine implementing the logic. There's a particular genius to logo design, that needs to be applied here. If this were instead a cartoon animation, Walt Disney dosed the coffee one day at the studio to see what his animators could do, we'd get a visualization that better used its cognitive bandwidth to establish an icon language with the viewer.

The abacus is the T-shirt for this idea. Like a musical instrument, one can imagine operating an abacus without holding one, and some shopkeepers were able to leave theirs in a drawer after enough practice. Beads, like balls, leverage our brains' ability to manipulate physical reality, but the idea of counting became more vivid as the abacus design grew more streamlined. It not only became easier to build and operate, but it became a more effective "thought worm" implant into our minds.

We make contact with aliens, and in their museums they have an abacus for regular expressions. What does it look like?

This is important to get right, because soon AI will communicate with us using Pixar-quality 3D animation, generated in real time for each conversational turn. Many people still think of AI communication as exchanging "text message" prompts, but for my math research I'm already getting illustrated PDF slide decks as AI's responses to me in single conversational turns. One just has to ask; the form AI communication takes is only limited by our imagination.

Feynman diagrams and the like as rare exceptions, scientific communication is still mired in "telegraph" form, math symbols one reads along lines of text that once lived on physical paper. As future generations adapt to visual symbolic reasoning, the animations AI will use to communicate with us will evolve from representational to abstract art, for a "fatter pipe" pouring symbolic information into our minds. They'll find the "thought worms" that best hack our minds, make us smarter.

I'd love to see what your regex viewer would look like, if you were imagining this future.

This is actually pretty satisfying, I wish it had sound. Maybe a dramatic follow cam for the balls

Seems the input text box is limited to 301 characters.

what was that big cloudflare outage regex again ?

Now I have 3 problems

Reply on news.ycombinator.com