Why are regular expressions difficult?
johndcook.com
johndcook.com
People think regular expressions are difficult because they're taught either not at all, or very badly. If you learn them deliberately, moving through the unfolding complexity of patterns, they're actually both beautiful and intuitive.
I think most programmers first come to regular expressions when they encounter a problem that can't be efficiently solved in any other way. The sort of problem that can only be solved by regex is also the sort of problem which is solved by complex regex. This means that the first introduction is to be thrown right in at the very, very deepest end. Regex becomes a confusing and arcane thing, and the programmer resolves to rarely use them.
However, it doesn't have to be like that. If you teach regex before they're 100% required - when they're used to solve problems that are biddable and have multiple possible solutions - you can start to get a feel for the syntax. That makes regex feel accessible, and soon you can start to work with even the complex ones.
The ergonomic problems of parser generators are worth quite a bit of analysis, but the basic difference is that people can learn how to use regular expressions without understanding how regexes are implemented but practically you have to know how parser generators work to use a parser generator. Anybody who can write a parser generator can easily work around the things that are difficult with parser generators.
What's hard is that even regular grammars, let alone CFG, let along non-CFG, are hard for people to grasp.
The problem is HARD, not the solution!
For example to have a regex that can remove // from inline comment of an INI-format text file.
(((\x27[ \!\"\#\$\%\&\(\)\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*\x27\s*)|(\"[ \!\#\$\%\&\x27\(\)\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*\"\s*)|(\/([ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*)|([ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*))*)*)([;#]+)*.*$
That regex works very well and it’s breakdown can be better explained using various online Regex testersOff-topically, I’ve got another next-level problem, and that is how to convert Regex for use within a bash shell script.
It’s nasty.
However, I have listed several online Regex testers so you can better understand the result of (my) Regex creation, one regex character at a time.
https://extendsclass.com/regex/4d49f33
https://www.freeformatter.com/regex-tester.html
More online testers that I couldn’t get to work with are detailed in https://github.com/egberts/bash-ini-file/issues/1
Let us see how to use the advanced technology of factoring out common expressions into "subroutines" makes it much easier to understand. Note, I have no idea of what the regex you are doing actually does, so I am doing things purely mechanically, and I have only basic working-memory knowledge of regexes.
First, let us restructure it:
(
(
(
\x27
[ \!\"\#\$\%\&\(\)\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*
\x27\s*
)
|(
\"
[ \!\#\$\%\&\x27\(\)\*\+\-\.\/0-9\:\;\<\=\>\?@A-Z\[\\\]\^\_\`a-z\|\~]*
\"\s*
)
|(
\/(
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*
)
|(
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*
)
)*
)*
)
([;#]+)*.*$
Now let us reorganize the clearly common expressions: (
(
(
\x27
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~\;\#\/\\\"]*
\x27\s*
)
|(
\"
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~\;\#\/\\\x27]*
\"\s*
)
|(
\/(
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]+
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*
)
|(
[ \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~]*
)
)*
)*
)
([;#]+)*.*$
Now let us factor out the common expressions: common = { \!\$\%\&\(\)\*\+\-\.0-9\:\<\=\>\?@A-Z\[\]\^\_\`a-z\|\~}
separator = {\;\#\/\\}
(
(
(
\x27
[common + separator + {\"}]*
\x27\s*
)
|(
\"
[common + separator + {\x27}]*
\"\s*
)
|(
\/(
[common]+[common]*
)
|(
[common]*
)
)*
)*
)
([;#]+)*.*$
At this point, going any further likely requires a deeper understanding of what it is actually trying to accomplish, but the purely mechanical transformations have dramatically reduced complexity of the regex to something in the realm of comprehensibility.The problem statement can be described as something like: how to remove trailing inline comment after properly skipping over any matching pair of quotes, single or double, that are found before the inline comment notation, whose removal is starting with the comment notation ‘#’ or ‘//‘ on each non-continued line within a .ini text file.
and most regex engines cannot handling labels of assignments for later reuse within its expression.
It is however still useful to leverage a higher language to do the constructing of a final regex pattern from repeated patterns (basic building block) prior to engine usages.
And now I’ve neglected to mention regex101.com as totally working just perfectly (for I’ve just tested them as WORKING WELL just now).
I do still plan to them.
Thanks for the heads up!