Hacker News Reader

Comment by coldtea | Hacker News Reader

>Best practices are a very wide category, don't parse HTML with regular expressions is the kind of thing you really should just say don't do that.

I call BS.

"Don't have a continuously running service parse arbitrary HTML with regular expressions" would be a bad thing.

Parsing specific, given, HTML files, with known structure to the dev, with regular expressions (e.g. as part of a one-off scrapping script) is totally, absolutely, fine.

If my HTML file is like:

  <html>
  <body>
  <a href="xxxxx">foo</a><br>
  <a href="xxxxx">foo</a><br>
  <a href="xxxxx">foo</a><br>
  <a href="xxxxx">foo</a><br>
  <a href="xxxxx">foo</a><br>
  <a href="xxxxx">foo</a><br>
  <a href="xxxxx">foo</a><br>
  </body>
  </html>

I can parse it just fine with regex.

And even parsing is the wrong word: I can extract the information I want is a better term. I won't be parsing anything in the AST sense.

View on HN

As someone who has created more hacks like you describe than I care to admit, you actually make the case of precisely why you don't parse HTML with regex.

Because that code works fine for 5 years until it blows up after something upstream changes. From a RoI calculation this might be perfectly fine if you're still around and remember how to fix it - it's a 5 minute fix after all! But if you've left, or if that system became old and crufty and fragile you may have just burned 3 days tracking it down.

If someone is asking this question, it is entirely appropriate to tell them "don't ever do that" - they obviously lack the experience and big picture thinking to know when it is appropriate to deviate due to business reasons. If you leave it at that I agree you're being a dick - whenever shooting someone's idea down you had damn well better have some appropriate alternative options.

Serious question... why would any other approach be exempt from the problems you stated?

Nothing is exempt, some things are far less fragile than others though.

But are also significantly slower and consume hundreds of times more memory, and will still break with the first little change in the DOM tree on the path that you rely on - which smart regexps can sometimes handle. For instance if you're looking just for a title and a price of a single product (common requirement for spiders), you can extract that with regexp without caring about the html page at all, just pick the <h1> and something that looks like a dollar value. I had spiders like that kept working after a full redesign of the site where even the platform was moved from WP to Magento. No DOM parser could handle that, plus they would be too slow in the first place. You need to pick the right tools for the job, everything else is BS...

"Because that code works fine for 5 years until it blows up after something upstream changes."

I believe the point here is exactly that nothing will change, because the regex in question isn't for a service, it is just for that specific file, right now, today. I've regexed specific HTML files myself too, because even though I am very comfortable with XPath and Beautiful Soup and tree representations in general, regexes are even easier on a static file like that.

We're not talking about one-liners used on a single file here (as far as I could tell) - if that's the case, use whatever quickest hack you can think of to get the job done. Use-once coding is entirely different - everyone loves some code golf once in a while, but it doesn't mean you commit that to git :)

For re-usable tooling though, after some time you tend to avoid design patterns that you've personal witnessed break down repeatedly and cause issues.

"We're not talking about one-liners used on a single file here (as far as I could tell)"

Well, we are, because to quote coldtea, the person you directly replied to, "'Don't have a continuously running service parse arbitrary HTML with regular expressions' would be a bad thing. Parsing specific, given, HTML files, with known structure to the dev, with regular expressions (e.g. as part of a one-off scrapping script) is totally, absolutely, fine." In context the first sentence clearly means that running the service like that would be a bad thing (hooray English and it's deep ambiguities in double negatives).

I still wouldn't necessarily be too upset about telling a junior dev to be suspicious of REs or avoid them (for instance, parsing HTML with REs typically requires using non-greedy matches, and there are some subtleties around that), but if you know what you're doing on a quick job it's fine. If you're a senior dev and you still haven't figured out what's likely to entrench itself and what really is a one-off job, well, that's your real problem. I haven't been surprised about what gets entrenched in quite a while.

When I was young, I was asked to print some address labels, so I wrote a quick super-short BASIC program to parse a specific file, right now, today, and got those labels printed. Made all sorts of assumptions, but it didn't matter, because it was a static file, and nothing could ever change in it.

Three or four years later, I was having lunch with the guy I'd done that for when he got a phone call. Turns out they were using that program of mine to print labels monthly now, and one of the completely-safe assumptions I'd made years earlier had bitten them. Fortunately, he knew how to resolve it easily, but I learned a very important lesson that day.

Was the lesson that there's a fortune to be made in software maintenance?

What lesson was that?

From my perspective, this other guy was using a program outside of its design spec. That can be fine, but such a thing is prone to the problem of safe assumptions suddenly not being safe. You wrote the program for the problem you needed it for, and for which, it worked fine. Unless part of the problem at the time was "we expect to use this program for a couple years", there is no reason to make it overly robust.

"right now, today" and "that specific file"

Code designed to be used once has a habit of sticking around.

https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it

There is premature optimization and then there is not being stupid. Using one of the 10,000 XML/HTML parsers to get to v0.01 takes about as long on average and avoid ridiculous problems.

The only thing regex has going for it is you might already know it. But, this is one of those time when learning something is worth it.

PS: XP is doing a depth first search, which can be EXtreamly wasteful.

There are plenty of disposable code situations where the code wont break after 5 years, because it does not exist after two months. Also, failing regexp on likely generated html like that likely wont take 3 days to find. That is ridiculously high estimate even for messy codebase.

You are trying to make estimates and decisions for project you know literally nothing about. That is about worst practice of them all.

In case the question was asked by someone lacking the experience - well then it is absolutely ok to learn regexp by trying to parse info few downloaded files of toy scrapped site. Having newbie fight like this just to learn makes no sense.

> failing regexp on likely generated html like that likely wont take 3 days to find

This is true, but only because you're implying that the reason (failing regexp on HTML input) has already been identified. Maybe the parsing job fails silently and then people down the line (who have no idea that the data was passed around as HTML at some point) wonder why no data is coming in anymore, or why they're seeing incomplete data. That could take weeks to trace back to the parsing problem.

Silent failure seems to be root cause of delay in situation you described. Ast based parsing can be made to swallow errrors too.

And parsing the HTML to an AST won't work either in 5 years, the format of the tree will probably have changed and you will be getting index errors.

That is a trap and why people give that advice so freely. Sure, today you just want to parse these files just the once an never again, ever, really I mean it.

Can you determine if it's <html><body>Yes</body></html> Vs <html><body>No</body></html> well sure and even more complex operations also 'work'.

However, if you control what's going on then don't use regular expressions it's wasteful overhead. If you don't control it then you can't tell if it will stay like that, which is the trap. Things that don't change the meaning like swapping the order Attributes which does not change the meaning will often force an update to your regular expression.

PS: It's also why these are called best practices not physical laws.

>If you control what's going on then don't use regular expressions it's wasteful overhead.

Not really, if you control what's going on it's a very fast option -- use awk, sed, or your scripting language's regex lib, and get the values you need. End of story.

Nobody cares if you can micro-optimize it with lower level string parsing that doesn't use a regex engine -- and the regex engine might end up being faster that that anyway.

>If you don't control it then you can't tell if it will stay like that, which is the trap.

No, but I already covered that in my original comment. One can do it to extract values for a specific, known html file, I said.

Besides,

(a) a lot of changes can still be caught by regex -- e.g. the order of attributes don't matter if the regex checks for that specific attribute alone (e.g. a href in an a tag).

(b) other changes will also fuck any non-regex, parser-based lib. E.g. if the nesting changes or an element is moved to another section. It's not like you don't have to update scripts written with e.g. BeautifulSoup or some HTML parser when the document changes.

>It's also why these are called best practices not physical laws.

That's up to those who advocate them to understand: and don't hand them out as if they were physical laws "NEVER PARSE HTML WITH REGEX".

I don't mean CPU overhead.

I mean you are creating a brittle process that's hard to verify it actually worked, unless you have access to the data some other way in which case it's mostly pointless. Regular expressions are not going to tell you if one of the files just happens to have bad data or any number of things that's going to cause problems.

In the case of processing HTML with regex there is a strong theoretical argument against it: a regular expression is a type 3 language while HTML is described by a type 2 grammar. Type 3 being a subset of type 2, regular expressions are not expressives enough to handle the grammar of HTML.

In most cases of best practices however, such a mathematical argument doesn't exist. So, it's more of a battle of (not so) informed technical opinions and experiences, where drawing a clear line is impossible.

Reading list:

https://en.wikipedia.org/wiki/Chomsky_hierarchy

https://stackoverflow.com/a/14207715

Reply on news.ycombinator.com