Hacker News Reader

Comment by phil21 | Hacker News Reader

As someone who has created more hacks like you describe than I care to admit, you actually make the case of precisely why you don't parse HTML with regex.

Because that code works fine for 5 years until it blows up after something upstream changes. From a RoI calculation this might be perfectly fine if you're still around and remember how to fix it - it's a 5 minute fix after all! But if you've left, or if that system became old and crufty and fragile you may have just burned 3 days tracking it down.

If someone is asking this question, it is entirely appropriate to tell them "don't ever do that" - they obviously lack the experience and big picture thinking to know when it is appropriate to deviate due to business reasons. If you leave it at that I agree you're being a dick - whenever shooting someone's idea down you had damn well better have some appropriate alternative options.

View on HN

"Because that code works fine for 5 years until it blows up after something upstream changes."

I believe the point here is exactly that nothing will change, because the regex in question isn't for a service, it is just for that specific file, right now, today. I've regexed specific HTML files myself too, because even though I am very comfortable with XPath and Beautiful Soup and tree representations in general, regexes are even easier on a static file like that.

We're not talking about one-liners used on a single file here (as far as I could tell) - if that's the case, use whatever quickest hack you can think of to get the job done. Use-once coding is entirely different - everyone loves some code golf once in a while, but it doesn't mean you commit that to git :)

For re-usable tooling though, after some time you tend to avoid design patterns that you've personal witnessed break down repeatedly and cause issues.

"We're not talking about one-liners used on a single file here (as far as I could tell)"

Well, we are, because to quote coldtea, the person you directly replied to, "'Don't have a continuously running service parse arbitrary HTML with regular expressions' would be a bad thing. Parsing specific, given, HTML files, with known structure to the dev, with regular expressions (e.g. as part of a one-off scrapping script) is totally, absolutely, fine." In context the first sentence clearly means that running the service like that would be a bad thing (hooray English and it's deep ambiguities in double negatives).

I still wouldn't necessarily be too upset about telling a junior dev to be suspicious of REs or avoid them (for instance, parsing HTML with REs typically requires using non-greedy matches, and there are some subtleties around that), but if you know what you're doing on a quick job it's fine. If you're a senior dev and you still haven't figured out what's likely to entrench itself and what really is a one-off job, well, that's your real problem. I haven't been surprised about what gets entrenched in quite a while.

When I was young, I was asked to print some address labels, so I wrote a quick super-short BASIC program to parse a specific file, right now, today, and got those labels printed. Made all sorts of assumptions, but it didn't matter, because it was a static file, and nothing could ever change in it.

Three or four years later, I was having lunch with the guy I'd done that for when he got a phone call. Turns out they were using that program of mine to print labels monthly now, and one of the completely-safe assumptions I'd made years earlier had bitten them. Fortunately, he knew how to resolve it easily, but I learned a very important lesson that day.

Was the lesson that there's a fortune to be made in software maintenance?

What lesson was that?

From my perspective, this other guy was using a program outside of its design spec. That can be fine, but such a thing is prone to the problem of safe assumptions suddenly not being safe. You wrote the program for the problem you needed it for, and for which, it worked fine. Unless part of the problem at the time was "we expect to use this program for a couple years", there is no reason to make it overly robust.

"right now, today" and "that specific file"

Code designed to be used once has a habit of sticking around.

https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it

There is premature optimization and then there is not being stupid. Using one of the 10,000 XML/HTML parsers to get to v0.01 takes about as long on average and avoid ridiculous problems.

The only thing regex has going for it is you might already know it. But, this is one of those time when learning something is worth it.

PS: XP is doing a depth first search, which can be EXtreamly wasteful.

There are plenty of disposable code situations where the code wont break after 5 years, because it does not exist after two months. Also, failing regexp on likely generated html like that likely wont take 3 days to find. That is ridiculously high estimate even for messy codebase.

You are trying to make estimates and decisions for project you know literally nothing about. That is about worst practice of them all.

In case the question was asked by someone lacking the experience - well then it is absolutely ok to learn regexp by trying to parse info few downloaded files of toy scrapped site. Having newbie fight like this just to learn makes no sense.

> failing regexp on likely generated html like that likely wont take 3 days to find

This is true, but only because you're implying that the reason (failing regexp on HTML input) has already been identified. Maybe the parsing job fails silently and then people down the line (who have no idea that the data was passed around as HTML at some point) wonder why no data is coming in anymore, or why they're seeing incomplete data. That could take weeks to trace back to the parsing problem.

Silent failure seems to be root cause of delay in situation you described. Ast based parsing can be made to swallow errrors too.

And parsing the HTML to an AST won't work either in 5 years, the format of the tree will probably have changed and you will be getting index errors.

Serious question... why would any other approach be exempt from the problems you stated?

Nothing is exempt, some things are far less fragile than others though.

But are also significantly slower and consume hundreds of times more memory, and will still break with the first little change in the DOM tree on the path that you rely on - which smart regexps can sometimes handle. For instance if you're looking just for a title and a price of a single product (common requirement for spiders), you can extract that with regexp without caring about the html page at all, just pick the <h1> and something that looks like a dollar value. I had spiders like that kept working after a full redesign of the site where even the platform was moved from WP to Magento. No DOM parser could handle that, plus they would be too slow in the first place. You need to pick the right tools for the job, everything else is BS...

Reply on news.ycombinator.com