Hacker News Reader

Comment by tannhaeuser | Hacker News Reader

> HTML in the wild was a mess, and required a lot of error recovery and empirical quirks

> HTML in the wild is not quite as bad as it was and is much more easily parseable as well

I beg to differ with the "HTML in the wild" concept and need for heuristics. HTML, including HTML5, was and is easily parseable using SGML, the markup meta-language on which HTML versions up to HTML 4 were specified. My DTD for HTML 5 (W3C's current HTML 5.2) can parse up to 97.31% of the test suite normatively referenced by the HTML spec (the rest being somewhat in a grey area containing constructs to make test automation happy, which is designed to never fail, but rather to produce something still in a predictable way) [1].

[1]: http://sgmljs.net/docs/sgml-html-tutorial.html (reported on the slides reached via the "TALK" link)

View on HN

"the HTML spec" is definitely not the same as "HTML in the wild", so your comment has relatively little to do with what the parent comment is saying. Real web pages contain all sorts of sins (or at least did, back in the 90's) such as unclosed tags and missing elements that are "required", and browsers were expected to parse those pages because other browsers did. As a simple example, <b><i>foo</b></i> is not valid HTML 4 (or even SGML) because the tags are not strictly nested, but it would be suicide to make a browser that rejects pages containing that.

"it would be suicide to make a browser that rejects pages containing that" Nonsense, browser should have never supported it. We can't decide after the fact yes indeed.

The idea that HTML is hard to parse isn't exactly controversial; the HTML spec has a long description of the exact rules you're required to implement to parse HTML and it's a bunch of special cases and error handling. Certainly I wouldn't call it "heuritics" any more since everything is well specified, but equally certainly the rules are derived from heuristics that browsers implemented and are now required to maintin compatibility.

I'm having difficulty unpacking your claim about a DTD "parsing" HTML5, but I find it hard to believe that a primarilly DTD-based parser would get all the rules right around, say, foster-parenting, or handling of -- in script elements, or pseudo-entities or any of the the other weird quirks that are specific to HTML and not shared with SGML languages. Perhaps you can link the code and details about the claimed test suite pass rate?

I'm enjoying finally meeting someone to challenge my claims ;) I haven't published my test scripts yet (they're not very exciting and just extract test data and add a DOCTYPE anyway), but for the time being you can just follow the instructions given in the tutorial I linked to see whether the cases you're concerned about are covered (the tutorial also explains how to deal with other common HTML parsing issues).

Nb: "<!--" in script element content isn't a problem for parsing as such and was just used to stop browsers rendering JavaScript code text during the introduction of JavaScript back in 1995 or so (see also [1] mentioning the consequences for treating script content as CDATA or #PCDATA). Predefined HTML entities are also fully supported by the WebSGML revision of SGML. Don't know about "foster-parenting", but SGML has very specific rules for tag inference to accomodate elements not allowed at a given context so I'm assuming you're referring to that (also described in detail in the linked paper below).

[1]: http://sgmljs.net/docs/html5.html#script-data

Reply on news.ycombinator.com