Hacker News Reader

Comment by bratao | Hacker News Reader

We have been using https://github.com/rushter/selectolax as a faster alternative to BeautifulSoup with html5lib because many malformed webpages in the wild don't work with lxml.

View on HN

The problem is that libxml2's 20-year old HTML parser never supported HTML5 [1], leading to more and more problems with downstream consumers like lxml, PHP or Nokogiri. PHP recently switched to Lexbor [2] and Nokogiri to libgumbo [3]. That said, I'm hopeful to receive enough funding to implement a HTML5 parser in libxml2.

[1] https://gitlab.gnome.org/GNOME/libxml2/-/issues/211

[2] https://wiki.php.net/rfc/domdocument_html5_parser

[3] https://github.com/sparklemotion/nokogiri/issues/2204

libxml is xml parser, html5 is not xml.

It's a bit late to be saying that to people already using libxml because "It should be able to parse "real world" HTML." https://gnome.pages.gitlab.gnome.org/libxml2/devhelp/libxml2...

Speaking of which, I don't understand why not. It seems like it would have been trivial to keep html5 a true xml. I do not understand what the actual technical reason for not doing that was. Naively, it just seems like breaking compatibility out of disdain rather than actually useful progress. Saving a couple of characters every once in a while does not justify the change, so I presume there must be a better reason?

There was XHTML and HTML5 is a direct result of finding out that was not the right solution. The main issue that was being solved there was that browsers do not parse invalid plain HTML consistently, which XHTML solved by requiring invalid XHTML to be rejected outright. This did not work. HTML5 solves this by defining the parsing rules such that there is a concept of document being invalid, every sequence of bytes deterministically maps to one particular DOM tree. This feature essentially precluded basing HTML5 on either XML (simply impossible) or SGML (that might be possible, but is in fact redundant formalism and describing the syntax in prose makes more sense, as everybody is going to hand-craft the parser anyway).

They specified how tag soup gets forced into HTML5.

They could have just as well defined how tag soup gets forced into XHTML.

I felt XHTML had fairly limited adoption on the web and in many cases web page authors seem to have preferred the »render tag soup« approach that in most cases did the intended thing than having to deal with XML namespaces, proper nesting and escaping, etc. Even though in most cases HTML nowadays seems to be authored as if it was XML with every element painstakingly closed and often even making elements that need no closing self-closing.

Probably because XML would need to be extended quite a bit to accommodate all of the multimedia stuff, attributes without values or quotes, special names for certain characters, optional or disallowed closing tags and whatnot that's in HTML5.

I think pushing in both the layout design conveniences and the strictness of XML data transfer in the same standard would be quite bulky at best. In practice we'd likely see a lot of nasty security issues in implementations and so on.

Ah this answers my question in another comment.

Thanks!

Reply on news.ycombinator.com