HTML5 does actually support an XHTML serialisation[1] if you need your content to be easily parsed with XML tools.

[1] https://www.w3.org/TR/html5/the-xhtml-syntax.html

XPath is defined as operating on an XML infoset, not a DOM. And there's no defined mapping from a DOM to an XML infoset anywhere. So really we're in undefined territory.

In reality in browsers, that coercion to an infoset never actually happens, and XPath is matched directly against the DOM, which leads to differences in edge-cases (notably, adjacent text nodes, which the DOM allows and the XML infoset does not).