The complexity in modern browser engines has nothing to do with parsing invalid HTML and everything to do with the vast, absurd feature set of the modern web platform, and the complex interactions between the components.
The bigger issue is backwards compatibility; the web has bent over backwards to maintain it, which is a blessing and a curse. On the one hand it makes the feature set absurd, but on the other hand the web is (relatively) easy to archive since format interpreters for it are widespread, and pages can be built and then forgotten about because they still work.
I'd like to see that experiment done.
This is talked about in a few videos and articles by it's creators.