Yeah, I also think there should be a more fundamental discussion about where the web is heading. Instead of fixing the tools perhaps one should think about a new foundation for everything. Not sure HTML etc. is it.
And HTML trundled along unwanted in WHATWG through those times, scorned, rejected as inadequate and not suitable. And yet, XHTML2 got closed down, everyone switched back to the HTML path.
And processes involving changing web development from an environment to an output format - Google's GWT and Dart, don't seem to have gained much traction.
I don't know what happened to Xanadu, either.
You probably should dig into the history of Rich Internet Applications (formerly XUL, before Mozilla decided to stamp out the naming confusion with their own XML vocabulary), in the days before Ajax really stabilised and things drifted back into the browser.
In an ideal world, now that the web has shifted from being a bunch of linked documents to complex applications, we'd have a development stack aimed at developing applications, with sane means of specifying UI layout and behaviour, low overhead client-server comm protocols etc.