There's a separate issue that a lot of stuff requires JS, but the JS mostly just calls JSON endpoints, so that's easy to scrape. The tricky thing is scraping ASPX sites that jump through a bunch of hoops instead of having a simple backend API.
Next, HTML2+SGML were also well-designed, and people weren't just doing regexp. The mess didn't come in until HTML3 and even more so, HTML4.
Today, it's easy to parse /specific/ pages. If I want to automate one web page, and it's well-formed, HTML5+AJAX makes that easy.
However, in contrast to HTML2, it's very hard to parse pages /generically/. That's why I gave the example of Altavista and a11y tools, which need to work with any web site.
Try to make something like that today: a generic search spider, a web browser, or an a11y tool. See how far you get talking to JSON endpoints. They're often easy enough to reverse-engineer for a specific web site, but you need a human-in-the-loop for each web site. With HTML2, one would build tools which could work with /any/ web site.
And boy were there a lot of tools. Look at all the web browsers of the nineties, and the innovation there.
The early web were the era of dozens, perhaps hundreds of competing web browsers which were made possible by simple, well-engineered web standards. Pages were served statically, or with CGI scripts. You had a whole swarm of generic spiders, crawlers, and bots which automated things on the web for you. Anyone could write a web browser, so many people did.
The dot-com boom/bust had companies doubling in size every few months, people who could barely code HTML making 6-figure salaries, Netscape imploding, early JavaScript (which, at the time, looked like a high schooler's attempt at a programming language), and web standards with every conceivable ill-thought-out idea grafted in.
If one of the first programs you ever wrote was a scraper for an ASPX site, you never saw the elegance of the early days. ASPX came out not just after HTML3, but after HTML4.
Most definitions of the early web include the PHP Cambrian explosion because essentially all websites today got their start then and only a few horseshoe crabs sites (mostly the homepages for CS profs!) predating it survive. Gopher sites were also probably really easy to scrape too. ;-)
But my original comment was 100% unambiguous: "I liked HTML2. I hated basically everything which went into HTML3 and HTML4."
Y'all responded by citing bad examples from the HTML3 / HTML4 era as examples of things going wrong...
---
Note: Before I get jumped on for "kid," it's the username.