Hacker News Reader

Agreed. That’s really my thoughts.

The internet or rather the crawlable WWW itself has changed.

The WWW content itself is no longer what it was.

Do you think the proliferation of web frameworks makes it harder to crawl?

Probably has some influence. Because everything needs to be an app, implemented as an SPA, rather than just a web page. Hypertext is dead.