Someone had success using GPT-3 to classify episodes of a podcast[1]. I imagine if you fed the HTML from the crawler into an LLM, it could come up with a usable classification for it.

[1] https://news.ycombinator.com/item?id=35073603