I'm downloading an entire forum now. I did not have much luck with tools like HTTrack. LLMs are very useful to create a custom scraper. I indexed the forum first (got URLs and titles of every thread) and then I supply the index to a second program that downloads the threads. I even managed to get third party embedded image assets pulled in, just one extra description in the spec before feeding it to the LLM. I keep getting blocked after some 200 fetches, but I have a dynamic IP.