I'm downloading an entire forum now. I did not have much luck with tools like HTTrack. LLMs are very useful to create a custom scraper. I indexed the forum first (got URLs and titles of every thread) and then I supply the index to a second program that downloads the threads. I even managed to get third party embedded image assets pulled in, just one extra description in the spec before feeding it to the LLM. I keep getting blocked after some 200 fetches, but I have a dynamic IP.

I have been using HTTrack is cases where I want a browsable local version of any site (not just a page)! Most of the applications mentioned here as comments only allow for saving a single page. But afaik, only HTTrack allowed me to save a local version of the website and not just a webpage. The “depth” argument allows me to do that when downloading a site through CLI

HTtrack was great when pages were static or server side generated but now that lot are rendered by JavaScript on the client HTtrack isn't much use.

https://github.com/archiveteam/grab-site works well too, but neither are great with JS