That used to work great for me, but recently it started to fail. It downloads a few pages but then it gets errors, as if it is detected and prevented by the server from scraping.
44 karma · joined October 1, 2024
That effectively keeps public domain works copyrighted, for most purposes, for most of us. Most libraries for instance have some EULA that says you can only use scans you order from them for non-commercial purposes, even if they scan from a public domain book. I do not know if that can be enforced in general, but it would not be fun to have to find out in court.
Unfortunately entire businesses are built around preventing people from using bots, for obvious reasons, so the only obvious way forward to make browsing the web a better experience will also mean ending up on the wrong side of that battle.