Hacker News Reader

This is really interesting, is there a way to provide downloadable caches for websites where that would be legal? I could Imagine just pre-downloading wikipedia, stack overflow, all kinds of documentation, etc. In a compressed and preorganized format instead of scraping it every time.

View on HN

The cache is pure and straight files-in-folders. This makes it trivial to browse the cache by hand:

cd .cache/offpunk/https/news.ycombinator.com/item cat "id=46943752"

So it could be trivially shared.

The "netcache" tool gives you the cached content or, with --path, returns the path were to find the contentd.

The only point is to preserve the file-modification attribute, which serves to know the age of a cached ressource.

Kiwix basically provides this. You can use kiwix-serve to serve the downloaded zim files locally.

I tried to add kiwix support to Offpunk in the early days but kiwix do not support incremental update of the wikipedia database, which makes it a huge pain point.

Yeah, that’s the biggest disadvantage of kiwix. It probably makes their hosting bill enormous as well. Zimdiff/zimpatch was implemented a decade ago so I’m not sure what the holdup is.

Reply on news.ycombinator.com