Hacker News Reader

Comment by jhrmnn | Hacker News Reader

What I'd love is a program that automatically downloads and archives any article-form website on which I spend more than a minute in a browser, and makes everything full-text searchable. Does something like that exist? Perhaps it could be solved with a local proxy and some smart heuristics on request timing in a browser-agnostic way.

View on HN

The auto-save feature of SingleFile browser extension [1] allows you to do this. You will just need to adjust the settings to change the delay after which the page is automatically saved. SingleFileZ [2] can also do this and produce full-text searchable archives.

[1] https://github.com/gildas-lormeau/SingleFile

[2] https://github.com/gildas-lormeau/SingleFileZ

Probably a little hacky but using a userscript & an archive tool could work. I have a userscript[1] that I've been using for years to archive some news sites but it just uses the Wayback Machine. It can probably be modified to wait more time and send the request to something like ArchiveBox[2] (or any archive tool that has a web interface). The downside is you'd need to whitelist specific sites (or whitelist everything and have a keybind or some content matching to activate the archive function? idk)

[1] https://gist.github.com/antiops/00a37a1de289415fa7cbd9b5d1d2...

[2] ArchiveBox, can have full text search: https://github.com/ArchiveBox/ArchiveBox

I built https://mitta.us/ to do this, then I added GPT-3 to it for discussions and tagging. There's a bookmarklet and a Chrome extension for this as well. It's not quite released, but I'll let anyone test it if they want!

I would love to test it please

Question is which search engine can handle this properly though? The first part can probably hacked together with something like DEVONthink and some browser extension but the search features aren’t useful when you have tens of thousands of docs that you just randomly browsed

I think what you are looking for is pinboard.in. I have thousands of bookmarks, and I've never opened it once.

Reply on news.ycombinator.com