gwern has a very involved post on archiving as well, https://www.gwern.net/Archiving-URLs
Somewhere on my to-do list is archiving everything I visit on the internet. It's frustrating to know that I've seen something, but be unable to find it again.
Somewhere on my to-do list is archiving everything I visit on the internet. It's frustrating to know that I've seen something, but be unable to find it again.
I simply use the browser's (Firefox) save page feature (ctl-s). Then every now and then, I convert the folder with these pages to a squashfs image (which de-duplicates all the CSS, JS, image files that are saved multiple times). I then use shell tools to search (ls grep locate etc.). This doesn't save the URL, but I also maintain a private "bookmarks" Git repository for more interesting bits where each bookmarked resource gets its own file (in a descriptive hierarchy), along with thoughts and notes.
What works well is that I am somewhat selective, complete trash doesn't end up being archived, it's pretty space efficient. It's also simple to wade through (in the shell), each saving action is just a file and folder pair. Also often I use the reader mode Firefox feature, and then save that (i.e. I save what I saw, not what was delivered). What doesn't work so well is that saving the page is often a bit of a hassle, often Firefox reports save errors then I just ctl-s again to see if the previous attempt succeeded, if not, hit the reload icon in the download task list which apparently forces it. Also, that it saves embedded videos, haven't figured out yet how to disable that, so periodically I remove videos again. Also, when saving multiple pages from the same site, links to other pages don't go to the local mirror. (In cases where it's important, I use wget -m -k.)
- You wouldn't have to bother with JS files, they are removed from saved pages by default.
- You couldn't (easily) de-duplicate resources because they are embedded in base64 into the page. However, SingleFile can detect all the hidden elements and unused CSS rules/declarations (by computing the cascade). So the HTML and CSS are optimized. It is also able to group duplicate images (by using CSS custom properties). There are other options to make the document size as small as possible. Most of the time, pages saved with SingleFile are smaller than Chrome MHTML files.
- The saving process would be much more simple, reliable and could be automatic.
- Videos wouldn't be saved (by default), a snapshot of the video would replace each video.
- Maybe the "filename template" in SingleFile would help you to organize things.
[1]https://chrome.google.com/webstore/detail/save-as-mht/hfmodl...
EDIT: it could be related to the fact that SingleFile is maybe more strict regarding the HTTP header "Content-Type". For example, it will discard images with "text/html" as content type value.
[1] https://blog.chromium.org/2018/10/trustworthy-chrome-extensi...
You could take this and post process your internet history on a rolling basis to accomplish your goal.
1. it's a security liability. some content i load in a web browser is private and I don't want to archive or duplicate it anywhere
2. it means a lot of crap. part of archiving content is curating what gets archived and what doesn't. i didn't touch on this in the article, but it's a key idea archivists need to address. for example, archiving the page I'm typing in right now (news.y.com/reply) makes no sense at all because it's solely dynamic content and would mean nothing when browsed later.
So instead I send specific links I want to keep in my bookmarks system, as I mentioned in the article. It's far from ideal, but it's a much better compromised than archiving every time I visit my weather service page. ;)
If you happen to stumble upon a solution (obviously self-hosted/local, due to the unfiltered access to page content), I might be willing to contribute with configuration scripting or client/server splitting so that the bulk doesn't have to stay on e.g. a laptop.