Hacker News Reader

Comment by drpfenderson | Hacker News Reader

Wholeheartedly agree. It's sad to slowly see the bitrot in my own bookmarks, trying to pull up important resources or media from only a few years ago only to hit a 404 or domain placeholder. And many times there is nothing on Archive.org or similar.

That ArchiveWeb extension looks excellent for custom captures. Picking the pages and elements you want during capture is a great feature.

For quicker one-offs, SingleFile gives you a single HTML with all the media of a page: https://www.getsinglefile.com

If you're making archiving a more involved part of your workflow, I highly recommend ArchiveBox. It can save pretty much every type of file from everywhere, in almost every format you can imagine, while building a searchable index of the archives. Been using it successfully for many years: https://archivebox.io

View on HN

I have recently started using hister [0] this index’s everything I read and allows me to half remember bits and have a LLM find it.

[0] https://hister.org/

I am using both Linkding and Linkwarden self-hosted and archiving my bookmarks as HTML, PDF, and screenshot (I might be overdoing it) without the need for any additional dependency like client software or extension. Linkwarden works better at archiving things like Reddit, but now with the anti-scraping mechanisms in place it might het harder to do anyway.

I realized I really need this when one website which was a treasure trove for PowerMac technical information and assembly guides just went dark. Some of that info I could never find elsewhere. Linkrot sucks if you're in the middle of an (admittedly drawn out) personal project and your source of information vanishes with no trace. Belt and suspenders from now on. Might give Hister a try.

Can vouch for Hister! In the fairly short time I've been using it, I already had a few sites disappearing or changing content and hister kept the juicy information accessible.

It was painless to set up and quick to use when needed, too.

I also use Singlefile and Obsidian Web Clipper a lot, but Hister has been great and is now a first-class citizen in my system. It helps it works passively for those things that don't look important when you read them, but become relevant later.

I just started using it, and to my surprise it only saves the sanitized html content, no scripts no css. I'm thinking of pairing this with archivebox. But hister is amazing. I love it. It's from the creator of searxng if I'm not mistaken which is another piece of great software.

By default it only keeps the latest copy of sites so I'm not sure how that prevents replacing with 404?

When it gets a 404 it doesn’t overwrite the existing copy

What if you get a 200 but just a "This is not available any more"?

I used ArchiveBox for years. It's really good. The self-hosting and maintenance gets annoying after a while though.

Link rot really bothers me and I want people to get into archiving web pages so that eventually there will be a decentralized mirror of the unslopped web sitting on people's hard-drives, ready to share whenever someone needs it. So, I made a worse version of ArchiveBox ( https://memoryhole.app ). My philosophy is that, if people see how it is easy to start archiving and how useful it is, then they'll keep it up with SingleFile, or ArchiveBox or anything else.

I've been using Archivebox for years also, but it recent years it started to have multiple issues. My queues are full of pages that failed to scan, and it's been several versions since I actually had anything new there.

I did small amount of troubleshooting a few times with no resolution, but now I'm thinking of switching my thought process. Maybe something like a "web clipper" is much better, as complex SPA pages often fail to archive well anyway. And I should focus on content itself, not the page. That SingleFile extension fits well here as well.

Did you run into similar dilemmas at some point?

It's funny - I actually went through a similar transition and also realized it was a little too heavy-duty for my personal workflow.

At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads. For larger, or multi-page, captures I still fire off an Archivebox command. I run into blocks occasionally, but it also just feels like the open web is becoming less and less open in general.

Have you tried ArchiveBox since v0.9 was released? v0.9 was basically a complete rewrite with a new archiving engine, and it ships with ArchiveWebPage built-in as one of its plugins now.

As of the latest v0.9.77 it's now tested to be able to handle 2 million snapshots / 2TB on a 4CPU server, archiving @ 1 URL/min consistently while serving all the web UIs @ ~100ms. Also has automatic importing from social media, google docs, notion, canva, figma, and a growing list of sites with self-healing AI scripts & cookie sync from your browser so it doesn't need as much babysitting.

> At this point I'm using Linkding (self-hosted bookmarks, which has an archive function) paired with Singlefile (which integrates with Linkding) which locally captures and uploads.

I second this workflow, exactly what I switched to from Archivebox. Linkding has a built-in "create HTML snapshot" feature but the Singlefile browser extension works much better.

I've used OneNote back in the day for that.

Nowadays it's Obsidian and Obsidian Web Clipper browser extension. The only caveat is what doesn't save the images locally by default and needs some manual action: https://obsidian.md/help/web-clipper/capture

SingleFile is great for single pages. All the tweaking I do is enable or disable saving scripts, and it works for >90% of webpages. Output being a single file (heh) simplifies things enormously, there’s no hierarchy or folders to deal with - hard to imagine your saved files not loading or breaking in some way as long as the original capture works.

SingleFile has a nice batch download feature as well. It's a little rough around the edges, but useful when needed.

I haven't tried archivebox, but self-hosted https://karakeep.com/ has been working well for me. When the server-side parsing fails I'll fallback to using SingleFile in my browser which then forwards it to my Karakeep instance.

I can vouch for Karakeep. With an LLM API key, the app even automatically tags and summarises each entry for better search.

I’ve been going over the art that I’ve saved in the past 15+ years and it’s quite sad when I go look for the original source of an image and find out that it’s completely gone from the original website. No indication that they’re on some other website, at least by reverse image searching it. Definitely has made me glad that I’ve been hoarding these images on my hard drives, with the total count being somewhere over 300k at the moment. I think the oldest ones have now been migrated across four different computers. Kinda sad for everyone else since they can’t see them anymore though.

Haven’t started saving web pages yet, but I definitely should though. My home server has some tens of terabytes to spare, so that’s at least not an issue. Been eyeing these comments to find a good self-hosted solution.

I'm using Linkding hooked up with SingleFile to bookmark+archive all important pages. Highly recommend! https://linkding.link/archiving/#using-the-singlefile-browse...

On MacOS, I use my own built app called HEAP which lets me archive in various formats locally (full page image, pdf, markdown, html, and webarchive) via a single right click of the menu bar button:

https://apps.apple.com/ca/app/heap-website-full-page-image/i...

Disclosure: I am the developer.

I use AnyBox on macOS to do this for bookmarks. It saves a WebArchive copy and a stripped 'reader'-esque version. SingleFile can also pipe into AnyBox, if you want to use that as the archive.

Avg bookmark is valid for 5 yrs.

My bookmarks file is a disastrous graveyard as I have 20 yrs’ old ones…

And as a publisher I don’t link fresh sites (don’t even mention Netlify, Lovable, Vercel…) … Vannevar Bush would be surprised how his Xanadu evolved.

singleFile is great, and the extention works on fennec/waterfox on android as well (not sure about firefox mobile)

another one i use when i only need to save the text of a webpage is markdownload, which formats the page using markdown

https://addons.mozilla.org/en-GB/firefox/addon/markdownload

Archivebox annoys me because (as far as I remember) it doesn't have any options for capturing a webpage on your end and uploading it to your self-hosted Archivebox instance. With numerous sites moving to Cloudflare to mitigate AI agent crawling, it makes it really hard (or outright impossible) to capture a page because it wants to do it server side.

The ArchiveBox Browser Extension https://extension.archivebox.io/ has this built in. It captures both full MHTML and screenshots and uploads it to your server automatically, it was added a few months ago.

Also works from iOS and Android so you can save logged-in content directly without having to set up cookie sync (which it also supports now too directly from the extension).

This is basically the main reason that I started building https://github.com/mfinelli/recueil. It _only_ works in your browser so you can bypass captchas or paywalls or whatever and then it uses singlefile (https://www.getsinglefile.com) to upload the result to your self-hosted archive.

Reply on news.ycombinator.com