Comment by enumjorge | Hacker News Reader

Comment by enumjorge | Hacker News Reader

Parent Full thread

Not only that, but I also wonder if we're overestimating the value of keeping all of this data around. Who's going to have the time to search and curate these mountains of information when we're generating tons more of it every day? I imagine the ideal goal is to allow future historians to learn about our past selves, but I think there's a tipping point where only those with lots of resources can afford to meaningfully consume it. Those typically are wealthy companies or individuals, and I'm generally less excited about what do with our information.

Obviously there's value in archiving some information, but a save all or even same most approach starts sounding a little hoarder-ish. Sure you might one day make use of that 1997 November TV guide, but chances are you won't and in the meantime you're paying the opportunity cost of storing it.

Maybe we need to take a page from Marie Kondo and only keep that which sparks joy and learn to let go of the rest. There's a chance someone will need a bit of info that no longer exists, but we'll probably be ok.

Part of the challenge here is that it's hard to know in advance what is or isn't worth archiving. It may only be clear a few years later that some big chunk of now-dead data was important.

In that sense, curating all of it doesn't really matter as long as you archived it. Someone trying to find the data later (or curate it!) can find their way to the right URLs using other sources, and then begin the process of curating this archived data after-the-fact.

The internet archive is most useful for when you click a link and it is dead which is very often. The wikipedia references are filled with dead links which now point to IA.

There is probably a lot of junk data on IA though especially video site archives but its worth keeping stuff that isn't needed if it means keeping stuff that was useful.

> Who's going to have the time to search and curate these mountains of information when we're generating tons more of it every day?

Presumably some sort of search engine, not a person.

Well, there are some tools that have been developed that have pretty amazing capabilities to crunch through staggering quantities of data and come up with useful insights. It's basically big g's core competency, and there are tons of other companies that do the same thing, as well as open-source solutions that can be used.

Reply on news.ycombinator.com