Hacker News Reader

Comment by Asparagirl | Hacker News Reader

If the destruction of all this unique user-generated content pisses you off -- and it should -- then feel free to join ArchiveTeam and help save this (and other) content for the Internet Archive. Check out the ArchiveTeam wiki at archiveteam.org, or come on over to #archiveteam and #archiveteam-bs on efnet throughout this week to watch a self-organizing data rescue mission happen in real time. (There will probably be a new dedicated channel spun off for this project later today or tomorrow.)

In the meantime, programs like youtube-dl and livestreamer are your friends.

http://rg3.github.io/youtube-dl/documentation.html http://livestreamer.readthedocs.org/en/latest/

View on HN

To elaborate, the ArchiveTeam distributes a very simple-to-use virtual machine image that you can run. When you do so, your machine will help download and collect from disappearing websites. I don't think they've got scripts for justin.tv yet, but it sounds like they are going to, so now would be a good time to get your VM ready.

It's easy, and the more people who run the machine, the more power the ArchiveTeam has at their disposal. I encourage everyone to give it a look.

You can get the virtual machine here: http://tracker.archiveteam.org/

Right, the idea is that justin.tv could be formatted as a new project for the ArchiveTeam Warrior -- which you can run on your laptop or a VM or a cloud server or a few Amazon instances, or whatever. But with only a week's notice before the shutdown, I don't know whether they can get the customized scripts up and running in time. They're working on extracting a list of user and video id's right now.

it has stopped working for me, what about you, it still works ?

Everything seems ok for me. What is going wrong?

It's worth noting that you don't have to use a virtual machine. If you're familiar with compiling basic utilities and installing Python packages, then you can run it without the overhead of a virtual machine. It's generally all on the wiki/Github.

I downloaded the VM today and spent a fair while trying to diagnose a CheckIP error. It turns out, after going on the IRC channel that none of the projects on the VM are up to date and hence the VM can't be used to help the project. Hence the error. If I see the website update to mention justin.tv I'll certainly fire it up. But it seems a shame that there is nothing I can do to help with their project as a whole; I couldn't even see any way to donate - I guess I should just donate to archive.org.

It honestly disturbs me that Myspace, Justin.tv, and these other companies don't seem to ever consider shipping their soon-to-be-deceased data to the Internet Archive. Why not help preservation a little? It'd likely be simpler to handle than the oncoming denial of preservation defense that the ArchiveTeam launches. Have there been any examples of companies doing this?

You might think that this is all useless information, but I guarantee we'll look back in likely only a few years and think in horror at all that we've destroyed.

The best art blog about a half decade including a kid who's dead now just disappeared. When Myspace deleted all of their content, I lost all the content of my dead high school friend (turns out you need to back up everyone and thing you care about, not just yourself[1]) and @maxfenton mentioned "The best art blog about a half decade including a kid who's dead now just disappeared.". It's not an isolated problem.

I almost feel like we need a name-and-shame registry and/or public awareness campaign. I'd love to see this written into the user rights of a website when it launches.

"If the website is to be shut down, all publicly accessible content will be sent to the Internet Archive. If you would like to opt out, tick this box."

[1]: http://smerity.com/articles/2013/deleted_digital_tombstones....

As expected, the dedicated channel for the data rescue has now been set up: join #justouttv on efnet. Right now, coders with Python, Lua, and/or Redis knowledge would probably be most in demand, given the existing toolset. Hopefully very soon there will be a new Warrior project formally set up, and then anyone who can run a VM or an AMI or install the tools from the command line would be able to join in to distribute the downloading.

During the Hyves.nl rescue in late 2013, ArchiveTeam did accept donations to cover Amazon instances earmarked for that project, so it's possible they may do something like that again this time.

In case people are curious, the Warrior code and seesaw code are here:

https://github.com/archiveteam/warrior-code2 https://github.com/ArchiveTeam/seesaw-kit

"If the destruction of all this unique user-generated content pisses you off -- and it should"

I'm not sure why it should. When did we get this strange idea that when you make some piece of content, no matter if it's quality or not, it deserves to pollute the cyberspace forever.

I realize this is an unpopular opinion, but one of the reasons nature "works" is that everything old is allowed to dissolve and become the base ground for the new that will come after it.

People naturally copy content that they find interesting. And content which isn't that interesting naturally gets lost. Maybe that lost content stuck around enough to inspire someone to do something in that spirit later or (which also counts), but it doesn't survive itself.

Imagine if everyone gets Glasses one day and starts doing non-stop 1080p video stream of their lives. Are we going to be "pissed off" at that being deleted too?

If you do the stats you'll notice that over 99.99% of the content in archive.org is never accessed. Nobody cares.

And that's how it should be. We should really allow content to die. I have the firm belief that information doesn't just "want" to be free. It also wants to be allowed to die gracefully at some point.

People sometimes make copies of things they find interesting, but those copies are rarely accessible to the public. It can be pretty hard to get your hands on something after it disappears from the web. For example, forums.truecrypt.org went down three days ago, it's of interest to some people, and you're going to have a hell of a time finding some guy who happened to launch HTTrack on it before it went down.

Valid points, but you need some strategy for pruning the archive.

We can't just keep everything forever. First it was text files, then html files, then html files with images, and now videos.

Without pruning, the task of archive.org will become impossible, so the question of what's important and what isn't will have to be decided at some point.

I'm currently running through a Peter Norton assembly tutorial from the 1980s. It's already helped me understand the code better than I have before. Should that content have died, just because it's old?

If you'd re-read my post, you'd notice my focus is on the content's quality, and not its age. Age serves as a useful threshold to review if something should stay or not, but it's not the sole criteria.

Important things will be copied and preserved. That's the natural order of things.

Despite I expressed myself very clearly, I expected two types of reactions: knee-jerk reactions like yours, which argue against something I didn't say; and getting down-modded, because the currently accepted wisdom of HackerNews is that we should hoard any type of information regardless of its merit.

Even though it's explicitly against the rules to down-mod to disagree with someone's opinion, every scoring system gets abused this way eventually. I find this sad.

>you'd notice my focus is on the content's quality,

But then you redefined it as 'popular' and 'interesting' content. While something that is popular and holds the interest of people might also be quality content the reverse is not necessarily true. How many quality on maths or chemistry or history or engineering or some remote tribal language or what have you are popular? (Lets assume some arbitrary popularity metric like a million views) Sure you might be able to popularize the 1-2 minute nugget of incomplete/dumbed-down information but I don't consider that quality. And I'm quite sure that you won't be able to actually do anything by knowing some random formula/effect/trivia without the multi-hour lectures associated with their fundamentals.

>Important things will be copied and preserved. That's the natural order of things.

I disagree. We've lost countless important historical documents because they were neglected/destroyed/etc. What you're defining as the natural order is just the 'winners' re-writing history. Its already easy to alter history by altering Wikipedia or buying a news publication and making certain articles un-crawlable. What if Google deleted the cache or Wikipedia did not keep a history of changes? "If you can't Google it, it doesn't exist" is pretty much how the future might unfold. I think its important to preserve even unimportant things.

You might have a point or two in what you're trying to say, but you should really consider the broader argument. Oh and I did upvote you. :)

Who's to say what we'll find important down the line? Quite often what seems mundane and not worth the effort to copy or maintain turns out, in hindsight, to have true value. One of if not the most important insights in studying previous cultures comes from going through their trash heaps. Similarly, while they could seem pointless, a lot of this content could provide unique insight on our own culture down the line that 'valuable' pieces saved and curated can't.

See, for example, some of the BBC shows that got deleted because magnetic tape was too valuable to just store popular culture on.

Recently the BBC has been asking people to search attics and sheds for home recordings (taped from tv or radio broadcast) or "lost" tape (probably meant for erasure and reuse that got ahem 'lost' on the way and ended up in a BBC technician's collection).

I find it weird that mega had petabytes of poorly de-duped content (very many copies of the same pirated movie in different rips) yet we're talking about deleting content because, well, because.

Having said that I'm not giving them any money to keep it archived let alone online so perhaps I need to shut up or put up.

There's value in preserving the mundane. If we only preserve things which are noteworthy, the cultural artifacts, the special occasions, the celebrations and wedding dresses, it's easy to lose track of the fact that it's worth remembering what everyday life was like, too.

Reply on news.ycombinator.com