Hacker News Reader

Comment by ineedasername | Hacker News Reader

I'm surprised IA relies on outside, potentially ephemeral services at all. I thought all of their material was self-hosted. Had that actually been true, it seems like IA wouldn't have needed to delete its account over an issue of whether their usage fit the intended use case for Flickr.

View on HN

The problem is, the internet archive is great at storing data, not not the best at making that data discoverable, viewable.

A partnership like this makes quite a bit of sense.

It seems like there'd be a lot of volunteers to help with building their own. On that scale, I'm not sure if/which FOSS option might be suited. But the sheer technical challenge of scaling a system to > 100 petabytes along with a compelling mission would attract a lot of talent. Heck, if they'd just worked with Flickr long enough to extract the metadata created surrounding these images so that all that effort wouldn't be lost then I bet there would be ample volunteers just to reimplement a way of utilizing it. To discard all that work... I can only.hope that no actual data is lost and they will be able to do something with it in the future.

Uncoordinated volunteers are worse than useless, sadly.

Why would it be uncoordinated? Volunteers work for FOSS after all, and if the work for IA was also under a FOSS license I don't see why it couldn't work. And IA is high profile enough to attract really talented people from the FOSS community that already understand how to work under that paradigm.

You can already do this without building a huge storage system or needing any sort of permission from the Internet Archive. Anyone can upload material to IA, and anything on IA can be embedded into other webpages. You can dream up the ideal user interface and build it without needing any permission or cooperation from IA.

Sure, but the point is to have IA be the owner of an archive w/ a good, well performing UI/UX. Because:

I might build an excellent interface to IA's millions of images, so good that it becomes the default way of viewing them. Countless bookmarks and links from other sites are published against the material through my site. But then one day I decide it's just not working out anymore and I pack it all up. I call this imaginary system Fleckr.

I'm not saying people shouldn't build what they want in this way, but IA shouldn't rely on other services to host anything that it wants to stick around. They appear to have done that with Flickr, because Flickr had a good system, but now it's all dust. Or, if not dust, it will require a complete system to be built to do something similar, migrating user contributions (if they were salvageable) to this new system. I'm saying this new system should be owned & operated by IA. Building it w/ volunteer work under a FOSS license like many other large projects seems a reasonable option.

We don't know this was the doing of Flickr though, I mean it seems like there could have been an intermediary step where IA could download all the comments and such? I mean nothing was stated at all, and that's a bit scary that the IA would just not say anything and allow all that to be deleted nonchalantly.

That's what I'm hoping. Although I doubt that IA, on it's own, would suddenly go to Flickr & say "Hey our use case doesn't really fit what you intended, let's figure out how to go our separate ways."

My guess is that the conversation was initiated by Flickr. Whether IA agreed with them from the beginning or was forced to "agree" probably doesn't matter, especially because I can see the merit of the argument that Flickr may not be an appropriate platform. I'm just hoping things were save in a way that they'll be able to implement a replacement at some point.

this is why archives should be independently duplicated, to head off a massacre.

what isnt clear to me is the survival of the content, was there a data migration or was it all just nerfed?

All the images still exist on the internet archive, just not in a format that can be explored.

What's been lost is the efforts of all the users to separate the interesting images from the chaff.. and organise them into collections. For such a large corpus that's a really valuable thing.

Also links people had to those images are just dead.

And all the images users had previously favourited have just disappeared from their collections.

what a task it would be, to reconstruct the relations and cobble up a db from user data.

>this is why archives should be independently duplicated

It's an interesting challenge with IA because they have > 100 petabytes of content, many thousands of $ in hard drives alone. Too much for random individuals to decide they want to host their own clone & become a mirror. So, we're be looking at large institutions or companies to do this philanthropically. Or, here's the tech challenge, some distributed mechanism where participants all host a portion, with enough redundancy that a large % could go offline and still keep integrity.

Reply on news.ycombinator.com