Hacker News Reader

Comment by sp332 | Hacker News Reader

I guess this answers why they're not doing archives anymore. 1,000 TB of videos with fewer than 10 views!

Yeah. Do we know what Justin.tv's hosting costs for all that data was, or what infrastructure they used to store and serve it? That's a lot of data to take care of for a service with no direct monetization.

It is really out of date but the high scalability article has some information http://highscalability.com/blog/2010/3/16/justintvs-live-vid...

Where do you see that statistic?

The rest of the tweets in the thread:

> What about grabbing the 9 views, 8 views, etc? We have more time!

-@erazmus

> The problem is that at some point, we end up with a LOT of disk space used for VERY obscure videos.

> Normally, archive team does not care, but in this case, the disk space will increase to over 1.1pb. PETABYTE

> That's space that can be used to save a LOT of other work out there that needs love. Tough decision.

- @textfiles

They said they archived 10TB (and that covered everything with at least 10 views), and that the total including all videos would be 1.1PB, ie 1100 TB. So there is over 1000TB of videos with less than 10 views.

1.1PB would cost $56,000 at the cost of Backblaze's latest pod, based on their recently posted pricing.[1] That would be for live storage in physical computers. If they put it on tapes at $0.01/GB, it would only cost $11,000 for the tapes.

1: http://www.tuaw.com/2014/03/19/backblaze-now-storing-100-pet...

In the posted thread.

Read the rest of the conversation:

  >What about grabbing the 9 views, 8 views, etc? We have more time!
  >The problem is that at some point, we end up with a LOT of disk space used for VERY obscure videos.
  >Normally, archive team does not care, but in this case, the disk space will increase to over 1.1pb. PETABYTE
  >That's space that can be used to save a LOT of other work out there that needs love. Tough decision.

I wonder if Usenet is at all a consideration for excess archival. Any NSP I've heard of allows for unlimited uploads. Sounds like a good use for the archaic system.

Not an option. Usenet doesn't preserve data long-term; it's effectively an epheremal medium. (Some archives do exist, but they almost all skip high-traffic groups.)

Most premium providers haven't deleted binaries in several years. At worst, it could at least be temporary archival until more permanent storage is acquired in the future. For example, place the 0 views Justin.TV vids there until ArchiveTeam and justify the space themselves.

0-view vids are over (estimated) 1,000 TB! I don't think most Usenet providers have that spare capacity.

Yea, that's a valid point. I'm pretty sure some could handle it, but many peers may not be able to. Using Usenet for mass archives would have to be done gradually. "more than 25 petabytes of Usenet messages on the Giganews servers." ... "Giganews is 50 percent bigger today than at this point last year and we continue to add servers with this kind of growth in mind." [1]

I just think it's an under utilized medium for this sort of media conservation.

1 - http://www.giganews.com/blog/2014/01/giganews-reaches-2000-d...

Reply on news.ycombinator.com