Hacker News Reader

Comment by JimmaDaRustla | Hacker News Reader

I'm probably explaining something most everyone here already knows, but:

I don't know what Glacier uses, but the "offline" storage is much the same how mainframes use tape - disk based files (datasets in mainframe speak) are migrated to tape after a given amount of time to decrease storage costs. These datasets needs to be recalled back to disk to be used again. This is nearly identical to how Glacier operates with S3 as explained by Amazon.

The reason tape isn't available immediately is because the tape drive, unlike a traditional harddrive, doesn't contain a reader. So multiple tapes typically sit in machines with only only a few readers that move back and forth based on requests made to read the tapes. If there are many requests, the amount of time to recall a dataset increases.

Glacier may or may not be using tape, but the behavior is nearly identical to how tape has been used in past systems.

View on HN

You're referring to https://en.m.wikipedia.org/wiki/Hierarchical_storage_managem.... AFAIK, most systems work at the file level, but if you address space is large enough, you could implement this as a virtual memory system that swaps RAM to disk, disk to slower disk, slower disk to tape, tapes to tapes stored offline, etc.

I don't know what Amazon uses, either, but I expect Glacier data to be pushed out to some layer that doesn't need power while idle at all, heavily checksummed and also replicated across the globe. The relatively heavy prices tag for accessing it probably is there to keep contention on some relatively thin pipe low. For example, if they use blue ray, they may have only a few readers online for every 100 or 1000 customers, both to keep prices low and to differentiate Glacier more from their other products.

if you address space is large enough, you could implement this as a virtual memory system that swaps RAM to disk, disk to slower disk, slower disk to tape, tapes to tapes stored offline, etc.

Funny to imagine,

  - My program has been stuck copying a variable for over half an hour!
  - Yeah, there was a page fault and the tape loading guy still hasn't come from lunch

Yes, but the other side of the coin is "why are those pages of the program still on disk? They haven't been touched in years". But such systems tend to keep parts under human control.

You rarely ever need your fire extinguisher, but that doesn't mean the LRU replacement algorithm that keeps your house clean can move them to the back of your attic.

True.

Back in the day, before automated tape readers, tapes were stored in silos, like a library, and employees would find the cataloged tapes and manually retrieve and load them into tape readers. My company use to hire co-ops to do this back in the 70s or so.

Reply on news.ycombinator.com