Hacker News Reader

Some have theorized that this is what's behind Amazon Glacier, and that they want to keep it to themselves: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bd...

I also crave for such cheap storage, perfect for archival/backups, especially when combined with modern snapshotting.

HVDs never really materialized, despite being promising, and actually on the market at some point, IIRC: https://en.wikipedia.org/wiki/Holographic_Versatile_Disc

A lot of other medium were developped, never commercialized: https://en.wikipedia.org/wiki/3D_optical_data_storage

A new optical storage medium is long overdue. This time, it will probably be without support from the media industry.

> A new optical storage medium is long overdue. This time, it will probably be without support from the media industry.

If anything, I hope that will massively help its adoption. Blu-Ray is a DRM cess-hole of unnecessary complexity. Players – which many people bought blu-ray drives to be – have to be cryptographically secure, contain "upgrade only" firmware (to enforce key revocation mechanisms) and contain a fairly complex microprocessor. There's no official linux support.

All of these anti-features directly compete with "low cost" and "widely available". If I could put 10 TB on a DVD-sized disc, even as a WORM storage medium, and the disc was cheap, I'd buy the thing tomorrow

I'd take an on-prem solution for cheap archival storage any day over Glacier.

With egress costs on s3 that may not be viable if you generate data on the cloud

Is the suspect for Glacier proven?

I can't see them using optical based on their price structure (compared to bluray media costs per GB). And I doubt they own a manufacturing facility to crank out super-secret multi-terabyte-sized optical disks. However it fits right in with the cost of LTO tapes. Quick back of the envelop: Their web site says 1$ per TB per month starting cost. That is $12/TB per year. A 12 TB LTO-8 tape costs around $70 retail, which puts it at $5.60 per TB. So the cost of the media used to store customer data is paid back in 6 months, or slightly more if they are using parity tapes.

That would be my guess, the customer data hits disk first, then gets enough parity and split among enough data centers to hit their reliability target.

The problem is that much of the demand for cheap, shelf-stable backup media has gone away over the last decade or two. Most backup jobs either go to cloud or to disk, not tape. And because of that, tape drives are more expensive than ever, which takes away much of the benefit of the cheap media.

LTO, for example, is relegated to archival use-cases nowadays, such as video production. You'd think that would make sense, because the media's shelf stable... but it's actually really awful for archival. Why? Because new drives can't read old tapes past two generations, and old drives stop getting made. So every time a new tape drive generation comes out, a movie studio can't just upgrade drives in their libraries and use the new tapes. Nor can they just hold onto their old drives forever - what happens when they start breaking, and they can't read their old tapes anymore? (Or worse, an old/failing drive eats a tape?)

Every movie studio with an archival program is on a constant upgrade treadmill, including costly media migrations. That is, they're copying data from old tapes onto new, so they can maintain compatibility with new drives. This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?

Now, let's say you're not a movie studio with a massive digital archival problem. You just want to backup 5 terabytes. Unfortunately, 5 terabytes is so little that tape vendors have forgotten how to count that low. It's far cheaper (not to mention, more performant) to just buy a bunch of disk drives and migrate data between them. Or just pay Amazon to do it. Which is what everyone wound up doing.

To add: LTO speeds are currently in the 300MB/s range: if you want > TB size storage you likely don't want ~100MB/s for regular use. I work on systems where we are regularly writing multiple tapes of 5+ TB and if you get only 100MB/s you're waiting hours and if something fails or gets stuck you can be thrown off schedule by more than a day very quickly. Even at the current 300MB/s speeds it's very slow. If personal archival or even small-scale archival is your interest getting network storage is probably easier, faster and cheaper. What's more, is that you have to keep some kind of inventory of your tapes so you know where to find data if you ever need it. Depending on the size of your collection you can be looking for even more expensive equipment to manage the archive of your archives.

Not to mention that 300MB/s is considered best-case conditions for sequentially stored data. Anything even remotely random is going to have far slower archival times, unless you have archival software that does parallel I/O. (In my experience, not many archivers do this.)

Most production tape installations include dedicated disk storage to act as a buffer for the drives. You archive to disk first, so that your drives aren't tied up, and then write the now-sequential data to tape.

Tape drive backwards compatibility is an issue for sure but optical drives have had decent backwards compatibility so far. DVD drives can read CD's. BR drives can read CD, DVD and BR. I think backwards compatibility is more feasible for optical as the larger pits on older media can still be read by narrower wavelengths. They'll just look like giant craters flashing by instead of nice neat pits.

AFAIK I believe that they actually do use different lasers for different disc types, but the cost to put two or three laser diodes in one drive is basically nil these days.

The big problem with optical storage is the same that tape struggles with - it's primary market shrank.

This was a bit of a self-inflicted wound. CDs and DVDs were basically ubiquitous, drives came down in price dramatically, and the storage densities and pricing was competitive with competing technologies. BD is where that all started breaking down. BD had a format war with HD-DVD, and both formats relied upon new and then-expensive technologies such as blue lasers and quad-layer stacking. PC manufacturers (notably Apple) balked at the cost of either format. Microsoft backed HD-DVD, but wisely decided not to bleed money by not sticking an HD-DVD drive in the Xbox 360. This created an odd situation where the Xbox 360 sold better as a game console, but Blu-Ray outsold HD-DVD because Sony had let the PlayStation division bleed out for a couple of years.

Of course, Microsoft still wanted the 360 to be able to play HD movies, and their response was to partner with Netflix to make streaming a reliable (arguably better) substitute for movies on optical media. In other words, Sony killed HD-DVD, so Microsoft killed the whole home video market by pushing digital distribution for everything.

If you're wondering, there are successor formats to Blu-Ray. Interestingly enough, the whole quad-layer stacking thing was rebranded as BD-XL and UHD-BD (which the PS4 doesn't even support), and there's also a proper successor format for data storage called Archival Disc. Except the latter is basically an LTO competitor - you can't buy bare AD drives and discs and use them like Blu-Rays. The technology only exists in archival systems like Sony's ODA system (Gen 2 and Gen 3; Gen 1 used Blu-Rays), which has discs stacked in cartridges that have to be loaded into special, very expensive readers and media libraries.

I believe streaming killed optical disc for video even if there are no format war. It's just convenient and quality is fine for most people.

> This entirely defeats the purpose of shelf-stable media; if I can't buy newly manufactured drives for old tape formats then who cares if the tapes themselves can sit in a box for 20 years?

Why can't you buy new drives for old tape formats? What new features are these new formats providing?

The new formats have higher storage density than the old ones. First generation LTO (LTO-1) stored 100GB on a tape: how much would you pay for a drive like that today? Unsurprisingly, they don't make them any more. LTO drives are read-compatible with tapes from two generations earlier, so you can read LTO-1 tapes on LTO-3 drives, but LTO-3's capacity was 400GB, so no one buys those any more either. The current generation (LTO-8) has 12TB per tape, so you can see the upgrade treadmill chugging along.

https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations

I have been talking and asking about this for years. And was told about Archival Disc [1] on HN.

500GB per disc right now, and only came out in 2019. 1TB per disc in the future but I suspect not in the next two years.

I wish they have a more consumer, or prosumer friendly version though. The current drive goes for about $9K [2]

There are Archival BluRay disc that offer 50 + year storage, but they are only in ~50GB per disc.

[1] https://panasonic.net/cns/archiver/optical_technology/

[2] https://www.bhphotovideo.com/c/product/1549175-REG/sony_odsd...

I have very little hope for the price for these things to come down to prosumer levels. The best I can hope for is for local libraries to have one available for use (although you need like 8 hours to fill the 5.5 TB maximum and they're so expensive you don't want to lend them out). However, I have very little hope for that as well since most consumers are just told to use the cloud.

>since most consumers are just told to use the cloud.

Yes and I dont like it this trend at all. But it seems we dont have any other way to fight it.

They remind me of m-discs, which seems to be cheaper as well. How is data stored on those Archival Discs?

In my opinion, it's best to adopt industry practices for home life here. Schofield's 2nd Law of computing comes to mind: if your data doesn't exist in two places, then it doesn't exist anywhere.

Offloading reliable backups to companies seems reasonable. It's a lot more money than archiving to physical media, but it's also a lot more reliable. NAS systems also make it easy to have reliable, hot on-site data that keeps a copy of the data backed up to a cloud service. Many of these cloud backup services are also zero-trust.

Nooooooooooo, you're supposed to back it up to the cloud and pay some startup jerk $XX/mo for the privilege! [/s]

Says whom? Most cloud storage have huge free tiers. I have unlimited storage on telegram.

Telegram upload speed is limited to 3MB/s.

Doesn't matter how high your connection speed is, it is capped from telegram's end.

So will they intake 10TB?

Never tried my upload is low speed and I don’t have that much data.

I bet you actually have "unlimited" storage.

Can you name something that is not "unlimited"?

I don't have a limit, I never hit the cap. By that logic nothing is unlimited (texts, calls, data, etc), but realistically it doesn't matter because I don't use it much. I don't care to test it, I use it sparingly and never had a issue with limits.

I assume that my S3 bucket is as close to unlimited as it gets.

Its also "unlimited" isn't it? There is no such thing as unlimted. As a subsciption you will also lose it if you stop paying.

I imagine it's as unlimited as it gets considering the sun is going to explode in some billion of years. However, I have no trust in having "unlimited" storage on a service where I'm not explicitly paying for it.

I burn 100GB backups via blu-ray. One or two is enough for my non-movie files. They started stagnating a few years ago when streaming ascended however. Shame as I'd like higher capacity as well.

Cheap USB flash drives are already moving into this space. I just got a compact 256GB one for $35 or so, just a matter of time. SD cards would probably be more convenient, but they aren't sadly aren't read in devices like stereos any longer.

Just remember to use those flash drives and re-copy them occasionally, like once every couple years.

The data is supposed to be valid for a decade but it is not a permanent storage system.

That much music (even at lossless compression that’s a lifetime of nonstop music) is completely impractical to navigate using a common stereo UI

Assuming you're referring to the parent comment's 100GB drive (or even the 256 GB one) that's really not true. A quick `du -sh` tells me I have right at 100 GB of FLAC music, and I've easily listened to every single track. There are, in fact, only 3770 tracks. At an average of 4.5 minutes a song (guessing), that's only around 11.8 days of music. About 8 hours a day of music for a month!

Or to calculate it another way, a typical FLAC stereo track might run about 800 kbps or 100 kB/s. 100 GB will let you store 11.6 days of music at that bitrate, which is just about the same total I got for my collection above.

To be fair, the parent said "impractical to navigate using a common stereo UI" which doesn't have much to do with the capaciousness of a library, more about how easy it is to browse/search/find/choose what you want.

Some stereos have pretty small, monochrome displays. For example this seems to get mentioned a lot in the UK, partly because they are advertised through mainstream channels at older listeners: https://www.brennan.co.uk/

Dunno if you call them "common".

Not really, it's about five clicks to start an album, which I'm pretty adept with, and plays for an hour. I have several hundred in Letter/Artist/Year,Album order. Less work than futzing with the radio constantly, which is generally terrible (unless you love tired top-40).

Why are tape drives fragile? I use them for my long term backups of personal files I thought they were robust ?

They are not fragile in an absolute sense, the poster meant that no matter what tape drive you have, it will break down many years before the tapes written with it would become unreadable.

After the tape drive becomes unusable, you may no longer be able to buy another compatible tape drive, because they might be discontinued.

For example I am using LTO-7 tapes. For the moment, I do not have to worry, because the current LTO-8 tape drives are able to read my LTO-7 tapes and the same should be true for the future LTO-9 tape drives.

However, when LTO-9 tape drives will be the norm, I will have to buy a LTO-9 tape drive and many LTO-9 tapes and copy all my archives, because if I would delay this I would risk to no longer find even LTO-9 tape drives. In that case my tape collection will become unreadable, even if the tapes themselves are guaranteed for at least 30 years.

I have already lost many years ago some archives that I had on QIC tape cartridges, because after my tape drive broke I was not able to find any other compatible tape drive.

What scale are you working at and how have you made it cost-effective versus a stack of external HDDs?

I am not that poster, but I also use tapes.

One may save about $14 per TB by using tapes instead of external HDDs (which strangely have become cheaper than internal HDDs).

In order to recover the cost of the tape drive, you need to write about 200 TB.

I have about 50 LTO-7 tapes, i.e. about 300 TB of compressed archives, so I have recovered much more than I have paid on a LTO-7 tape drive, 5 years ago.

Nevertheless, even if would have written only 100 TB, I would still have bought the tape drive, just for the peace of mind, because after the optical discs became too small I have used HDDs for archival storage, but after a few years in cold storage they become unreliable. If I had not stored duplicates for each HDD, I would have lost a lot of data.

Of course, someone who needs to archive just 20 TB or even 40 TB of raw data, can buy just 2 or 4 HDDs and store each file on 2 HDDs and that should be OK.

For 100 TB or more of raw data (i.e. 200 TB, when counting the redundancy, because also for tapes you should store at least 2 copies, preferably not in the same location), tapes become cheaper.

My brother got a cheap LTO tape drive on eBay and he bought a few LTO tapes so I think I bought one off of him for a few dollars more.

It’s only a few hundred gigabytes and the tape holds about one terabytes .

Note this is my long term cold storage I have backups made on my MacBook Air (hard disk) and my sold MacBook Pro (ssd which I will use on my new 2021 MacBook Pro eventually).

I also have backups using blu ray discs and I have one long term storage blu ray that I have been meaning to use

iomega stuff was generally rather precarious. LTO drives are solid.

Optical discs aren't torn apart when the reader malfunctions?

https://forums.tomshardware.com/threads/my-cd-shattered-insi...

I've had a CD-ROM back in the day shatter on me like this. I'm pretty sure its a common enough story that everyone 'knows' it could happen... but was rare enough that not everyone personally came across it... but I never got to the bottom of it (why it happened, or what conditions led to it). It only ever happened to me once after all.

It sounds like x56 speed drives were just too fast for some CD-ROMs. We went from x1 speed in the late 80s to x56 speed by the late 90s, so it makes sense that some CD-ROMs just weren't designed for the speed increase and maybe shatter.

Alternatively: maybe some physical damage existed. We beat up CD-ROMs back in the day: storing them in backpacks without protectors and whatever. They were very reliable in terms of reading, but maybe those kinds of mistakes would lead to fractures that would literally blow up on you later.

Even then: the most common protector were paper-sleeves, because dust / scratches were what we were most worried about.

---------

We used to carry around stronger plastic cases in the early 90s. But by 00s, it was all paper-sleeves. The plastic-cases would themselves shatter when carried around, but the CD-ROMs / DVD-ROMs were stronger, lol.

I remember my x52 drive would make at least 2 or 3 "test" spins every time I'd insert a disk, just to see how fast it can spin it. It felt like an airplane taking off every time I'd put a new scratch-less disk.

I had one year warranty on that drive. This "feature" stopped working exactly one week after my warranty expired, it would just go full speed regardless of how beaten the disks were.

That's how I got two of my heavily scratched disks shatter.

A few more weeks down the road, the function that's supposed to slow down a disk before a full stop, also stopped working... so now even if a disk was spinning at x52 inside and you'd press the eject button, it would instantly stop the motor and eject it!

That's when I finally threw it away and got myself a "normal" x46 or x44 (don't remember) drive...

Never seen it but have seen many ruined tapes over the years.

Optical disks degrade due to bit rot when the substrate of the disc starts to peel off