AWS Tape Gateway
aws.amazon.com
aws.amazon.com
Sometimes people build things/products/companies/whatever not because of the chase of money, but because they want to put their idea in the world. Maybe it's just a case of someone believing they can build something grander :)
Some abstractions can be proven to be objectively-better solutions, along some axes you care about, for your use-case, such that the existing abstraction you were relying upon is then provably sub-optimal. Switching to the more-optimal abstraction is not then an "homage to the brilliance" of the company that created the abstraction; it's a simple "pay CapEx to lower OpEx in a way that will pay itself back after N years" decision. If you can budget it, you do it.
I'm not saying that every abstraction that the IaaS vendors invent is this kind of better mouse-trap, mind you. You can usually distinguish the ones that are, because they get independently reimplemented and used by people who aren't just aiming to build "cloud-native" software, but instead just really want/need the semantics of that particular abstraction, independent of where it runs.
Object storage is a great example of an abstraction where that is the case. There are many things that do object storage now — both cloud services, and regular deployable software. In fact, even the more-traditional backup vendors, like Backblaze, now also offer object-storage APIs.
I'm fairly critical of Amazon's engineering culture and practices but as a customer I have to admit they are head and shoulders above their competition and frankly most SaaS providers as a whole.
They've secured millions of dollars in revenue from me alone just by being so customer friendly.
I've worked with some of the smartest people in the tech field, building products from scratch for billion dollar customers and learning every day.
I've always told people that it's probably the best place you can start out of college.
EDIT: This was back in 2016, I'm not sure how much has changed without Bezos and Jassy at helm of AWS
Unless your willing to put up with a huge ration of shit on a regular basis, or you make it into management, your time is limited by the up and out mentality of the organization as a whole.
Amazon has created the perfect scenario to spend shitloads of money getting talent up to speed, only to part ways with them shortly thereafter.
Management at AWS arguably puts up with much more shit on a regular basis.
* working in infrastructure means doing a lot not quite so exciting things. most people aren't actually excited by ops, and a lot of your dev work will be on ops. its also the case that ops was historically not as well rewarded, though thats been fixed
* every service needs to be held to the highest tier of operational excellence, and so you can't just chabuduo when things aren't working as well as they should. in the worst case, this means throwing bodies to the ops meatgrinder.
* delivery speed is slowwwww. some of this in endemic to the space, but aws at the end of 2021 (when i left :) ) was a lot slower than than when i joined due to so much risk aversion and bureaucracy
* comp wasn't as good compared many tech peers, and comp could very easily not keep pace if you weren't tippy top every year. this is imo the most self-defeating part of amazon culture as a whole because it incentivizes people rated between ~p10-p20 and p50 to leave.
I feel like this is true for every single large company I’ve ever worked for. Things always feel like they’re regressing from a high point. I have no idea why it it.
All of them had free coffee when I joined, and they’ve slowly removed the vending machines and just never replaced them.
I've worked at Amazon for over 4 years. During this period, I saw around 80% of all team members being promoted to customer, perhaps half of them voluntarily. Amazon provided psychological service to those who felt they needed it, and a couple who quit confided in me they used the service. One guy overtly burned out and never showed up again.
Long work hours were the norm. Plenty of CRs were posted after 10PM. Even SDE3s frequently posted CRs on Sundays. The senior manager was hell bent on proving leadership he could deliver projects in impossible deadlines and not rarely did he warned SDEs that failing to deliver on those promises were firable offenses. This in turn fostered a work environment that led to some backstabbing and less than ethical stunts from team members prone to those and desperate to make a good impression.
It's true that Amazon's salary structure is designed to push out people. It is designed to impose a hefty pay cut after the second year or so. Nevertheless this is not the problem. The problem is the awful corporate culture that's designed to hold SDEs responsible for not delivering things even in impossible scenarios, and the backstabbing culture it sprouts by design.
But I'm sure some people might have a good time working there.
> The senior manager was hell bent on proving leadership he could deliver projects in impossible deadlines and not rarely did he warned SDEs that failing to deliver on those promises were firable offenses. This in turn fostered a work environment that led to some backstabbing and less than ethical stunts from team members prone to those and desperate to make a good impression.
is an extreme situation and one I never saw across ~3 orgs between amazon and aws.
I've been there 5 years, haven't worked more than 40 hours a week ever, and a lot of the people that started at the same time as me are in the same situation. And of those that aren't, they left because they wanted new experiences, not because they didn't like Amazon.
I never had to put up with any "shit", whatever that could mean.
It can’t be all bad, but it’s clearly not all good either.
It kind of makes sense that the people still there wouldn’t have bad stories to tell. What do you think is the worst part about working for Amazon?
Spending the money to recruit the employee, then pay them for a year or two, only to lose this talent is a great way to light cash on fire. That ain't cheap when your talking about hiring software developers!
It's relatively common that people don't make the cut during their probation period for example. Amazon (and other FAANGs) clearly isn't for everyone.
So I would question anyone that takes the easiest job they can find (their old job) after leaving Amazon in less that a year or two.
Isn't the average tenure at Amazon less than 3 years? "Most" of the people who you'd ask would be fresh from the interview rounds and with only one or two performance review rounds under their belt.
But I agree their support is very responsive.
IIRC, they may not be pretending. It's not public knowledge what storage medium AWS glacier is built on, but many speculate it is actually tape.
Edit: found in another thread here, the inventor linked the patent (https://image-ppubs.uspto.gov/dirsearch-public/print/downloa...) which confirms it's virtual tapes, not actual tapes.
No, just like it is not weird and definitely not illegal for the things they call “virtual machines” to end up being run on actual machines.
There's still isolation, there's just no virtualisation abstraction.
Is that not misleading?
They physical tapes, if any, would be many layers down.
At announcement time Glacier was using super dense racks of hard disks but only powering a subset of them. This led to some tape-like behaviors, like your restore request might take a long time to process because your data was likely stored on disks that weren't powered on and you'd need to wait for them to come online.
We talked about how we want to change our design to copy AWS. Because it's standards, and our users (internal users) know how to use it.
You can have your shit packed into a snowball and shipped to you.
https://aws.amazon.com/snowmobile/faqs/ > Snowmobile does not support data export.
https://aws.amazon.com/snowmobile/
Quickly retrieve data from the cloud whenever you need it.
What does it mean?
Feels a bit like intentional misdirection, but it's more likely a case of sloppy writing.
https://docs.aws.amazon.com/snowball/latest/developer-guide/...
> Snowmobile does not support data export.
Q: Can I export data from AWS with Snowmobile? Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary.
The snowball “device” is pretty neat:
* 45-foot long High Cube shipping container
* [each snowmobile container] comes with a removable connector rack with up to two kilometers of networking cable that can directly connect to the network backbone in your data center.
* AWS can provide the auxiliary chiller if needed based on the site survey findings [if temperature exceeds 85F/29.4C]
* A fully powered Snowmobile requires ~350KW [sic] Generators can be provided by AWS if needed
* Snowmobile pricing is based on the amount of data stored on the truck per month. Provisioned Snowmobile Capacity: $0.005/GB per month
Isn't the whole point of tape that it is a physical thing, and may be taken offsite in a truck / stored in a vault, and all that jazz.
"Just reconfiguring" your backup software sounds like a business might just bypass all that without necessarily realising the consequences.
Then "we got hacked and our local backups are gone" -> "restore from offsite tape" -> "oh, actually there are no tapes, it's all the cloud now" -> ... ??
Taking tapes to a secure offsite location means they're air-gapped from the source data, so even in a situation where an entire network is compromised, remotely-stored tapes can't be wiped/encrypted.
I'm sure AWS has thought about this issue while designing Tape Gateway, but it's not clear to me how it could know whether a request to retrieve and overwrite a virtual tape was legitimate or not.
S3 offers WORM (Write Once Read Many) storage with Glacier and Object Lock. The slow retrieval for Glacier hints that AWS isn't using hard disks.
I'm not entirely comfortable with the trend towards ever more esoteric and seemingly "all or nothing" technologies.
Tape (and other physical things) may have their downsides, but I think there's something to be said for something that could (theoretically) be forgotten in a closet and read 50 years later.
Likewise, AM radio may not be the best quality, but it seems like you can cover more area with a single installation than any other communication technology, which might come in handy after a serious disaster like a nuclear war (e.g. crank up the nighttime power of some undamaged countryside station running on generators to tell survivors where it's still safe).
There's a sense in which tape is water in which backups swim.
The fastest way to enable large enterprise access to S3 and Glacier for backups was to meet them where they were. We did this by virtualizing a tape library.
Background: I'm one of the original inventors - https://image-ppubs.uspto.gov/dirsearch-public/print/downloa.... I am no longer with AWS.
This is a fantastic example of how broken the patent system is.
This is not an invention it’s a good implementation of a well understood problem.
It is also priced ridiculously - it is at least an order of magnitude more expensive than operating a physical library and remote physical storage that is beyond cyber threat by virtue of it being disconnected.
A real backup is offline and offsite.
Some of the solutions that external vendors produced were nightmarish and left customers up the creek without a paddle in a disturbing number of situations.
Your disaster recovery plan will need to budget for extra egress costs if you ever need to restore something big from one of those virtual tapes. AWS charges for egress -- downbound -- bandwidth but they don't charge for ingress -- upbound -- bandwidth.
(And I think it's funny their icons and images look like old school DECTapes.)
Closed, works as intended: This is the industry-standard price model for all ransomware.
/light sarcasm
The last time I dealt with tapes was in the 1990s, and it was dead simple. We used tape backups at a company with only 150 people. I was the most junior admin so I had to be the tape monkey, but it was easy. Just run backups regularly, change the tape every couple of days and label it, ship a stack to the offsite storage and request the old stack back, and then do a test restore each time a stack came back.
The whole thing took maybe two hours a week.
When you're juggling 12 plates, a couple are gonna fall.
Everything else got easier, bigger, and faster.
Two decades back, you needed tape to cost effectively back up hundreds of gigabytes. These days, pretty much unless you have a spare room with an automatic tape library (robots moving tapes), you're too small for tape to make sense.
So, yeah - virtual tape library, while obviously having downsides, is massively useful for a small enterprise.
That’s a lot of 9s.
> Feynman was disturbed by two aspects of this practice. First, NASA management assigned a probability of failure to each individual bolt, sometimes claiming a probability of 1 in 10^8, i.e. one in one hundred million. Feynman pointed out that it is impossible to calculate such a remote possibility with any scientific rigor.
https://en.wikipedia.org/wiki/Rogers_Commission_Report
Unless S3 can survive nuclear war or an asteroid hitting the earth, I don’t buy it.
I'm personally fine with the claim as I understand the methodology and assumptions but it has to be interpreted carefully.
You can use rclone to backup to an S3 bucket, and then have the S3 bucket set to "instantly" move files to deep archive.
To retrieve them, you will need to run an extra command to "unarchive" them for some period of time.
I find this easier than trying to interact directly with any Deep Archive API.
It is very cheap, and AWS's durability guarantees are impressive.
Main S3 page, linked to a note about Deep Archive https://rclone.org/s3/#glacier-and-glacier-deep-archive
The only thing I am not sure on is how to quickly restore a large number of files in a bucket from GDA to S3. It obviously can be scripted, but I don't have that handy. I only keep a small number of large, previously encrypted files in it, so I manually restore from the GUI.
(By the way, rclone can transparently encrypt files and filenames client-side!)
https://www.backblaze.com/b2/cloud-storage-pricing.html
edit: it integrates with TrueNAS (formerly FreeNAS) so it's been pretty much set-and-forget, but I can check in and see my stuff through their management interface. I do encrypt my docs before uploading, which TrueNAS also supports.
You should be able to accomplish the same, but you may need to either re-organize your stuff or set up multiple tasks.
edit: for clarification, "/mnt/main/media" contains various other subdirectories of Linux ISOs etc., which is why I push /mnt/main/media/Photos specifically.
The pro is that it has a client and, once setup, it just works.
The con is that it's a client running on a single system and the personal pricing plan doesn't support a NAS I believe. I periodically copy my files over from a RAID NAS to a local USB.
Found it a lot easier than rolling my own AWS Glacier solution.
I'm using CrashPlan for Small Business, so I can backup select NAS shares as well. $10/month.
You can use rclone as one simple tool.
1TB = 3.2€
5TB = 10.9€
10TB = 20.8€
...
https://www.hetzner.com/storage/storage-box
No transfer fees.
Why doesn't Iron Mountain or some other competitor offer a service like this?
FWIW, Amazon isn't 'doing tapes' with this product. It's just a compatibility shim laid on top of object storage so that legacy tape backup/archival systems can eliminate physical tape libraries.
That being said, kudos to Amazon for having a solution for those who aren't in my circles and still use them.
Only huge companies like Amazon can be this dumb. They don't even mention tar or pax, the two most common tape backup applications.
Also, how will this magic Amazon "Tape Gateway" back up petabytes over slow links? There are many data heavy businesses that don't necessarily have tons of Internet bandwidth. Fully saturating a 100 Mbps outgoing connection will only get you 1 TB a day, so what happens when you have a tape's worth of new data daily and an already heavily used Internet connection?
Easy: it won't. If you have more data than bandwidth, you won't use this service. Not sure why you seem to think otherwise.
Then hauls it all away for ingestion into its cloud.
Or, if it will fit in your station wagon, you can haul your data down the highway to one of Amazon's clouds.
Or put it in a box and ship it Fedex.
I started from Tanenbaum’s hurtling station wagon and used my poetic license without worrying too much that the parent commentor would buy a forklift before talking to AWS sales.