I want to have an AWS region where everything breaks with high frequency
twitter.com
twitter.com
Secondly, this would help make CE better. I actually asked Amazon for an API to do this ten years ago when I was working on Chaos Monkey.
I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.
I also asked for an API to slow down networking, set a random packet drop rate, EBS failures, etc. All of these things can be simulated with software, but it's still not exactly the same as when it happens outside the OS.
Basically I want an API where I can torture an EC2 instance to see what happens to it, for science!
Pretty much every option you have from the OS will let caches flush, will have in-progress writes complete.
Yank the power and none of that will happen. It'll let you actually see what level of fibbing your OS and hardware are telling you.
Oh you got a result from that flush to disk? It completed? Are you sure? Really really sure? Lets find out...
I have no idea! Maybe everything is actually great. Or maybe someone else will be oncall the week you hit that one in a million failure case. Or maybe it will happen to your competitors instead of you! Without testing, all we have is hope.
Let’s say a hard power off causes data corruption that can’t be fixed. For lots of applications downtime is better than corruption, so in that case you at least will be able to test when you should take the system down and recover from a known good backup.
I disagree.
While you can't control the bugs in the hardware or drivers or whatever, you can definitely ensure that your application detects corrupted data and can at least warn that something is wrong rather than starting back up and silently serving from corrupted data.
It also ensures that a node suddenly disappearing without a graceful "I'm leaving" message, and potentially with open transactions or other operations can be handled by the clients and nodes that ARE still around.
I get it for desktop users... they want to open some benchmark they downloaded and get a higher number than their friends (RAID-0-esque sharding data across individual flash chips), and they also don't have access to software that can add error correction codes (because the Windows installer can't recognize an SSD with 4 flash chips as 4 drives in a RAID array)... so the disk has to do it itself. But why is it a thing in datacenters? Just so you can install Linux+ext2+MySQL and call it a day? That seems crazy to me when better storage software exists.
I'm not the GP, but I assume it is because the GP would like to run his "production" setup (or even production setup itself) under such circumstances. So far as I remember, the origin of the idea came from the Chaos Monkey which would sabotage certain services in a production environment to ensure than everything was redundant and fail safe.
Wouldn't just running "halt -f" do the same?
I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.
If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.
echo c > /proc/sysrq-trigger
A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.
https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...
Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off().
This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting.
Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.
Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.
To my mind simply turning off the networking, physically cutting a cable, or hardware spontaneously combusting are not discernible events to an outside observer. What am I missing?
Even killing a VM instantaneously on a host only simulates the loss of the guest OS's cache. The host OS still has its cache.
And even killing the host OS only simulates loss of the host OS's cache. The drives still have their caches powered, so might behave differently on power loss than host OS crash.
And killing a drive while the host keeps running is different again.
(And in terms of high-level software-observable data effects, abruptly killing power to a drive is not the same as slowly lowering the voltage or limiting the current to the drive, or seeing corrupt data on the I/O bus while the host system loses power (which is never instant), or... you can go quite deep with this.)
In the networked storage environment at AWS, who knows what events are possible to observe on EBS in the event of a system-level failure such as power loss in a data centre.
For example, EBS is a distributed system with replication, so if it has an implementation error that only affects certain untested, complex failure modes, it might lose recent writes in flight (as expected), but then some time later they might come back, or some of them might come back if it's sharded. Both are bad if software using it has already started state recovery after an outage. Distributed systems often have surprising incorrect recovery patterns, because it's a complex and subtle problem; that's why the Jepsen tests keep finding bugs in databases people have been using for years.
The observable events can differ depending on whether it's VM hosts, switches, block storage units, drives failing and in what ways.
For Linux guests we have abstracted away most differences we care about under fsync() or various combinations of O_DIRECT and virtual HDD cache disabling, or relying on known features of filesystems, but there's no easy way to be certain those abstractions actually provide the expected semantics on different kinds of system failure. Stress testing higher layers in the virtualization stack would go some way to verifying that they do in practice, not just in theory.
And one day there will be PETA [0] for EC2 instances!
https://en.wikipedia.org/wiki/Plurale_tantum
That is, it seems to be grammatically plural but not have a distinctive singular.
4. pl. A piece of real estate; a building and its adjuncts;
as, to lease premises; to trespass on another's premises.
[1913 Webster]The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ...
I wonder the degree to which this functionality would end up becoming relied on as a "in worst case hard kill things to recover" behavior that folks utilize for bad engineering reasons as opposed to good ones ...
Edit: Another term is Switched PDU
On top of keeping people who don't want it out, you'd need it to be expensive to offset the lower utilization.
I'd be okay with a garbage-tier, too. It's not failing on purpose, but only because no one gives a shit. Maybe that's just standard AWS?
This would be more effective if so many other AWS features weren't hidden in non-obvious places in the docs.
In a previous job we actually had to make a decision like this: our server software was generally stable enough, but we reset it whenever we did an update. Every once in a while it would break when running for long enough.
I opted for forcefully restarting the software every day, because I'd rather relive the pains of restarting, than to discover in production what it is like for the first time to run the system for the 501st day of uptime.
(To be honest, the system in question was mostly written in Erlang, so controlled restarts were already baked into its philosophy.)
Yeah, but did he win the Putnam?
2007 HN was highly entertaining
https://en.wikipedia.org/wiki/William_Lowell_Putnam_Mathemat...
I don't pay much heed to AWS Heroes. My city has an "AWS Hero" who hoarded the organization of the official AWS meetup for themself, and who is infamous for making liberal use of Cunningham's Law [1] for everything.
Honestly, I don't either. There's a handful of us who are technical and a huge number whose contributions are primarily marketing.
# cat > dropme.sh <<'EOFILE'
#!/bin/sh
set -eu
read -r SLEEP
tmp=`mktemp` ; tmp6=`mktemp`
iptables-save > $tmp
ip6tables-save > $tmp6
for t in iptables ip6tables ; do
for c in INPUT OUTPUT FORWARD ; do $t -P $c DROP ; done
$t -t nat -F ; $t -t mangle -F ; $t -F ; $t -X
done
sleep "$SLEEP"
iptables-restore < $tmp
ip6tables-restore < $tmp6
rm -f $tmp $tmp6
EOFILE
# chmod 755 dropme.sh
# ncat -k -l -c ./dropme.sh $(ifconfig eth0 | grep 'inet ' | awk '{print $2}') 12345 &
# echo "60" | ncat -v $(ifconfig eth0 | grep 'inet ' | awk '{print $2}') 12345
All jokes aside, I actually asked my google cloud rep about stuff like this; they came back with some solutions but often the problem with that is, what kind of failure condition are you hoping for?
Zonal outage (networking)? Hypervisor outage? Storage outage?
Unless it’s something like s3 giving high error rates then most things can actually be done manually. (And this was the advice I got back because faulting the entire set of apis and tools in unique and interesting ways is quite impossible)
Just firewall off the real s3, and point clients at a proxy which forwards most requests to the real s3 and returns errors or delays to the rest.
Mostly because programmers seem to have an easier time thinking about errors, and their programming language might even encourage them to handle these kinds of errors, but arbitrary delays often slip by unanticipated.
The simple concept is all services/API's should take a deadline parameter. The call should either complete before that time, or return an error. If you are writing code for a service, and you get an incoming request, you send the same deadline for any requests to other services.
Host A calls host B
Host B runs SQL+COMMIT
DB commit finishes
Host B deadline expires
DB success reaches host B
Also would need fanatical time synchronization. Maybe even PTP rather than NTP?
I think one might be able to make this approach work with poor clock sync by having the request contain the number of milliseconds left, rather than an absolute time. It isn't great if network delays dominate, but typically it's queueing delays that dominate most webservice responses.
Clock sync doesn't really help, especially if a request can trigger multiple independent action that can individually succeed or fail.
The simplest example where this goes wrong is posting a post on Reddit. You write your text into a text box and submit. You get an Error 500. You cannot know if your post went through (even if you go check whether your post is visible, sometimes the replica that you got your response from hasn't gotten your post yet). And they don't do any deduping, so if you just retry, you may end up with multiple copies of your reply.
The chaos aspect of this would certainly increase the evolutionary pressure on your systems to get better. You would need really good visibility into what exactly was going on at the time your stuff fell over, so you could know what combination(s) to guard against next time. But there is definitely a class of problems this would help you discover and solve.
The problem with the testing aspect, though, is that test failures are most helpful when they're deterministic. If you could dictate the type, number, and sequence of specific failures, then write tests (and corresponding code) that help make your system resilient to that combination, that would definitely be useful. It seems like "us-fail-1" would be more helpful for organic discovery of failure conditions, less so for the testing of specific conditions.
Let's not let `perfect` get in the way of `good`.
Certainly having a 100% traceable system would be ideal, most systems are not that.
There is still a TON of low hanging and easy to find issues that would automatically fall out of a system of random fails. Even if engineers have to spend some time figuring out what the hell is going on, it would overall improve their system because it would shine a bright shiny flashlight on the system to let them know "Hey, something is rotten here". From there, more deterministic tests and better tracing can be added.
Error conditions you already know about are easy to test and code against. I would guess most system failures come from conditions nobody expected. When you have a randomly failing system you can discover them. Fixing problems and testing for them then will be easy in comparisons.
For example: Years ago I worked on a video streaming solution. Some day we got a device that would garble our traffic randomly, slow it down and so on. things started crashing left and right. Within a month we squashed hundreds of bugs and had a rock solid system that basically impossible to crash.
I always wondered about AWS for other cloud systems how you could prepare for problems. It's hard to predict what can fail in what ways and you can't really force error conditions. I really like this idea of a cloud where everything breaks.
The Azure group would ask them to send large loads of data their way, so they could get some "real" load on the servers. There would be issues at the infra level, and the team had to detect this and respond to it. In return, the data team would also ask the Azure folks to just unpug a few machines - power them off, take out network cables - helping them test what happens.
Unfortunately, this was a one-off, and once the data center was stable, the team lost this kind of "insider" connection.
Howerver, as a fun fact, at Skype, we could use Azure for free for about a year - every dev in the office, for work purposes (including work pet projects). We spun up way too many instances during time, as you'd expect, and only came around to turning them off when Azure changed billing to charge 10% of the "regular" pricing for internal customers.
I always wondered how many people got some questions from the storage team, if they really needed all those exabytes.
This is the reliability engineering equivalent of building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it.
Not handling failures correctly is a time honored tradition in programming. It is so easy to miss.
For example, how often have you seen a malloc check for `ENOMEM`?
Even though that is something that could be semi common. Even though that's definitely something you might be able to handle. Instead, most code will simply blow chunks when that sort of condition happens. Is the person that wrote it "wrong"? That's debatable.
Some languages like Go make it even trickier to detect that someone forgot to handle an error condition. Nothing obvious in the code review (other than knowledge of the API in question) would get someone senior to catch those sorts of issues.
So the question is, HOW do you catch those problems?
The answer seems obvious to me, you simulate problems in integration tests. What happens when Service X simply disappears. What happens when a server restarts mid communication? Is everything handled or does this cause the apps to go into an non-recoverable mode?
This are all great infrastructure tests that can catch a lot of edge case problems that may have been missed in code reviews. Even better, that sort of infrastructure testing can be generalized and apply to many applications. Making rare events common in an environment makes it a lot easier to catch hard to notice bugs that everyone writes.
It's basically just Fuzz testing but for infrastructure. Fuzz testing has been shown to have a ton of value, infrastructure fuzzing seems like a natural valuable extension of that. Especially when high reliability and low maintenance is something everyone should want.
Consider over-commit. It means that the malloc call will succeed even if there isn't available memory. Instead the system just hopes you wont make use of the memory you asked for. And if you do make use of it, well then the OS might kill you. Just like that. No opportunity for error handling.
When you don't handle it, you get something closer to undefined behaviour instead.
I've never concerned myself with that, but it's apparently a lot more complicated than just doing that check.
See:
https://news.ycombinator.com/item?id=20143277
Some of the thread is hilarious, for instance:
"malloc is not allowed to return a non-null pointer to a memory block that cannot be written to. Linux does it anyway, and in doing so blatantly violates the standard.
...
That is my point, it doesn't fail. Either the kernel finds out a way to map the memory and it succeeds, or it kill the program and the instruction never runs. Code that doesn't run can't violate the standard."
Hi Bob, I can't tell you what reports I want or what we'll do with the data until you've first collected the data for analysis. Thanks!
When you do finally figure out why you are in business, you may find that all the data you gathered is worthless. Sometimes it's really innocuous stuff like storing datetime values without enough resolution or accuracy. Only when you know how you plan to use the data can you ensure you are gathering it properly.
Teaching developers to collect all the data they can in hopes of being able to write arbitrary reports down the line is reckless behavior. Hyperscalers love pushing this ideology with inane bullshit like "Datalakes", so you can see where a conflict of interest may arise in the industry.
I've worked on a project (in a peripheral role, fortunately) where the entirety of the CIO's mandate was to "use AI", full stop. So the team was tasked with putting together a data warehouse that would collect random logs and app metrics, and then sprinkle AI fairy dust of their choice onto it to generate Insights(tm).
This kind of thing happens all. the time. in large companies where everybody involved wants either money or bullet points for their resume, and the actual outcome is nearly irrelevant.
I don't see how I could do it differently. I can only truly know which metrics are useful from experience with past incidents, but I can only gain this experience when the metrics were already collected during the past incident.
You're right that you'll never be able to cover 100% of all cases, but using that logic, your specs will never test 100% of scenarios, so you shouldn't write specs.
I think the problem is you assumed it was "a magical box that people can rely to cover 100% of test cases of".
That's a poor leap. It's clearly not. It's a good system to test a set of network failures at varying degrees. Just like any engineered system, it needs to be documented what it can and cannot do.
I also eagerly await my do it all, 100% magic box.
One, this is not how AWS regions are designed to work. What they're thinking of is a virtual region with none of its own datacenters, but AWS has internal assumptions about what a region is that are baked into their codebase. I think it would be a massive undertaking to simulate a region like this.
(I don't think a fail AZ would work either, arguably it'd be worse because all the code that automatically enumerates AZs would have to skip it, which is going to be all over the place.)
Two, set up a region with deliberate problems, and idiots will run their production workload in it. It doesn't matter how many banners and disclaimers you set up on the console, they'll click past them.
When customer support points out they shouldn't be doing this, the idiot screams at them, "but my whole business is down! You have to DO something!" This would be a small number of customers, but the support guys get all of them.
Three, AWS services depend on other AWS services. There are dozens of AWS services, each like little companies with varying levels of maturity. They ought to design all their stuff to gracefully respond to outages, but they have business priorities and many services won't want to set up in us-fail-1. When a region adds special constraints, it has a high likelihood of being a neglected region like GovCloud.
The evil configuration should randomise anything that isn't specified. No string comparison type selected? You get Turkish. All I/O and networking operations fail randomly. Any exception that can be thrown, is, at some small rate.
Or to take things to the next level, I'd love it if every language had an interpreted mode similar to Rust's MIR interpreter. This would tag memory with types, validate alignment requirements, enforce the weakest memory model (e.g.: ARM rules even when running on Intel), etc...
We are forced to use Azure for business reasons where I work, and the frequency of one off failures and outages is insane.
I'm on the fence though if I would want a separate region for this with various random failures. I think I'd be more interested in being able to inject faults/latencies/degradation in existing regions, and when I want them to happen for more control and ability to verify any fixes.
Would be interesting to see how they price it as well. High per-API cost depending on the service being affected, combined with a duration. Eg, make these EBS volumes 50% slower for the next 5min.
Then after or in tandem with the API pieces, release their own hosted Chaos Monkey type service.
Unreliability.io - Shitty Performance as a Service.
We hook your accounting software up to api.unreliability.io and when a client account becomes delinquent, our platform instantly migrates their entire stack into the us-fail-1 region. Automatically migrates back again within 10 working days after full payment has cleared - guaranteed downtime of no less than 4 hours during migration back to production region. Register now for a 30 day Free Trial!
There's extensive research and works on all things resilience. One could say: if one build a system that is proven to be theoretically resilient, that model should extrapolate to real-world resilience.
This approach is probably intimately related with pure-functional programming, which I feel has been not explored enough in this area.
Yet to figure out how to test any of those cloudwatch alerts/rules. I've had them deployed in my dev/test environments for months now, after having to manually deal with a handful of them in a short time period. They've yet to trigger once since.
Umbrellas when it's raining etc.
There have been some talk internally of doing chaos engineering to help improve the reliability of our company's products as a whole. Unfortunately, the most easily simulatable failure scenarios (e.g. entire containers go down at once instantly, etc.) tend to be the least helpful since my team designed the service to tolerate those kinds of easily modelable situations.
The more subtle/complex/interesting failure conditions are far harder to recognize and simulate (e.g. all containers hosted on one particular node experience 10s latencies on all network traffic, stale DNS entries, broken service discovery, etc.).
Doing this constantly for a all products in a single region would be absolutely exhausting for SRE teams
(Discliamer: I work for ^GOOG and my opinions are my own)
(I also used to work for Google, as an SRE. Fun times.
Many of our dirt exercises usually included the stipulation: 'by the way, assume that <scarily over-competent coworker> is on vacation and not to be disturbed'.)
Sure you can. Simulate an Ec2 outage? Turn off your VMs. Lambda down? Turn off your serving version. Etc...
I think the only things that difficult to simulate would be control plane outages for testing CI/CD pipelines--though CI/CD pipelines generally are not production critical and probably aren't worth testing directly. If you want to test recovery/repair operations when CI/CD is down you can just prohibit the victim from using it to fix things.
For non-automated API calls, you could just say that X API is down for the duration of the test
I don't work at Google but I imagine just working on making it a dedicated goal to have a continuously faulty region would reduce the burden of everyone involved.
This sounds nice in theory but I think you are underestimating the vast complexity of large cloud platform internal systems. The amount of work to make this happen reliablely would be immense. You would probably need a few people from each cloud SRE team with some SWE partners working for a year or longer to get this up and running. Then after that we would need to work out all of the bugs caused by these contrived failure modes and build automation to repair damage.
It's just too much work to justify when there is so much work to be done for regular prod already
This would be about getting them to actually match the real world behaviour to the SLA
https://landing.google.com/sre/sre-book/chapters/service-lev...
Google introduced exactly this to one of their internal services so that downstream dependencies can't rely on its extremely high availability.
Setting it up requires some effort (you can't just choose a region in your AWS config), but it's available now and can be integrated with tests.
As for things like EBS failing, or dropping packages, it's a bit tricky as some things might break at the OS levels
And given sufficient failures, you can't swim anymore, you'll just sink.
we've had a ton of instances fail at once because they had some kind of rack-level failure and a bunch of our EC2s ended up in the same rack. :(
The "real" challenge is not creating chaos but managing it and verifying that your apps are resilient to said chaos.
This would be great for testing the pieces of the stack that the provider is responsible for, but you may still want to inject chaos into the part of your stack that you do control.
The technician updates the ticket after a while with "cannot find road." The folks back in the office try to send them directions, but then the technician clarifies, "road is gone." Our fiber, and the road it was buried under was totally demolished in the few hours it took to get someone out there. The developing world can develop at alarming rates.
Other tales from the middle of nowhere: People shoot at arial fiber with guns. Or dig it up and cut it for fun. One time out technician was carjacked on the way to doing a repair.
Though not very relevant to your stories about Brazil, it's a neat technique in its own right:
https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...
I'm a happy AWS user and I'll stay a happy AWS user not for their prices or features, but service. Which was the reason I was a Rackspace fan before it was sold and went down the tube.
They were very reliable, imo. Aside from general Netscaler bullshit, We only ever had issues with them when we'd try to get them to do too much so that the CPU or Memory was overloaded.
We tried on a few occasions to get more cores allocated to them, but no. This made terminating large numbers of SSL connections on them problematic.