Yea, I looked it wrk2 but it was a no-go right out the gate. From what I recall the changes to handle coordinated omission use a timer that has a 1ms resolution. So basically things broke immediately because all requests were under 1ms.
1,163 karma · joined September 3, 2015
> I always disable C-states deeper than C1E
AWS doesn't let you mess with c-states for instances smaller than a c5.9xlarge[1]. I did actually test it out on a 9xlarge just for kicks, but it didn't make a difference. Once this test starts, all CPUs are 99+% Busy for the duration of the test. I think it would factor in more if there were lots of CPUs, and some were idle during the test.
> Try receive flow steering for a possible boost
I think the stuff I do in the "perfect locality" section[2] (particularly SO_ATTACH_REUSEPORT_CBPF) achieves what receive flow steering would be trying to do, but more efficiently.
> Would also be interesting to discuss the impacts of turning off the xmit queue discipline
Yea, noqueue would definitely be a no-go on a constrained network, but when running the (t)wrk benchmark in the cluster placement group I didn't see any evidence of packet drops or retransmits. Drop only happened with the iperf test.
1. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processo...
2. https://talawah.io/blog/extreme-http-performance-tuning-one-...
The c5 instance type is about 10-15% faster than the m5, but the m5 has twice as much memory. So if memory is not a concern then switching to c5 is both a little cheaper and a little faster.
You shouldn't need the c5n, the regular c5 should be fine for most use cases, and it is cheaper.
Nginx in front of nodejs sounds like a solid starting point, but I can't claim to have a ton of experience with that combo.
On the surface they are undercutting Cloudflare in terms of price ($0.10 vs $0.15 per million), but with such differences in functionality it probably isn't a fair comparison.
The architectural approach is what I find most interesting. There was another post that went into more details on the architecture and security, but it seems to have been taken down.
From my understanding of the test scenario, you are using a single TCP connection to run the throughput test, and hitting AWS' documented[1] throughput limit for a single flow: 10 Gbps if the two instances are in the same placement group and 5 Gbps otherwise. The reason some of the network-optimized instances were "slower" than the non-optimized ones is most likely a random draw of whether both instances in the test were physically close to each other (basically whether or not they are accidentally in placement groups).
To show the true throughput you would need to use multiple connections/flows, 5-10 would probably suffice. If the single flow test case was important then maybe you should have mentioned that AWS has a specific limitation around this. Personally I don't think a single flow test case is particularly realistic for a throughput test. Either way, how it is presented is pretty misleading.
1. "Single TCP flow is limited to 10 Gbps for instances in the same placement group and 5 Gbps between instances anywhere else." https://docs.aws.amazon.com/whitepapers/latest/ec2-networkin...
On the throughput side, all this test does is demonstrate the documented[1] throughput limit for a single TCP connection. 10 Gbps if the two instances are in the same placement group and 5 Gbps otherwise. The reason some of the network-optimized instances were "slower" than the non-optimized ones was because it was simply a random draw of whether both instances in the test were physically close to each other.
If they wanted to do a proper throughput test they would have used placement groups and multiple connections/flows. If they felt like the single flow test case was important they should have mentioned that AWS has a specific limitation around this. Personally I don't think a single flow test case is particularly realistic.
The fact that the obvious discrepancy between their results and the documented (multi-flow) limits didn't cause them to dig deeper is enough to make me very skeptical of the purpose of this paper.
The latency results are also basically a random spread. It is essentially distribution of all the different latencies you might randomly get between two instances if you don't use placement groups. It says absolutely nothing about the networking capabilities of different instances used in each test.
1. "Single TCP flow is limited to 10 Gbps for instances in the same placement group and 5 Gbps between instances anywhere else." https://docs.aws.amazon.com/whitepapers/latest/ec2-networkin...
They are using this fallout as an opportunity to gauge the market. Risky to be sure, but they didn't do it without considering the risks.
From the article:
> I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the endIf anybody from Cloudflare is listening I would love to get on the beta asap. Would also be great if you had some ready to go simple, clean blogging templates :)
I'm generally impressed with the direction Cloudflare is moving, they have some really powerful and innovative infrastructure primitives (workers, durable objects) and a killer network. Going to be interesting to see how they take advantage of that as they build out higher level offerings.
By the way, if anybody has recommendations for simple, clean, modern looking blog template (something like medium) that they have actually used, I am all ears. So far I have found theses two:
- https://github.com/jekyll/minima
- https://github.com/wowthemesnet/mediumish-theme-jekyll/
Minima is a little too minimal and mediumish seems a little muchish, but I don't want to spend too much time searching for the perfect theme so unless something else lands at my feet I probably just pick one and start writing.
io_uring looks like it will be a big deal, it will probably be a while before it really hits mainstream, but I was waiting on 5.10 before trying it out.
I definitely recommend checking out the video. Even if you have seen it before, rewatching it in the context of this post-mortem really makes it hit home.
The relevant resiliency pattern in this case would be what they refer to as cell-based architecture, where within an AZ services are broken down into smaller independent cells to minimize the blast radius.
They specifically mention in the write-up that this was a gap they plan to address, the "backend" portion of Kinesis was already cellularized but that step had not yet been completed on the "frontend".
Celluarization in combination with workload partitioning would have helped, e.g. don't run Cloudwatch, Cognito and Customer workloads on the same set of cells.
It is also important to note that celluarization only helps in this case if they limit code deployment to a limited number of cells at a time.
This YouTube video[1] of a re:invent presentation does a great job of explaining it. The cell-based stuff, starts around minute 20.
I did a quick search in the GitHub repo for WireGuard and didn't get any results so I guess you aren't using it.
Looks like whoever took over the account is not some random kid with a friend at Facebook, but a "professional" account hijacker.
Whether they compromised the account via hacking, bribes or social engineering is anyone's guess, but it seems like this ain't their first rodeo.
I have tried doing similar things in the past,using API Gateway and SQS without Lambda, and it was a huge pain.
I wish they had skipped coining the "storage-first" buzzword though. I mean, I get it, but I feel like it hurts more than it helps.
My first assumption was that they were just talking about writing data to S3 and then using Lambda to process after.
VSCode, GutHub and Docker. Three peas in a pod.
FYI if your instances are behind an Application Load Balancer or Classic Load Balancer then they are protected, but NOT if they are behind a Network Load Balancer.
A patched kernel is available for Amazon Linux 1 and 2, so you won't have to disable SACK. You can run "sudo yum update kernel" to get it, but of course you have to reboot. Updated AMIs are also available.
Amazon Linux 1: https://alas.aws.amazon.com/ALAS-2019-1222.html Amazon Linux 2: https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html
For Amazon Linux 2 the fixed kernel is kernel-4.14.123-111.109.amzn2. Looking at my instances, it look like I have been on that version since Friday.
I think GitHub Actions will become a major force in the CI market in short order, it has so many things going for it
a) Everyone already has an account and lots of code already lives there. One less extra thing to worry about.
b) I trust MS/GitHub with my Cloud secrets more than I trust the various other CI providers.
c) The financial backing of MS to provide a significant free tier
d) The fact that actions can so easily be shared on GitHub is a killer feature. More are more projects/companies will build actions for their end users.
I would love to get a behind the scenes look at the process of gradually improving the components of DynamoDB with better technologies, while still maintaining reliability and performance.
Customers are paying AWS so that their SREs don't get called, they don't care if the AWS SREs do as long as the system keeps running.
Based on the supporting quotes at launch from Capital One, Dow Jones and WaPo it sound like enough customers are ok with vertical write scalability and (pretty awesome) horizontal read scalability for now because it fits their use case and is better than what they had before.
Also consider that since the cluster management overhead has been removed from the customer, they can essentially "shard" by using a separate cluster for each sufficiently large service/org/dept, which might actually work out better for them in some respects.
Perfect is the enemy of good enough, the architecture might be laughable to you, but it is probably miles ahead of what the customer was using before.
The Aurora storage subsystem is much more limited in terms of horizontal scalability and performance, they probably chose it because it was a better/quicker fit.
I kinda expected them to build it on top of DynamoDb's backend and provide the same kind of "Serverless" on demand experience, but I guess the architecture didn't fit, or maybe this was just faster.
1. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...