So true it hurts
I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.
I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea.
Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?).
One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".
Glad that works for you but for my game servers edge computing is not ideal and generally currently not even applicable.
At some point in time they'll have to start thinking about profitability.
Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.
The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere.
This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry.
In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations.
Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has.
Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete.
The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.
When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.
Only the most trivial business systems can operate without any state.
If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same.
For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.
I point to Cloudflare (what I'm most familiar with). Across their entire stack of compute, storage, db, etc product lines you won't see a hint of region anywhere. It just works, everywhere. Same with Fastly/Oracle (although they like Fly are quite a ways behind Cloudflare).
I suspect (hope) that as Fly matures their apparent true goal of edge and region-less applications will mature more. Their compromises on region awareness for DB has always struck me as a more of a resource restraint compromise on their part than end goal. As you say these challenges are hard (expensive).
https://developers.cloudflare.com/d1/learning/data-location/
Hint Hint description
wnam Western North America
enam Eastern North America
weur Western Europe
eeur Eastern Europe
apac Asia-Pacific
The fact remains this is somewhat down in the docs because it is automatic and invisible to the point it works well enough where most developers don't know, don't care, and aren't exposed to it in any meaningful way. Contrast this with AWS where the first thing you are presented with is picking a region where everything lives, while not clearly and explicitly explaining that many things are ultimately dependent on us-east. Cue the shock and confusion when us-east has yet another issue and those in all other regions don't understand how and why they are impacted.
You are correct about the free lunch, information theory, and physics at the end. I don't think a reasonable person can think when you write data to a pop in Tokyo that it could possibly show up in Chicago instantly. That is fundamentally impossible and always will be. I think CF does a pretty good job making it "magic" for most devs and use cases while explaining these fundamental realities and limitations to those who need to know (potentially) for whatever their application is.
Here's the Durable Objects documentation [1]. Don't those "location hints" look like regions?
Also, here's the "Data location" page for Cloudflare D1 [2]. See "available hints" at the bottom.
I think it would be more accurate to say that Cloudflare's architecture is designed around having lots of regions, automatic migrations between them, and no guarantee about which one you'll get. But they're still there.
[1] https://developers.cloudflare.com/workers/runtime-apis/durab...
[2] https://developers.cloudflare.com/d1/learning/data-location/
I actually argued against adding these hints at all, since it confuses the messaging and most people don't need them... but they were easy to add and solved an immediate problem for a few customers, so we did.
In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...
In any case, most people don't have to think about regions on Workers, and for the few that do, our goal is to change that.
(I'm the tech lead for Cloudflare Workers.)
That is an excellent way to design a product. Great job.
Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible. I haven’t worked with GDPR requirements but I have heard they are similar. AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.
All lower plans maybe get PCI DSS and that's it.
It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data.
We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day.
I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers.
Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.
N-2 companies ago I was a tech lead in 2015 where we were migrating to AWS. AWS was constantly explaining which underlying services were HIPAA compliant and what you build on top of it was your responsibility. I had to answer to separate non AWS compliance folks about my architecture.
AWS also gave guidance. But made no promises that your implementation was compliant. That’s where the entire “shared responsibility model” comes in.
Now, I work at AWS in ProServe. I am very familiar with our messaging regarding HIPAA compliance. We are very careful not to rubber stamp things as compliant. I know the best practices in and out. But when I’m asked if something is compliant, I say this is the guidance we are given. I’ll do a presentation before an Architecture Review board. But I would never make anything that hints as a guarantee.
The messaging/communication was more like what I described - "Oh yeah here's BAA you're good to go".
If you are a smaller organization, you may be able to request an SA.
My personal experience is from when companies pay for consulting hours from ProServe.
At the n-2 company I referred to above, the only thing we got from AWS is a BAA and we had an outside consulting company and compliance auditors. I was just learning AWS then.
In the Cloudflare Workers platform, we have a number of features to control "jurisdictions" where compute runs or where data is stored. These are meant for compliance, not for performance. Our goal is that no one has to think about regions for performance reasons, but certainly compliance issues are not going to go away -- in fact, they are getting more onerous over time. So features to control jurisdiction are likely to expand.
(I'm the tech lead for Workers.)
But it is rarely true for enterprise customers. They want the guarantees their data never leaves certain regions even in a request along with other security and compliance guarantees.
But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.
Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.
I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough).
Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.
AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.
For example, make your app talk to two backends in different EC2 regions, and now none of the smart placement helps you.
The poster we're replying to seems to be gleefully ignorant of speed of light limits and fundamental distributed systems challenges.
(The comment I'm responding to said something about people in glass houses not throwing stones, but was then edited).
Raising a bunch of money just isn't a "good" announcement to most of us. When unpopular things have happened with the company, your communication usually directly acknowledges and focuses on the concerns of the userbase. This serves two purposes:
1. It gives you a chance to show concrete steps to prevent specific concerns we have
2. Perhaps more importantly, it shows we're all on the same page and you're actually in touch with how your userbase thinks
Failing on #1 is something you can fix later, but I've never seen a company rebound from losing the trust gained by #2. It loses confidence that future decisions made can also be equally out of touch.
I've never seen a company rebound from losing that confidence
There are things that happen that disappoint customers that we care a lot about. Reliability issues several months ago absolutely fit that bill. So does hiring: we paused hiring for most of our engineering roles because we weren't happy with the experience candidates were getting. I don't know how "honest" and "transparent" we are about those things, but I do feel like when we're asked about them, we give straight answers.
There are other things that happen that disappoint people that we don't care so much about. Being unhappy that we took external funding to build out our fleet and improve reliability would count as one of those things. Again with the straight answers: I wouldn't expect us to spend too much time trying to "rebound" from this announcement.
Speak for yourself. Announcements of adequate funding inspires confidence fly will have longevity and resources to scale. Other concerns can be tackled in other blog posts, but nothing about the announcement of funding should inspire less confidence, what sense does that make?
VCs in particular will expect much more than their money’s worth, or (most of the time) for the company to die trying to earn it, and they won’t hesitate to force the latter in pursuit of the former. This increase in variance is literally their investment model.
So if VCs expect their investments to fail most of the time, and I see a company take VC money, I must have very strong reasons if I’m to disbelieve the VCs and expect the company to succeed—and I’m not following a venture investment strategy when I choose a tech supplier.
IPOs are somewhat milder in their intrinsic character, but they still usually lead to professional boards of directors and eventually professional execs, where “professional” usually means “no subject matter expertise”. And hardly anybody runs a public company on customer satisfaction nowadays, unfortunately.
So when I see controlling equity sold, my outlook changes from “will this go to shit?” to “when this inevitably goes to shit, will it be in the distant enough future that I don’t care or can plan for the loss?” (that being maybe a decade or so for infra, or implausibly long—like the rest of my natural life—for anything involving a community).
As usual, the question remains—why does none of this seem to be priced in? I don’t know.
ARE YOU SAYING YOU INVENTED THE OFFICE?
Ashburn is literally like the Embassy Suites next to the Five Guys, the Aloft next to the taco place, Equinix, Digital Reality, Verizon, NTT, CenturyLink, etc.
Take a look at this map here: https://www.loudoun.gov/DocumentCenter/View/171141/2022Elect...
Ashburn is not literally just one shopping plaza. It's a giant area zoned into different parts of datacenter, suburbs, and commercial sections.
Even forgetting all of this, just type "restaurants in Ashburn" into Google Maps see the plethora of results.
Just because someone has some imaginary boundary of what constitutes as Ashburn doesn't make it definitively small.
I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I think I'll be avoiding Fly on this alone.
I'm confused, when did any of Fly's employees say as much?
That said, the point I'm trying to make is that it is in fact a lot nicer to use than simply YOLOing whatever onto servers. This model of PaaS is basically what App Engine should have been. It's difficult to want to go back once you've used it.
Money doesn't contribute all that much to solving it though. It's usually time to make the early mistakes, continuous improvements, etc. But AWS had several decades to solve this, whereas fly clearly hasn't, so I'm not sure I'd count them out yet.
Their brand marketing is impeccable as a result. Every post on their blog comes through with this very clear, slightly snarky voice.
If you want to be a famous hacker, you must also be a slightly (industry) famous author.
Their tone of voice reminds me of early Slack before that became rote and boring.
Snarky is passable in open source, a genius whose abrasive manner is the cost of the amazing product.
In a corpo it's sleazy
Doing stuff isn't enough - you also need to communicate to the Earth that you have done stuff.
Was it just articles or more?
can you explain this part of the joke?
It's not intended to be taken literally (I assume).
Must be gerrymandering or something, maybe aided by voter suppression by English speakers against those in the county whose native language is x86-64.
https://www.loudoun.gov/177/Coat-of-Arms
"Where Tradition Meets Innovation" appears to be a branding push by the county dating to 2019 rather than an official motto. https://www.loudoun.gov/ArchiveCenter/ViewFile/Item/8563