The 2002 mandate for internal communication systems at Amazon
sametab.com
sametab.com
>3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.
API first... except if you want to be a number of certain new services that somehow managed to get away with not presenting an API, even though an API would make every service team's life easier.
> 5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
Except, similar to above, where teams apparently decided they didn't want to think that way at all and management just let them.
> 6) Anyone who doesn’t do this will be fired.
Unless your exception is perceived providing value to the company. Then you'll get lauded, and everyone is told they'll need to use your js laden, web only interface, and to hell with any automation.
Mostly those exceptions just codified further in my mind about just how right the Bezos email Yegge paraphrased actually was.
Screen scrape the other service and do data exchange via a Selenium script.
Directly interact with the other service's database.
CSV files and nightly batch jobs.
The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction.
Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement.
But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
- The page has automatic history & merge conflict resolution - There's built-in role-based security to control both visibility and actions (read-only vs. read-write) - You can work on a draft and then "deploy" by publishing your changes - You can respond to hooks/notifications when the page is updated
Considering that the alternative is either editing a text file on disk or teaching business users to use git, a Confluence page is not so bad.
Apparently it was so hard to deal with the developers that instead of exposing an API they would automate clicking around on Internet Explorer browser windows.
Thankfully I haven't heard much about it lately.
But enough about the web...
There's also:
- Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another
- Dump some file in a directory and hope something is treating that directory like a queue
- Read/write from the same database (/ same table)
Or the classic Unix trajectory of increasingly bad service communication:
- Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers)
- that, but with some mmap'd region (because the next team of developers doesn't know how to select())
- that, but with a local file (because the next team of developers doesn't know how to mmap())
- that, but with some NFS file (for scaling!)
- that, but with some hadoop fs file (for big data!)
Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.
This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
"Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."
On-hand work experience.
My personal experience comes from ingesting product feeds from online stores. Misapplication of \ from other encodings was the most common sin, but I'm pretty sure I saw about three dozen others, from double-comma to null-terminated strings to re-encoding offending characters as hex escapes. (And, of course, TSV files called CSV files, with the same suite of problems.)
This gets abused even within one service. If I could get my coworkers to FFS stop using rows in a database as a degenerate kind of communications channel between components (with "recipients" slow-polling for rows that indicate something for them to do), I'd be a lot happier.
> rowhammering the data you want into the next job.
I hope the aforementioned coworkers don't read HN.
- Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.)
- Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.
https://www.ibm.com/products/datapower-gateway
It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling.
It has the added benefit that you can add security for outside facing clients.
Disclaimer: I helped develop this appliance (but I no longer work for IBM)
So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.
You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".
Yeah it is (though not so "micro-": it's really one service per team, which is the scale that actually makes sense IMO). This is kind of a "seinfeld is unfunny" case: this memo was in 2002, and predated (arguably helped create) most of the modern microservice approach.
> Also isn't this terrible for latency and debugging ?
When two codebases are owned by different teams (and potentially in different languages), it's far better to have them separated by a network connection and a well-defined HTTP API than living in the same address space where they can corrupt each other. Since the interfaces are well defined, you can isolate any problem down to a request that's not getting the correct answer from the service it's calling, and then hand that over to the other team to investigate. Since the request and response are plain text, it's easy to see what's happening.
If responding to a single request from the end customer requires chaining through several independent codebases maintained by different teams, you have bigger problems than request latency. Each team should own a piece of end-to-end functionality, not a horizontal layer.
Considering 'microservices' wasn't a well-known term for another decade [1], maybe better to ask: isn't 'microservices' just this ?
[1] https://trends.google.com/trends/explore?date=all&q=microser...
I think the core of the Amazon invention was to mandate that the API be a strict boundary around a service. No cheating by making assumptions about the internals of the implementation. Force teams to work off the published API documentation, and make teams the guardians of each service. Microservices probably aren't developed exactly like that in many companies.
> Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..].
> He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS.
This is quite a strong and opinionated statement. I'd agree that Jeff made this change to improve communication infrastructure for Amazon.com. I wouldn't agree that he made it to enable AWS - for two reasons.
First, Amazon started thinking about EC2 and S3 way way later than this. Second, Kindle and SimpleDB (now Dynamo) were similar independent bets. These projects were happening at the time of moving to SOA, but they weren't tied to it.
The one thing common across all of these products is - making informed bets on market fundamentals and enabling teams to deliver on them.
Now, I know of a few fairly senior Amazonians who read those forums. They have more context than I do. So, if they chime in and say that Jeff sent this email to enable AWS, I'll gladly take it. :)
Until then, I wouldn't extrapolate Steve Yegge's post to mean something about Jeff B.'s intentions, and build an overarching argument over it.
Given the recent 20,000 startup ideas post, this is the perfect algorithm to cut through the noise
Minor nitpick, these are two separate services. SDB is still independently accessible and probably will be for a while. That being said they definitely don't encourage any new use of SDB and push Dynamo instead.
Kind of a shame because a modern NoSQL store with SQL support would be great for rapid prototyping before a concrete data schema is established.
sdb is build on the same principles outlined in the dynamo paper: https://www.allthingsdistributed.com/files/amazon-dynamo-sos...
ozgune is correct as far as I remember.
The beginnings of AWS, though, were for the retail site. We exposed search and browse and item metadata via an API first for a deal with AOL to provide them with product search and then later we opened it up to everyone. People built stuff like Simple Amazon on top of it, which I thought was pretty nice. I don't think this part of AWS exists anymore (please correct me if I'm wrong).
The first service exposed as what we'd today consider AWS was SQS and at the time it was kind of a head scratcher. Only later did we understand that it was glue for other services.
S3 and EC2 were quite a bit later I think.
Between 2000/2003 Amazon started to learn how to properly make something that resembled AWS (launched in 2006), it was a necessity targeted towards scalability. What Yegge letter meant is that Amazon was doing SOA to all its depts, making these be useful both internally and externally
>"5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world."
So you see, it wasn't to enable AWS as the digital service as we know it, it was to build a scalable company that could use everything it had both to serve its needs, and also to sell it as a product (then, all services we all know EC2 etc). Bezos applied SOA as a company, not only to its digital philosophy. And this corporate move, with no precedent, enabled not only AWS, but the whole Amazon as it is today.
Initially, during the good old days of Amazon, there was what you'd call a single datawarehouse. It made sense initially for every system that processed an order to access the data by querying that data warehouse—this meant that the processes would be distributed (different services), while the data would be centralized. It also meant that any change to the way the data is stored in the datawarehouse meant deploying code to a hundred places.
The most important problem that this addressed was however different. A centralized datawarehouse meant that every customer request bubbled up into N queries to the datawarehouse (where N is the number of services that needed access to the data—billing, ordering, tracking...).
The mandate summarized in one line would be this—"the data is the one that should go to the services, not the other way round." Voila, microservices.
Yeah we use services heavily, but there’s plenty of teams dumping data to S3 or using a data lake.
There’s also the “we need to do this but management doesn’t see value so let’s dump it on the intern or SDE 1, who we won’t really mentor or guide and then blame, forcing them to switch teams as soon as they can.”
If you work at another company and think we have our stuff figured out at Amazon, we really don’t. We have brilliant people, many of who are straight up assholes who will throw you under the bus. We have people who are kind and will help you gain all kinds of engineering skills. We also have people who are scum of the Earth shit people who work at Amazon because I don’t think any other sane company or workplace would tolerate them. We have extremes on the garbage people end of the spectrum, unfortunately.
Sorry long rant - point being - it’s good to learn how we do things. The internal email on services is pretty unique. I learned about it when I was an SDE 1 back in the day. But - don’t take it as gospel. It doesn’t mean you need to build services.
I can think of any number of examples where we follow anti patterns because no one gives a shit about the pattern, whether it’s a service, a bucket, a queue, or a file attached to the system used for scrum tasks, or shit passed over email... we care about value at the end of the day. If you don’t provide sufficient value at Amazons bar, they have no problem tossing you out the window.
> Having teams acting like individual APIs and interacting with one another through interfaces over the network was the catalyst of a series of consequent actions that eventually made possible the realization of AWS in a way that couldn’t have been possible otherwise.
Google has worked this way since time immemorial. That’s what protocol buffers are for: to create services and pass data between them using well defined interfaces.
Do internal Google services exclusively use the Google Cloud Platform APIs? The implication is that internal Amazon services exclusively use AWS APIs. I've never worked at either company though, so I don't know if it's true. Perhaps someone could clarify.
Untrue. Some even use older version of AWS "core" services (S3), not the latest and greatest version of AWS services.
"Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers."
- from Steve Yegge's Platforms Rant[1]
Also, 2002 is time immemorial. Google was founded in 1998.Protocol buffers were invented in 2001.
https://developers.google.com/protocol-buffers/docs/proto#se...
"Why would we sell our cloud platform? We can always make more money and have higher leverage by running our own services on it and monetising with ads; merely selling hardware and software services is a comparatively uninteresting and low margin business."
Selling Google's platform (and it really is a platform) is an obvious idea that occurred to everyone who was there. It didn't happen because of explicit executive decision, not because Bezos was some kind of savant.
I think Google could have really dominated the cloud space if they'd been a bit more strategic. The problems were all cultural, not technological. For instance they are culturally averse to trusted partnerships of any kind (not just Google of course, that's a tech industry thing). There are only two levels of trust:
- Internal employee, nearly fully trusted.
- External person or firm, assumed to be a highly skilled malicious attacker
There's nothing in between. So if your infrastructure can't handle the most sophisticated attack you can think of, it can't be externalised at all. If it can't scale automatically to a million customers overnight, it can't be externalised at all.
There's really no notion in Google's culture of "maybe we should manually vet companies and give them slightly lower trust levels than our employees in return for money". It's seen as too labour intensive and not scalable enough to be interesting. But it'd have allowed them to dominate cloud technology years earlier than AWS or Azure.
1. Everyone is super excited for other teams to share their data
2. Everyone wants an exception from sharing their own data because it's too hard or too sensitive to share.
3. Eventually everything gets shared, but it takes 3-4 times longer than it really should.
Hilarity ensued.
His follow up post explaining this, and with an interesting anecdote about presenting to Jeff Bezos, is archived here (seeing as G+ has, ironically (or not) given the context, shut down): https://gist.github.com/dexterous/1383377#file-the-post-retr...
https://aws.amazon.com/solutions/case-studies/amazon-fulfill...
I suppose it’s nice that the email, or really any amazon emails, has not been leaked.
So I've never worked at a company over 150 people. Is this... a normal thing for an email? Maybe I'm just one of those softies but an email with that line would throw me off my day and cause a serious hit to my morale and confidence of working there.
I have much respect for what he has achieved, so I didn't interrupt to question such a fear-inducing mindset.
The part about McCord firing an employee is sad and funny at the same time. To begin with, it looks like a page from Vonnegut’s Player Piano:
“McCord mentioned letting go a product testing employee who “was great,” but eventually lost her job to automation.”
It isn’t mentioned if the employee contributed to the automation that eventually replaced her, but it may as well be so.
Nevertheless, her depiction of the conversation is a rare mix of sad and funny:
“So I called her up. I’m like, what part of this is a surprise? … And she goes, yeah, but, you know, I’ve worked really hard; this is really unfair. I’m like, and you’re crying? She’s like, yeah. I’m like, will you dry your tears and hold your head up and go be from Netflix? You’re the–why do you think you’re the last one here–’cause you’re the best. You’re incredibly good at what you do. We just don’t need you to do it anymore.”
Of course, this strategy has its downsides. You can't ever hire juniors. You can't really hire people in and train them up at all, because everything has been built under the assumption that only experts will ever touch it. This makes an organization that operates like this inherently parasitic to the industry, only capable of hiring in experienced employees from other companies.
Incidentally, a couple of years ago, I was a contractor for Amazon for a very short time. One thing that stood out was that customers would email Bezos directly, or problems would bubble up to him in some other way, and he would respond by forwarding the email to an appropriate email list. Bezos’s email would be just a question mark, but he always got a fast and complete reply. Once people knew he was paying attention, they would do anything to resolve the problem. He didn’t need to formulate an actual question, or even a general “what’s this about?”, just a literal question mark would do the trick.
People don’t actually get credit for addressing the issue. People just get dinged for not addressing the issue quickly.
It is actually a red flag if a CEO or founder doesn’t behave this way since it means that they aren’t showing care and attention or that they are operating at the wrong level.
6) Anyone who doesn’t do this will be fired.
7) Thank you; have a nice day!
"There are two schools of thought about how information should flow within companies. By far the most common way is chain of command, which means that you always flow communication through your manager. The problem with this approach is that, while it serves to enhance the power of the manager, it fails to serve the company.
Instead of a problem getting solved quickly, where a person in one dept talks to a person in another dept and makes the right thing happen, people are forced to talk to their manager who talks to their manager who talks to the manager in the other dept who talks to someone on his team. Then the info has to flow back the other way again. This is incredibly dumb. Any manager who allows this to happen, let alone encourages it, will soon find themselves working at another company. No kidding."
Handling these complexities of scale are exactly what managers are for.
I acknowledge that it's probably a joke anyway.
No, that sort of overly forceful yet flippant language it's not normal.
Yegge has punched things up a little, but I can translate #6 into what I believe was the original text: "This represents the new direction for {company}, and anyone who does not wish to align with this new direction should pursue other opportunities".
The day-to-day employee at a large company becomes numb to executive emails, as 90% of the time they will be countermanded a month later. A sentence like the above is inserted to wake people up and indicate "no, seriously, this means you".
You can't sell to customers effectively if your flagship product only works because it has access to resources the customers will never have... and it is designed around that flagship's needs and not your customer's needs.
Edit: I've taken a rather lame crack at it and am open to improvements.
Amazon's not really big on "mandates" in general, but the term seems to fit Yegge's characterization of what happened. "Internal goal" would be another way to phrase it. E.g., "The single most important technical goal in the history of Amazon".
A single hour-long in-person meeting can forgo a month of back-and-forth online.
"Did everyone get a chance to read the doc I added to the calendar entry last night? No? Ok, I'll just review the main points. So for starters, let's go talk about why we're here...."
It doesn't happen at many other companies, even ones stocked to the gills with Amazon refugees.
Yegge has another famous blog post about presenting to Jeff.
I'm not following what he means. What is the thing he is describing as "an internal communication system" here? That made all the rest possible? What is/was this internal communications system?
I read this as saying different teams/services don't have to use the same thing either. That doesn't sound like an "RPC framework" or "an internal communications system" at all. It seems to leave the door open to everyone doing things in a diverse mishmash. Which isn't what I'd call "an internal communications system" at all.
But was/is there in fact an Amazon-specific "RPC framework", that all Amazon services use, some consistent framework used consistently accross services? I haven't heard much about this before so am curious to learn more. I haven't heard of an Amazon 'RPC framework' before, or what it's called, or what. And OP doesn't specify it either; does the rest of the audience know what's being talked about, and I'm just missing context?
If that is the thing that the OP thinks is really what Amazon got right... then the interesting thing is figuring out how it went from the paraphrased email, which doesn't actually demand such a thing, to.... such a thing. Who designed or chose this "RPC framework"? When? How? How'd they get everyone to use the same one? If that's the thing Amazon got right, there are some steps missing between the Yegge-paraphrased email and there, since the email doesn't actually even call for such a thing.
Or is that not what happened at all, and I'm still not sure what OP means by "an internal communication system" being the thing Amazon got right.
1) getting developers to talk about formalizing (to any degree) cross-system responsibility at all rather than just hanging code where it's easiest at-hand;
2) getting developers to think about external inputs/outputs at all and everything that entails, e.g. namespacing, versioning, forwards and backwards compatibility, validation, access control, ...
3) teaching developers to pick the right kind of interface for their data and processing, i.e. more or less queue vs. pubsub vs. RPC vs. REST vs. query language; of the three this is the only "technical" one, the others will be harder sells.
Once you have done these, the question of _which_ queue/etc. you take is largely irrelevant; there will be some natural pressure to standardize and even "unnatural" pressure from CTOs/management will be straightforward to implement.
By the time I arrived, however, there was a cross-language RPC framework that integrated with Amazon's monitoring, request tracing, and build infrastructure (for building and releasing client versions). It was very full-featured and the de-facto system for creating a service. Most of our communication in my organization was done using this framework, and systems that violated the "only communicate over a service boundary" mandate were real problem children.
Does anyone know if there's been much written on how this came to be and what it looked like? If not, it would be a useful thing to write about!
Cause it does seem like a really important thing, without it, the narrative seems to be that you make a decree like Bezos', and bing bang magic, you get what AWS got. Where in fact, succesfully pulling off that RPC framework seems to be really important, and undoubtedly took a lot of work, good succesful design, and social organizing to get everyone to use it (perhaps by making it the easy answer to Bezos' mandate). But none of that stuff just happens, some have failed where AWS succeeded, the mandate alone isn't enough.
The AWS story is particularly interesting because a lot of the internal setup I was doing at the time was on old fashioned metal. There was an internal project called Move to AWS (MAWS) that encouraged using newly-developed integrations with the AWS systems that the public was using.
In other words, AWS lived alongside old-fashioned provisioning practices up until even the early 2010s.
My guess is this was the thinking behind the mandate, limiting all inter-service coupling to a single, well-documented interface prevents the whole system from devolving into a big ball of mud; primarily because it forces all dependencies to be explicitly declared and documented in the same way.
* Every service must provide latency and error-rate metrics.
* Every service must be capable of generating and/or responding to backpressure when things become overloaded.
* Every service must be prepared to support multitenancy.
I would have so much loved this approach in the last corporate job I had. It would have changed so many things in such a short time...
https://techcrunch.com/2016/07/02/andy-jassys-brief-history-...
So a jdbc interface and published schema would count?
Right, motivating everyone.. check..
Unfortunately it removes ALL of the top navbar but I've found it really useful to get around HN's damaging and useless gamification metric.