APIs, robustness, and idempotency
stripe.com
stripe.com
An ex-colleague pointed out to me on Twitter today that there are other APIs out there that have developed a concept similar to Stripe's `Idempotency-Key` header, the "client tokens" used in EC2's API for example [1]. To my knowledge there hasn't really been a concerted effort to standardize such an idea more widely, but I might be wrong about that.
[1] http://docs.aws.amazon.com/AWSEC2/latest/APIReference/Run_In...
I don't have a perfect history of events, but I suspect that our current design is basically the result of two things:
1. Idempotency keys as a concept were introduced quite a bit later than the API was originally conceived, so it made sense to make them an optional augmentation to existing integrations.
2. Our resource IDs have a fairly specific formats including a prefix (i.e. `acct_` for an account or a `ch_`). It would still be possible to generate this client-side, but it's a little extra trouble.
There are certainly a few things around HTTP semantics that Stripe got wrong. e.g. Most updates should probably be `PATCH` instead of `POST`, but it's probably not worth changing at this point.
> "On a response failure (i.e. the operation executed successfully, but the client couldn’t get the result), the server simply replies with a cached result of the successful operation."
Have you considered having the server respond with different http-status-codes for the initial successful request as opposed to any nop-retries? This provides the client with additional information that may prove useful. A lazy client could simply choose to treat both response codes identically, as a success indicator. And a more diligent/sophisticated/paranoid client could choose to act upon this information in some other way.
From a side-effects/persisted-data perspective, returning a different status-code would have the exact same effects as what you described. But it would also give clients additional information that they can choose to act upon if desired.
We use similar tokens in our API for idempotency as well, though we called them nonces. Standardizing a bit on terminology and how to implement them properly would be really helpful.
From a more practical sense, the naming is most likely to stay unchanged just because a lot of people are used to already.
But the concept as it applies to HTTP APIs is a great one that I think could have a broadly recognized implementation. Whether folks start adopting the "Idempotency-Key" header or some other approach, having awareness and maybe even a standard could be great.
Oh yes, hugely agree there! Part of the reason that I wrote this is that I couldn't find much prior art around idempotency keys (there is some, but not much). I am at least partially hoping that people will remember this article (or that it'll show up in a Google search) as they're implementing their own similar concept, and re-use the naming.
By the way, WebDAV extended this mechanism with a general If header [4] for all your precondition needs. I’m kinda glad it didn’t catch on though...
[1] https://tools.ietf.org/html/rfc7232
[2] https://tools.ietf.org/html/rfc7232#section-4.2
It's always a bit of a fine line as to what makes the final cut in this sort of article (I tried to stay on message without getting too off track), but HTTP conditional requests are definitely something that could have been a good fit.
I should point out though that using `ETag`/`If-Match` generally has a slightly different use on updates compared to Stripe's `Idempotency-Key`. A server sends back an `ETag` that's correlated to the current state of a resource, and clients make conditional requests using one so that they can get a guarantee that they're not changing state where they didn't expect to.
Because every HTTP request stands by itself, it's very possible for a client to fetch a resource and go to update it on a second request only to accidentally clobber changes that were made by a different client. It's this sort of "mid-air collision" that `ETag`/`If-Match` help to avoid. Mozilla's documentation on the subject is quite good:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ET...
Or put another way, is the only difference that ETags are generally computed based on the server's stored state of a particular resource so it's possible to have multiple clients with the same ETag, while the 'resource' backing an Idempotency-Key is the entire state of a particular user that encompasses all the user's resources?
Using `If-Match` is essentially indicating that you want to make a request conditionally as long as the server's state matches a nonce that you're holding. Presumably that nonce was handed to you already by the server on an initial request that you already executed.
You can hand Stripe's API an `Idempotency-Key` on the first request that you make against it. Furthermore, you'd never say that the first request made in this way was meant to be conditional, even if subsequent retries (after an initial failure) might be.
I think it wouldn't be a problem in practice to just retrofit `ETag` to do the same thing, but doing so is (arguably) semantically wrong, and I think there's something to be said for the clarity that just using your own header with an obvious name like `Idempotency-Key`.
I'm open to be persuaded though :)
Instead of POSTing to /transactions, I GET /transactions/fresh (optionally a URL linked from /transactions to avoid assumptions about URL structure and capabilities) which generates a unique ID and redirects to /transactions/{some-unique-id}. Attempting to GET that transaction would return 404 as it doesn't exist yet, but I can PUT to it to write a transaction. Now I'm using only idempotent methods instead of POST, and I don't need to figure out how to properly construct and manipulate a token in a header, and proxies don't need to know about this special header to know requests are idempotent since the methods I'm using communicate that already.
This adds a minimum of one extra request to all two-phase resource creations. If your goal is to retry safely though, the number of retries could dwarf that overhead. It all depends on how likely it is that failures actually occur. Client-side ID generation removes the extra request but brings back the problems of clients needing to understand URL and ID formats and construction logic.
https://stripe.com/docs/upgrades
As to how they achieve it Amber has a good blog post over at http://amberonrails.com/move-fast-dont-break-your-api/ but it means it really is integrate once, and never touch it again :)
API versioning is definitely a debatable subject, and I don't think that anyone at Stripe would claim that the current state of affairs is perfect by any means, but it's one that we think provides a good compromise between the stability of client integrations and our own ability to iterate on the API's design and make progress.
The classic problem with web APIs is that unless you have a good versioning scheme, once you've published them, you can never make a backward incompatible change (like removing a field) unless you're okay with breaking some people's integration. Especially when it comes to payments, people tend to have strong feelings about having their integrations broken, so we try to take as many precautions as possible to make sure that doesn't happen.
An approach to versioning that you'll see in many places is to do "major" API versioning where you do something like prefix your URLs with `/v1/` or send in a special `Accept` header. A problem with that approach though is that you'd need an incredibly good reason to ever build out a `/v2/` because if you ever bump that major version you're going to leave an incredible number of users behind on the original. Most people want to integrate one time and not have to worry about upgrading (ideally ever).
At Stripe, we've tried to build a compromise by introducing minor, date-based versions that include only a fairly constrained set of changes, but which we can bump more liberally. As others have mentioned here, your account gets locked into a version on its first request, and if never want to worry about API versioning at all, you can leave that version untouched essentially indefinitely.
If we realize that we made an API design mistake somewhere, we can fix it relatively easily and keep the API's design more cohesive for new users, while also leaving current users unaffected. It's also much easier to maintain for us because we only have to build a small compatibility module instead of having to maintain two (or more) completely divergent major API versions.
Anyway, I hope that helps explain some of the thinking behind this versioning scheme :)
Even amongst those that did do a major API revision, I suspect that you're usually left with the no-win situation of either leaving users lingering on your previous version ~forever, or enforcing a deprecation schedule and annoying a lot of people (Twitter's V1 retirement for example).
Say I come in with a request for V2. How does that get directed to the V2 code path? What about services that are identical in V1 and V2. Do you have 2 copies of the same logic? Sorry for the naive question but API versioning is something that has been on my mind recently.
This information has been talked about publicly before, so I don't mind explaining at all.
For the most part, the core API endpoint logic is all coupled to just the latest version. For each substantial API change in each new API version, logic is encapsulated into what we call a "compatibility gate".
Before responding, a merchant's current version is looked up, and the response is passed back through a compatibility layer that applies changes for each gate until it's been walked all the way back to the target version, then the response is sent back.
I'm glossing over a few details here of course — versioning can affect request parameters and even core logic in many places, so some gates need to be embedded throughout core code. We try to keep that as clean as we can.
This is basically the only way to work with any payment service integration, in my experience.
A lot of these services concentrate on documenting their current versions but not historical ones, so maintenance of integrations using older versions is unnecessarily difficult, even if you only want to use other features that were already supported at your current version.
It's often also painful to update to a more recent revision, with limited documentation or tools for identifying breaking API changes and how to convert your integration systematically to work with the newer version. You basically have to do a full rewrite against the new API from the start, and of course that probably comes with a significant risk of regressions in other areas.
In some cases, there are also confusing and/or poorly documented rules about which version you actually get both on active requests to the API and in any webhooks you get back, depending on what's pinned to what, whether you've updated a default version in a dashboard somewhere, whether you send any extra headers with your request, whether any testing/sandbox environments are linked to the same version as production, etc.
It's all very unfortunate, because every payment service we use has added potentially useful new features since we first integrated, and using those features would probably bring in more revenues for us and by extension more fees for the payment services. However, the risk of breaking such an important part of our system is just too great for us.
Stripe's API versions come with absurdly detailed changelogs, documentation that's complete for each version and automatically shows you the right doc for the version you're on, and consistency/forethought in API design that means very, very rare breaking changes that cause real problems for an application.
I've used them for almost 3 years now in an extremely complex payments system -- we use nearly every feature they offer in several different ways -- and it's always been a pleasure.
I was being polite. Stripe is my go-to example for integrations with payment services sometimes becoming write-only by default as time passes and their API moves on.
Stripe's API versions come with absurdly detailed changelogs
Stripe's API has a basic changelog. This is helpful, but also the minimal requirement to be useful at all.
It would be more useful to provide a migration guide that also shows how to change an existing integration to work with newer API versions when they make significant changes, similar to the documentation available for setting up a new integration in the first place.
documentation that's complete for each version
As Stripe's service has grown, unfortunately their documentation has grown less reliable. There have been quite a few errors and omissions in recent years, which we've pointed out to them from time to time.
As for "each version", where have you found official documentation for any API version except the latest?
automatically shows you the right doc for the version you're on
No, it doesn't. I've just checked this.
consistency/forethought in API design that means very, very rare breaking changes that cause real problems for an application
Breaking changes aren't particularly unusual with Stripe. Fortunately, they are pretty reliable at supporting older API versions as well, which is to their credit.
Again, I would consider this a minimal requirement to be useful in this industry, but it's not something that everyone does as well as they do and I know it's part of the reason some businesses stay with them.
I've used them for almost 3 years now in an extremely complex payments system -- we use nearly every feature they offer in several different ways -- and it's always been a pleasure.
Unfortunately that doesn't mean the limitations of how their system works are any less limiting.
In addition to the various points about, I'd add that although you can override your current API version on a request, you can't similarly override it temporarily on webhooks, so testing an updated integration is difficult.
There's also, as far as I've ever seen, no documentation that specifies exactly what the effects of that API version setting on the dashboard are. For example, will changing it affect both production and testing environments simultaneously, and can it be downgraded again when testing against a certain API version is finished or if a regression is found? You can only find these things out by trial and error in my experience, and it's a brave developer who'd try that on their first integration when it might compromise their production system.
It's still not possible to set up a fully automated integration test suite either. This would obviously be a very useful facility if you wanted to check whether your integration would still work properly against a new API version.
I'm happy that you're happy, but please consider that there are plenty of us who have also been working with Stripe and maintaining integrations with their systems for a lot longer than 3 years, and maybe we've simply encountered different problems than you have.
[Edit: Removed some unnecessary snark. The point of this comment was not to have a dig, because Stripe do better than many in this area. But the problem of effectively becoming version locked as a current API gets further ahead of the current integration is ubiquitous with web services, including payment services. Almost all of them, at least among the ones I've used professionally, could do more to help their users not just integrate initially but also maintain or update those integrations, possibly months or years after they were first written.]
By looking it up I learned the concept of jitter, which is how Ethernet works, and I think it's really cool.
PUT example.com/orders/abc123/charged
The first PUT starts charging. If the client disconnects and retries while the original charge process is already running on the server, the server can just await this already running process. And if the order is already charged, the server can just return the outcome of the charge.
As long as there is a unique URL for every state a resource such as an order can be in, you can uniquely identify every action on it on the server-side and re-attach to running processes.
Using unique URLs for each state is the main idea behind REST (REpresentational State Transfer). REST namely means just this: representing server states as resources (URLs).
So you want your endpoint to only do something once. Fine, does that mean I need a table in the DB with every key I've ever seen?
The most pleasant way I've solved this has been to think of this as rate limits where the limit is once per forever. After that they fit nicely in a token bucket and rate limit caching solution. This is one of the most useful thing that I think https://www.ratelim.it/documentation/once_and_only_once does. (I built RateLim.it)
Do the Internet a favor, and file an issue with your favorite HTTP library asking them to implement exponential backoff.
I haven't found anything in JS that does this properly though. Do people really just write apps that crap out upon the first HTTP request failure?
The best library I have come across is actually SquareUp's OkHttp (the payment processing companies seem to be the only ones getting this right).
- Should they be the default?
- What requests should be retried? (as much as we wish GETs were idempotent... they're not) see https://lists.w3.org/Archives/Public/ietf-http-wg/2017JanMar... for an intro to the complexities here
- What should get reused? (e.g. do new dns lookup? reuse connection? reuse proxy connection?)
- How to handle non-replayable pieces? (e.g. request bodys coming from a fifo)
- Usually a request has a timeout/deadline. should it be restarted for the retry? (probably not)
I haven't implemented retries yet, as the above questions seem hard to answer without knowing application/server specific qualities (and hence should be left to the user of the http library). Please feel free to file an issue :)
As a library user I need to work around the peculiarities of the particular service endpoints I'm integrating with. Backoff and retry aren't specific to the application/transport protocol through which a service is consumed.
For example, (in the java world) the approach in libraries like hysterix, guava-retrying, and failsafe https://github.com/jhalterman/failsafe#retries
Most of the colleagues I've had have rolled their own one off solutions when needed.