Idempotence: What is it and why should I care?

This is solid advice. Another common trick is to disable the submit event when the submit button is clicked for the first time, preventing two requests from firing, when the user double clicks. Then re-enable the event, if the request fails. Ideally this is done in addition to server side nonce validation, and not as the only preventative measure, because browser differences, or network issues could cause a double request (more likely if the HTTP GET method is used, instead of a proper POST). Regardless server and client side techniques should be used together to create a better UX, similar to using both client and server side email format validation.

Yup. This is actually an interview question that I go through to see if they know mechanisms to prevent the double charge, fulfillment, etc. problem.

Good point, but I'd add that not only when charging money, but any operation that can have side effects if executed more than once (emails, external calls, etc.).

Having idempotent queue consumers/listeners can help with overall systems resilience (someone already mentioned this below). It helps as much as you don't have to worry when restarting a queue broker and/or apps with consumers.

It adds complexity, but I think it pays off in the long run.

I apologize for the noob question, but... is it a good idea to have the client be in charge of generating UUIDs? When dealing with idempotence, they have to be truly random, not just pseudorandom, right? I looked into this recently, and my understanding (from the cursory amount of research I did) was that the UUID generation method depends on the random number generator used by the browser, which can vary a lot.

Why not generate the UUID on the server and send it to the client along with the page request (if using SSR)? I.e. the server generates the UUID, sticks it in a database field, then sends it to the client. When the client responds with the UUID, you can check that against the database to make sure its valid.

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but:

- Express all operations as log messages (ez pz distribution)

- Ensure all operations are idempotent

- Record the operations (this is the log you can distribute if you please)

- Disallow API code modification, only allow accretion/use of new API endpoints.

- All APIs that come up have their own databases, a bit of the CQRS model here (but without events -- just the actions performed)

- When you need to stand up new API servers, start the new ones (with handling code for old operations completely unchanged) next to the old ones, and update the http-server code (like request handlers) to output the new commands. Older servers that don't understand the new commands will ignore (or redirect), and new servers that do understand will process and add to the distributed log. New nodes just stream the replications of the already existing nodes and no one spends any time with an inconsistent view of the database

Of course, writing to a distributed log is slow (pick whichever consensus algo you want, you either have durability with a quorum or best-effort without), but this only is a huge deal if you're doing lots of writes, and for most web applications, that's not what's happening, the vast majority is reads.

CRDTs might even fit in here, because if you want a multi-master setup, you could literally keep the log as a set (I'm not quite sure how truncation of super old records would want) keyed by transaction ID -- Assuming the same request doesn't go to multiple servers, their logs should be easily combinable at the end of the day -- API1 is gonna see events A B and E, API2 might see C and F, and API3 will likely see D G and H.

Honestly everything I've described here is really more like moving the coordination/distributed log problem to the application level (up until now all this action would just happen @ the Postgres/DB level), but I'm not yet convinced it's a terrible idea.

I haven't found the time to actually try to make what I'm describing here a thing but would love to hear thoughts

There was a story about toggles in apis on HN a few months ago [1][2], basically the toggle function was called for the garage door, which is clearly not idempotent, rather than an action like open/close so that is wasn't repeated. The toggle action led to opening and closing of the door repeatedly. Non idempotent requests are especially bad when tied to real physical machinery.

[1] https://twitter.com/rombulow/status/990684453734203392

[2] https://news.ycombinator.com/item?id=16964907

spotify> For example it is hard to see the difference between Ω and Ω even though one is obviously a Greek letter and the other is a unit for electrical resistance and in unicode they indeed have different code points

This surprised me, because the correct Ohm symbol is in fact the Greek letter, so why does Unicode have a special code point for it?

Unicode also does this for Kelvin, where the correct symbol is a capital K but Unicode has a separate code point for it, and for ångström where the correct symbol is a capital A with a circle above it but Unicode gives it a separate code point.

They do not do this for Newtons (capital N), Joules (capital J), Watts (capital W), or anything else I can see where the standard symbol is an ordinary letter or group of letters.

In all three of these cases the Unicode Consortium recommends NOT using the separate code point.

So...what's special about Ohms, Kelvins, and ångström that (1) gives them their own place in Unicode, and (2) what is the point since we are not, according to the Unicode Consortium, supposed to use them?

This is about idempotent operations, which are basically state changes that have the same affect if executed multiple times as if executed once:

  $ chmod a+x file
  $ chmod a+x file

This is linked to mathematical idempotence in the sense that an operation is a function which takes some inputs i and the state of the world S and produces a new state S':

S' = f(S, i).

So then if f(f(S, i), i) = f(S, i) then f is idemponent mathematically, and the operation is idempotent in the software sense.

This differs from the definition in the article. I think the article's definition is the significantly more common one for software engineering.

In category theory we also say h is idempotent iff h^2 = h whereas h is an endomorphism.

One of my favorite parts of Python is the idempotent type casting functions.

Say you have a function that takes a list as input, but sometimes a tuple is passed. Wrapping the input like lvalues = list(values), ensures that lvalues will always be a list (or it throws a type error), so you won't get annoying attribute errors, when trying to access the list's methods. In the case of a list type(list([])).__name__ == 'list', as does type(list(list([]))).__name__, etc.

This can get a bit tricky with situations where type casting may be undesirable, say casting a string to a list. So judicious use is necessary.

Amen. And if you combine idempotent operations with atomic operations wherever possible, it gets even better!

How do you do it without throwing an error?

My philosophy has been to ignore the error if it's a re-application of the migration because the migration _is_ idempotent. So ALTER TABLE foo ADD COLUMN bar INT either succeeds (because there is no bar column) or it fails (because there is a bar column) with no harm done.

Database migrations are inherently stateful so I'm not a fan of indompodence here, it can leave the schema in some arbitrary states. I much prefer tools like flyway (https://flywaydb.org/) that are more deterministic, each migration will only be run once so you're going from known state to known state.

One thing to note about this - idempotency is a very simple concept, but the implementation is actually quite hard. Case-in-point: the kafka producer API recently gained support for "idempotent/transactional capabilities": https://kafka.apache.org/documentation/#upgrade_11_exactly_o... . That is to say exactly-once persistence by using an idempotency key.

Everything looks bulletproof unless you take a step back. To connect with the example you gave: just having the kafka producer guarantee that one call to send() is idempotent is not enough. Your application needs to be able to be idempotent - e.g. if the same RPC/web request has to be retried on a different server. You need to be able to pass an idempotency key TO the kafka API - which you currently cannot do. The API currently allows for either a global(ish) lock or duplicated messages - so it's not quite idempotent.

Idempotency needs to be end-to-end, otherwise it doesn't work. Unfortunately that's very rarely the case - almost nobody tries to idempotently make XHR requests to their servers. In effect it's almost always easier to de-duplicate idempotently on read rather than attempt to write idempotently. It's a really hard simple problem with lots of corner cases.

And in 2005 Google came up with Google Accelerator, which has shown many sloppy programmer what happens when you use HTTP method which is supposed to be idempotent—GET—for others purposes. What happened that GA crawled all the links it found for prefetching and some of them were "delete" links in admin interfaces. I think that was the biggest push not to use GETs to modify data :)

That's okay, I read "impotence" and couldn't believe that anyone would need it explained as to why they should care.

> you want two timestamps. 1 for when the row was created and 1 for when it was last updated. The upsert should change the ts_updated value, and the attempt should be logged.

If timestamps are worth recording they're worth recording in the normal way you'd record business data. Associate a timestamp with when the user made their request, not on when your server happened to process it.

> A gas station pump resets because of a blink in power. It remembers all the information about the transaction except for the uuid because the programmer was smart and wanted that to be, well, unique. And he wanted the pump to be smart and retry the failed attempt. Same txn; different uuid.

Don't do that. If it's a transaction ID, generate the ID for the transaction and keep it with the data for the transaction.

> The user hits the refresh button in the middle of the txn, but the rest of the form data is cached. Not the uuid.

Again, keep the ID with the data it goes with.

> Trusting a client to only ever retry with the same uuid is not safe. So when I say that you need a uuid in the same context, I mean the uuid created in your database when it receives an auth request, inside the SQL transaction used to create the entry for that attempt.

That's an unscalable approach. If you can afford to run everything on a single database with monotonically increasing time then sure, knock yourself out, it makes everything easier and is good enough for a lot of cases. But in these days of client-side UI, like it or not you are working on a distributed system where the client and server are separate nodes, and distributed system techniques are your best hope of getting sensible behaviour.