Hacker News Reader

Comment by pbhowmic | Hacker News Reader

Idempotency is great to have and almost always non existent. It is just infuriating as a dev to keep coming across this mirage of an idea. I don’t get to reprocess the same bill twice without bad consequences. I don’t get to ship the same order twice.

View on HN

Not everything can be idempotent, nor immutable, nor stateless. But if you adopt the core assumption that you will achieve those states in the places where they can be achieved (and attempt to make them easily achievable with patterns that support them), then when you do isolate situations that cannot be, like physically shipping a product, you can pay more attention to to making them correct. Most of the innumerable in memory, business logic, database operations, etc., behind the shipment can be made idempotent, which gives you more confidence that you didn't trigger a physical shipment because of an issue.

In other words, the decision to ship can be made idempotent. And that, in a modern software app, the result of a pretty complex set of things that can individually go wrong.

You could argue that Amazon made themselves something of a success by bringing at least the principle if not the actuality of idempotency to the shipment itself. By making it easy to get a replacement shipped.

> Not everything can be idempotent

Any concrete examples?

I was recently toying with an idea of a language where all side effects would be idempotent. Couldn't find any counterexamples.

Working with the OS-level semaphore seems like a pretty easy counterexample. Its the "least idempotent" thing I can think of in modern programming.

https://man7.org/linux/man-pages/man3/sem_post.3.html

-----

In fact, multithreaded synchronization in general is non-idempotent. You don't want to send a synchronize signal twice, you need to send it exactly as many times as necessary. (Ex: mutex lock / unlock, which is probably the simplest version of a sempahore that you can get. Semaphore of size 1)

Agreed, though I think the GP was looking for a situation you can't refactor into becoming idempotent. Increments and decrements like traditional semaphores are definitely not idempotent, but couldn't you make an idempotent semaphore by changing the signature to be increment/decrement(source, from, to), such that it only applies the operation if the current state is equal to from, and the source is one particular subscriber? Then from the semaphore's perspective it's receiving a time series of messages some of which have the same identity, until all locks have been released and the owner gets notified.

> but couldn't you make an idempotent semaphore by changing the signature to be increment/decrement(source, from, to)

Think about a mutex lock: if the semaphore (aka: mutex) is already locked, your signature is: sem_wait(mutex, 0, 0).

Which is not idempotent. Two "sem_waits(s, 0, 0)" mean that you need two (other) threads to unlock you. One sem_wait(s, 0, 0) means that only one other thread needs to unlock you.

------

Semaphores used in this manner are how you implement reader/writer locks, as well as thread barriers. (If 100 threads are in existence, you wait for 100 semaphore_posts from those 100 other threads).

None of the semaphores or mutexes are idempotent. And never can be.

Hmm, not really, I was more giving the GP the benefit of the doubt that they have some, so maybe they'll chime in. I think they were focusing on the fact that idempotent systems often need to interact with other systems that are not idempotent, but then the act of dealing with those systems, the surface area, can often be made idempotent. To raise the stakes very high, I picture software operating a military robot, where it is quite important that it fire its gun only the specific number of times it is cleared to by external logic. Instead of attempting to guarantee that the robot is sent a 'fire gun' message only the requisite number of times, the system should focus on putting the robot into a 'fire gun' state, the messages of which that sent the robot into that state can be made idempotent...

I certainly support your toying, because I think a lot of impedance to idempotency is the lack of easy patterns to achieving it.

That's why you have unique invoice numbers - it converts the (paper) protocol into an idempotent one

You need more than just a unique invoice number, you need a unique identifier for any actions that can modify an invoice.

The dunning chain is dark and full of terrors. It's more Kafka (append-only log of unique operations) than some Platonic idea of idempotent operations.

Why? What happens if I update the invoice address twice?

If that is all there is to it, then the op is idempotent.

If on the other hand, there is a side effect such as modifying the last_updated column then it isn't and should be coded appropriately.

Working on an ERP software atm, A lot. Often invoices HAVE to be immutable for accounting purposes, so changing the address has to keep an old copy around for the old invoice, while attaching it to any new ones/ones not printed yet. Along with that, some other data may be tied to that address that cant be changed. Like tracking a package in transit that has been sent to the old address.

Idempotency is a godsend for such actions.

Like Rust, you keep the truly non-idempotent actions minimal, isolated, and well understood. Then you wrap everything around those models and state changes with idempotent behavior.

As a trivial instance, you can guard a non-idempotent email send event with a database table and idempotency key. When you attempt to send the same message twice, you'll see that you have already done so.

With diligent engineering, this type of thinking can scale to non-trivial active-active concurrent writes and more.

Reply on news.ycombinator.com