Hacker News Reader

Comment by dharmab | Hacker News Reader

Postgres as a queue is one of the worst decisions I've seen made and I and others I've worked with have spent years of our lives unwinding the consequences.

View on HN

It’s almost as if we were in an ordered line, all waiting to learn the same thing, and had the lesson delivered no more than once.

saying so without context or requirements at all reduces the effectiveness of the insight

Sadly my work is subject to regulatory requirements that prevent me from discussing specific details :(

Surely if your complaint is general enough to give such advice to everyone here, there must be a way to express generalized technical details of that complaint without disclosing anything more privileged than you already have. If that kind of abstraction and generalization isn't possible, then it sounds to me like something specific to your work's implementation of the ideas and not something that applies broadly.

Oh, okay. It’s great that you’ve let us all know that you hold The Secret. Thank you.

Completely disagree for our use case. If your messages aren't send-and-forget but rather represent work that needs to be tracked, it is incredibly difficult to manage the state when your database and queue are separate. Using postgresql as the queue and leveraging the same transactions as your business logic solves many many issues.

How is it incredibly difficult? I’ve seen this work very well.

Is your code handling different varieties of event states stored in your database?

Why implement this as a monolithic data source? Was it not possible to separate different states by additional queues?

Curious to hear the decision making process here. Maybe I can see it for low volume?

Admittedly we're not using it for anything high volume.

dralley is saying your queue push and database write aren't transactional. You have to be ok suffering some small % of message loss or messages sent without a database commit

Depends on your queue semantics, ie is it just an at least once delivery or is there an ack back to the queue service.

If there is a two phase process then you just need idempotence and can safely transact with the db.

It doesn't matter what your queue semantics are. If it's not part of the transaction, delivery and push cannot be guaranteed to be atomic.

At the end of the day, the ONLY way to reliably do this is to hook into the databases native journal/log anyway. Postgres gives you better primitives to work with than installing Qlik for example.

Why do you think so?

As a person who has implemented worker farms at scale, I don't understand which part is "incredibly difficult", or what having a dedicated queue server prevents you from updating database state within "the same transactions". If your worker process has to update some kind of task state and calculate some business logic, it can still do so whether if you use rmq, redis or whatever.

I assume they mean the dual write problem

So atomic updates are incredibly difficult?

Not atomic updates per se, but transactional updates. Ie doing two atomic updates as one consistent atomic action.

Care to elaborate?

years seems like hyperbole

Reply on news.ycombinator.com