Hacker News Reader

Comment by ramraj07 | Hacker News Reader

Always hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?

View on HN

Well, they are just a mechanism to push jobs between different components where you treat Redis as a message queue. In Redis you can implement one using Lists, Sorted Sets, Stream or Pub/Sub; though they are commonly implemented through the first two. Though if you end up using first two, to guarantee at least once semantics you need to take care of acknowledgement in a not so pleasant way.

You can replace Redis with pretty much any message queue though like RabbitMQ which has a better consumption story. The main advantage of using any one of these would be the throughout you can achieve and it decouples your database at that point which a lot of people prefer.

Tables as queues is perfectly legit, one less component to worry about.

One more advantage it gives is you can have pretty easy job guarantees through the database transactions.

Does anyone have any rough performance numbers on each approach? I'll only use DB tables when I don't want to deploy Redis (which I usually do, as a cache or whatever), and I'll use RabbitMQ for more "serious" projects. On small projects, I'll even forgo Redis and use the DB as a cache too, which works pretty well.

What do y'all do?

My job queue requirements have been very modest (less than ten jobs per hour) and no more than 2 or 3 workers. I also generally have redis deployed but it never occurred to me to use it given it's "ephemeral" nature - always remained paranoid that services on the cloud should always be assumed to go down at any time

I generally agree, but I have Redis as part of my main stack now, and it's been pretty solid. I'm fairly distrustful of "cloud" services, though, so I run my stuff on Hetzner and manage them myself. For my revenues, where a day's downtime might cost me $5/mo in lost revenue, I think it makes more sense.

Reply on news.ycombinator.com