Delayed Job vs. Sidekiq | Hacker News Reader

Delayed Job vs. Sidekiq

blog.appsignal.com

112 pointsunripe_syntax65 comments

I agree with most of the findings in that article, but I miss one very important one: consistency.

Jobs in your queue in the database can be committed in the same transaction that makes the mutation. With redis you have no such thing. Leading to (been there, it hurt) potentially missing jobs or queuing jobs that have no associated mutation in the db.

Furthermore, a big downside of redis is that it's promise of consistency is low. Mostly by design: redis is not meant to be the primary, one and only store of truth. Commonly, people store stuff in redis that can be re-built from database or some eventlog or so. But not sidekiq. If your redis starts failing (been there, it hurts), e.g. though OOM errors, you are loosing data. Jobs, with their parameters, are dropped, often irrecoverably. And with Rails' common set-up, this means you are irrecoverably loosing data: there simply is no way to find what emails you did not send, nor any way to re-queue them. There is no way to re-enqueue those "generateReport" jobs using the data at that point in time and so on.

Basically: Redis is quite certainly not the best tool to store your job-queues with associated data, in. Sidekiq is the less sturdy approach of both.

With listen and notify in Postgres, it's a fantastic choice for a job queue if you're not crazy scale (which is most apps, honestly).

Especially with transactions as mentioned, with elixir and ecto and Oban, you can ensure your jobs are inserted properly. And it's easily testable.

I have strong opinions on this, but IMO if you are thinking about using PG pub/sub, you'd almost always be better served by just running a worker that reads from a logical replication slot. Little bit heavier lift, but there's plenty of reference code out there, and you get consistency + delivery guarantees.

You can run Redis in a persistent setup. So each Redis instance has a replica and both master and replica write out to disk. You still don’t get transactions, but depending on your scale a transaction for every job you enqueue may be impossible.

If you’re small enough that you don’t knock over your DB by putting jobs in it then the transactions are nice!

As you say, looks like redis-raft is capable of solving durability and consistency in a replicated environment as of 2020, which is welcome news: https://jepsen.io/analyses/redis-raft-1b3fbf6

At Betterment we use our OSS mostly-compatible fork of Delayed::Job referenced elsewhere in the comments to enqueue and work millions of jobs a day and sleep much better at night with the at-least-once delivery semantics if-and-only-if the related transaction commits which you can’t get without some form of integration with your primary database.

Our solution was to build middleware/interfaces that only push jobs after a transaction has committed. There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.

Agreed on the Redis health being critical and if it's unhealthy you can't trust anything. We over-provision compute and memory and enable persistence as a result. We are also mindful of our payload sizes, which we also compress. Additionally any jobs scheduled beyond a certain time threshold are committed to the DB, not Redis, and we have a periodic job that adds them back to Redis as we approach their scheduled time to run. This is good for both durability and keeping the surface area of Redis work smaller.

It's not perfect and there is probably a day where we switch it out but for now we have developed a deep understanding of Sidekiq and Redis and get to leverage the larger Sidekiq ecosystem. This has worked well for us to the tune of about 35 million jobs a day.

> There is still room for some inconsistency -- a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.

If it happens though is it a big deal? I’ve seen Sidekiq jobs used to move money, for instance. This can result is Bad Times. Right tool for the job, if consistency is a requirement Sidekiq I think gets incorrectly applied in a lot of applications.

I don't move money, but if I did I would do it like we do for most objects in our system (like interacting with S3 objects for example), I'd represent that transaction (in the money sense) in Postgres with a state and the job would be expected to do work and update the state. The transacting job itself could be queued immediately for convenience but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency.

In this sense the job acts on data, it doesn't hold data, and you can build in backstops in the event of any kind of failure, not just Redis. You'd want to build this into your architecture regardless of what your background job tooling and infrastructure you use.

We worked on the same problem, and what you described works, however it also happen to be exactly what delayed job does.

Which bring me to the next point: Sidekiq should never be the default choice. It brings distributed systems complexity to an audience that's trained to work on a monolithic database.

Delayed job or similar should be the default, with sidekiq relegated tp jobs ready to be lost

I agree with you that people often overlook the complexity of tying changes to their database, with enqueuing jobs in a completely separate system.

https://www.betterment.com/engineering/delayed-resilient-bac... does a great job explaining this.

> but ultimately there should be a periodic job that attempts to transition states of any outstanding transactions and the job that does that work should be able ensure uniqueness (in the runtime sense) and idempotency.

You've re-implemented a job queue in the database.

No I didn't.

You kinda did though. Your models in states they shouldn’t be are jobs. These kind of headaches are avoided using a job queue that provides strong consistency.

Everything is a trade off.

Transactional consistency is nice. Maybe worth it for some low-volume jobs.

The downside that comes with it is using your primary DB as your queue is that the queues in an app tend to absorb all the exceptional conditions (queue grows to millions during worker outage, backfill which needs to process millions of items). Using your primary DB means all that disk usage, memory usage, and IO hit your primary DB in exceptional ways and impacts your app in more ways than just queue delays. It also scales much less linearly than other queue options.

A nice middle ground is to store all your data and state in your DB transactionally, and only queue tiny lightweight jobs with database primary keys (database: sentWelcomeMailDate=nil, queue: sentWelcomeMailForUser:1234).

This is another case for neither DJ or Sidekiq. A fully hosted option like SQS addresses the durability concerns of Sidekiq, and the scaling concerns of both. I’ve queued 1B+ jobs in SQS and watched the workers slowly burn through them over the course of a month.

DJ and Sidekiq can go quite far. I’ve used both and patched some perf issues on DJ. The lesson learned was queues at scale are hard distributed systems problems - you probably don’t want to have to know how they work, or host the distributed system yourself.

> Maybe worth it for some low-volume jobs. > ... > The lesson learned was queues at scale are hard distributed systems problems

I'd go so far to say that when you need such scale, Rails or Job Queues like Rails' are not the solution. Event-based architectures (event-sourced, event-buses, actors etc) are far better suited.

Rather than trying to solve all the problems that those architectures were explicitly designed for in redis/sidekiq/rails, just move to those architectures.

You won't have the nice "rails generate CreditCard" anymore /s, but you'll gain a lot of consistency, simplicity, performance and architectural sanity.

Totally. Or use rails for your web app, and something else for your queue processing. Rails is nice for rails things. Rails is scales for serving web apps much better than DJ/Sidekiq do at serving queues.

I've been using Sidekiq for years, rock solid and no complaints.

However, this recent addition to the space seems to be gaining traction and appears to have an excellent feature set -

https://github.com/bensheldon/good_job

Yeah I'm a big fan of Good Job too.

As long as you stick to pure ActiveJob features it really shouldn't be too difficult to scale from in memory based queue to PostgresSQL backed Good Job and then to Redis backed Sidekiq if and only if you need to.

Yes, I wanted a PG-based one and reached for Good Job on a new project since it was recently created and with Active Job in mind from the start. Check out the intro:

https://island94.org/2020/07/introducing-goodjob-1-0

The problem with postgres-backed queues is they can't be used with connection pooling, so your workers are unnecessarily using up DB connections.

I thought that was an ActiveRecord problem. We use MYSQL and have a lot of long-lived connections.

Not sure how it works with MySQL, but with postgres the most common pooling options use transactional mode, which doesn't support LISTEN/NOTIFY.

Without the pro version it's not reliable:

> If a Sidekiq process crashes while processing a job, that job is lost. [0]

We use Sidekiq Pro at work but I find it unfortunate that reliability is a pro feature. It's really hard to tell a product owner: Yeah, we might lose a few jobs once in a while.

[0] https://sidekiq.org/products/pro.html

Seems like a good reason to pay Mike.

agree. its not a lot of money, or it didn't used to be last time I looked. If your org relies heavily on it just pay

I just looked this up. It's $995/year, or approximately 4% of the fully loaded cost of an FTE software engineer at most companies. i.e. about 3 weeks of a developer's time.

I understand paying extra for niche features. Isn't reliability a core feature of background job software?

We do pay for it.

Perhaps you could address my point? Open source background job software that isn't reliable by default? That makes sense?

rocket job is pretty solid too, has a lot of the pro sidekiq features for free https://rocketjob.io/

One aspect of this article that's touched on but not really explored in the article is the greater Sidekiq ecosystem. Beyond just the core Sidekiq application working well there are a host of well maintained open source add-ons to Sidekiq that are tremendously helpful.

Statistics https://github.com/davydovanton/sidekiq-statistic

Failure Handling https://github.com/mhfs/sidekiq-failures

Unique Jobs https://github.com/mhenrixon/sidekiq-unique-jobs

All of which also extend the web UI for Sidekiq which is incredibly useful for both debugging and having a handle on what's with your queues.

Finally, if you're going to be using Sidekiq in any serious way I'd recommend Nate Berkopec's "Sidekiq in Practice" - https://nateberk.gumroad.com/l/sidekiqinpractice

Beyond being an incredibly useful resource on its own - you get access to a very active private Slack that is filled with other very helpful developers who are using Sidekiq.

Delayed Job seems to have been abandoned, a better alternative that is a fork with many improvements is https://github.com/betterment/delayed

Yes sorry, please use betterment's fork. I got busy with other projects since writing delayed_job and had to prioritize.

like running one of the top 100 companies in the world :) thanks for everything

Consider adding this to the top of the README of delayed_job?

I helped maintain delayed_job_web as a UI. It has fallen behind substantially now though. If folks are looking for a UI - it would be great to update it for modern versions of Rails. https://github.com/ejschmitt/delayed_job_web

very cool! thank you

Another fork is [0], which adds debouncing, serialization, recurring jobs, and other useful things.

(Disclaimer: I used to work at Instructure.)

[0] https://github.com/instructure/inst-jobs

One big difference if you're changing forks is to take note of "at least once delivery": https://github.com/betterment/delayed#at-least-once-delivery

If you're not aware of that, simply switching repos may cause some surprises.

Just to note that at least once delivery is the best case scenario if you configure Delayed::Job correctly, we just made it impossible to configure otherwise in our fork. The only alternative is at-most-once delivery, which puts you in Sidekiq territory with the potential for silent job loss. The semantics of exactly once delivery can only be achieved by domain-aware code, and the way to do that with any form of background job system is building jobs that are internally idempotent.

Seems maintained to me:

https://rubygems.org/gems/delayed_job

https://github.com/collectiveidea/delayed_job

    > 4.1.10 - January 18, 2022 (40 KB)
    > 4.1.9 - December 09, 2020 (40 KB)

Ok last I checked it was before January and I wanted to upgrade to rails 7 and couldn't so ended up moving to Sidekiq

Rails 7 support only just got added https://github.com/collectiveidea/delayed_job/pull/1161

Sets a bad precedent, I'm on rails's main branch and this was the big dependency holding me back for a while. Sidekiq works with rails's main though

https://github.com/que-rb/que

This one seems to be the most performant. By a lot too, from my understanding (haven't ran any benchmark myself, but the readme shows some good postgres knowledge)

This article matches up with most of my own findings as well. I've been using Sidekiq for almost 10 years now. I currently use it with a project that just hit over 1 billion jobs.

To me, Sidekiq is the perfect example of keeping something simple. A basic Redis + Sentinel deployment with persistence enabled and multiple read replicas has allowed me to achieve this.

Scaling Sidekiq workers is also incredibly easy. I have dedicated job queues for various categories, and I simply create a new Kubernetes deployment for each category. Each deployment is set to only process a specific queue. This allows me to throttle how quickly each queue gets processed, simply based on replica/pod count.

I've been using sidekiq (paid, free) for years, the next app (mostly rails apps, but not exclusively) I implement background processing probably will not include it.

Reasons to avoid Sidekiq not mentioned:

- Sidekiq does not implement all of or discourages use of activejob abstractions (retry_on, globalid for example)

- Paid support can be curt, hostile, or down-right "works on my machine, sorry!"

- Sidekiq's better performance has never materially mattered for my applications; they're almost always waiting on IO from the db or some net connection.

I'm sorry if any of our previous interactions were less than professional. I often answer support queries from my phone when out and about, so tossing off a curt, one sentence reply will happen.

I generally get multiple app issues per day so I can't afford to reproduce every one: just yesterday I asked a customer to provide a reproduction, they worked on it and found the issue was in another gem their app used. At my scale with ~2000 customers, this happens almost every day.

One thing I would love to see is a Fiber-based job runner for mailers and other non-cpu-intensive jobs that spend most of their execution time waiting (say, doing API calls). It seems like that would be one of the absolute sweet spots for the new Fiber scheduler stuff in Ruby 3.0. Preferably it'd be sidekiq compatible but you could have the fiber runner listening to one queue and the normal sidekiq runner(s) to another queue.

I've been toying around with this idea for a while and the main blocker is the incompatibility of Rails (mostly AR) with some of the Fiber stuff. That is slowly getting cleared out though.

I'd like to see it as well. However It's not immediately evident to me by how much this will improve things; afaik you will hit a bottleneck that is in the number of available db connection (since each fiber, just like threads, has to borrow a db connection from the pool).

Fibers have a lot of potential but not in the one request per thread/fiber model we have now. Or perhaps I'm thinking about this wrong?

I have in one of my applications a background job that basically does the following:

- Some quick database lookups (~ 10 ms total)

- An API call (depends on the time of day, but between 500-800 ms)

- Another quick database insert (~ 20 ms usually)

The job only needs a db connection for about 5% of the total runtime of the job and can give the connection back to the pool while the API call is running. Many mailer jobs in the system have a similar db-to-api-call ratio.

(Also, I am of the opinion that all but the very biggest sites should not have any problems with db connections. Get a database that handles tons of connections gracefully like MySQL or use something like pgbouncer when using Postgres. Having low connection limits should not be a problem in 2022)

I've recently discovered jemalloc, specifically when used with Heroku.

"Using jemalloc instead of regular malloc helps too. The exact way to do this depends on the platform you use, but it is pretty simple on Heroku. Just set heroku-buildpack-jemalloc as the first buildpack (ahead of the heroku/ruby buildpack)."

FYI, remember to set JEMALLOC_ENABLED=true in your env to actually turn it on.

https://github.com/gaffneyc/heroku-buildpack-jemalloc

Surprised Resque isn't thrown in the mix here. It's performed very well over the years, even as the workloads and scale of the application has grown (from 100s transactions a day, to 10k/minute). The main advantage over sidekiq is it uses a forking model, so you don't need to worry about memory leaks or thread-safety as closely.

Been pretty happy with delayed jobs for a long time. I’ve used it a number of times at jobs for small to medium businesses and it’s alway worked well. And it’s nice you don’t have to setup any additional infra. Just use your database.

No disrespect to sidekiq. I see the use case for it and it seems like it is better supported.

If all you need is lightweight jobs like transactional email, I highly recommend suckerpunch.

I can only wish there were more options like that in Python space. Celery is just a horrible tire fire and everything else seems to be either pretty minimalist, somewhat immature, or both.

Interesting content but AppSignal lacks lot of features. I've tried to use it once for a Rails app and I haven't foud not usefult insights. Would be interesting: - Monitoring performance and suggest how to fine tune the app (# Threads, #Words, RAM, CPU etc..)

- Identify memory leaks

- Identity memory leaks on Postgres

Why spending so much time writing content when you could have focused on your product instead?

Do anyone (NewRelic or DataDog) actually do that? I have found New Relic focuses on large scale observability (distributed tracing) not actually understanding low level implementation specifics and building tuning profiles. NR doesn't even do request level object allocation tracking like AppSignal.

DataDog offer memory profiling for Java, I think, but not for Ruby.

Memory profiling is hard, especially for Ruby.

To echo prescriptivist's comment, neither DataDog nor New Relic's profilers offer memory profiling for Ruby applications.

We chose `delayed_job` at Avo for it's simplicity and ease of use.

https://avo.cool

I am constantly reminded how lame ruby and the rails ecosystem is. Consistency was solved long ago in the .Net ecosystem and using standard tools would not lead the author down a dark road of ‘it just works’ but it really doesn’t. All this ‘you could use xxxx but I haven’t setup my instance that way’ is just fan boys trying to pull the wool over your eyes to hide the fact that the tools are really crap. Consistency is paramount in the enterprise and anything less is just a lie that the system works. It doesn’t.

OK, ha

Negative tone aside, indeed: the problems that Rails "solves" here have long been solved in computing. Not just .net. In Ruby too, just not in Rails.

Event-based architectures such as event-sourcing, -listeners, -subscribers, message-buses, actor pattern. All of them solve the problems that Rails tries to solve with delayed jobs. But does it with different tradeoffs.

Rails is first and foremost an MVC setup with a heavy and tight coupling to the database. That is a poor fit for apps or domains with lots of sequential logic, messaging, services or orchestration. Rails is perfect for reasonable simple "CRUD", its less perfect for "CRUD+business-logic" and a very poor fit for complex domain models with (eventual) consistency and/or message passing.

For the middle-scenario, Id -personally- forego Rails, and choose or build an archictecture that focuses more on logic than on "getting stuff in and out of a database". But I see why people will choose Rails there too, and then bolt the logic and consistency on top of it.

As a counterpoint, I used to work on an automated testing tool for websites and mobile apps, which was built entirely using Rails (with Resque for background jobs).

We had customers setting up tests which could cover thousands of pages and run for hours (or even days).

Sure, Rails probably wasn't the ideal tool for the job, but it worked fine.

Most of these problems are solved in Rails, too. If you want consistency, there are options for it. This is an article about two non-enterprise (aka, non-paid) options.

Reply on news.ycombinator.com