Don't add a message queue until you really need one. And if you do make sure you account for it being down, running locally, back pressure if the queue gets full, monitoring, and logging at least.
When I try to estimate ROI on those features it's usually miniscule. A/B test are unconvincing and the aggregate reports synthesized from the data look more like numerology than rigorous science. More-over, senior leadership usually doesn't realize how much additional liability they take on by collecting and storing so much customer data.
It's not 2010. The large tech firms are your direct competitors and regulators across the world have caught up, including in many US states. If you have a primary product for which people will pay money, then the "surveillance agency as a side-business" model of the 2000s-2010s is probably a bad idea. It's a net expense that exposes you to liability and distracts your team.
Additional benefit: without real-time analytics and personalization, you can go even further without a message queue.
That's not the architectural reason for message queues in my experience.
Primarily, a message queue is used when there is a potential for a bottleneck in the overall application throughput where there is high cost of reacting to some event so instead of reacting to the event synchronously, you queue the event and react to it asynchronously.
Some very common use cases:
1. Your UI. Operating systems use a message queue to capture and forward inputs. Your mouse and keyboard events are all queued in a message queue.
2. Webhooks. Many webhook origins have timeouts on responses and your application must respond within a given window or it will be throttled or downgraded. In this case, the best practice is to queue the event to be processed asynchronously (that queue can be a simple database table with your own logic and wrapper around it or something like AWS SQS, Azure Service Bus, or Google Pub/Sub).
3. Mitigating Throughput Bottlenecks. Most applications have a read/write asymmetry so it makes sense to optimize your architecture to scale for reads. But what if your application occasionally has to handle a burst of writes? Should you size your infrastructure for that case? One approach is to proxy the writes through a queue so you can size the infrastructure for a maximum throughput that is managed by the queue. For example, instead of 1000 concurrent writes per second, a queue can capture the write mutations and trickle out only 100 concurrent writes per second. Instead of sizing your application to scale to handle 1000 writes per second, you only need to size your queue to handle that scale.
4. Resiliency. If a message fails, it can be retried according to whatever heuristics make sense for the domain. Sure, you can use a simply loop to retry, but every message queue provides some mechanism for handling retries, failed message delivery, and so on. If you decide to roll your own and log a failed call into a database to try it again later...well, you've effectively captured a message in a custom queue.
UI - Logging mouse movement, keyboard events, and other types of attention proxies.
Webhooks and/or Bottlenecks - calling out to either internal or third party classifiers, or more recently even generative models, for personalization based on user tracking data.
And I don't think this is just my unlucky experience. The fine article includes user tracking as one of only three explicitly enumerated reasons for queuing:
> Why? Because users increasingly expect a real-time experience. In use cases like order flows, webhooks, user tracking, etc. users expect to be able to see the new data in the user interface instantly, instead of having to wait for some background batch processing to periodically reload.
Of the explicitly enumerated motivations in the article:
1. "Order flows" - tracking/modification is often one of the higher latency items in order flows ("you might also like" / "what to order next" features).
2. "Webhooks" - often used for tracking/personalization
3. "User tracking" - ...this one is easy :)
Note: though message queues are great, they make asynchronous operations and interacting with external services much more resilient.
So, you can go very far without a message queue... by using a simpler message queue?
Lets draw the line somewhere the other side of applications that advertise themselves as having that function?
it’s basically a cache with a fancy API so you can incrementally update data structure instead of having to constantly write the whole thing you are trying to cache.
If you mean “it doesn’t have every feature of Postgres” and behaves a little differently then sure but it absolutely is a database.
Watch is a poor man transaction and stored procedure does not work when you need to read value interactively from somewhere else (user/filesystem/another database) while inside a transaction.
I don't disagree with you about WATCH being less powerful than PG transactions. SQLite data types are less powerful than PG types but it's still a database.
All of this to avoid spinning up an SQS queue?
I also question whether you really bought a horse in this analogy.
i’m not sure if this pattern can be called a type of saga.
you basically have a tiny queue in you db using a special table that you drain to an external queue server.
What you might mean is that you might not need a complicated server setup e.g. Kafka for simple message queues?
it allows you to make the « add message to queue » part of a transaction that update other table which is nice!
A real queue have an API similar to JMS, ex rabbitMQ, IBM MQ, Microsoft MQ, …
The difference is you have a mailbox of message from which you remove messages you have consumed by acknowledging reception. Message will be sent to subscribers until they are acknowledged.
So publisher write message m1, m2, m3
subscriber receive m1, m2,m3 and ack only m3.
it will later receive m1 and m2 again but not m3.
Kafka is more like tcp It’s a stream.
Then most of your statuses are Complete and finding an ever decreasing percentage of Incomplete rows will result in essentially a table-scan.
Then that table scan will start to take longer than your cron interval or will biff through your IO or CPU and you've got problems.
Then you might try adding indexes or beefing up your queries and then you'll realize you've just traded 6 for 12/2 because now your index updates on writes are hurting you more than your scans on reads.
Then you'll have the brilliant idea to cull your data regularly or have different tables for Complete vs Incomplete.
And then you'll want to do this culling or row-move in a transaction because you've never bothered making the job idempotent because it was implicitly so in the database all along, and now your transaction locks will compete with your cron scanning or index updates and now you'll maybe put the status table(s) in a different database and work on idempotency.
And then maybe you'll take a step back, have a nap, and wake up to realize you've just implemented a queue but not very well and have tied its design to your specific business-flow and good luck adding retry logic or new status fields.
If you're gonna use a database as a queue, make it a different database server (or schema!) from your business data. Don't tie business transactions (completing the job) to database transactions (since ultimately there will almost always be more than just one data/processing system involved), and maybe really just adding a simple Redis or SQS thing from the beginning isn't going to cost you that much so might as well squeak it in while the costs are still low.
> Then most of your statuses are Complete and finding an ever decreasing percentage of Incomplete rows will result in essentially a table-scan
This is absolutely not true. Performance of your lookups will depend on the total number of "pending" items and never on the table size.
Generally indexes on boolean columns are useless due to low cardinality, but in this specific example they'll work just fine. If you have a 100 "pending" items in a table with billions of rows, search on "pending" status will never trigger a fullscan, only a 100 lookups. You can even make your index partial, so essentially your index will be your mini-queue.
Idempotency is a valid point, but it is as hard with queues as it is with databases. Exactly-once delivery is generally not possible in distributed systems (in the real world at least, where you don't own every node). Queues won't magically solve it for you, they need to implement the same locking mechanisms as the database.
Partially this is why AWS SQS boasts to be almost infinitely scaleable, but limits you at 300 (!) rps for FIFO queues.
If this were just a discussion about backend architectures or message queue architectures or Chiselstrike it might feel more cohesive.
I think it's challenging to describe simply, because you've built a system that masks over these different architectures.
Super minor things: 1) personally I find memes in technical articles distracting but I know not everyone agrees. And 2) the differing graphics backgrounds and borders on the gray background, coupled with Medium's (two) sidebars is also a little distracting.
Related possibly helpful, I get the feeling these days we write for 2 distinct set of eyes on the internet: those looking at a large screen and those who digest articles on a 3x4 inch screen. For example, one can see your diagrams and read the text discussing them on a pc at the same time.
Examples:
"But often, especially with nascent projects, we can’t afford."
"From full backend-as-a-service solutions like Firebase, to tools like ORMs to abstract and simplify dealing with databases."
"Simple, with an architecture that already internalizes all of the common issues seen when handling events."
You also sometime use sentences that are too verbose or hard to parse.
Example:
A) "In this post we’ll show how the lack of good abstractions leads to build-your-own backends lacking a key component that is present in most modern backends: a message queue."
B) "In this post we’ll explain why message queues are important to a good back-end, and why developers may overlook them. (part that doesn't need to be said: because of the focus on database-centric abstractions.)"
A) "If you ask most people, a backend is comprised of a database where the data can be stored, a business logic layer to mediate access to that data, and a couple of authentication components. But if you ask modern backend engineers, the answer differs. Yes, you have all that, but there’s a major component missing in this picture: a message queue.
From modern alternatives like Redpanda, to established systems like Kafka, message queues are an integral part of most professional backends in use today."
B) "Most modern back-end engineers will attest that a message queue, such as Kafka or RedPanda, is crucial to a good back-end."
Generally I'd get rid of the conversational, quirky tone, I think it'll improve things stylistically.
For the content itself I'd write a clearer plan with titles and subtitles, and the idea you want to convey in each paragraph.
Also a good tip is to always write the introduction at the end, it makes it clearer in your mind what you're actually trying to convey.
It's like you showed up just to dump on the author. You added no value here.
You fail to articulate why it was hard for you to read. You failed to articulate what might have been better. You failed to have a basic level of compassion and human decency in a way where you could even make a base-level attempt at wording your post as anything other than a snarky attack.
I'm personally sick and exhausted of commenters who behave like this. The people who contribute nothing but put in the extra effort to show up in comment sections to dump on people who are trying their best to share or build something.
Factually, you are the party introducing personal insults into this conversation.
I didn't know I was trying to sell a message queue, thanks for letting me know!
I don't get it but there's a good chunk of people who are ready to jump on anything that looks even slightly comercial by even one single aspect. I've been told my personal blog was AI generated blog spam once in the past when I had literally nothing to sell. I also don't understand the fundamental issue with having a commercial involvement as long as you aren't trying to BS people or shove advertising down their throats.
It's an abstraction over a message queue, which is only one feature among many.
But btw, as a pointy-haired dude I am becoming (although I am bald), if someone really wants to buy a message queue from me, I'll happily sell it!
I don't think there is anything wrong with your article, you are quite up front about what you are selling, so personally it is no big deal.
My personal opinion is that if you don't want to come off as selling something because your genuinely interested in tech, just publish under a different domain or pseudoname