I don't understand how you can make queueing high-performance. Queueing things is literally the opposite of making things fast.
You likely need to read about message queues, why they are used, and why they have performance constraints. Message queues are a system, and the system to maintain those queues and content has an overhead. The project is building a solution that is higher performing than comparable systems.
Queuing things often makes things faster (!) as you are usually dealing with limited resources (ex: web and job workers) and need to operate on other systems (ex: payments) that have their own distinct capacity.
If you have 1000 web workers, and your upstream payment provider supports 10 payments per minute, you'll quickly have timeouts and other issues with 1000 web workers trying to process payments. Your site won't process anything else while waiting for the payment provider if all web workers are doing this task.
A queue will let you process 1000 web requests, and trickle 10 requests to your payment provider. At some point that queue will fill, but that's a separate issue.
Meanwhile, your 1000 web workers are free to process the next requests (ex: your homepage news feed).
Would you recommend something to read about that?
You'd usually want your queue system to fail the enqueue if it is full, and you'd want monitoring to ensure the queue isn't growing at an unsustainable rate.
It also forces you to think a bit about your message payload (rich data or foreign keys the worker loads).
RabbitMQ, Redis-based queues (ex: ActiveJob or sidekiq), Gearman, and others will all offer different mechanisms to tackle full queues.
- you need to ensure that your producers don't produce at a faster rate than your consumers can keep up
- if the consumer is unable to handle the request now, queueing it and handling it later is often not the right thing to do. Most likely by the time the resource is available the nature of the request will change.
TCP itself is already a queue. All these message queue systems also make the silly decision of layering their own queueing on top of TCP, leading to all sorts of problems.
Basically those things only sort of work if you have few messages and are happy with millisecond latency.
> Basically those things only sort of work if you have few messages and are happy with millisecond latency.
Not really... queues are great to defer processing or have longer running job than your HTTP and TCP timeouts will allow. Building large data export won't happen within a single HTTP request.
ex: https://shopify.engineering/high-availability-background-job...
1. They're referring to the queueing itself. They're not saying "this will make some other thing fast" they're saying "this is fast at queueing".
2. Queuing is not the opposite of making things fast.
But what I think the other post is alluding to is the fact that in-order delivery is often not a requirement, and can be an anti-feature. I know that in every use of Kafka I have personally encountered the relative order of messages was quite irrelevant but the in-order implementation of Kafka converts the inability to process a single record into an emergency.
That is decent but still below the speed of the network, which should be the main blocker.
TCP is just not a good fit for anything that needs high-performance or bounded sending times. What it's good for is giving you a simple high-level abstraction for streaming bytes when you'd rather not design your networked application around networking.
Building the right logic for your application is not difficult (most often, just let the requester resend, which you needs to do anyway in the age of fault-tolerant REST services, or just wait for more recent data to be pushed) and easier than tweaking TCP to have the right performance characteristics.
UDP also has very small message (datagram) size limits, so you would also need to implement some way to fragment and re-combine messages in your application.
At this point you've built an ad-hoc re-implementation of 80% of TCP.