I wish companies would stop inflating their numbers by citing "per day" statistics. 4 billion messages per day is less than 50k/second; that sort of transaction volume is well within the capabilities of pgsql running on midrange hardware.
It can in theory work, but the real world would make this the most unstable platform of all the messaging platforms.
Just one vacuum would bring this system down, even if it wasn't an exclusive lock... Also I would be curious how you would implement similar functionality to the URL deep linking and image posting / hosting.
Mind you the answers to these will probably increase average message size. Which means more write bandwidth.
Some bar napkin math shows this would be around 180GiB per hour, 24/7, 4.3TiB per day.
Unless all messages disappeared within 20 days you would exceed pretty much any reasonable single-server NVME setup. Also have fun with trim and optimizing NVME write performance. Which is also going to diminish as all the drives fail due to write wear...
That's my point. Per-second numbers are far more useful than per-day numbers.
Useful for what exactly? You're basically accusing the company of inflating their numbers (??) as if they are deliberating lying about their scale.
1) avg mean (for throughput estimates), 2) p50 (typical customer / typical load), 3) p90 / p99 / p99.99 (whatever you think your tail is) 4) p100 (max, always useful to see and know, maybe p0).
Or throw in a histogram or kernel density estimate, sometimes there are really interesting patterns.
I’ve never seen a technical blog give such traffic or latency details though.
Edit: reading other comments, please do not read this as diminishing this blog post or Discord, great clear writing and impressive solution to an interesting problem.
It's a blog...it IS marketing.
It's not even the most interesting metric about our systems anyway. If we're really going to look at the tech, the inflation of those metrics to deliver the service is where the work generally is in the system --
* 50k+ QPS (average) for new message inserts * 500k+ QPS when you factor in deletes, updates, etc * 3M+ QPS looking at db reads * 30M+ QPS looking at the gateway websockets (fanout of things happening to online users)
But I hear you, we're conflating some marketing metrics with technical metrics, we'll take that feedback for next time.
> 50k/second
Yes, 50k/second for every minute of the day 365/24/7. Very few companies can quote that.
Not to mention:
- Has complex threaded messages
- Geo-redundancies
- Those message are real-time
- Global user base
- Unknown told of features related to messaging (bot recations, ACL, permissions, privacy, formatting, reactions, etc.)
- No/limited downtime, live updates
Discord is technically impressive, not sure why you felt you had to diminish that.
I don't see anything on their messaging specifically, just assuming they would have something similar.
https://discord.com/blog/how-discord-handles-two-and-half-mi...
Data storage is going to be multi-regional soon, but that's just from a redundancy/"data is safe in case of us-east1 failure" scenario -- we're not yet going to be actively serving live user traffic from outside of us-east1.
1. Cherry pick a piece of info 2. Claim it's not that hard/impressive/large 3. Claim it can be done much simpler with <insert database/language/hardware>
>Our databases were serving around 2 million requests per second (in this screenshot.)
I doubt pgsql will have fun on mid-range hardware with 50k writes/sec, ~2 million reads/sec and 4 billion additional rows per day with few deletions.