3,448 karma · joined November 4, 2013
formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/
[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]
My full time job is writing Rust for large-scale production systems. The language is great - and as an organization we're doubling down on our investment in Rust this year.
We have a bunch of operational experience with C*/ScyllaDB.
We hit the performance ceiling of redis, and rather than deciding to go with redis-cluster, and dealing with the headache of managing persistent redis, we went this way, so we could consolidate data storage to scylladb/cassandra (it was cassandra at the time of rewrite, scylla now since we've migrated our data stores over).
The service itself is pretty simple, so it wasn't a gargantuan task. So it was either, rewrite this again in Go (using off-heap data structures to avoid GC pauses, which makes the software both harder to maintain, and also less idiomatic) or try a new language where we won't have these issues - where we don't have to resort to clever hacks to work around the runtime system.
Since then we've continued our investment in Rust, after success of this project (and many others we've yet to blog about as well.)
BTW, here's some places I've spoken about Discord's Elixir usage:
- a podcast: https://smartlogic.io/podcast/elixir-wizards/s5e8-nowack-hei...
- interview with José Valim (elixir's creator): https://elixir-lang.org/blog/2020/10/08/real-time-communicat...
But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)
You can solve the first hurdle with better hardware (for the same price as our usual specced macbook pro, I opted to custom build a beefy workstation with a 3790X).
The second, you just solve with time. I remember when I first started with rust I hated it. I was working on a native project, and felt like it would have been easier to do in C++. But internally now we have more mindshare and knowledgeable folk who understand Rust - this has helped tremendously in onboarding engineers to the language. Also, having a working editor (via rust-analyzer) and the quality of compiler error messages have improved significantly in the past few years.
In terms of immature ecosystem, I think the rate at which the ecosystem is advancing is incredibly promising. Tokio hit 1.0 in Dec 22 of last year, which will finally put an end to the dependency upgrade thrash we've had to deal with. Going from tokio 0.1 -> 0.2 -> 0.3 -> 1.0 has been a journey. But with 1.0, finally, libraries will have a stable API to target, so we won't have to worry about this much for a while. Additionally, in the following months we intend to start funding critical open source rust projects that we use to ensure their success.
https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users?
Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!
Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go.
Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands.
And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.
I'm not sure this is entirely noteworthy unless you somehow think CPython has a JIT. Would be much more interesting to compare to pypy.
EDIT: Am wrong, see below! Maybe we can update the original post to one of the links below.
Well for the real time side, I can't tell you how big a boon it's been to build our platform on top of Elixir/BEAM. Hands down the best runtime / VM for the job - and a big big secret to our success. Where we couldn't get BEAM fast enough - we lean on rust and embed it into the VM via NIFs.
2021 is the year of rust - with the async ecosystem continuing to mature (tokio 1.0 release) we will be investing heavily in moving a lot of our workloads from Python to Rust - and using Rust in more places, for example, as backend data services that sit in front of our DBs. We have already piloted this last year for our messages data store and have implemented such things as concurrency throttles and query coalescing to keep the upstream data layer stable. It has helped tremendously but we still have a lot of work to do!
To help scale those super large servers, in 2020 we invested heavily in making sure our distributed system can handle the load.
Did you know that all those mega servers you listed run within our distribution on the same hardware and clusters as every other discord server - with no special tenancy within our distribution. The largest servers are scheduled amongst the smallest servers and don't get any special treatment. As a server grows - it of course is able to consume a larger share of resources within our distribution - and automatically transitions to a mode built for large servers (we call this "relays" internally.) At any hour, over a hundred million BEAM processes are concurrently scheduled within our distributed system. Each with specific jobs within their respective clusters. A process may run your presence, websocket connection, session on discord, voice chat server, go live stream, your 1:1/group DM call, etc. We schedule/reschedule/terminate processes at a rate of a few hundred thousand per minute. We are able to scale by adding more nodes to each cluster - and processes are live migrated to the new nodes. This is an operation we perform regularly - and actually is how we deploy updates to our real time system.
I was responsible for building and architecting much of these systems. It's been super cool to work on - and - it's cool to see people acknowledge the scale we now run at! Thank you!! It's been a wild ride haha.
As for scale, our last public number perhaps comparable to Slack is ~650 billion messages sent in 2020, and a few trillion minutes of voice/video chat activity. However given the crazy growth that has happened last year due to COVID - the daily message send volumes are well over the 2 billion/day average.
- it only looks at system time at system startup, then relies on instant for monotonic clock source. unfortunately for setups that do leap-second smearing (ours does), this would deviate slightly out of sync after operating for a while.
- does not handle seq overflow within a given millisecond window - this i think is very unsafe, and can cause the generation of duplicate IDs if `next_id` is called at too rapid an interval.
We originally wrote this service in Elixir but have since rewritten it in rust. Actually very similar to this crate. Maybe I should open source ours too haha.
See the section "Specifying Throughput with Provisioned Mode" here: https://docs.aws.amazon.com/efs/latest/ug/performance.html
tldr: the html is stuffed in the exif data!
If you are a real human, and not a spam bot, our general policy is to unban and tell you "hey don't do that again." Which is the outcome that should have (and eventually did) happened here.
As for the initial unban, a bug had caused the "successful appeal" macro to apply - but it did not actually unban your account. Hence the confusion.
We are definitely working on improving this and the experience - and I'd like to apologize for the quite honestly shitty experience you had. 2020 has been a wild year for us, and we are still playing catch up operationally on some things - with the rapid surge of usage on the platform.
Unfortunately the people handling these appeals don't have all the information necessary to explain exactly why you were banned - and as a general policy we try to keep those exact details under wraps to thwart attempts from people to try and reverse engineer / circumvent our anti spam. (Of which there are many, so... so many people trying to do this.)
We are working hard on improving our process around this - but it's a rather gargantuan effort. From both the improvement of automation and tooling, and also staffing in general. But we are actively working on it - and a lot of my day job is to make sure we are handling these kinds of cases in a timely manner and also making sure nothing slips through the cracks due to a bug like your tickets have.