Ended up rewriting it in house. We ended up with a SQRL variant that is a restricted type-checked subset of python. It is not python however, just syntactically similar.
3,448 karma · joined November 4, 2013
formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/
[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]
Ended up rewriting it in house. We ended up with a SQRL variant that is a restricted type-checked subset of python. It is not python however, just syntactically similar.
I work at discord and specifically have been tracking this issue. We recently released support for AV1 hardware encoding for go live streams. However, the process of probing the GPU for what codecs it supports has triggered a latent nvidia driver bug. We are releasing a workaround some time today, and nvidia is releasing a driver update to address this issue as well.
Apologies for any inconvenience this has caused.
Except in France - CNIL does not agree with NIST. :(
You mean rustls? https://github.com/rustls/rustls
We are not even near this scale yet at my place of work, and we are moving towards this methodology of strong automation to orchestrate a cluster. We hire software engineers to operate our database clusters, and the expectation is to somewhat be selfishly motivated to write programs to remediate issues so you don't get paged constantly. We do not expect to grow our headcount proportionally to the number of clusters or nodes we operate.
You must treat your nodes like cattle not pets. If a node fails, automation kicks in and re-bootstraps it. It is not worth figuring out how to nurse it back to health. When you are performing rolling or scale up operations on the cluster you are just invoking automation to do everything for you.
We have solved these issues in a few ways, mainly:
- working with the relevant product teams to implement appropriate rate limiting or improving data modeling.
- introducing our own query layer, written in Rust that sits in front of Cassandra that uses a form of micro-caching called read coalescing, and also other forms of query throttling/load shedding to reduce work the database must do for hot keys/pathological patterns of access. We expose a GRPC interface from this - and this lets us centralize control of the client driver and tune it appropriately, while also getting to leverage the ever growing open source grpc traffic routing solutions (envoy, etc...)
and ultimately,
- switching to ScyllaDB, a C++ rewrite of Cassandra which is of course void of any garbage collection issues, and features faster overall performance and lower latencies.
Scylla, however, is not without its own set of issues - and somewhat strict hardware requirements[0] thanks to the seastar engine it is built on top of. Their team however has been delightful to work with, and our platform is markedly more stable in current year than it was in years past thanks to the above factors.
Operationally, however, Scylla and Cassandra are quite easy to run, the trickiest part is repairs. Common operations such as cluster expansion, or replacement of node are so common an operation that they are at this point mundane. Be wary however about read/write amplification issues inherent to LSMT databases, choosing the correct compaction strategy and tuning it appropriately can be quite key. Additionally tombstones can be quite bad for performance.
In current day we offer a new more generic solution that sits on top of scylla (it would work with Cassandra too) that provides a simple interface to query KKV based data, without having to worry too much about problems like large partitions, hot keys, or tombstones! With a design like this, the underlying cluster thus far has been issue free and very easy to operate.
[0]: https://discord.com/blog/how-discord-supercharges-network-di...
To say that the success of discord was due to "pouring VC money into marketing" completely misses the whole "we also spent a lot of time and effort to build a product that was better than what was out there and solved a problem for our users."
I don't know when those plans might materialize, but it is very much top of mind for us that there is ever growing knowledge on Discord that isn't accessible via search indexes.
And no, we do not use an ORM like diesel, since we primarily use ScyllaDB. We wrote our own - it's very nice. We have a fully type-safe scylla/cassandra client, that does static validation of the queries (which we define as annotated structs), and also, when the service starts up, it validates that the schema that the service has agrees with the schema the database has. Insofar a making sure that if you say `WHERE id = ?`, and you pass a i32 but it wanted a String, the service will not start.
It also does request coalescing and tracing/telemetry, and will soon do rate limiting, circuit breaking and speculative retries as well.
Once you understand how the borrow checker works and how to write/structure your code in a way that the borrow checker is happy with it sort of becomes second nature.
I think by side effect coding in this style does lend itself to simply writing better to understand code. Nowadays I just write my code this way without much thought. It happens effortlessly. I find myself writing programs in other languages like I would in Rust nowadays.
This is compared to when I started and ran into borrow checking issues many times a day. Additionally the borrow checker has improved since then and NLL borrow checker will accept more programs as valid than the previous one.
1) We want to write software that is fast and inexpensive to run. Our rust services are quickly growing, but cumulatively, they remain in the sub-1000 core count, while processing many millions of requests per second across our service with excellent latency.
2) We want to write software that won't break and page us in the middle of the night, or blow up when we deploy it. We operate some of the core persistence primitives at our company, so the consequence for error in the worst case could be corruption or loss of user data. Our experience thusfar with Rust is that we're able to confidently iterate and deploy our services. The developer velocity is indeed higher, and continues to grow with your experience in the language.
Our team of 10 (with 3 engineers having started in the last 2 weeks) owns all of the database clusters, and all of the services surrounding them. Our on-call is surprisingly quiet, with maybe 0-3 low sev incidents a week, with most of them being due to the underlying database/hardware, and not the rust services. We are actually writing a Rust control plane for our databases soon, which seeks to automate most of the response and remediation of most of the incidents we experience in a given week, and also to automate a bunch of manual work that we currently do around operations of the databases.
The learning curve can be a bit rough, but our team of engineers is motivated to learn, and we have a great culture of mentorship and teaching from the engineers who are familiar with Rust.
Yes, incredibly error prone. Bad sector reads were observed at an alarming rate over a short period - well beyond what is expected if you were to just buy an enterprise nvme and slap it into a server.
It's also worth mentioning that in a cluster of N, to recover a node, it simply needs to stream 1/(N - 1) of the data from its neighboring nodes. So when you look at the cluster as a whole, and the strain on each node that is UP and serving traffic, it's insignificant.
We can either re-build the node by simply wiping its disks, and letting it stream in data from other replicas, or we can re-build by simply re-syncing the pd-ssd to the nvme.
Node failure is a regular occurrence, it isn't a "bad" thing, and something we intend to fully automate. Node should be able to fail and recover without anyone noticing.
It's actually quite nice - because since everything is stored on GCS/S3, it is mostly self healing, we can treat the historical as cattle and not pets.
We also run clickhouse, and unfortunately the above is not true - at least in our setup.