HNHacker News
TopNewBestAskShowJobs

jhgg

3,448 karma · joined November 4, 2013

currently: Lead of Supercomputing at SpaceXAI, building the biggest AI supercomputers in the world. https://grok.com, https://x.ai

formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/

[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]

submissionscomments
jhgg··on Show HN: SQRL – Anti-spam rules language Twitter acquired in 2018, open-sourced
We used to use Smyte at my place of work prior to the twitter acquisition. Very cool stuff.

Ended up rewriting it in house. We ended up with a SQRL variant that is a restricted type-checked subset of python. It is not python however, just syntactically similar.

jhgg··on When Discord is open in the background, NVIDIA card will not reach full speed
Correct.
jhgg··on When Discord is open in the background, NVIDIA card will not reach full speed
Hey all,

I work at discord and specifically have been tracking this issue. We recently released support for AV1 hardware encoding for go live streams. However, the process of probing the GPU for what codecs it supports has triggered a latent nvidia driver bug. We are releasing a workaround some time today, and nvidia is releasing a driver update to address this issue as well.

Apologies for any inconvenience this has caused.

jhgg··on Elastic Cloud password complexity has an “OR” condition
This is what we do at my place of work. Minimum 8 characters - and the password cannot appear as one of the top compromised passwords in HIBP list.

Except in France - CNIL does not agree with NIST. :(

jhgg··on Tell HN: Google Cloud lets anyone add you to a project without your permission
Does this apply with https://cloud.google.com/resource-manager/docs/organization-... enabled?
jhgg··on Build your front end in React, then let ChatGPT be your Redux reducer
This is by no means a practical exercise... and is to demonstrate the capabilities of a large language model, not to reduce boilerplate.
jhgg··on Rewriting a high performance vector database in Rust
> production grade and pure rust tls library

You mean rustls? https://github.com/rustls/rustls

jhgg··on Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
What is wrong with CQL? The only gotcha I am aware of is UPDATE/INSERT really just meaning UPSERT.
jhgg··on Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
We did with some. Although I don't recall the exact specifics but there were improvements we were able to make with GC tuning and switching from CMS to G1. We were on old version of Cassandra for our most problematic cluster - and it was our last cluster to migrate to Scylla so we didn't really do much to explore the problem space of JVM tuning when we were already moving away from JVM already.
jhgg··on Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
At this scale you automate away 99.9% of the things you respond to.

We are not even near this scale yet at my place of work, and we are moving towards this methodology of strong automation to orchestrate a cluster. We hire software engineers to operate our database clusters, and the expectation is to somewhat be selfishly motivated to write programs to remediate issues so you don't get paged constantly. We do not expect to grow our headcount proportionally to the number of clusters or nodes we operate.

You must treat your nodes like cattle not pets. If a node fails, automation kicks in and re-bootstraps it. It is not worth figuring out how to nurse it back to health. When you are performing rolling or scale up operations on the cluster you are just invoking automation to do everything for you.

jhgg··on Cassandra at Apple: 1000s of Clusters, 300k Nodes, 100 PB
Our biggest issue with running Cassandra was related to pathological read / write patterns by some tenants on our system causing outsized availability impact due to triggering garbage collection pressure that would cause whole node GC STW pauses and severe tail latency / query degradation.

We have solved these issues in a few ways, mainly:

- working with the relevant product teams to implement appropriate rate limiting or improving data modeling.

- introducing our own query layer, written in Rust that sits in front of Cassandra that uses a form of micro-caching called read coalescing, and also other forms of query throttling/load shedding to reduce work the database must do for hot keys/pathological patterns of access. We expose a GRPC interface from this - and this lets us centralize control of the client driver and tune it appropriately, while also getting to leverage the ever growing open source grpc traffic routing solutions (envoy, etc...)

and ultimately,

- switching to ScyllaDB, a C++ rewrite of Cassandra which is of course void of any garbage collection issues, and features faster overall performance and lower latencies.

Scylla, however, is not without its own set of issues - and somewhat strict hardware requirements[0] thanks to the seastar engine it is built on top of. Their team however has been delightful to work with, and our platform is markedly more stable in current year than it was in years past thanks to the above factors.

Operationally, however, Scylla and Cassandra are quite easy to run, the trickiest part is repairs. Common operations such as cluster expansion, or replacement of node are so common an operation that they are at this point mundane. Be wary however about read/write amplification issues inherent to LSMT databases, choosing the correct compaction strategy and tuning it appropriately can be quite key. Additionally tombstones can be quite bad for performance.

In current day we offer a new more generic solution that sits on top of scylla (it would work with Cassandra too) that provides a simple interface to query KKV based data, without having to worry too much about problems like large partitions, hot keys, or tombstones! With a design like this, the underlying cluster thus far has been issue free and very easy to operate.

[0]: https://discord.com/blog/how-discord-supercharges-network-di...

jhgg··on Cloud Spanner now with a free trial
You can email hn@ycombinator.com and request a title change. I have seen this done before.
jhgg··on Forum Channels: A space for organized conversations
I am curious about this as well. We take privacy very seriously at Discord. Earlier this year, we entirely re-wrote our privacy policy as well to be more plain English and less legalese. I think that it's actually one of the more well written privacy policies out there.
jhgg··on Forum Channels: A space for organized conversations
All the marketing in the world isn't going to move the needle if your product doesn't solve a problem or offer a better experience for users.

To say that the success of discord was due to "pouring VC money into marketing" completely misses the whole "we also spent a lot of time and effort to build a product that was better than what was out there and solved a problem for our users."

jhgg··on Forum Channels: A space for organized conversations
This is actively being worked on right now actually :)
jhgg··on Forum Channels: A space for organized conversations
There is talk about allowing public/discoverable servers to be indexed.

I don't know when those plans might materialize, but it is very much top of mind for us that there is ever growing knowledge on Discord that isn't accessible via search indexes.

jhgg··on Pingora, the proxy that connects Cloudflare to the Internet
They mentioned they were using tokio, so naturally, yes.
jhgg··on Pingora, the proxy that connects Cloudflare to the Internet
We had the same experience at work deploying rust services that serve many billions of requests a day as well.
jhgg··on Rust – A hard decision pays off
We use a mix of axum, and tonic/tower, depending on if the service is internal (grpc) or external (http).

And no, we do not use an ORM like diesel, since we primarily use ScyllaDB. We wrote our own - it's very nice. We have a fully type-safe scylla/cassandra client, that does static validation of the queries (which we define as annotated structs), and also, when the service starts up, it validates that the schema that the service has agrees with the schema the database has. Insofar a making sure that if you say `WHERE id = ?`, and you pass a i32 but it wanted a String, the service will not start.

It also does request coalescing and tracing/telemetry, and will soon do rate limiting, circuit breaking and speculative retries as well.

jhgg··on Rust – A hard decision pays off
No. Not anymore. 4 years in and I maybe run into a borrow checker issue very rarely in a given week.

Once you understand how the borrow checker works and how to write/structure your code in a way that the borrow checker is happy with it sort of becomes second nature.

I think by side effect coding in this style does lend itself to simply writing better to understand code. Nowadays I just write my code this way without much thought. It happens effortlessly. I find myself writing programs in other languages like I would in Rust nowadays.

This is compared to when I started and ran into borrow checking issues many times a day. Additionally the borrow checker has improved since then and NLL borrow checker will accept more programs as valid than the previous one.

jhgg··on Rust – A hard decision pays off
Writing Rust at scale, I concur with some of these findings. Nowadays, my team exclusively writes Rust. Our rationale is two-fold.

1) We want to write software that is fast and inexpensive to run. Our rust services are quickly growing, but cumulatively, they remain in the sub-1000 core count, while processing many millions of requests per second across our service with excellent latency.

2) We want to write software that won't break and page us in the middle of the night, or blow up when we deploy it. We operate some of the core persistence primitives at our company, so the consequence for error in the worst case could be corruption or loss of user data. Our experience thusfar with Rust is that we're able to confidently iterate and deploy our services. The developer velocity is indeed higher, and continues to grow with your experience in the language.

Our team of 10 (with 3 engineers having started in the last 2 weeks) owns all of the database clusters, and all of the services surrounding them. Our on-call is surprisingly quiet, with maybe 0-3 low sev incidents a week, with most of them being due to the underlying database/hardware, and not the rust services. We are actually writing a Rust control plane for our databases soon, which seeks to automate most of the response and remediation of most of the incidents we experience in a given week, and also to automate a bunch of manual work that we currently do around operations of the databases.

The learning curve can be a bit rough, but our team of engineers is motivated to learn, and we have a great culture of mentorship and teaching from the engineers who are familiar with Rust.

jhgg··on How Discord supercharges network disks for extreme low latency
> Are GCE's SSDs particularly error prone?

Yes, incredibly error prone. Bad sector reads were observed at an alarming rate over a short period - well beyond what is expected if you were to just buy an enterprise nvme and slap it into a server.

jhgg··on How Discord supercharges network disks for extreme low latency
ScyllaDB only supports XFS due to its requirement on aio.
jhgg··on How Discord supercharges network disks for extreme low latency
No dedicated NICs, the link speeds are fast enough to not really worry about this.

It's also worth mentioning that in a cluster of N, to recover a node, it simply needs to stream 1/(N - 1) of the data from its neighboring nodes. So when you look at the cluster as a whole, and the strain on each node that is UP and serving traffic, it's insignificant.

jhgg··on How Discord supercharges network disks for extreme low latency
Scylladb is luckily much faster. We can rebuild a ~5TB node in a matter of hours.
jhgg··on How Discord supercharges network disks for extreme low latency
On GCP, live migration moves the data on the local-ssd to the new host as well.
jhgg··on How Discord supercharges network disks for extreme low latency
We have two ways of re-building a node under this setup.

We can either re-build the node by simply wiping its disks, and letting it stream in data from other replicas, or we can re-build by simply re-syncing the pd-ssd to the nvme.

Node failure is a regular occurrence, it isn't a "bad" thing, and something we intend to fully automate. Node should be able to fail and recover without anyone noticing.

jhgg··on Instagram can track anything you do on any website in their in-app browser
This is why https://developer.apple.com/documentation/safariservices/sfs... exists and why the blog post advocates for using it.
jhgg··on Druid: A Real-Time Analytical Data Store
Strange - we use Druid at scale at work, and it doesn't really require much maintenance. In fact, for the most part aside from some incidents caused by our own misconfiguration, it buzzes away just fine. Runs on Kubernetes. Most annoying thing was tuning segment compaction.

It's actually quite nice - because since everything is stored on GCS/S3, it is mostly self healing, we can treat the historical as cattle and not pets.

We also run clickhouse, and unfortunately the above is not true - at least in our setup.

jhgg··on Android removes much of Fuchsia-related code as Starnix project progresses
https://www.usenix.org/system/files/hotcloud19-paper-young.p...
← PreviousPage 4 of 17Next →