HNHacker News
TopNewBestAskShowJobs

jhgg

3,448 karma · joined November 4, 2013

currently: Lead of Supercomputing at SpaceXAI, building the biggest AI supercomputers in the world. https://grok.com, https://x.ai

formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/

[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]

submissionscomments
jhgg··on Amazon S3 quietly deprecates BitTorrent support
Plain ole bit torrent. Run some seed nodes on dedicated hardware with unmetered bandwidth. It'll get you pretty far.
jhgg··on Rust heads into the kernel?
This is a ridiculous post.

My full time job is writing Rust for large-scale production systems. The language is great - and as an organization we're doubling down on our investment in Rust this year.

jhgg··on Tell HN: Don't put your status page on the same domain as your app
We'll fix this. Looks like we missed a few references to the old status domain! :)
jhgg··on gRPC benchmark results
I would look into re-running the rust benchmarks w/ jemalloc or mimalloc instead of the system allocator and see how it performs.
jhgg··on Why Discord is switching from Go to Rust (2020)
We don't have any operational experience with Aerospike.

We have a bunch of operational experience with C*/ScyllaDB.

jhgg··on Why Discord is switching from Go to Rust (2020)
The original implementation of this "service" was in Redis using lua scripts.

We hit the performance ceiling of redis, and rather than deciding to go with redis-cluster, and dealing with the headache of managing persistent redis, we went this way, so we could consolidate data storage to scylladb/cassandra (it was cassandra at the time of rewrite, scylla now since we've migrated our data stores over).

jhgg··on Why Discord is switching from Go to Rust (2020)
We kept dealing with GC pauses in go, and eventually after investing a bunch of time trying to fix this - we decided to research what it'd be like to rewrite it in another language.

The service itself is pretty simple, so it wasn't a gargantuan task. So it was either, rewrite this again in Go (using off-heap data structures to avoid GC pauses, which makes the software both harder to maintain, and also less idiomatic) or try a new language where we won't have these issues - where we don't have to resort to clever hacks to work around the runtime system.

Since then we've continued our investment in Rust, after success of this project (and many others we've yet to blog about as well.)

jhgg··on Why Discord is switching from Go to Rust (2020)
I think you're missing the point. We use Elixir to solve one class of problems which its excellent in doing (notably: real time distributed systems.) In this area, there is nothing even remotely close in terms of capability. So, we choose the runtime over the language (elixir is a pretty cool dynamic language though!). Also, due to the unique nature of Erlang/Elixir, a lot of problems that exist in C/C++/Rust/Go don't exist in that language. Of course this has meant writing a lot more tests to test things that a typechecker should be able to catch. But it is what it is. We're following Gleam though to see where that ends up.

BTW, here's some places I've spoken about Discord's Elixir usage:

- a podcast: https://smartlogic.io/podcast/elixir-wizards/s5e8-nowack-hei...

- interview with José Valim (elixir's creator): https://elixir-lang.org/blog/2020/10/08/real-time-communicat...

jhgg··on Why Discord is switching from Go to Rust (2020)
No. React/TypeScript is too productive for our project engineers.

But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)

jhgg··on Why Discord is switching from Go to Rust (2020)
The only major cons to rust imo are the slow compile times, the steep learning curve and the immature ecosystem.

You can solve the first hurdle with better hardware (for the same price as our usual specced macbook pro, I opted to custom build a beefy workstation with a 3790X).

The second, you just solve with time. I remember when I first started with rust I hated it. I was working on a native project, and felt like it would have been easier to do in C++. But internally now we have more mindshare and knowledgeable folk who understand Rust - this has helped tremendously in onboarding engineers to the language. Also, having a working editor (via rust-analyzer) and the quality of compiler error messages have improved significantly in the past few years.

In terms of immature ecosystem, I think the rate at which the ecosystem is advancing is incredibly promising. Tokio hit 1.0 in Dec 22 of last year, which will finally put an end to the dependency upgrade thrash we've had to deal with. Going from tokio 0.1 -> 0.2 -> 0.3 -> 1.0 has been a journey. But with 1.0, finally, libraries will have a stable API to target, so we won't have to worry about this much for a while. Additionally, in the following months we intend to start funding critical open source rust projects that we use to ensure their success.

jhgg··on Why Discord is switching from Go to Rust (2020)
We will continue to use beam/otp where it makes sense. In fact some rust runs in BEAM now thanks to Rustler. We wrote a blog about this!

https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...

jhgg··on Why Discord is switching from Go to Rust (2020)
Elixir was not us chasing a trend - but rather betting on a language and runtime system we knew would be amazing if it worked.

Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users?

Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!

jhgg··on Why Discord is switching from Go to Rust (2020)
Not a chance. We are going hard on rust in 2021 after some very successful projects in 2019 and 2020. our engineers have ramped up on the language - and we have good support internally (both in terms of tools, but also knowledge) to see its success.

Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go.

Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands.

And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.

jhgg··on Why Discord is switching from Go to Rust (2020)
We are hybrid. We use bare metal for our RTC workload (voice video streaming) and GCP for the rest.
jhgg··on Why Discord is switching from Go to Rust (2020)
We were experimenting with Go as a "fast" backend language to implement stuff like data services and high throughput systems in.

Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.

jhgg··on Why I Built Litestream
A bit more literature on the N+1 situation in SQLite: https://www.sqlite.org/np1queryprob.html
jhgg··on Node.js 14 is over 20x faster than Python3.8 for fib(n)
An interpreter with a JIT is obviously faster than one without. Especially when dealing with CPU bound work.

I'm not sure this is entirely noteworthy unless you somehow think CPython has a JIT. Would be much more interesting to compare to pypy.

jhgg··on Teamspeak 5 to be based on the Matrix protocol
Oh real interesting! Didn't realize that was the case - and the source linked was just a new account O:
jhgg··on Teamspeak 5 to be based on the Matrix protocol
I don't think this is true, sorry. TS5 is definitely not built on Matrix.

EDIT: Am wrong, see below! Maybe we can update the original post to one of the links below.

jhgg··on Slack is down
Our secret sauce is Elixir/BEAM and Rust :)

Well for the real time side, I can't tell you how big a boon it's been to build our platform on top of Elixir/BEAM. Hands down the best runtime / VM for the job - and a big big secret to our success. Where we couldn't get BEAM fast enough - we lean on rust and embed it into the VM via NIFs.

2021 is the year of rust - with the async ecosystem continuing to mature (tokio 1.0 release) we will be investing heavily in moving a lot of our workloads from Python to Rust - and using Rust in more places, for example, as backend data services that sit in front of our DBs. We have already piloted this last year for our messages data store and have implemented such things as concurrency throttles and query coalescing to keep the upstream data layer stable. It has helped tremendously but we still have a lot of work to do!

To help scale those super large servers, in 2020 we invested heavily in making sure our distributed system can handle the load.

Did you know that all those mega servers you listed run within our distribution on the same hardware and clusters as every other discord server - with no special tenancy within our distribution. The largest servers are scheduled amongst the smallest servers and don't get any special treatment. As a server grows - it of course is able to consume a larger share of resources within our distribution - and automatically transitions to a mode built for large servers (we call this "relays" internally.) At any hour, over a hundred million BEAM processes are concurrently scheduled within our distributed system. Each with specific jobs within their respective clusters. A process may run your presence, websocket connection, session on discord, voice chat server, go live stream, your 1:1/group DM call, etc. We schedule/reschedule/terminate processes at a rate of a few hundred thousand per minute. We are able to scale by adding more nodes to each cluster - and processes are live migrated to the new nodes. This is an operation we perform regularly - and actually is how we deploy updates to our real time system.

I was responsible for building and architecting much of these systems. It's been super cool to work on - and - it's cool to see people acknowledge the scale we now run at! Thank you!! It's been a wild ride haha.

As for scale, our last public number perhaps comparable to Slack is ~650 billion messages sent in 2020, and a few trillion minutes of voice/video chat activity. However given the crazy growth that has happened last year due to COVID - the daily message send volumes are well over the 2 billion/day average.

jhgg··on Show HN: Sonyflake – distributed unique ID generator implemented in Rust
No I didn't but after looking at it it looks like its implementation falls short.

- it only looks at system time at system startup, then relies on instant for monotonic clock source. unfortunately for setups that do leap-second smearing (ours does), this would deviate slightly out of sync after operating for a while.

- does not handle seq overflow within a given millisecond window - this i think is very unsafe, and can cause the generation of duplicate IDs if `next_id` is called at too rapid an interval.

jhgg··on Show HN: Sonyflake – distributed unique ID generator implemented in Rust
The one that says "wip; use at your own risk" with no updates in the last 5 years? Or is there another?

https://crates.io/crates/flake

jhgg··on Show HN: Sonyflake – distributed unique ID generator implemented in Rust
Discord uses snowflake based IDs to great success. It's actually been one of the rock solid parts of our infrastructure and the fact that time stamps are embedded in the ID are great. Every channel, message, stream, attachment, user, server, etc... gets a snowflake.

We originally wrote this service in Elixir but have since rewritten it in rust. Actually very similar to this crate. Maybe I should open source ours too haha.

jhgg··on Tokio 1.0 – async runtime for Rust
Congratulations on 1.0! Tokio and the ecosystem surrounding it (tracing, hyper, prost, tower) are incredibly well thought out and a pleasure to use for building fast performant services with solid latencies.
jhgg··on Post Mortem of Google Outage on 14 December 2020
To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"
jhgg··on Post Mortem of Google Outage on 14 December 2020
They do that as well - see "provisioned throughput"

See the section "Specifying Throughput with Provisioned Mode" here: https://docs.aws.amazon.com/efs/latest/ug/performance.html

jhgg··on Show HN: This website is valid JSON
Prior art (rather similar, but instead of being json, it's a JPEG): https://news.ycombinator.com/item?id=12262470

tldr: the html is stuffed in the exif data!

jhgg··on Scaling Datastores at Slack with Vitess
It is a pretty high QPS. I work on a chat service that is quite popular (even more popular than Slack) and looked at our QPS to our persistent data stores. It is less than an order of magnitude compared to slack's when you normalize for DAU. Really wonder what they're doing over there.
jhgg··on Discord bans me for using their official client
We are not banning people for using Ripcord. We are banning people for using our API in ways that our official client doesn't. This is paramount to our spam detection strategy, which is detecting quirks in requests that try and emulate our official client, in order to send spam. There is a very thin line between "scripts meant to try and emulate our client to send spam in an automated fashion" and "third party client" both try and emulate our client, but perhaps for different intent. However, our systems currently treat these as equals. It is for this reason that we have maintained that 3rd party clients are against our ToS - and most warn you before you use them. Given the vanishingly small user-base of 3rd party clients relative to the tremendous amount of spam-bots that constantly are attacking our platform, it's an unfortunate reality that this is bound to happen.

If you are a real human, and not a spam bot, our general policy is to unban and tell you "hey don't do that again." Which is the outcome that should have (and eventually did) happened here.

jhgg··on Discord bans me for using their official client
Huge apologies on this one. Most of your tickets were still in the queue - and no one has gotten to them yet. Some other tickets were closed in error because it looked like the issue was resolved - as your original ticket had mentioned there was an unban.

As for the initial unban, a bug had caused the "successful appeal" macro to apply - but it did not actually unban your account. Hence the confusion.

We are definitely working on improving this and the experience - and I'd like to apologize for the quite honestly shitty experience you had. 2020 has been a wild year for us, and we are still playing catch up operationally on some things - with the rapid surge of usage on the platform.

Unfortunately the people handling these appeals don't have all the information necessary to explain exactly why you were banned - and as a general policy we try to keep those exact details under wraps to thwart attempts from people to try and reverse engineer / circumvent our anti spam. (Of which there are many, so... so many people trying to do this.)

We are working hard on improving our process around this - but it's a rather gargantuan effort. From both the improvement of automation and tooling, and also staffing in general. But we are actively working on it - and a lot of my day job is to make sure we are handling these kinds of cases in a timely manner and also making sure nothing slips through the cracks due to a bug like your tickets have.

← PreviousPage 6 of 17Next →