HNHacker News
TopNewBestAskShowJobs

jhgg

3,448 karma · joined November 4, 2013

currently: Lead of Supercomputing at SpaceXAI, building the biggest AI supercomputers in the world. https://grok.com, https://x.ai

formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/

[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]

submissionscomments
jhgg··on My favorite Erlang container
This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable.

We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.

jhgg··on My favorite Erlang container
No, it's entirely in user-space. Our websocket protocol can just negotiate a resumption. https://discord.com/developers/docs/topics/gateway#resuming
jhgg··on Building a Cache in Elixir
One thing you can do is use :ets.slot/2 to do random probing of the cache and evict entries. The way to do so is simple, just have a process every few milliseconds check a random set of keys (say, 50 keys), and check if they expire. If >25% of keys were about to expire, repeat this process again instantly, until the % of keys expired in the batch is <25%, then poll periodically.

This is how Redis versions prior to 6.0 implemented key expiry. It's a very simple algorithm and can prove quite efficient at evicting expired keys without having to constantly scan the entire range.

jhgg··on My favorite Erlang container
At Discord, we maintain long lived web-sockets, and need to be able to deploy our real-time elixir based services (which runs on the BEAM VM just like Erlang) without any disruption to user traffic.

To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed.

We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.

jhgg··on Deno raises $21M
Competition is great - sure, but I want to know what Deno gives me over its competitors.
jhgg··on Deno raises $21M
Deno deploy seems cool and all, but I haven't seen any great rationale for using their service over say Cloudflare Workers.
jhgg··on Cloudflare had a partial outage
They're not down worldwide, we're still seeing traffic from some POPs, but it looks like a majority of their POPs are dead.

This feels like a bad config push.

jhgg··on Transparent memory offloading: more memory at a fraction of the cost and power
You should read the whole article, because TMO does differ from traditional swapping. Notably, swapping usually occurs during periods of extreme memory pressure, where as TMO will offload memory much sooner and more intelligently.
jhgg··on Bugs that the Rust compiler catches for you
The key difference however being that finalization of an object is guaranteed when it falls out of scope, and not at the whims of when the garbage collector decides to run.

It would be a nonsensical idea to leave up to the garbage collector releasing an acquired lock on its own time. Such a situation could very easily cause a deadlock, as a program may be waiting on a lock that the garbage collector must release, but while the program is waiting on the lock, no additional garbage is being generated that might trigger a garbage collector run, nor is there any guarantee that even if that run does complete, that the objects you wanted finalized would be.

jhgg··on Why Discord is switching from Go to Rust (2020)
We use Elixir + Rust in a few select places. But the majority of our Rust code is services that are purely written in Rust.

We use Rust over Go, not only because of the garbage collection issues, but because it's truly a better language in almost every way (once you learn it!)

I will say, Go is much easier to pick up, but in exchange you pay in the long term having a language that actively works against you when you start working on more advanced programs, and a mountain of code that's accumulated over the years that you have to maintain.

We work on high concurrency systems here, and I very much enjoy not ever having to think about "is this thing thread safe" because the compiler is checking that for you. I love being able to use the type system to offer my co-workers powerful, but difficult to misuse libraries. I like having sensible abstractions around concurrent execution.

Like, for example, if you create a channel in go, and for whatever reason, don't try to read from the channel, or give up (because you're racing a timeout), then the goroutine that tries to write to that channel will block forever and leak. In Rust, if you try to write to a channel where there is no longer a receiver, the write to channel will return an error, which you can then choose to handle, or simply ignore depending on your use-case. Of course, you can be wise and allocate your channels with a capacity of 1, but you can also just completely forget that, and start a steady leak of goroutines for the lifetime of your program that the garbage collector won't save you from!

Want to execute many futures with bounded concurrency in Rust and collect the results back into a Vec, but give up if any of the futures fail, or if a timeout is elapsed, and also make sure that all allocated resources are properly dropped and closed in the event that any errors happen? Just combine a futures::stream::StreamExt::{buffer_unordered, collect}, and a tokio::time::timeout, and in a few lines of code you've done it.

Want to do the same in Go? Spawn a pool of goroutines I guess, distribute two channels, one for sending them work, one for receiving work, and don't forget to throw in a WaitGroup, pass a context along to all the goroutines, make sure you don't forget any defers, if you are using a shared resource, make sure it's thread safe, or make sure you're locking/unlocking the appropriate mutex, make sure you size your result channel appropriately or you might leak goroutines and any allocations they hold if your main goroutine that's spawned all that work timed out waiting for the results to come in. Is there a library that does all this for you in Go? I googled "golang run many goroutines and collect their results" and looked at the first page of results, and it's basically the above...

It is no surprise then that we've picked to use Rust pretty seriously. When you're looking to build reliable systems with serious speeds and massive concurrency, you pick the best tool for the job. That for us is Rust, not Go. And for our real time Distributed systems, we pick Elixir, because BEAM/OTP is just so dang good.

jhgg··on Why Discord is switching from Go to Rust (2020)
Yes we are using rust in a big way. We have multiple teams now full time working on Rust. It is being used on both the client and server, as native modules, web assembly, and also native rust services and NIFs that embed themselves in our elixir services.

It has been an incredible success. I plan to blog more about it in the coming months. Our usage of Rust is continuing to grow, and if you check out our jobs page, you might notice all backend / infra jobs list Rust in them now :)

I think probably 40% of requests are handled directly by rust services now, with the rest involving one or more rust service called from our Python API layer.

jhgg··on Ask HN: Are you using actors in production? Why/Why not?
At Discord our entire real time system is built ontop of Elixir. Everything is an "actor" in that model. Every single Discord server, websocket connection, voice call, screenshare, etc... distributed using a consistent hash ring.

It's an incredibly great model for these things. We've been able to scale this system from hundreds to hundreds of millions of users with very little changes to the underlying architecture.

jhgg··on Who's Attacking My Server?
Yeah? You can't really spoof TCP connections... unless you are talking about doing a BGP hijack. Can you elaborate more on this?
jhgg··on Incident report: Spotify outage on March 8
We suffered from similar issue to Spotify. Traffic Director pushed a mostly empty configuration to envoy. Very frightening indeed. It continued to push that invalid configuration over the course of the outage.
jhgg··on Celery in production: Three more years of fixing bugs
Celery is not great. At my place of work we begrudgingly are using it right now, with plans to replace it with something else this year. I am at the very least amazed we got this much use out of it. Considering replacing it with a much much more simple version that just uses google cloud pub sub and some simple python routing of the messages to functions.
jhgg··on Why is my Rust build so slow?
It allows for things like https://docs.rs/typed-html/latest/typed_html/ to exist.
jhgg··on Dutch forensic lab says it has decoded Tesla's driving data
I mean in the same event that you list, Elon literally opens with: "Tesla is much more than an automotive company, and we have lots of deep AI activity..." I think it is fair to say that a company that is literally inventing its own AI training supercomputer is probably an AI company as well :)

https://youtu.be/j0z4FweCy4M?t=2833

jhgg··on We analyzed 425k favicons
I committed a fix, it's now 24k uncompressed! :)
jhgg··on Hacking YouTube with a MP4
We try to grab the first frame to show a preview. But if we can't for whatever reason, that's when the sad poop appears :(
jhgg··on Hacking YouTube with a MP4
This is how the video element works in chromium. I suspect it looks at the same metadata field. Beyond leading to a bit of absurd UI state though it's not the same kind of issue that this post describes, which deals with trying to transcode these kinds of videos which could multiply storage utilization on the backend.
jhgg··on Hacking YouTube with a MP4
We don't transcode video, so no.
jhgg··on MangaDex infrastructure overview
They mention a $1,500 budget per month but then omit things critical to understanding how they achieve that cost point.

What is actually more interesting is to understand what portion is spent on servers versus bandwidth - and what hardware configuration they use to host the site. For example, Is $1,500/mo just paying for colo costs + bandwidth, with already owned recycled hardware (think last gen hardware that you can get at steep discounts from eBay / used hardware resellers...)

That would have been way more interesting to know given the blog title than the choice of infrastructure software they use.

jhgg··on How Discord Stores Billions of Messages (2017)
Upstream blockers in scylla + its our biggest dataset, and is a quite large project.
jhgg··on How Discord Stores Billions of Messages (2017)
We're well over 12 nodes in current year :P
jhgg··on How Discord Stores Billions of Messages (2017)
Yea. Everything but messages is on Scylla.
jhgg··on Reining in the thundering herd: Getting to 80% CPU utilization with Django
So, in guincorn default mode (sync), the mode I'm assuming you're using. This means you really have 1 process handling 1 request at a time. The "thundering herd" problem really only applies to connection acceptance. Which is to say, that in the process of accepting a connection, it is possible to wake all idle processes that are waiting for a connection comes in (they will wake and hit EAGAIN and then go back to waiting.) Busy processes that are servicing requests (not waiting on the accept call) will not be woken, since they aren't waiting on a new request to come in. The "thundering herd" problem as I understand it, can indeed waste CPU cycles, but only on processes that aren't doing much anyways. I do however believe that `accept()` calls have been synchronized between processes on Linux for a while now to prevent spurious wakeups. You should verify you're actually doing spurious wakeups by using `strace` and seeing if you are seeing a bunch of `accept()` calls returning EAGAIN.

In gunicorn, `sync` mode does exhibit a rather pathological connection churn, because it does not support keep-alive. Generally, most load balancing layers already will do connection pooling to the upstream, meaning, your gunicorn processes won't really be accepting much connections after they've "warmed up". This doesn't apply in sync mode unfortunately :(. Connection churn can waste CPU.

Another thing to also note is that if you have 150 worker processes, but your load balancer only allows 50 connections per upstream, chances are 100 of your processes will be sitting there idle.

Something just doesn't feel quite right here.

EDIT: I do see mention of `gthread` worker - so you might be already able to support http-keepalives. If this is the case, then you should really have no big thundering herd problem after the LB establishes connections to all the workers.

jhgg··on Reining in the thundering herd: Getting to 80% CPU utilization with Django
Our app is also a messaging app. So lots of small & fast responses.
jhgg··on Reining in the thundering herd: Getting to 80% CPU utilization with Django
This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless now that we're using envoy, so that'll probably go away soon.

Works amazingly well! We run our python API tier at 80% target CPU utilization.

jhgg··on Remote code execution in cdnjs of Cloudflare
Still can't beat the speed of light...
jhgg··on Mumble: Open-Source, Low Latency, High Quality Voice Chat
A pass-through mode where we disable all audio processing is on the roadmap, for those with really good setups that don't need the post-processing :)
← PreviousPage 5 of 17Next →