HNHacker News
TopNewBestAskShowJobs

jhgg

3,448 karma · joined November 4, 2013

currently: Lead of Supercomputing at SpaceXAI, building the biggest AI supercomputers in the world. https://grok.com, https://x.ai

formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/

[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]

submissionscomments
jhgg··on Discord bans me for using their official client
No, this was not the case. A 3rd party client was very much involved here.

> So i could imagine a user wants to do something that Discord makes it difficult to do (perhaps they want to search channel logs locally, or delete all of their messages in a channel, or amny other things), and if I wanted to do that myself I might just throw some JavaScript into the console in a for-loop, which would obviously result in irregular API requests.

We don't really weigh sus requests too highly (to the point of acct termination) unless they are likely to impact other users, or be used to move laterally on the platform (join servers, send friend requests, open new DMs... etc.)

jhgg··on Discord bans me for using their official client
Hi,

I lead the anti abuse team at Discord. I've looked into this case a bit more in-depth to see what was going on. Although I can't get into specifics, given the poster was kind enough to provide us with ticket IDs, I was able to look into this a bit further to see what actually triggered the account ban.

> Discord bans me for using their official client

This statement is false, unfortunately - and our abuse detection flagged the account for using our API in ways that our official clients do not. In this case, there was indeed a 3rd party client involved. Although our rules and heuristics do not specifically target 3rd party clients, it does target things that "don't look like what our official client does" which is very effective at targeting spambots that are constantly waging war against our platform - but has the unfortunate side-effect that using a 3rd party client in certain ways may lead to account termination.

In general, appealing the ban would lead to an unban (we're humans after all, and we're trying to stop spammers/bad actors, and that's what this system is meant to stop.) I am going to following up with our T&S team to understand why this case saw such a rocky conclusion.

jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users (2019)
We needed to support arbitrary index access into the sorted set - which is why stdlib btree map doesn't work.
jhgg··on How Discord Won
We don't have an open req for our chat infra team right now which heavily uses Elixir. We are quite committed to Elixir, actually, we were interviewed by José Valim about how we use Elixir :)

https://elixir-lang.org/blog/2020/10/08/real-time-communicat...

jhgg··on How Discord Won
Discord does not sell user data. We've stated this many times, unequivocally. Our monetization puts the user as the customer, not say, the product to be sold to advertisers (read: other social networks.) We think that this alignment of incentives sets us up to do the best by our users. Our monetary success depends on how great an experience we can build for our users, such that they convert to a paid subscriber - while not gating everything behind a paywall. Hence why our premium subscription service generally includes cosmetic incentives, that are optional to the core experience of chat, and as well, and unlocks stuff like higher quality streaming (since it costs us more to run, we unlock higher quality once you pay us.)

As for where it's stated that we don't sell user data, it's in our privacy policy: https://discord.com/privacy

> The Company is not in the business of selling your information. We consider this information to be a vital part of our relationship with you.

> No Sale of Personal Information: The CCPA sets forth certain obligations for businesses that sell personal information. We do not sell the personal information of our users.

> Because Discord does not sell the personal information of our users to third parties, this is not applicable.

And also in our safety center: https://discord.com/safety

> We do not sell your data, nor do we share it with third parties for advertising purposes.

jhgg··on Discord Desktop App RCE
I'm sensing a trend in the replies here, which is that this was too low a pay out, perhaps we should have a discussion about what is a worthy payout, because we based this off of what we saw other companies paying out.

Our bug bounty page on H1 (invite only currently) lists critical vulnerabilities at $3,000 for a bounty, we decided to up the pay-out to $5,000 for the thorough report on the exploit - even though the vulnerability was not in our code, we paid extra (from our perspective...) because the vulnerability did after all ship in our software.

I compared this to similar payouts for RCE's, for example, slack paid out $1750 for a similar RCE earlier this year, due to lack of context isolation (https://hackerone.com/reports/783877) - although this did get published after we paid bounty - so it did not influence our decision.

Clearly there is some discussion to be had about what is fair. For context (which might be missing), this was not a 0-click drive-by RCE, and to exploit involved 4-clicks you had to be sent the embed, (1) click the "play" button to launch the iframe [we never load iframes with 0click], (2) click play within sketch-fab to load the 3d model, (3) click a small "1", and then (4) click a "+" in the tooltip that pops up in order to trigger the exploit. I'm curious if anyone has prior art or opinions on what would be a more appropriate bounty?

jhgg··on Discord Desktop App RCE
No. The vast majority of our updates are just new features & bugfixes of existing features.
jhgg··on Discord Desktop App RCE
Hi all! Discord Employee here that was involved in the remediation of this exploit! I just wanted to clarify with a timeline, and explanation as to why we had context isolation disabled!

9:21 PM on July 16, 2020 we received a very detailed report from Masato outlining this exploit.

9:34 PM: Ticket acknowledged - and we began a deploy that would disable sketchfab embeds within the app, to remediate this known attack vector.

10:00 PM: Update pushed to stable to disable all existing sketchfab embeds.

Thanks to the detailed report, we were able to go from a report to a fix deployed to stable in ~40 minutes!

Following that, the next day we deployed a better update as we understood more about the issue (which was the sandbox attribute on the iframe.) In addition, we also paid out $5,000 for this bounty, even though the main fault that lead to RCE was due to a bug in Electron (CVE-2020-15174) which allowed for a bypass of our CSP, by allowing the main window to be navigated to a different domain.

----

As for context isolation, a lot of the code that had been written was not compatible with contextIsolation - and required significant work to refactor. For example, due to the way that objects needed to be cloned to pass through the bridge, the internal APIs that existed needed to be entirely reworked, as they were not really compatible with this model. We began this work in April shortly after we worked out all the quirks required to upgrade to Electron 7 which is when contextBridge would be available for us to turn on contextIsolation. It was not as simple as flipping a boolean from false -> true, and required a re-work of our native modules and their internal APIs, and also doing so in a way that would be backwards & forwards compatible with the various app versions that we had shipped in the wild - in addition to dealing with some performance regressions that needed work-arounds in the new context isolated world.

In August, we shipped context isolation to our Stable release channel and gave Masato the green light for disclosure - which leads us to today!

jhgg··on Discord Desktop App RCE
Once rearchitected there was negligible performance hit. But context bridge etc... basically requires cloning all objects that need to be passed to the renderer/browser window from another context (the native module). If your message passing is frequent and objects are large this can have an adverse impact to performance.

Our video rendering implementation needed to copy frame buffers to the renderer. Doing this over context bridge would have been quite expensive.

A more ideal approach which the team is working on keeps the buffers on the GPU - which should solve all the performance woes and more.

More on this here: https://www.electronjs.org/docs/api/context-bridge

jhgg··on Discord Desktop App RCE
Enabling contextIsolation was a lot of work, and required a re-working our native modules (around voice and video) while still maintaining performance.

Getting everything to work in a context isolated world was a many month effort, which had begun long before this bug was reported to our team.

(The majority of this code was written before context isolation even existed :P)

jhgg··on Cloud Traffic
We "use" traffic director at work. We thought we'd be able to use it for service discovery across our mesh, but it does way more than that in problematic ways that leak internal google abstractions :<

I asked one thing of the product team (roughly), "we like the bits where it tells envoy the nodes that are running the service, and what zones they are in. we don't like all the other weighting things you try and do to act like the google load balancer."

More in-depth we we've had traffic director direct traffic to instances within a specific zone by weighing that zone higher before instances in that zone would even start up, causing service degradation.

We considered writing our own XDS proxy which would run as a sidecar to envoy, connecting to traffic director and stripping all the weights.

After some back and forth with the TD team, we came up with a solution to instead fool it into not weighing things by setting the target CPU utilization of our backends to 1%...

jhgg··on Migrating Dropbox from Nginx to Envoy
discord is also on that list - although we have not spoken much about it yet!
jhgg··on Algolia introduces pay-as-you-go pricing for search
Why the indirection in the pricing page. Like, if a unit is 1,000 searches/month, why not just put that in the matrix.

Free = 10,000 searches/month.

Standard = $1 per 1,000 searches. ...

Pay as you go:

1,000 - 10,000 searches = Free

11,000 - 100,000 searches = $1.00/ 1,000 searches

The whole to unit conversion really just adds a level of indirection that I don't understand. This was further confused by units having different colored dots depending on the plan, making me think there were 3 different kinds of units.

A slider would be nice, let me slide it to what my search volume for a given month would be, and tell me how much that would cost, factoring in volume discounts.

Additionally, this is a huge red flag:

> If you exceed your committed usage, there are overages that will be charged.

What are the overages??! Why is it not just sliding back to pay-as-you-go pricing, like reservations for say EC2 work.

----

As an aside, we use Algolia to power some search features at Discord. This new pricing structure looks to be an order of magnitude more expensive (we fall under the "contact sales" usage here...) Luckily we're grandfathered in or we'd have to consider putting a cloudflare worker in-front of this and leveraging that to do caching of common hot queries to reduce cost.

jhgg··on How the biggest consumer apps got their first 1k users
Discord's first cohort of users came from the final fantasy XIV subreddit - after a friend of the cofounder posted there after trying out the app. It's been all uphill from there.
jhgg··on Show HN: Lynk – Securely expose local TCP and HTTP services to the web
This product offering is a bit confusing. Not that I don't get the offering - but that the marketing around it confuses me.

It's designed for forwarding applications for development purposes? But then why the in-depth monitoring and alerting (something that seems to imply a production system might be running on top of this.)

The "meaning your website performance won't take a hit no matter where your users are" doesn't really make sense in this context either. My "users" are hopefully my coworkers/the client I'm demoing to. And if I'm tunneling up my site over wifi from my laptop, a "highly available" and "globally distributed" load balancer setup really doesn't make sense, nor is it relevant for the development use-case.

And if it's "6x more performant" than ngrok- is that really relevant if you're using it for a trivial amount of traffic in a development environment? Why is this a selling point to me?

Further - if I'm tunneling TCP, does "end to end" really matter? If I'm tunneling something that's not encrypted on my application (let's say a plaintext redis connection) the link between this service and the person connecting to the tunnel is still unencrypted. Which means the whole "end to end" marketing really doesn't make much sense - unless of course I don't trust the network I'm running on - but for some reason maybe trust the network the person accessing the tunnel is on (super unlikely situation...)

The example also irks me - exposing a database directly to the internet (if it's a database for development, hopefully - it's probably fine?)

Is this trying to compete with ngrok, or something like Cloudflare's Argo tunnels? Because the marketing material leads me in two different directions, which is pretty confusing.

jhgg··on Build a real-time Twitter clone with LiveView and Phoenix 1.5
You can kill them. Just like processes on your computer!
jhgg··on Secor: A service persisting Kafka logs to S3, GCS and Azure Blob Storage
You are looking for Druid: http://druid.apache.org/
jhgg··on Discord outage postmortem
There are two fixes listed. One is to improve client behavior to reduce the likelihood of an incomplete request to be handled. The other is to fix the crash bug that occurred when an incomplete request is handled. The fix would be incomplete with only the first mitigation.
jhgg··on Why Discord is switching from Go to Rust
In practice, for our service, most of our CPU time is not spent in data mutation, but rather networking and serialization (this is btw, the same conclusion Redis came to when they added "multi-threading".)

You can scale-out by running multiple instances of the service (shared-nothing, N many depending on how cores you want to run on.) Or, you can do message-passing between cores.

In this case, we have 2 modes of scale-up/out (add more nodes to the cluster, or add more shared-nothing LRU caches that are partitioned internally that the process runs, allowing for more concurrency).

We however only run one LRU per node, as it turns out that the expensive part is not the bottleneck here, nor will it probably ever be.

jhgg··on Why Discord is switching from Go to Rust
Assuming only one thread at a time needs to access the LRU cache (not hard with the shared-nothing message passing architecture which we employ here), the lifetime of the object being checked out from the cache is able to be understood at compile time, and we can just use the borrow checker to ensure that it remains that way (we've got a mutable reference to the LRU, and we can use that to get a mutable reference to an object within the LRU. By the time the function that is mutating the data in the LRU finishes, the references to the objects must be dead (the borrow checker will enforce that.) Since all this information is available during compile time, runtime ref-counting (via rc/arc) is not necessary.

This is made possible by rust's memory model, where it understands ownership of data, and the lifetime of each reference that's being taken from that owned data. This means that the compiler can statically determine how long an object needs to live, and that references to the object don't outlive the owned data. For use-cases where the lifetime of references are able to be statically understood, an arc/rc is not required. This blog-post goes into it in much better detail than I can: https://words.steveklabnik.com/borrow-checking-escape-analys...

jhgg··on Why Discord is switching from Go to Rust
I think this is a statement that is ignorant to our development efforts, and how our team is staffed, and what our objectives are.

First and foremost, we do care deeply about desktop performance. We shipped this week a complete rewrite of our messages components, that come with a boatload of performance optimizations, in addition to a new design. We spent a lot of time to do that rewrite in addition to applying new styles, because given what we know now (and what's state of the art in React world), we can write the code better than we did 3+ years ago. In terms of total engineering time spent, the rewrite of messages actually took much longer than the rewrite of this service from go to rust.

That being said, the desktop app does load much slower than we'd like (and honestly than I'd like personally.) I commented in another thread why that is. That being said, the person who is writing backend data-services, is not the one who's going to be fixing the slow boot times (our native platform team). These are efforts that happen independently.

As for our motivations for using rust, I think saying that "a little GC jitter on the server requiring a complete rewrite" is one of many reasons we wanted to give rust a shot. We have some code that we know works in a Golang. We want to investigate the viability of Rust to figure out how it'd look like to write a data service in rust. We have this service that is rather trivial, and has some GC jitters (that we've been fine with for a year.) So, an engineer (the author of this blog post) spends some time last year to see what it'd look like to write an equivalent service in rust, how it'd perform, how easy it'd be, and what the general state of the ecosystem is like in practice.

I think it's easy to forget that a lot of work we do as engineers isn't all about what's 100% practical, but also about learning new things in order to explore new viable technologies. In this case, this project had a very clear scope and set of requirements (literally rewrite this thing that we know works), and a very well defined set of success criteria (should perform as-good or better, see if a lack of GC will improve latencies, get a working understanding of the state of the ecosystem and how difficult it would be to write future data services in rust vs go.) Given the findings in our rewrite of this service, running it in production, and now using features that have stabilized in rust, we're confident in saying that "in places where we would have used golang, we consider rust viable, and here's why, given our exercise in rewriting something from go to rust."

jhgg··on Why Discord is switching from Go to Rust
A lot of the startup cost right now is in our really ancient update checker. There are plans to rewrite all of this, now that we understand why it's bad, and have some solid ideas as to what we can do better.

I do think it's reasonable to get the startup time of Discord to be near what VS Code's startup times are. If we remove the updater, it actually starts pretty fast (chromium boot -> JS loaded from cache) is <1s on a modern PC. And there's so much more we can do from there, for example, loading smaller chunks in the critical "load and get to your messages path" - being able to use v8 heap snapshots to speed up starting the app, etc...

The slow startup time is very much an us problem, and not an electron problem and is something I hope we'll be able to address this year.

jhgg··on Why Discord is switching from Go to Rust
When we investigated, there was no way to change that that we could find - barring compiling go from source (something we could have done, but wanted to avoid.)
jhgg··on Why Discord is switching from Go to Rust
More people at our company know Rust than Kotlin. It's used across multiple teams (from our game SDK, native encoder/capture pipeline, chat infra team for Erlang rust NIFs.) where as Kotlin is only used by our android team.

We are willing to adopt early technologies we think are promising, and contribute to or fund projects to continue to advance the ecosystem. Yes, this means the path less traveled, but in the case of rust (and in the past Elixir, and even React Native) we think the trade offs are worth it.

Also the tokio team uses Discord for their chat stuff, so it's nice to pop in to be able to ask for and offer help.

jhgg··on Why Discord is switching from Go to Rust
It was perhaps too deterministic. What's not mentioned in the blog is that after running for long enough, the cluster would line up it's GCs, and each node would do the 2 minute GC at exactly the same time causing bigger spikes as the entire cluster would degrade. I'm guessing all it takes is a few day night cycles combined with a spike in traffic to make all the nodes reset their forced GC timers to the same time.
jhgg··on Why Discord is switching from Go to Rust
tokio 0.1 was definitely less efficient, when we compare go to 0.2, tokio uses less cpu consistently, even when compared to a cluster of the same size almost a year later with our growth over the time since we switched over.
jhgg··on Why Discord is switching from Go to Rust
Correct. This is internal time it takes to process the message. Since once a node is "warm" thanks to their large caches, it's mostly in memory operations and queueing for persistence which happens in the background.
jhgg··on Why Discord is switching from Go to Rust
This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019, and wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchronous rust ecosystem.

This blog post kinda internally matches our upgrade to std::futures and tokio 0.2, away from futures 0.1.

jhgg··on Why Discord is switching from Go to Rust
Your reply misses the point. We were already doing so few allocations that the GC only ran because it "had to" at every 2 minute mark. The issue was the large heap of many long lived objects.
jhgg··on Why Discord is switching from Go to Rust
checked in objects in a sync pool gets cleaned up on GC. It used to clean the whole pool, but now I think it does half each GC cycle. If you want to say "objects checked in should live here forever and not free themselves unless I want them to" sync pool is not the tool for the job.
← PreviousPage 7 of 17Next →