HNHacker News
TopNewBestAskShowJobs

jhgg

3,448 karma · joined November 4, 2013

currently: Lead of Supercomputing at SpaceXAI, building the biggest AI supercomputers in the world. https://grok.com, https://x.ai

formerly: Principal Engineer of "Core Technologies" at Discord. https://discord.com/

[ my public key: https://keybase.io/jhgg; my proof: https://keybase.io/jhgg/sigs/NfC1mAm4fGJgKsLzYcshaSlYAfvMnxgLUZgee3nWqzI ]

submissionscomments
jhgg··on Why Discord is switching from Go to Rust
This is one of multiple, we did not blog about this one, but switching a Python http service for analytics ingest that was purely CPU bound to rust resulted in a 90% reduction in compute required to power it. However, that's not too interesting because it's known that Python is slow haha.

We have 2 golang services left, one of them has a rewrite in rust in PR as of last week (as a fun side project an engineer wanted to try out.)

Additionally, as we move towards a more SOA internally, we plan to write more high velocity data services, and rust will be our language of choice for that.

jhgg··on Why Discord is switching from Go to Rust
This is a bit unclear. The root map is still a hash map, but it's a "map of maps" the inner map is a BTreeMap - this is for memory efficiency, as the inner map is relatively smaller and we wouldn't have to deal with the growth factor of a hash map (and having to manually manage that.) where as the root hash map is pre allocated to its max size.
jhgg··on Let's Reverse Engineer Discord
This transport mode is also publicly documented here: https://discordapp.com/developers/docs/topics/voice-connecti...
jhgg··on A lot of complex “scalable” systems can be done with a simple, single C++ server
SO is really not a great example of a high traffic site. 5500 req/sec of mostly read only traffic is not that crazy at all, and their hardware footprint is incredibly over-provisioned for the workload. I don't really think their example stands well here.

For example, at work, our entire analytics ingest workload (HTTP) for a few hundred million users runs on 8 core VMs on GCP, written in Rust/Go, each node doing ~40k events/second.

Our RTC infra sustains >1m PPS per node on 4 core 3.9ghz 2014-2017 xeons.

jhgg··on Facebook says a bug caused its iPhone app’s inadvertent camera access
Looking at the videos, it looks like the camera opens when the main view is swiped to the right. And you can see a ~300ms delay between when he swipes down and when the camera starts. This makes sense, because the way you open the story feature on FB is to swipe right. It's a good UX feature to start booting the camera as soon as you initiate the gesture, rather than when you've fully navigated so the camera is ready when you've finished your swipe.

Their layering system also pushes the view slightly to the right to create a perspective effect when it transitions back to the app from viewing an image.

Of course, this triggers the component that thinks the app is starting to pan to the right - and starts the camera and renders to the view behind the main view.

This seems totally plausible to be a bug - and I'm not sure why other commenters on HN aren't bringing this up.

Their app is fairly complicated, and it's totally reasonable that the team that worked on the story feature assumed that the only thing that would cause the view to move right was a user gesture. However, it looks like their layering system also caused it to move right as part of a perspective transition. This sounds like a bug to me.

If you really wanted to keep the camera open in the background, there are other ways to do so, such as literally not rendering the image to a visible framebuffer upon initiating the AVCaptureSession. iOS does not require you to attach a AVCaptureVideoPreviewLayer to the capture session - and you could very easily just take those frames and process them without ever showing a preview to the user.

The bug is not that they were capturing in the background, but that the perspective transform of the main view caused the view behind it (for the story camera feature) to think the user was swiping the main view to the right and to start up the camera to make sure it's ready asap. Running the camera in the background is actually pretty expensive - you don't want to suck power doing so. As much as facebook wants your data, it also wants you to regularly interact with the app. If users think the app is draining battery too fast - they'll use it less throughout the day to preserve battery life. Making sure it isn't a power suck is important to their core business.

jhgg··on Apple Is Sending URLs to Tencent?
Chrome does something like this, you can learn more about it here: https://codereview.chromium.org/6286072/
jhgg··on A multithreaded fork of Redis that is faster
This is pretty interesting. At work we run a few dozen cache nodes serving several million QPS during peak. We were debating switching some of this workload to memcached from redis - but this may be a viable alternative.
jhgg··on Which companies are using Erlang, and why?
Although not in Erlang, Discord uses Elixir which runs on the BEAM VM to much success as well for our real time distributed system as well as VOIP signaling components. Looking back years later - it’s safe to say we wouldn’t have chosen anything else.
jhgg··on Guess I'm Done with Discord
>My impression would be that an aged account with a good reputation would be held to much less scrutiny than a new account, regardless of my method of accessing the service.

"Good" accounts turn bad pretty quick. We have some betterments to make around taking account age into consideration - but it's also a well observed event that a prior good account gets compromised, moves between continents and starts sending out spam. We've also observed spammers register accounts, sit on them for a while (we've observed some age for over a year) before using them for spam. So, if we notice an "account traveling around the world at an unreasonable speed" we use that as a signal as well - and it is a very common pattern, almost exclusively exhibited by spam accounts, but also the few users whom connect via tor.

>That doesn't really mean much when Discord openly detests third-party FOSS clients and will not make its server available at least in a similar capacity to GitHub's self-hosted solution

In an ideal world, it'd be nice to support 3rd party clients - but unfortunately - we've observed on many occasions where 3rd party clients have malicious plugins that lead to account compromise. Additionally, having to support 3rd party clients can be problematic from an anti-spam perspective, as it muddles the line between "here's an obviously fake client" and "here's a legitimate 3rd party client." I actually wonder if this is why twitter struggles at anti-spam so much (but I don't know nor have talked to anyone at twitter to verify this.)

I also don't really understand why we have an obligation to offer a self-hosted solution. An advantage of our business is our server infrastructure - and although we occasionally blog about how we do things, maintaining an open source release is neither good for business, nor is it for product velocity - and definitely not something we can support given the available engineering resources. We are a very small team of engineers. For the first 3 years of the product, the infrastructure team at Discord was 2-4 people, in the current day, the IC's on the Core Infra team at Discord is less than 5.

I think a lot of people have this misconception that we are a huge company with a bunch of engineers - however, unlike a lot of valley startups, we actually hire very slowly, and deliberately - and relative to other products in our space, our team is exceptionally small. From what I hear, our entire engineering department is the size of the mobile department at another company in the voice/text chat space. As such, we work efficiently and deliberately - with the goal to build a good product, and also to ensure that we're successful as a business in the long term. These values mean that we do have to make trade-offs. But we do so in the interest of our users. Discord as a product is one that I'm passionate about working on, and a product that I use daily to play games with and talk to my friends.

> If they have my phone number, they have my phone number.

Have you considered using a burner phone? Very easy to pick one up from your local convenience store for a few bucks - and will work with phone verification on our product just fine - and will work with others that employ similar anti-spam solutions.

> Despite what we're stuck with, I do genuinely believe Discord could tune their spam and login mechanisms such that false positives are kept to a minimum.

I do agree! We are actively hiring for this position: https://discordapp.com/jobs/4286902002 - there are many betterments to be made, but we need more people such that we can work on em!

jhgg··on Guess I'm Done with Discord
No, I don't believe that adding the ability to reduce the security of your account is necessarily a good idea.
jhgg··on Guess I'm Done with Discord
We are not aware of non-compliance with the OpenSSL license. We attribute usage of OpenSSL in our licenses page (along with all the other OSS projects we use.) https://discordapp.com/licenses - reproducing the copyright notice as required by condition 2 of the license:

> Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

As for condition 3, I am unable to locate any marketing materials which mention the usage of OpenSSL. However, if any such material exists that is in non-compliance we are very interested in ensuring that we are in compliance.

jhgg··on Guess I'm Done with Discord
Hey. I work at Discord - and actually, this system is a thing I work on - and code my team wrote caused your account to be locked. If my team is doing a good job, you won't notice us. If we're doing a bad job, you might get some spam, or your account may be blocked for false positives.

Discord gets a lot of spam. We've disabled, and/or challenged millions of accounts for trying to use our platform for unsolicited spam (trying to advertise their service, sex bots, crypto spam, etc...). Our anti-spam systems continue to evolve - just as the spammers who target our platform continue to evolve. The spam attacks against our platform vary in terms of how elaborate and skilled they are. Some are very obvious in terms of a detection perspective, and some are not. As such, we use a blend of signals, heuristics and machine learning algorithms to determine whether someone is spamming on our platform. Additionally, we look at where spam is originating from as an input to our heuristic.

One such source is TOR exit nodes - and as such, our system considers content created (DMs opened, etc..) from people using TOR exit nodes with more stringency than other sources. As such, if you are using TOR, it is definitely more likely that you may get challenged either via captcha, or phone verification. The system is definitely not perfect - and unfortunately in OP's case, it flagged the account for phone verification.

To address the 3 demands in OP's email:

> 1. Discord's anti-spam isn't so anal,

I'm not entirely sure what this means, nor what actionable steps I can take. You are using TOR, a source of a great amount of spam/attempted spam on our network.

> 2. my account (and other accounts in good standing and with proper 2FA) is exempt from such checks

Having 2fa is not a strong signal as to whether or not an account is legitimate. It is very trivial to automate setting up 2fa on an account. https://github.com/pyauth/pyotp can be used to both generate and validate 2fa codes. It'd be trivial to hook that up to the registration flow to enable 2fa - and if that was a way to 'bypass' our anti-spam measures, it'd surely be exploited.

> 3. I don't have to solve a Google reCAPTCHA for an account I have taken every step to protect against bruteforcing. Using Tor is not a crime; don't treat it as such.

Malicious actors constantly attempt to brute-force logins on our system - generally from public password dumps or other leaks. A lot of these brute-force attempts come from TOR, and other public proxies. In order to avoid information disclosure, we always captcha logins from these kinds of IPs, regardless of whether or not an account exists with the e-mail in question, whether the login credentials are correct, or there is 2fa enabled on the account. So, the "captchas" you notice are not really specific to your account, but rather, the origin of the login. Using TOR is not a crime, you are right - but - it's also our responsibility to our users to make it reasonably hard for their accounts to get compromised on our platform (even if they don't employ the best security practices - and reuse their passwords across the internet.)

Finally, I'd like to address: "Discord has shown to be hostile toward FOSS and privacy for a while now" and understand why that is.

As a company, we have tried to give back to open source software (either by financial sponsorship, or by contributing our bugfixes/changes upstream.) We also attribute all open source projects we use in our software here: https://discordapp.com/licenses. Additionally, we host many open source communities on our platform: https://discordapp.com/open-source. And finally, we try to open source software we make which may be useful to the eco-system in general: https://github.com/discordapp/.

As for privacy, we've stated that we don't sell your data. When you verify your phone number, we ONLY use it for the purpose of anti-spam, and it is never shared with anyone (aside from twilio, which sends you the SMS), especially for the purpose of financial gain. We're pretty up front about how we make money (freemium model: https://discordapp.com/nitro, in-app commerce: https://discordapp.com/sell-your-game). We provide privacy controls: https://support.discordapp.com/hc/en-us/articles/36000410991..., and allow you to request an export of all the data we have stored on your account: https://support.discordapp.com/hc/en-us/articles/36000402769...

I know this reply won't satisfy everyone, but hopefully, being truthful and upfront about this will help!

jhgg··on GCP to introduce an additional charge for publicly addressed VM instances
I think this nets to a < 0.01% delta to our bill. Is this really that big an issue in practice? For hobbyist projects, maybe? But for larger customers, this is a drop in the ocean.

I actually think this is a (somewhat) smart move. It incentivizes you to use a NAT gateway to egress traffic to your instances. You can either run your own fleet (it's very easy - but you gotta fix it if it breaks), or use their managed NAT solution (more expensive, but with less worry hopefully) - and not just tack on a public IP to every instance that needs internet connectivity. We did this from the get-go (running our own NATs) - and only have a few public IPs relative to the # of instances we run.

jhgg··on “Why was Rust chosen for Libra?” in congressional hearing [video]
I don’t think anyone actually runs nightly against what’s on the latest / master. I know at work, we use nightly rust, but we pin to a specific date release, usually when the around latest stable is cut.
jhgg··on 1Password: Standalone / Local Vault Option Gone?
The cloud component is only for synchronization of passwords between devices. No network connectivity is required otherwise.
jhgg··on Client side caching in Redis 6
Yes, and my point is, if the client is connecting to a redis server, it can just run MONITOR to see all the "sensitive" data you are talking about, and thus, this new feature poses no additional privacy risk. If you cannot trust your client to talk to your database, this isn't a problem with the database, but a problem with the client.

In your specific example, your client is also probably the one issuing the commands to store and retrieve the private messages in Redis - meaning, it already has access to all the data required for a breach. If you have an unrelated workload that is accessing a shared redis, but it does not need access to the sensitive data, perhaps, use an ACL to limit clients from the alternate workload to a different redis DB (which provides some level of isolation), or simply use a different redis entirely, and don't allow clients running the alternate workload to connect to the database that contains sensitive data.

jhgg··on Client side caching in Redis 6
What? MONITOR is a command like all other commands - you can run it from a client, or really, anything that can establish a TCP a connection to the Redis server. (Eg telnet.).
jhgg··on Client side caching in Redis 6
No more risk than using MONITOR.
jhgg··on Google Cloud networking issues in us-east1
Cloudflare took us down this morning, but also shielded us from the impact of this fiber cut, due to direct peering with google (I’m assuming over different fiber paths.)
jhgg··on Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
We do rollups into bigquery where we have a bunch of dashboards to look at stuff historically.

I did really like Kibana, ultimately, we had to ditch it (because of ditching ES). Of course, this was a good thing, as I more than once degraded ingest the ES cluster by just using Kibana to do some aggressive filtering. Clickhouse handles these without problem.

I think a more complete world view may be to pipe logs into kafka, and ingest them into Clickhouse/Druid for different types of analysis/rollups.

Our current logging volume exceeds ~10b log lines per day now. Clickhouse handles this ingest almost too well (we have 3 16 core nodes that sit at 5% CPU). This is down from a... 20ish node ES cluster that basically set pegged on CPU... and our log volume then was ~1b/day.

For more ad-hoc, we just use the clickhouse-cli to query the dataset directly. We are tangentially investigating using superset with it.

jhgg··on Show HN: Vector – A High-Performance Log and Metric Router Written in Rust
We use a rather bespoke syslog -> clickhouse log sink (https://github.com/discordapp/punt/tree/clickhouse) we wrote in house because logstash (and then subsequently elastic starch) was too slow. Would love to switch off of it and to this! Hopefully a clickhouse sink comes soon! Maybe will contribute one upstream!
jhgg··on Degraded performance on GitLab.com
Running Redis on GCP? I’ve seen CPU and latency spike on our production Redis nodes before - solely due to noisy neighbors causing memory bandwidth contention.
jhgg··on Show HN: Guardscript – Detect any changes made to your JavaScript files
>GuardScript is usefull when Subressource integrity is not feasible or not implemented, which is most of the time for SaaS services.

Can you explain what this means?

jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
This data structure has been operating in production for almost a year now, we have not touched it for any modification after writing and deploying it. We had 0 issues deploying it, and thanks to the comprehensive testing suite and benchmarks, we were confident it would meet the performance objectives. It was actually a drop-in replacement for the pure elixir data structure we had written. Deploying it was literally just a find + replace of OrderedSet to SortedSet in our code, running our test suite and then gradually deploying the new code to production.
jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
ETS has no facility to insert/remove an object and get the index at which the object was inserted to/removed from. This is needed in order to synchronize windows within the sorted set to our clients. We would have loved to use ETS, and use it in many places actually, and have open-sourced a few:

- https://github.com/discordapp/zen_monitor

- https://github.com/discordapp/gen_registry

- https://github.com/discordapp/ex_hash_ring

jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
Cartesian trees do not provide the ability to get items at arbitrary indices within the data structure in an efficient member from my understanding. In order to get the Nth item in the tree, a linear traversal is required. Furthermore, to get the index at which an item is inserted or removed requires the same traversal to accumulate the index.
jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
They are most definitely sorted lists. This is mentioned very clearly in the documentation: http://erlang.org/doc/man/ordsets.html

>An ordset is a representation of a set, where an ordered list is used to store the elements of the set.

jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
The cost of making a system call and network serialization + deserialization still makes this infeasible.
jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
On a given node, we maintain a few million sorted sets that see a query and insertion velocity in the orders of millions of operations per second. When we are meticulously measuring operation time in the microsecond range to meet a performance target, a networked solution is out of the question.
jhgg··on Using Rust to Scale Elixir for 11M Concurrent Users
One thing that our blog-post didn't mention too well was the need to be able to access arbitrary slices of data within the sorted set, as well as being able to know the index at which an item was inserted as well as removed. This is necessary for our usage of sorted sets, as clients can subscribe to a given window of the sorted-set (e.g. the top of a member-list), and in order to compute the delta operations to keep the client-side list in sync with the one on the server, we need this information.
← PreviousPage 8 of 17Next →