Why Discord is switching from Go to Rust (2020)
blog.discord.com
blog.discord.com
Every time I see a company post about how they’re changing programming languages, I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. But, of course, you’d be lambasted by your colleagues if you didn’t provide technical reasons, so you go and hunt for technical reasons that seem insurmountable and build a case for it, turning a blind eye to perhaps less exciting solutions that don’t require a programming language change.
(It’s sort of like how a cop in America can find you doing something illegal in any circumstance if they look hard enough. Except in this case, it’s a matter of finding technical deficiencies.)
This strategy highly depends on clout though, which is why I think technical arguments are frequently really a façade. If, say, a junior programmer doesn’t have clout, even presenting an equivalent technical argument won’t lead to a change in programming language, especially if it’s the senior’s favorite. I’m sorry to say, but most tech organizations aren’t meritocratic or egalitarian in this way (and perhaps they shouldn’t be).
As it pertains to the article—which again I don’t claim is all “fake” politicking (I don’t know anything about or anyone at Discord)—they mostly ripped on garbage collection. They specifically talked about how GC was bad because their LRU cache had to be scanned, which was part of why they observed spikes in their metrics. I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?
I think using any language with any serious and large project will find deficiencies in the language. At some point in a product lifecycle it’s more profitable to just work with and ameliorate those deficiencies than do rewrites where the new deficiencies will be completely unknown.
So far though rust-analyzer >>>>> JetBrains' plugin
Ah yes, because what I really want out of an ostensibly memory-safe language like go is to have to resort to C, or what amounts to C, to get anything done. Or I could use a language which doesn't have this garbage issue. But that would just be political nonsense apparently.
The article is about a very specific service handling state for if you've seen a message in a channel. It has nothing to do with processing user content.
"Principled policy and software design" is exactly what's needed to get rid of memory bugs in C too! So we can conclude that these things are impossible, or else we would not need memory safe languages in the first place. Since it's increasingly apparent that go requires these mythical safeguards after all, it's safe to say that it is not a memory safe language in the useful sense.
And here the cache is essentially the entirety of the micro-service:
> [The service's] sole purpose is to keep track of which channels and messages you have read.
so it's a giant map of (user, object) -> read flag or something along those lines. The DB commit and service input would be very light, and not really your main concern.
We hit the performance ceiling of redis, and rather than deciding to go with redis-cluster, and dealing with the headache of managing persistent redis, we went this way, so we could consolidate data storage to scylladb/cassandra (it was cassandra at the time of rewrite, scylla now since we've migrated our data stores over).
We have a bunch of operational experience with C*/ScyllaDB.
As others have said, however, a big factor certainly is that an internal lead developer lobbied for Rust, which they experimented with.
Plus most of the organizations still write code in matured languages such as C,C++ and Java.
Also mature languages, even though they're more stable, are just more of a pain to work with since they don't have a lot of the new shortcuts.
For something like Discord, Rust is perfect.
Just like how they did in this case completely switching from Go to Rust
I'm sure we're glossing over probably hundreds of hours of investigations by dozens of people.
The service itself is pretty simple, so it wasn't a gargantuan task. So it was either, rewrite this again in Go (using off-heap data structures to avoid GC pauses, which makes the software both harder to maintain, and also less idiomatic) or try a new language where we won't have these issues - where we don't have to resort to clever hacks to work around the runtime system.
Since then we've continued our investment in Rust, after success of this project (and many others we've yet to blog about as well.)
Maintaining a codebase written in C is far far harder than one written in Java and that's far harder than one written in Go.
Personally i think developer skill has far more to do with maintainability than language choice. And I have worked with a lot of shit code.
To get actual product work done with optimal efficiency, a mature ecosystem will always win.
It seems like everyone is just applying their subjective view. Which is fine. Just don't assume your equation works for everyone and every business.
C++, C and Java are old, so some mistakes can be forgiven. However there are problems with these languages that new languages have addressed.
Do you think a car from 1985 is better than one from 2015? The same applies to programming languages. Given the explosion of programming, it is absurd that the best ideas from PLT are not in mainstream languages.
A complete re-write (preceded by a very deep analysis) is exactly the opposite of productivity.
> problems with these languages that new languages have addressed.
And new problems were introduced... case in point: this very article.
> Do you think a car from 1985 is better than one from 2015?
Depends on the car...
I think this analogy would be better: "Do you think a hammer and screwdriver from 1985 are better than the ones from 2015?"
Compare Go and C++, and I bet C++ has had more language changes since Go's 1.0 than Go has. That makes large portions of C++ "relatively new and immature" compared to Go.
Furthermore, A lot of companies to this date have there performance critical code written in C++ , C or Java
Such as you can take HFT for example they use C++ for most of their trading applications instead of newer languages
I'm definitely not saying C++,C or Java is better than Rust or Golang each language has its own pros and cons
One of the big issues with Go is that it's got a steep learning curve. (There was an article about this posted on HN yesterday, in fact)
Rust, by contrast, is an extremely easy language to learn. Of all the languages I've picked up at least partially in my ~12 years of professional programming, Rust was probably second only to to PHP in "get up and start running with it".
If Discord is trying to pick up new developers, or cross discipline seniors to improve their talent pool, Rust is definitely a better choice than Go.
I wish we had a good way to evaluate this question in a quantitative way, but I haven't seen one yet. So reasonable people are going to disagree on the answers here.
I am sure rust is great but it is at the very least more difficult to learn then go.
I don't get Rust. I got Go very easily. Rust just doesn't make sense for me.
In Go I can just write down my thoughts and they translate to code easy. I don't even know which mental game I have to play in Rust. And all the conventions in Rust are another step. The final nail is the un-community of Rust. I still have no idea how to write a simple RESTful http server with a mongodb backend. I've asked only to have been met with downvotes on Reddit. The official MongoDB discourse didn't answer either.
Time and again I want to do something with it but every time I just quit frustrated and irritated of the complexity of everything in Rust land.
Think of Result as (foo, err) bundled into one thing. result.unwrap() is then equivalent to:
if err != nil {
panic(err)
}
if err != nil {
return ..., err
}
Don't get me wrong, there are complicated things in Rust, I just don't see how Result-based error handling is one of them. This is also not a new idea, many languages have something similar.
This is more about scale and saving computing costs, than proving out a business idea or implementing new features.
High performance systems in GC languages often preallocate or use object pools to reduce time spent in the garbage collector. LMAX's Disruptor, for example, allocates all memory at startup; no additional memory is allocated so the garbage collector never runs. Generational GCs make this strategy more effective because long-lived or large objects are traced rarely or not at all. Another option for trading systems is to run garbage collection after hours when the market has closed.
Go's GC is conservative, not generational, and has a hard coded collection interval of 2 minutes; you pay for a GC even if no memory has been allocated. There are only a few configuration options for Go's GC; if it doesn't work for your use case then you can either fork or use a different language.
Rewrites of services to decomp into different services, migrate to different datastores to scale up etc. is completely justified and is a good problem to have.
Scala has already found its niche in Big Data, streaming and distributed systems. Usually, the stack that you can find in my area, consist mostly of Java codebases, sprinkled with some Scala services that are purely data-oriented. There's a high probability that Discord is already using Scala.
It reads like a tool (Rust) looking for a problem.
I would say it's cool _because_ of the technical reasons, though! You make it seem like people are inventing technical reasons to make rust seem cool. I could make a long "technical reasons" list why I would personally prefer to work in Rust, but it boils down that it makes me more productive, and the code more performant with fewer gotchas and less sacrifices/compromises (compared to say C++ or C). Do I think Rust is cool? I guess... But that which attracted me to Rust was there already; I didn't have to make it up[1].
I understand not every language is for everyone. I've had colleagues dismiss rust simply because they prefer C++ (and we work with C++). Some will even agree Rust is better when presented with technical reasons, but still not like Rust "just because."
> I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?
I write Go intermittently since I participate a little in the Kubernetes ecosystem, and haven't dwelled that deeply into it. How would this be done without reaching for C, though? Or would it be fitting to swap languages in this "subset" of the implementation?
[1]: I feel like "make up" is the wrong phrase here, but I can't think of a better one right now. I guess what I want to say that Rust's "cool" reputation is well deserved. I don't think the technical reasons are made-up to serve as office-politics cannon fodder.
The most recent example was a replacement for a complex YAML DSL, served by spring config sever and interpreted by a python program on every instantiation. This thing was literally using 20% of our CPU cycles over the whole compute cluster and estimated to cost around €6k per month. Long story short I provided a fully working pre-processing solution with effectively no runtime cost, no dependency on the spring config server and a more powerful & type-safe DSL. It was dhall with a bit of bash and python, although you could also have used jsonnet or any configuration language. Another developer with more clout pushed me out in favour of their own solution, which I believe was to keep me away from his political pedestal. A few months later I left and from what I've heard the company is still burning 20% of it's compute budget on this...
Such a theory is more robust if it predicts how the organization will work in general. Thus one would also be partial to suspecting that neo-languages are declined for political reasons as well -- that a few powerful voices simply prefer to find other solutions, and then find post-hoc technical arguments.
Such a theory works less well if it is only used to explain some particular (ostensibly) technical decisions.
now its php->rust so a widely different migration but rust was measured for its latency cutting value, not just fad factor
I'd believe discord and others are on rust for this and not just the new shiney (that would be to their loss)
https://www.techempower.com/benchmarks/#section=test&runid=8...
High-level C# (asp.net) is almost twice as fast as GO in the benchmark... Rust is also in the top 3.
So, why should I use GO instead of C#, if ASP.NET/C# is so blazing fast and requires a lot less lines of code to achieve the same? -> It makes absolutely sense to switch to C# or Rust.
So these perfect situations where you only do the same thing and call same code over and over never happen; never!
Anyways, GO isn't popular for its RAW perf, GO is popular because:
- Simple language
- Native code
- GC
- Cross compilation
- Fast iteration (build/rebuild/deploy)
- Relatively tiny, statically compiled executable
Stop using serde. please.
And on top of that iteration time is hurt because of both the language, the compiler perf, and the lifetime problems that you need to constantly think about
The benefits of using rust on other hand is non-negligible, i wish the team would focus on solving that kind of immediate problems
So simple it can't do Object.keys()
Don't get me wrong, I enjoy writing code in Go, Rust, and TS. But my god, Go literally is a pain in the ass every time I want to do something that isn't a map reduce problem. And even for map reduce problems it's annoying due to lack of generics.
Maybe generics will finally give Go something that every other language has. A strong, useful, standard library for collections.
Simple on paper, but what's the experience as a result? some are okay with it, some love the restrictions and limitations, but i personally don't enjoy them
But again, i can understand the choices and the benefits
That doesn't sound right. .Net has good support for caching of generated native code. It's well ahead of Java in this regard.
You can use ReadyToRun[0] to pre-JIT portions of the code for faster startup. Full AOT compilation is in the works too[1] albeit it's still experimental.
[0]: https://docs.microsoft.com/en-us/dotnet/core/deploying/ready...
[1]: https://github.com/dotnet/runtimelab/tree/feature/NativeAOT
That said, C# and JVM are both mature and have decades of GC improvements, which could explain why golang is falling behind. Time will tell.
I use Go because its decently fast, I like the syntax, has a great stdlib, and I feel like it’s very easy to share a Go codebase with other engineers.
The other day we reviewed some oldish and complex code and it was so plain to see what it was doing and get back into the flow of that logic. I don’t find this happens with other languages as much.
I’m not saying Go is the right tool everywhere for everyone. I do love that about it though.
Code legibility is starting to become a primary if not the primary factor for a lot of software these days, because that's what allows it to be maintained.
I think more analysis should be done on why this is for go. Java and C# should be similarly legible, they are not complex languages either. It's possible that go encourages certain paradigms.
I have never had that same familiarity from Python, Ruby, JavaScript. I think there's something to be said about that in Go's favor.
That makes it very hard to read go code and build a mental model in your head.
Verbose does not equal complex usually.
Something that's a single method call on a built-in type may be 3 lines in Go. This doesn't make it hard to understand or keep in your head. "This iterates over a map" is the same regardless of the lines of code. You can very quickly skim verbose code, especially in a language like Go, where idioms and conventions are identical across many code bases.
Go's interfaces in particular feel like a really easy way to build clean, composable abstractions that make sense and hide complexity in the right way. The single-function-interface pattern is really one of my favorite things about Go lately.
It may take some scanning to go through all the dumb code, but you typically won't get fooled by abstractions. Go offers so little in terms of abstractions that you can usually see right through them.
This won't be everyone's experience, but it has been mine very consistently across internal and 3rd party code.
peope who say that they can just save money in their colo are probably never even working in the scale of discord, which is fine.
Perhaps Go folks should just use inline assembly to demonstrate how useless these benchmarks are.
These benchmarks aren't intended to show the performance of everyday code. They show what experts can achieve on each platform. That's why they take pull requests instead of writing the code themselves from tutorials.
I'd still rather see the popular way of doing things be the benchmark.
If we’ve learned nothing from Java and Python, it’s that managing runtimes and environment can be a huge pain point.
EDIT: swipe keyboard put ringtone instead of runtime and I missed it. Corrected.
This confused the hell out of me until I realized it's supposed to be runtime.
It's never been clear to me why someone would want this.
Python OTOH...
Not true since .NET Core 3.
$ dotnet publish --self-contained --runtime osx.11.0-x64 -p:PublishSingleFile=true
In .NET Core 3, you can build and distribute a single-file executable which, when run, will actually unpack an executable and a ton of DLLs into a temp directory, and run it from there.
In .NET 5 they changed it, so that when you build a "single-file" executable, you actually get an executable plus a few DLLs, which must be distributed together.
However, time to market is almost always a priority. Unless my margins are razor-thin, spending twice as much on servers for shorter development cycles is a net win in most cases.
aspcore 7,016,966
gnet 7,010,982
People share results of benchmark where some languages are missing.
In Go, the scheduler is smart enough to stop something even when it's not blocking on IO. This makes it more robust.
Of course, use what you need for when you need it. Rust definitely shines in some circumstances.
C# grew the `await` operator in C# 5.0, released in August of 2012. Go altogether hit 1.0 on 2012-03-28. If the one is new, the other is new.
You have to understand that the programming language is a product in same way as anything else and there are influencers, marketing bribes, paid advertisements, forum/hn/fb/... paid spammers, etc.
Just remember what spam spike the Rust (I am again not talking about rust as a language... just about the noise everyone made) had here when it emerged, they threw the shadow on the Nigerian spam, oh the amount of noise from Rust-Distributed-Evangelist-Task-Force (they will probably downvote me to hell for mentioning this :D), I had a lot of fun reading HN at that time, the most used word was 'and', the second was 'rust', it was so exaggerated that it was like watching Monty Python.
In today world the only review, or if you want, benchmark, is the one you make. Nothing else counts and if you feel that C# is faster then enjoy using it.
And quite frankly, who cares about speed today (except some very strange people like me), people are using very strange languages and want to run them on backend. And they actually do. Ignore the benchmarks, no one cares (except those that would like to advertise themself like "faster than C" :D). Just threw in some more vapor on the cloud and you have "fixed" the speed issues. /s
We're still using and investing in rust at Discord. For example, in the past year we successfully pulled part of the core messaging system out of python and into a rust service. Rust has been a boon for developer productivity and correctness of code.
We've also been very happy with the way the rust ecosystem is evolving and the amount of adoption the language has seen.
I'm happy to answer questions here.
P.S. We're still growing and hiring (including remotely). You can find my email in my profile.
We also combo Elixir NIFs and rust to speed up parts of the Elixir services https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
I'm really glad this blog post is being discussed on HN. I've learned a lot. But there are also a few foolishly critical comments. You and jhgg should keep in mind: Illegitimi non carborundum.
Cockroach also wrote about GC optimization: https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c...
For caching specifically there is this library as well: https://dgraph.io/blog/post/introducing-ristretto-high-perf-...
Nothing wrong with Rust, exciting project, but personally think Go is better for building performant micro services like this. Rust seems more suited for building browsers, drivers etc. Rust's performance is slightly better, but it's not as refined around go routines and developer productivity.
Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.
This is too cynical. A large re-write will not be approved at any large companies without millions of meetings and documentations filled with justifications.
I'm not saying that all companies do it this way, and that there is no "resume building" or "magpie-ing" by some engineers in many companies. But it is perfectly possible to state a thesis "Rust is better for our X and Y and probably for Z". and to then research and prove that thesis.
Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go.
Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands.
And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
BTW, here's some places I've spoken about Discord's Elixir usage:
- a podcast: https://smartlogic.io/podcast/elixir-wizards/s5e8-nowack-hei...
- interview with José Valim (elixir's creator): https://elixir-lang.org/blog/2020/10/08/real-time-communicat...
https://gleam.run/book/tour/try.html
It also looks like there are currently some serious limitations:
Besides, Elixir/Erlang has the best dynamically-typed equivalent of algebraic types (tuples + atoms + pattern matching) :-)
Not having to worry if this random reference you're using has been overwritten somewhere else in the code is absolutely refreshing.
It allows me to bring back the mental map of my code and the variables to the exact scope I'm writing in.
This is an underrated feature of rust. Especially for newer devs who do not have that mental map of the code yet, and are worried that any small change they make might cause cascading failures. That's just something you don't think about in Rust.
But you couldn’t help yourself even after being corrected by someone who knows what they’re talking about.
That's an unusual claim :) Could you elaborate? How is the learning curve, especially for fresh graduates who have been trained only on languages like C, Java and Python?
> The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
Agreed. I like Go, but that's big drawback of goroutines and threads.
Frankly, I don't consider myself a particularly good programmer - but when I use Rust, I feel confident because of being backed by the compiler. That's something I can't say about any other language, and I only have less than a year of Rust experience compared to 5 years of total experience programming.
Imo Rust is easier to master than other languages, but it's definitely harder to do basic stuff in it because of the upfront costs I mentioned.
I have worked with multi-threaded C++ code long enough to see that it's impossible to work in a team while keeping the code safe. Usually servers crash and canaries catch it in production, but if we have type system improvements that solve these problem, why not take advantage of it?
> Rust moves many potential program problems from runtime to compile time; it has a high bar to meet, but once you meet it the program is rock solid
And then somebody replies with:
> Oh my favorite language doesn't make me do all that work up front, therefore easier (TM). Why Rust so hard?
But the reply misses the point: Make no mistake, you're still paying the cost (with FASTCASHNOW-interest rates), but the cost is just smeared out over days or weeks as you fix, debug, and tune it during use. You may choose whether to do that work up front or while the program is in active use, but avoiding the work is not a choice.
One way to look at it would be that Rust forces you to conform to a specific ownership model, and makes you be explicit about resource management and how things are shared and accessed. Having to do all that can make certain patterns awkward or hard to write, especially for those new to Rust who haven't internalized what the compiler wants yet.
But on the other hand because so much is explicitly spelled out it can end up reducing the amount of things you need to consider or keep track of while reasoning about code, making it easier. Especially for code which is essentially bolting libraries together with a bit of data transformation.
It's really... not. For people that have actually written big things in Rust. We keep repeating it, but we try to cautiously, tread carefully, lest we be accused of being part of this dreaded Evangelism Strike Force we keep hearing so much about.
Ease of use != ease of learning.
Rust is hard to learn (unless you have a deep ML/Haskell + C background). But once you're up the curve, it's a very expressive + productive language. Refactoring and maintaining large projects confidently is a breeze, and creating powerful abstractions that can be safely leveraged by large teams of programmers... just works.
Rust is still a very low level language, but if you like Haskell, you won't go into anything else when writing some low level program.
When I looked at it, the favourite quicksort example was so elegant, I wanted to learn the language. Then after I started looking at the run time (after wasting a lot of time going through learning Haskell), I realized that it was all a lie, ant that sort wasn't quick at all.
I don't see the same overpromising problems with Rust: you usually get the performance that you think you will get when you're writing the code.
The default libraries are extremely inefficient.
Generally if I use Python/Ruby I can get a 50x slow down that is accepted, but with Haskell it can be more, as it's very hard to think about what the compiler does (memory usage can blow up easily).
(Also the fibonacci sequence using 'zip'. Blew my mind.)
Haskell folks are fairly upfront about possible performance issues. Lazy evaluation comes with its own set of tradeoffs.
You can write very inefficient code in C++ if your algorithm isn't efficient.
,,The purity of Haskell code makes it easy to fuse chains of functions together, allowing for performance benefits.’’
Having something ,,beautiful’’ and inefficient is the problem I’m writing about. The language itself is guiding people to inefficiency.
I haven’t felt this with C++, so I’m not sure what you are talking about: usually simple / pretty code runs fast there, and more complex code runs slower.
Haskell can have a reasonable high performance, but if it's you main concern, or if predictability is important, you need a low level language.
We call this project farming where I am. I'm not a dev and don't work in IT industry, but I do work for a large company and every few months some new project is rolled out to 'improve' things and most of the time we all collectively groan and wait for the person to get promoted.
The most frustrating part is when once in a blue moon the project turns out to be useful, and the person gets promoted off of it and abandons the project. Nobody then wants to pick it up since they won't get any credit for running it and it ends up getting abandoned.
In other words, discard the technical arguments laid out in the article and just speculate that it was an HR/management decision.
Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense.
Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite.
[0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
From the essay, the issue discord had was that they were not triggering enough GCs. The ballast approach serves to tune the GC frequency down.
In Discord's case they were getting screwed by a periodic mandatory GC trigger.
> along with smaller cache's per service
They did use a smaller cache, but that increased their 99% since… there was now less stuff in the cache.
Now, it's completely the opposite.
Now, when a company successfully leverages technologies as a competitive edge and becomes a multi-billion dollar success, the thread is dominated by stodgy criticism for not picking the absolute most mainstream technologies possible.
Big takeaway here for those who are married to their tools. Rust happens to be just another tool for this company, and it works well for them; where it makes sense of course.
Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users?
Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!
It's really weird that such a game adjacent piece of software completely hardcodes shortcuts, especially when so many of them either conflict with or are near-misses for common chords, especially for very disruptive actions like starting voice calls. Even more so as it already has configurable keychords, just not for most of the useful stuff.
Also they should fix the labelling around these, either it's just not working or it doesn't work reliably, but trying to override built-in shortcuts via the configuration (in order to disable them) has never worked for me.
High-contrast helps the rest of us, too.
a one line patch in main() with something like
ballast := make([]byte, 10<<30)
might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so.
https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
[0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430
No. Twitch's ballast is because
> At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute).
Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below.
But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.
The past color the idea that a language switch is a waste of time, because in the past the choice of languages was from worse to bad or for same to same-ish.
But MODERN languages have something that make this compelling: Actual good practiques baked in the language, as defaults.
---
So, if a codebase is in trouble you can:
- Try to refactor it with "better" idioms...
but if this is not the norm in the language/ecosystem you depend in discipline, and that have proved to fail at scale.
- Change to "shinny lang" where do the right thing is the default...
And now if the developers try to do wrong, them will fight against the current.
---
Switch to Go is a great way to solve a project with concurrency issues: Go is good at that, and make a compelling case of build good concurrency apps. Similar if switch to Elixir/Erlang.
The case of Rust is similar: it make the "best" practiques the default, and very painful to do wrong (and sometimes to do right, but this is not the focus here :) ).
So, a language switch is the ultimate clean up for a project.
I have done several in the past (I'm contractor) and let me tell you: Any codebase I get is in deeeeeep trouble.
The best results have been a language switch, when I move from "bad" to "better". Just the fact the new language have something that is truly better make the move great.
Now, the opposite is true: If you are switching and NOT because the new language have that something that is truly in need or worse, is a language with the need of a LOT of discipline (like C, PHP, JS,...) then yeah, is doomed...
After reading this, I still don't understand. But I find https://codereview.appspot.com/5451057#msg38 particularly entertaining: there was going to be an environment variable to control the interval, which might have allowed Discord to keep using Go, but Russ Cox shot it down.
I still don’t like Electron, but Discord is doing a good job with it. Even their React-Native mobile apps feel decently snappy (where usually large React-Native apps are buggy and slow as fuck).
I always find it bizarre when python (and to some extent java) are lumped in as languages that you found Go or Rust a suitable replacement for. They're just such different ends of the spectrum - Go/Rust will just never be able to match the expressivity and rapid development ease of python because they have such different design goals.
If you find porting a production application from python to Go/Rust a satisfactory outcome, then I suspect you should never have been using python for it in the first place or you've been using python All Wrong.
Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.
Just to check that I understand what you're suggesting here, this is comparable to how we embedded systems folks might pre-allocate (either statically or at the very start of a program execution) a large chunk of memory and use it instead of using traditional malloc/free (or language equivalent), correct? So you bypass the GC and avoid any potential performance penalties introduced by having a smaller number of massive allocations that never get moved, freed, or reallocated.
this sounds like a recipe for disaster in every other company i have worked with. are Discord's engineers just THAT good or is there something else going on here?
Here I would recommend the following twenty year old article by Joel Spolsky, and Joel was merely reciting knowledge that was already well understood, but in an entertaining way.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Why Discord is switching from Go to Rust - https://news.ycombinator.com/item?id=22238335 - Feb 2020 (640 comments)
also https://news.ycombinator.com/item?id=23874336 - July 2020 (7 comments)
That all said, I've noticed that people here are really off golang in favor of rust. I believe I sort of understand. I've recently switched from python to go at work, and I frequently bring up why I like it so much (caveat I'm overly enthusiastic about the things I like related to my work). But again, that all said, at the end of the day, both Python and golang will definitely get the job done.
I also wonder how well Rust has continued to work for Discord, if it's been a problem to keep up with the rapid movement particularly in the async ecosystem in the last year or so.
Go 1.12 might have addressed this particular issue [0, 1], but it's possible that that came out too late for Discord to try before deciding to rewrite the service.
[0]: From Go 1.12 patchnotes: "Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live."
[1]: https://old.reddit.com/r/golang/comments/eywx4q/why_discord_...
> The service we switched from Go to Rust is the “Read States” service.
And the conclusion doesn't indicate "switching", but rather Rust is an "option" for Discord, and already used in the stack:
> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.
- Real-time stuff (websocket gateway, etc.) is Elixir with some Rust NIFs
- REST API is Python
- Other stuff is Rust or Python afaik? Probably wrong here
Obligatory "I don't work there"
Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.
But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)
You can solve the first hurdle with better hardware (for the same price as our usual specced macbook pro, I opted to custom build a beefy workstation with a 3790X).
The second, you just solve with time. I remember when I first started with rust I hated it. I was working on a native project, and felt like it would have been easier to do in C++. But internally now we have more mindshare and knowledgeable folk who understand Rust - this has helped tremendously in onboarding engineers to the language. Also, having a working editor (via rust-analyzer) and the quality of compiler error messages have improved significantly in the past few years.
In terms of immature ecosystem, I think the rate at which the ecosystem is advancing is incredibly promising. Tokio hit 1.0 in Dec 22 of last year, which will finally put an end to the dependency upgrade thrash we've had to deal with. Going from tokio 0.1 -> 0.2 -> 0.3 -> 1.0 has been a journey. But with 1.0, finally, libraries will have a stable API to target, so we won't have to worry about this much for a while. Additionally, in the following months we intend to start funding critical open source rust projects that we use to ensure their success.
https://stackoverflow.blog/2021/02/22/choosing-java-instead-...
I do believe that there is nothing better tuned for performance vs developer productivity than the JVM.
The language semantics may be a question for personal distaste..but then you do have Kotlin (https://jooby.io/ ) which ranks fairly near the top on techempower benchmarks https://www.techempower.com/benchmarks/#section=data-r20&hw=...
Coming from scheme where everyone and their dog has an R5RS compatible REPL that's pretty disturbing.
The essay is about Go which is pretty much in the same position: gccgo is a non-entity.
> and the singular complete one isn't even self hosting.
What are you talking about? Rust was self-hosting before it was even stable (a common source of problems as the language can then get optimised for implementing the language, thankfully it doesn't seem to have harmed it too badly).
Unless you have an existing code base or a necessary integration or requirement that demands it, I honestly wouldn’t start a C/C++ project in 2021 all other things being equal.
Discord rewrote this service in Rust because they had already had success using Rust for some performance critical services; it seems likely that they chose to invest in Rust and not C++ for those services because their developers pushed harder for or were more interested in Rust.
C just doesn't offer the same feature set at all.
This brings me to my next questions C++ has been there for a long time compared to Rust but still people decide to go with Rust
Just curious to know
Also, good C++ devs are decidedly not easy to come by. Especially when you get into performance sensitive work, they can command huge salaries; and a bad C++ dev can do more damage to a codebase than any other language I know of.