How Discord Scaled Elixir to 5M Concurrent Users
blog.discordapp.com
blog.discordapp.com
If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites.
Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level.
Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions.
I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!
Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).
That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).
But then the question is... how do you find high quality libraries without something like download count? The issue is signaling good libraries, with poor discovery. I subscribe to many "libraries worth looking at" mailing lists, the awesome lists, etc. but that is still a pretty coarse net. Searching for libraries on GitHub if you don't know the name already or don't really know what you're specifically looking for is an even worse experience.
Combine that with relatively poor fuzzy search on Hex, where even if you DO know what you're looking for, it leads to a very subpar experience. For example, "e-commerce" leads to different results than "commerce". [0][1] These then exclude [2], which is an ecommerce library available on Hex without that keyword.
[0] https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sor...
But regardless of how good search is it wouldn't find ryal_core when searching for "commerce" if that word or any similar is nowhere in the metadata. The description for the package ryal_core is "The core of Ryal." which doesn't do the search any favours.
I think indexing GH READMEs would go a long way to helping the discovery problem, perhaps utilizing the new GH tags thing in some capacity, even letting authors tag their own packages. That would help in this specific case though this is of course a pathological example I happened to find recently. Maybe also having a "people who have X as a dependency also tend to download Y", though I'm not sure if you have that information outside of direct dependency graphs on Hex packages.
I feel like many newer languages have this problem, but the weird thing about Elixir is the 20-30 years of Erlang libs that are out there that are great that people have issues finding. It's worse if the libs aren't on Hex, and anecdotally I think people have been moving to Hex more lately for Erlang libs.
https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468...
Building a solid standalone monitoring/metrics package is apparently next up on the "todo list" for Chris McCord (creator of Phoenix): https://www.youtube.com/watch?v=pfFpIjFOL-I
Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language.
About the only thing I found irritating was a lack of language support for mutable hash tables, but I thought they fixed that at some point.
And Erlang's bit packing/unpacking borders on magical in terms of expressiveness and speed. I wish other languages would adopt it.
I think it's a lot harder to sell people on a language that looks syntactically unintelligible to a lot of web developers. Most people I know working with Elixir now are/were Rubyists, so it's easy for them to make the leap from that and start to understand all of the awesome things they get from BEAM and OTP.
Then again, FP for me was not difficult at all, and actually made several of the issues I had with my Ruby code not being expressive/obvious enough, or difficulty in composition, disappear overnight.
I've noticed that pattern-matching and guards just completely eliminate a stupid* amount of boilerplate logic in Ruby code
* "stupid" in hindsight, of course
With that said, I could write basically anything Elixir gives me with Ruby. The real value I get from elixir is the blazingly fast performance of BEAM and the speed at which I can build complex and stable distributed systems with OTP.
... I'm not like that. I took one look at Elixir and said "HOLY CRAP IT'S A FUNCTIONAL SCALEABLE RUBY!" (of course that was merely scratching the surface, but...)
Robert Virding, a co-creator of Erlang, creator of LFE, also wrote Luerl - Lua implemented in Erlang.
So if syntax is important you have many choices to use the BEAM VM/OTP in a Ruby-like language (Elixr), a Lisp (LFE) or in Lua (Luerl)!
I am now looking at Robert's videos of a game whose logic is written Lua, but it runs an Erlang process for each ship of thousands he spawns in the demo. Pretty cool. There's even Torchcraft - Luerl and Torch ML for learning AI/ML in Starcraft.
I still prefer LFE because I like Lisp syntax, biased as yourself about such things, however, I still think Elixir macros are not as far reaching as LFE's since they are handled after parsing in Elixir vs. LFE [1], although I wouldn't mind being proved incorrect on this.
[1] https://groups.google.com/forum/#!topic/lisp-flavoured-erlang/ensAkzcEDQQI think this kind of attitude is unhelpful at best, and alienating at worst. There are a ton of "serious" programmers, myself included, who care quite deeply about the enjoyableness of the tools we use, and aesthetic/expressive qualities absolutely come into that.
I believe the recent popularity of elixir kind of proves the case. There are many improvements to the package managers & tooling, etc, but the most obvious is to the syntax - which transforms what previously seemed unapproachable into a genuine option. Dismissing any and all such interest as merely "superficial" seems uncharitable, to say the least.
It's not. Not for me at least. I prefer Erlang. Maybe I am strange like that.
I found initially the core concepts are hard - that is using using processes for concurrency, functional (immutable) data structures, functional patterns like recursion instead of for loops, the library ecosystems those are harder things. Those are the same in Elixir as well.
Erlang the language itself also simple. Think a bit like C and C++ if you're familiar. Erlang is like C, the language spec is small. Elixir has additional features which make it more expressive but also more complicated (macros, pipes). It's a bit like C++ having templates and classes. You can do more and with them, but it's a also a bit more to learn. I am exaggerating as Elixir is a lot more elegant and consistent than C++, I am just using the analogy to illustrate the idea of simplicity vs power.
It is now getting better. But it was a really steep curve to adoption.
Call me unserious or shallow or whatever, but for me - it was a tiny bit about the syntax. More than a tiny bit, actually. The other things you mention, all valid, were just additional barriers to something already fairly aesthetically unpleasing.
It's improved and improving rapidly, as you say, which is really great to see.
The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.
Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.
I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."
Under what circumstances does it get triggered (since you seem to be more knowledgable about this than I am!)? I expect records would not fall under this.
But it doesn't help with cases where you are constructing a term (dynamically) in a process and sending that term. There is more meat in the blog post of mine: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio...
Distributed Erlang/Elixir has known limitations. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.
The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow is a feature.
And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.
https://www.manning.com/books/the-little-elixir-and-otp-guid...
A better architecture for a distributed system has a strong composability property. It should be possible to modify every possible aspect of it on a running cluster without introducing downtime.
Write your own standard well documented distribution layer and become independent of underlying technologies.
1) you are misunderstanding the grand parents points. Better to have primitives that scale you to X rather than having to get to X on your own.
2) The grand parent wrote the Elixir language.
3) Making sweeping statements like "A better architecture for a distributed system has a strong composability property" is very easy.
For a fast time to market, I do agree with you. But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.
> 2) The grand parent wrote the Elixir language.
I didn't know him, nor I care.
> 3) Making sweeping like "A better architecture for a distributed system has a strong composability property" is very easy.
I'm considering this as an excuse, than a technical argument.
This requires having distributed system experts on your team or having the money to call them from day one, before you even reach the market and before you are even sure you will have custom needs. If you have the need, the expertize and the time, then surely. But writing a distributed system from scratch should certainly not be the first choice (IMO).
You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.
How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.
And what about process placement? On which side of CAP do you want your registries to sit?
If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.
And most times, it will be good enough.
As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.
But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear search is too slow and needs to run faster will be disastrous to your architecture. Likewise, assuming communication is free is equally disastrous.
The problem with building a distribution layer in another language is the effort involved. Most of the companies which pull that off and get stability in addition are usually large, multinational, and has ample amounts of engineering resources to throw at the effort required.
You can move it out just the same as you can elsewhere, you just add an unnecessary layer of complexity and lose a lot of the capabilities already built into the language. It boils down to a simple question of, what would be the purpose behind that decision? What improvement would you get from deciding to move any of that elsewhere? Do you gain speed at the cost of complexity?
Clustering is a non-trivial problem. It can't happen naturally in any language that includes mutable data without explicit oversight and thus most of the needed tooling can't be built into the system and guaranteed to work everywhere. As soon as some part of the system goes from "send message, get response" to "send message, change variable referenced in memory on this machine, get response" you break your ability to naturally cluster.
That leads to a dependency in central relay points for things like setting up web sockets and then having code behind them communicate. The code behind them in whatever language is usually going to be talking to other central relay points like load balancers or pub sub databases rather than directly to other specific servers.
As soon as you have central relay points, they become something else that you have to monitor and scale even if they are very high volume.
There's nothing wrong with this approach and it will certainly work but if you want to avoid that and build a distributed system in another language, you've got to invest a lot in other areas.
It is supposed to be easy. But they are not using language agnostic messaging. Instead they rely on Erlang distribution protocol. This is what makes it difficult to integrate.
> what would be the purpose behind that decision?
1. To achieve composability. I can replace one web socket server with a C++ implementation to see if I can handle 10 million connections per server. If I fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm moving to native code. The beauty of it is that I can do it one step at a time. Even I can implement web socket layer in multiple languages and experiment all possible options in production and no one even notices it.
2. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5 of them to write the core and the rest of the team NodeJS guys.
If you're going for your C++ layer, try https://github.com/saleyn/eixx for example.
That said, good Erlang architectures definitely knows when you should fan out and use a common layer for distribution. At $work we have a Kafka cluster for some event handling simply because it is the right tool for the job, and because it makes it easier to interface other programs on top of it.
In my experience, the distribution features are best used as orchestration however. Your distribution layer is Erlang, and then you use other languages for "leaves" in the architecture. If the right library is present in e.g., Go, then by all means use it. Erlang easily speaks protobuf for instance.
Sure, you want a beautiful lover, but is your object of desire within your reach? One needs to be practical when playing engineer.
Let's try this: Composibility is a highly desired property for distributed systems however correctly scaling composible semantics is a hard problem.
This sort of duality props up all over the place in general engineering, and certainly in software systems engineering.
In truly amazing open source fashion, they also made some libraries for other companies to leverage!!! Super big props to Discord for that one. Seriously can't thank their team enough for going above and beyond.
Fast forward several years, and isolated processes turns out to be a great fit for large SMP systems, and Erlang/beam is now doing signal path work in a lot of places. Erlang tends not to put explicit limits, but some techniques are going to fail at large scale; ex: if you have 50,000 processes across many nodes, sending the same message to each of those processes is going to be slow; sending one message to each node and fanning out from there is going to be faster; in no small part because you've reduced the network bandwidth you're using.
The nice thing when you hit Erlang scaling limits is that almost everything you need to fix is going to be in a pretty simple state. You're not going to find many things that are layers of optimizations on top of hacks on top of optimizations --- they do a good job of keeping things simple, and not optimizing until it's needed (and even then, they usually pick simple optimizations). Keeping things simple goes a really long way (especially with today's enormous servers).
Edited to add: I don't think they've even needed to tweak the vm yet either, just their user space code. That's pretty huge too.
Sending messages to processes on other nodes has the same syntax as sending to a process on your node, which makes it easy to run a distributed system. (Ports are different, you'd have to setup a proxy process on the remote node in order to send/receive from that).
Of course, with the base of process to process messaging you can build a higher level messaging queue (see RabbitMQ for a popular message queue built in erlang).
We haven't really had to. Really only args we use are "+sbt db +zdbbl 32000 +K true" and increasing the default process limit.
Give that the whole selling point of Erlang/Elixir is scalability at the price of the rest, the article is really telling me to avoid the tech.
They likely have much more than 5 million users.
To put this amount in perspective, you get 5 million connections after receiving 3000 connections per second, which are never dropped and remain connected, for ~28 minutes. The majority of websites do not get even 300 requests per second.
Few projects actually need the level of scalability that Erlang brings, and the cost of building and running an Erlang project as an enterprise application is non-trivial. People need to understand this if they want Elixir to succeed where Erlang hasn't.
In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even better, you can often postpone errors in a system which are non-fatal. I often wait until I have a couple of bugs in a system and then I'll work on that system for a few days fixing them all. In systems where even the smallest amount of brittleness destroys them, this isn't really an option.
Another experience of mine is that Erlang systems tend to take a bit longer to write compared to cranking out a fast solution in another language. But the payoff is that your Erlang solution tend to have better robustness as load increases and people start using the system.
The community is small however, so you'll need to write more code yourself in-house as a result. For some problem spaces, this will hurt a lot because there are off-the-shelf solutions in other languages. On the other hand, larger projects which are more specialized can benefit from you knowing your own code better.
Scala on the other hand is not remotely small and easy to understand.
However, the BEAM is not very fast on executing actual code, it is like Python in that way where it is good to slave out CPU-heavy work; the BEAM is built for async IO, and yes, running a JVM (or C++) system as a node on a BEAM mesh is fantastic for that (or a port, or NIF for small work).
My only concern is their long term viability and I don't just mean money wise. I'm concerned they'll have to sacrifice the user experience to either achieve sustainability or consent to a buyout by a larger company that only wants the users and brand. I hope I'm wrong, and I bought a year of Nitro to do my part.
But will you pay for Discord though? The features and quality Discord is able to provide are artificially propped up by VC funding. When it runs dry, we will be left with open source offerings, or the next product to take it's place and repeat the cycle.
But they seem to refuse suggestions to do so continuously, so they seem to have some business plan somewhere.
Also, we're looking for Erlang folks with payments experience.
cGF0cmljaytobkBmaW5peHBheW1lbnRzLmNvbQ==
I also see a lot of job postings asking people to decode some string for bonus "points"... 90% of the time its just rot13.
It's not really a superset. The JVM has a single global heap, whereas BEAM has per-process heaps. Global state is the enemy of concurrency, and the best GC algorithm is one you don't need to run at all. The "several hundred millisecond HTTP response latency and horde (sic) memory" problems you note are in part a result of having a global heap.
Of course I think part of the problem is most people in Java land just don't have the 5m concurrency requirements so it doesn't get the love it probably should.
yeah this is pretty much exactly why I turned away from Clojure and Scala and went for Elixir. Also, super ugly Java stacktraces that were not hard to trigger at all. The vast majority of errors in Elixir are wonderfully explanatory.
Thankfully, Erlang and Elixir tend to make this a lot less painful, in no small part due to their respective declarative/functional programming traits.
Both VM are completely different beast.
Erlang is one of the few language that does Preemptive scheduling.
https://jlouisramblings.blogspot.com/2013/01/how-erlang-does...
It was built from the ground up to do concurrency in an arguably superior way than Java. Java can't do this.
edit:
Wow Erlang have a really active community from the notable people that are commenting here.
To be fair, there's no reason JVM servers can't have sub 10ms responses, that sounds like a problem at the application level like you mentioned (spring). Nothing wrong with a JVM language if it solves your problem.
It's not a superset until it has a non-sharable memory heaps between threads, complete and easy hot code reloading, dynamic tracing (being able to log into a node and update code at will as it the application is running).
The safety and fault tolerance is the #1 advantage Erlang has and that it's hard to get with other frameworks that claim to be Erlang-like. Almost all of them focus on "We have a thread and a queue and we send messages between them so we have 90% of Erlang but faster". Sure they can spawn OS processes to get the same effect or even whole servers but it's more awkward and can only be done so many times before memory or CPU resources are exhausted.
Oh you can think of it another way. There is no point in having a distributed system with 5M concurrently connected users if it crashes 2 or 3 times per day and it has to be restarted and all those users lose their connections. So as the system gets more distributed and more scalable the fault tolerance aspect starts to move to the front alongside speed and performance. And that's just where Erlang starts to shine so to speak.
The first two I can definitely see -- particularly for robustness and debugging -- but I'm a bit surprised by the last one. Do people actually really log into running production systems and update code like this? It seems like it would be an incredibly dangerous thing to do. (Akin to using direct DB connections and typing in DELETE statements directly rather than e.g. putting them in SQL scripts first.) It could potentially also make it extremely hard to know what's actually running in production.
Yes, I've done it once in a while. Cases could be is to deploy a fix and the customer's system is up and running. If say it's something urgent that can't wait until it goes through the full deployment pipeline. Because hot code reloading works so well in Erlang it's not risky as doing it in Java for example.
In fact upgrading by hot code reloading is also a common thing Erlang world. So there are cases where it is done routinely. It takes some preparation and so on:
http://learnyousomeerlang.com/relups
Another case is if you see an issue happening but don't have enough logging or tracing ability in that part of code. You can upgrade the code with an additional log statement or save extra info to a file for debugging. Then remove the patch. The alternative is to try to replicate that on a separate system which sometimes might not be easy - don't have the exact access pattern, exact data and other factor that that would duplicate the original environment.
But you're right doing it haphazardly and just sprinkling hot patched code updates everywhere is a path to disaster. So it's possible to monitor and record these updates to them them visible and managed better. It's up to the team / organization to handle that.
The bottom line don't do it routinely, but when you have it can really save the day. And it's something that many (most!) frameworks / runtimes / languages don't support as well as Erlang does.
Thanks for the perspective!
What many people may be interested to know about Erlang is that you can log in to a production system, start a new shell running a tracer that listens on a localhost TCP socket[1], and use the dbg module in the production VM to trace calls and messages (and more) between any functions in any processes - in the running production system - and send them to the tracer node.
Done judiciously, the overhead is negligable, and the benefits are great. You can zoom in on bugs in real time.
I find the syntax of dbg match specs to be ugly, but it has saved my bacon so often it is so worth it, and it doesn't get mentioned that much, even though I feel it is almost as much a superpower as hot code loading.
[1] You use the separate shell to avoid accidentally crashing the production VM; if you do something boneheaded in the port-based shell, you can kill it and the production VM will just stop sending trace data to the dead TCP socket.
That's not the platform's problem, you're doing something wrong. I run my web services on Spray[1] and they start in maybe 10s, average HTTP request latency is 4ms, and they run forever at ~100-200MB RAM. That's without putting any effort into tuning.
[1] now replaced by akka-http but I haven't ported yet, and it works
This is the classic case (and I'm/we are guilty of it as well) of Java apps that are probably traditional monolithic apps that use frameworks with extensive reflection and class loading (ie spring component scanning).
You can get ridiculously fast loading time with Java if you use Dart2, a reasonable web framework and no ORMs. I'm talking 500ms... sort of depends on your JVM settings. Sure not 10ms but still way faster than 2 minutes.
How often are you booting your webservers that 500ms isn't fast enough?
> mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data
There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node.
Jesper L. Andersen (jlouis) talked about it in his blog: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio...
> After some research we stumbled upon :ets.update_counter/4
Might not help in this case but 20.0 adds select_replace so can do a full on CAS (compare and exchange) pattern http://erlang.org/doc/man/ets.html#select_replace-2 . So something like acquiring a lock would be much easier to do.
> We found that the wall clock time of a single send/2 call could range from 30μs to 70us due to Erlang de-scheduling the calling process.
There are few tricks the VM uses there and it's pretty configurable.
For example sending to a process with a long message queue will add a bit of a backpressure to the sender and un-schedule them.
There are tons of configuration settings for the scheduler. There is to bind scheduler to physical cores to reduce the chance of scheduler threads jumping around between cores: http://erlang.org/doc/man/erl.html#+sbt Sometimes it helps sometimes it doesn't.
Another general trick is to build the VM with the lcnt feature. This will add performance counters for locks / semaphores in the VM. So then can check for the hotspots and know where to optimize:
http://erlang-in-anger.com/ for introspecting it in production
Seriously, though, I might need your services in the future, if you're available.
Also, I'm pretty sure the security of your career is pretty guaranteed at this point lol (looking at Indeed data, interest in Elixir has risen 20fold in the past 3 years and the slope of that line is far steeper than all other languages in that space)
However you can subscribe to the Erlang mailing list. That's often a good place to start for tricky or interesting questions you might have:
http://erlang.org/mailman/listinfo/erlang-questions
To contact me directly check my profile.
Cheers!
if send/2 takes 30us to 70us, I'm guessing blocking as well, either on distributed communication or something else along those lines. For local message passes to take that long, my something-is-amiss-sixth-sense is tingling.
Ah good point. I didn't look at the code much. I was thinking of cases of passing any of those literals in gen_server calls and such and just getting extra performance from upgrading to OTP20 as a side-effect.
WhatsApp + Erlang was one of those cases (watch this talk and imagine trying to recreate that system with only a handful of server engineers using any other tech: https://www.youtube.com/watch?v=c12cYAUTXXs). Discord + Elixir appears to be another.
Curious if anyone has any examples that spring to mind from outside the highly concurrent messaging space.
It is probably not a solution for current Discord as they rely on linearizability, but I toyed with building an IRCd in Erlang years ago, and there we managed to avoid having a process per channel in the system via the above trick.
As for the "hoops you have to jump through", it is usually true in any language. When a system experiences pressure, how easy it is to deal with that pressure is usually what matters. Other languages are "phase shifts" and while certain things become simpler in that language, other things become much harder to pull off.
That is cool trick though. So it's basically sending the port itself around and changing its ownership, with something like port_connect(Port,NewOwner)?
And btw, thank you for writing https://www.erlang-in-anger.com and http://learnyousomeerlang.com !
The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly).
Also recently I've gotten http://propertesting.com/ out, you might enjoy it :)
> Also recently I've gotten http://propertesting.com/ out, you might enjoy it :)
It might be just what I need to understand and start using property tests. I've tried twice and gave up.
Oh and recon! Thanks for that too. Use it almost every day.
[0] https://github.com/elixir-lang/elixir/releases/tag/v1.0.0
That's why I'd like to hear more about productivity and ease now. Is it faster and more fun to scale things in certain languages then others. Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and procudctive way to scale, that doesn't matter.
Anyways, learning how team scaled is interesting, but it's clear to me now languages aren't limiting factors to scale.
Just like any language vs language debate each one has benefits for various particular use-cases. Any meaningful comparison of languages must be prefaced with the use-case scenario.
One of the strongest use-cases of Erlang/Elixir has always been building large distributed apps that need to scale (async web apps, telecom, chat servers, messaging mobile apps, etc). The ability to build these large distributed systems are baked into the very primitive parts of the language and standard library - to a degree that few other languages can compare to it, if any.
With Erlang/Elixir you design ALL applications in a way where scaling is rarely an after thought but rather a natural extension of the program.
> Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and productive way to scale, that doesn't matter.
People often make the mistake of trivializing Erlang/Elixirs as merely programming with actors. It's development not only predated the actor model but it also goes well beyond that to being the standard programming style you use when developing any program when using the language - the same way Rails embraces MVP. When this is fundamental part of every Erlang application then the means of scaling to a large distributed system are also a fundamental part of each program.
This built-in scaling is gained without any significant costs in terms of development time but also provides many benefits beyond scaling, such as highly modular and extensible code. There are real benefits even if you don't plan to scale to a large distributed system. similar to Rails it creates a predictable program design which makes joining new projects easier and deters NIH syndrome that is far too common in Java/C++/etc. And ultimately, regardless of what you are building, it provides very high performance by default for the type of async style applications that are popular on the web today.
So the key point here is not that the end goal was achieved (that you can scale) but how you get there.
If by languages you mean syntax, perhaps. If you mean platforms, then it does matter. And it's non obvious things such as fault tolerance for example - ability not just to have lots small concurrent processes but that they have isolated heaps.That's not just gimmick but it allows designing systems and operating them in a different way. For example having whole subsystems crash and restart safely without affecting the rest of the service.
Or even silly things like being able to hot reload code or log into a live node and add a dynamic trace or hotpatch a module to get extra debugging info without stopping.
Now you can sort of do that with other frameworks, it's just it's much nicer in Erlang because it comes built-in and it just feels like using the right tool for the job. As in using a hammer to hammer nails vs say using the pliers to hammer nails.
Hopefully more companies see success stories like this and take the plunge - I'm working on an Elixir project right now at my startup and am loving it.
FastGlobal in particular looks like it nicely solves a problem I've manually had to work around in the past. I'll probably be pulling that into our codebase soon.
In practice, Discord hasn't been completely reliable for my group. Lately messages have been dropping out or being sent multiple times. Voice gets messed up (robot voice) at least a couple times per week and we have to switch servers to make it work again. A few times a person's voice connection has stopped working completely for several minutes and there's nothing we can do about it.
I don't know if these problems have anything to do with the Elixir backend or the server.
EDIT: Grammar
Voice issues should not be happening. Please contact our support with more information and we will gladly investigate.
We love Cassandra, and hate it at the same time.
Check out this nasty bug we got. https://issues.apache.org/jira/browse/CASSANDRA-13004
It is by far the best at doing what it does
It has plenty of room for improvement
And thank you again with helping us with that bug :) we really appreciate it.
And most importantly:
It actually works at scale. Huge scale. Thousand node cluster and hundreds of thousands of instances scales.
Because a good chunk of the active maintainers actually run this shit in prod.
Our message storage clusters have a very large set of data that keeps increasing and using Scylla without incremental repair will suck so we are waiting on that.
The VM is an absolute marvel of engineering, and it's insane to me that it doesn't have more adoption yet in big tech companies.
My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again.
elixir (and winter) is coming
> My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again.
you conflated "top" with "large".
It makes sense too - say it's 10x easier to write something in language X than Y. If there's only 10 other people that might interact with the thing / have to read the sources, that's a great tradeoff. If there's a thousand other people that might have to at some point understand how some part of your code works, suddenly all of them have to learn the new language X.
As soon as you introduce standard message format, then all nice features such as built-in distribution, automatic reconnect, ... are almost useless. You have to do all these manually. May be I'm missing something. Correct me if I'm wrong.
For a fast time to market it seems quite nice approach. But for a long running maintainable back-end it not enough.
http://erlang.org/pipermail/erlang-questions/2011-June/05946...
Foreign nodes in C, Java, Python, etc. can also join an Erlang cluster:
http://erlang.org/doc/tutorial/cnode.html
http://erlang.org/doc/apps/jinterface/jinterface_users_guide...
That being said, a more typical architecture is to have Erlang spawn external processes and talk to them via stdio.
There are many ways to interoperate with other languages. Including libraries for other languages to claim to be erlang nodes (they will have to upgrade too when you want to upgrade to a newer version of erlang with distribution protocol changes, for example when maps we're introduced in r17).
You can easily do standard dist messaging within your erlang cluster and serialize to whatever makes sense at the boarder. You can serialize to erlang term format, if you like; it's well specified, but not terribly compact.
You're going to have the same questions with any other language too. Very few companies get to write clients, servers, and everything else in one language that never updates.
Then I don't see much difference between Go, Java, ... vs Erlang except they are simpler to learn plus finding Java, NodeJS, ... devs is much easier. What was the point of using Erlang? It was supposed to solve a problem for us, but we end up of solving the problem ourselves.
What I'm really trying to say is: integrating a mainstream language like Go, Java, C++, ... with a messaging layer like ZeroMQ (or something else) and adding some reliability features is going to be easier than introducing a totally new language (with a totally different paradigm) into the stack.
For the first several years, my team of Erlang devs had zero Erlang experience prior to being hired; they were just smart and flexible developers. I was ahead of the curve because I sort of remembered seeing a Slashdot post about Ericson opening the language way back when.
A lot of people end up using RabbitMQ as their distributed message queue, which is built in Erlang. If you went with that instead of ZeroMQ; and then slowly added more things, there's a reasonable path to writing more Erlang.
I'm not sure if you can really bolt on messaging and reliability features and get the same results; just like bolting on security later, if it's something that you need, it works better if you have it from the beginning.
But certainly, if you're happy with your stack, don't change it.
The same is true for serialization: new features are often introduced and then put to use a couple of major releases later. This ensures backwards compatibility. If you couldn't upgrade your cluster one node at a time, safely, then you would have to stop the system. The serialization format is also built to be machine-agnostic: You can run data from a 32bit little endian windows machine to a 64bit big endian sparc machine if you want, and it will work seamlessly. Of course that flexibility doesn't come for free and has an overhead. Another benefit is that data-at-rest can be safely decoded for every Erlang version back to at least release 6. This is quite useful in many situations.
So the path is usually to upgrade the OTP version first and then start using new features once the cluster is upgraded.
In the OTP20 release, a change happened in late RCs of the release because serialization stability was brought into jeopardy. It was reported by RabbitMQ and the serializer was changed so it is properly backwards compatible. Upgrade paths are important.
http://www.erlang-factory.com/upload/presentations/558/efsf2...
This seems to happen a lot when you are switching between wireless networks (E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on mobile (Seems to happen regularly, even if you're not moving around).
It's terribly annoying though and makes using the app via the mobile client to be very tedious.
Pretty funky :-)
When you are trying to get help w/ a specific task, you check checkout the mix tasks docs:
https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content
And honestly, I often just look at the source code:
https://github.com/elixir-lang/elixir/tree/master/lib/mix
The Elixir Slack channel is pretty amazing, too:
If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like https://socketcluster.io doesn't provide ?
But seriously, Discord actually benchmarked 5 million concurrent users, horizontally distributed, and having to ferry messages across the cluster, with specifically tailored fanout patterns (rather than just a global pub/sub. I.e., who a message goes to varies, rather than just "everyone").
Socketcluster.io only has benchmarks for a single machine, capped at 42k concurrent connections (though to be fair that was due to them running a single client, rather than a limitation of the server). They don't out of the box support horizontal scaling; you're required to spin up your own message queue solution for that.
So, basically, you're advocating a technology that solves -the simplest part of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any of the custom tweaking Discord describes, solve that AND more of the actual problem Discord had. So...yes, and no. Yes, there is plenty here they've describe that is not available in socketcluster.io, but no, nothing they've done here is no generally solved by an off the shelf system, because they're -using- an off the shelf system, Elixir + Phoenix.
SC does support automatic horizontal scaling across any number of machines out of the box if you're running it on Kubernetes.
There's also a CLI tool to deploy it automatically to any Kubernetes cluster: https://www.npmjs.com/package/baasil
See https://github.com/SocketCluster/socketcluster/blob/master/s...
From my understanding, you're basically saying "You can combine SocketCluster with the MQ of your choice (the installation and configuration of which is left as an exercise to the reader) and then between Docker, Kubernetes, and Baasil you can orchestrate and deploy it across a cluster". That sounds a bit more complex than just using SocketCluster, which is what the OP seemed to be indicating was all you needed, and is also including the DevOps story, which I don't think either he or I was intending to include.
I was not trying to indicate that SocketCluster can't be -used- to scale websockets horizontally, but that it's not just an off the shelf solution that would have solved Discord's problem either. It requires other parts, as both the docs and you mention.
I'll also reiterate from my post, SocketCluster has no benchmarks pertaining to what happens when you -do- scale horizontally (per docs here - http://socketcluster.io/#!/performance ). That lack alone would kill my interest in it (as would scc-state being a single instance, which would make fault tolerance a real concern to me, but it looks like you know that already). Is performing horizontal scalability tests on the roadmap?
It should only take a few minutes to deploy a cluster across hundreds of machines. The only limit is the maximum number of hosts that Kubernetes itself can handle (which is I think is over 1000 now)? SCC is self-sharding and runs and scales itself automatically with no downtime.
You can easily handle 5 million concurrent users with a small cluster. SC's problem isn't scalability, it's marketing.
For me to pick Socketcluster for a distributed solution (or more broadly, what I'd want for -any- technical solution) I'd want to know what else I need to pair it with (which the docs actually mislead me on), what else I can benefit from (which the docs don't tell me, but which does exist per your links), and what benefits I stand to get from using it (the docs tell me only marketing claims, but with no metrics, performance, data, etc, for what happens in a distributed context, well, I would avoid it).
Ideally, set up a clustered performance test, and then make as many of the artifacts (docker images, configs, readme, etc) available so others can conduct the same performance test themselves (as well as have a reference architecture for their own solution). Heck, if you're doing it in AWS, consider making the AMIs available along with whatever modifications need to happen. -That- would be very convincing for someone looking to adopt a solution in this space, if they could literally just spin up some EC2s and immediately start throwing load at a fully configured cluster.
Also, to make it clear, is this handling message passing between instances in the cluster?
Yes, it handles message passing between instances in the cluster. That means if you publish a message on a channel whilst connected to one host, the message will also reach subscribers to that channel which are on any other host in the cluster. It shards all channels across available brokers, when you scale up the number of brokers, it will automatically migrate the shards across available brokers with no downtime.
Possibly also consider some testing and documentation around geographic distribution; what happens if the nodes are located in different datacenters with non-trivial latency between them? Is that an issue? In the event of netsplits, does it split brain (probably not, given scc-state, but addressing that might cause it to)? That might be fine, it might not, depending on the use case, and just documenting what happens (by default, at least, if it's to be tunable) would be helpful as well.
[1]. https://blog.discordapp.com/how-discord-stores-billions-of-m...
It was super promising 3 or so years ago. But I haven't seen an update.
Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that.
The big gains would be in inlining across module boundaries and type speculation. But I hold that if we could compile bundles of modules in HiPE, we would have the same gain for a fraction of the development and maintenance effort.
The biggest lure of native code generation would be that we could get rid of a lot of C code in the system as the native cogen would be able to rival the C code in speed. Many Erlang programs spend shockingly little time in the emulator loop, especially if they are communication heavy.
If you need speed today, don't underestimate a port-program. My test is that you can pipeline about a million requests back and forth to an OCaml program per second per core. So if your work is on the order of 1+ milliseconds, this is usually a feasible strategy. Espcially because OS isolation means you can handle exceptions in the OCaml program from the Erlang side by restarting the port.
The reason I ask is because we're running a Erlang+JInterface program and the performance advantage the JVM has over BEAM is less than I would have expected. Even batching requests up into big pieces, we still see it's about 30% slower than running the same stuff in Elixir, without so much copying. But the reason we're doing JVM stuff at all is so we can re-use a whole bunch of code we already had written, and I would have expected it would have been a marginal performance win as well, but it's not.
Perhaps we're doing it wrong, too.
Do you use an external system like zookeeper? Or do you have very reliable networking and consider netsplits a tolerable risk?
I'm not implying in anyway that Elixir is bad. I just think there are too many horses these days to know which to bet on. Elixir? Node? Go? Rust? Something else?
Rust is lovely for security-sensitive code - I'm writing a customer identity management system with Rust+Postgres+Redis.
Go is... acceptable... for "glue" code where PHP would've been used a decade ago and Perl before that - all the successes of it I've seen fall into that sort of pattern.
Elixir is great when you need to think about networked, stateful systems on the scale of a rack of machines - it provides many of the components to help you design systems at that scale.
So... as ever, they all do quite seriously different things. I don't think many people need to build the sorts of systems Elixir is good for - it'll always have its niche in large-scale communications systems, though. A good many webapps fall into the patterns that Go is good for - take user input, munge it, send it to some backend system. A fair bit of code that drives the foundations of what those webapps are built on will eventually be written in Rust.
As an example Erlang and Elixir both fit essentially the same bill so it seems to me what was already a small niche is just getting fractured.
Then as you stated yourself Go is being used in areas where PHP & Perl were used before (as were also Ruby, Python, and Lua). So now there is just one more fracture there. Of course this is especially interesting since Go was supposed to be a better C or C++ then that should have placed it pretty squarely in systems programming land, yet is has managed to gain more traction as a glue and web services languages, ergo it ends up competing in this space when it probably should have competed more with C/C++/Rust.
Rust; as a systems programming language still has to compete with C/C++/ObjC in this space and considering that "all the systems" already run on these languages that is one huge challenge to change the guard there. I'm not saying that Rust is bad, rather just that I think it's long term outlook may actually be rather bleaker than other languages due to these challenges. Quite frankly D had some great concepts and improvements as a systems language over C++, but it is still barely a blip on anyone's radar.
If an individual is just trying to learn and expand their horizons then any of these languages are great to pick up. If, on the other hand, they are banking their future career on one then none is a sure fire bet right now. One would honestly still have more luck with a tried and true like Java.
Personally I enjoy picking up new languages just to see things from a different perspective and continue learning. Heck I spend much of my free time coding in Crystal which hasn't even hit a 1.0 launch so that adds pretty much zero improvement to my career prospects. :)
However, as Elixir is just another skin for Erlang, switching from Elixir to Erlang or LFE or whatever the next language for that runtime is will be simple - the issue is really BEAM and OTP, as few of the skills learned around them are cleanly transferrable to other runtimes and frameworks.
I suspect that Go will be around for a while by sheer force of inertia at this point - we're 5 years since 1.0 now and it still seems like there's new major projects being started in it every day.
Rust is... interesting, but I think it'll always have its niche. As much as people like to riff on the RIIR crowd, there's actually a somewhat decent ecosystem of reimplemented libraries and systems in Rust already, and now that Rust code is part of Firefox it seems unlikely that Mozilla will stop supporting it for a long time to come. It also seems to not be a zero-sum game - rather than stealing devs from C++, it's brought a lot of developers from "non-systems" languages into the "systems" space.
Difference between hiring "someone that knows it" and "a good dev interested by it"
My workflow for trying out a new language involves using the language for a small side project and gradually would try to scale it up. So, here's my summary, my experience of all the languages so far:
Scala - It's a vast academic language (official book is with ~700 pages) with multiple ways of doing things and it's attractiveness for me was the JVM. It's proven, robust and highly scalable. However, the language was not quite easy to understand and the frameworks that I've tried (Play 2, Lift) weren't as easy to transition to, for a Rails developer like me.
Nevertheless, I did build a simple calendar application, but it took me 2 months to learn the language and build it.
GoLang - This was my next bet, although I didn't give up on Scala completely (I know it has its uses), I wanted something simple. I used Go and had the same experience as I had when I used C++. It's a fine language, but, for a simple language, I had to fight a lot with configuration to get it working for me - (For example, it has this crazy concept of GOPATH where your project should reside and if your project isn't there it'll keep complaining). Nevertheless, I build my own (simple) Rails clone in GO and realized this isn't what I was looking for. It took my about a month to conquer the language and build my (simple) side project.
Elixir - Finally, I heard of Elixir on multiple HN Rails release threads and decided to give it a go. I started off with Phoenix. The transition was definitely wayy smoother from Rails, especially considering the founding member of this language was a Rails dev. himself (the author of "devise" gem). At first some concepts seemed different (like piping), but once I got used to it, for me there was no looking back.
All was fine until they released Phoenix 1.3, where they introduced the concept of contexts and (re) introduced Umbrella applications. Basically they encourage you to break your application into smaller applications by business function (similar to microservices) except that you can do this however you like (unopinionated). For example, I broke down my application by business units (Finance, Marketing, etc.). This forced me to re-think my application in a way I never would have thought and by this time I had finished reading all 3 popular books on this topic (Domain Driven Design). I loved how the fact that Elixir's design choices are really well suited for DDD. If you're new to DDD I suggest you try giving it a shot, it really can force you to re-think the way you develop software.
By the end of two weeks after being introduced to Elixir, I picked up the language. In a month and a half, I built a complete Salesforce clone just working on the weekends. And this includes even the UI. And I love how my application is always blazing fast, picks up errors even before it compiles and warns me if I'm no using a variable I defined somewhere.
P.S there IS a small learning curve involved if you're starting out fresh:
1) IF you're used to the Rails asset pipeline, you'll need to learn some new tools like Brunch / Webpack / etc. 2) Understand about contexts & DDD (optional) if you want to better architect your application. 3) There is no return statement in Elixir!
As a Ruby developer, here are my thoughts:
1. So, will I be developing with Rails again? Probably yes, for simpler applications / API servers. 2. Is Ruby dying? No. In fact, I can't wait for Ruby 3.
Some drawbacks of Elixir: 1. Relatively new, so sometimes you'll be on your own and that's okay. 2. Fewer libraries as compared to the Ruby eco-system. But you can easily write your own. 3. Fewer developers, but should be fairly to onboard Ruby developers.
Cheers.
What I like about this article is that they shared everything they learned with the community. Thank you for that excellent experience report.
Could this be possibly be the cause of the message reordering and dropping that I experience when I'm on a spotty connection?
Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this?
May be a matter of semantics, but the concept of ORM simply can't exist in a functional language as there aren't objects. That said, Ecto is pretty much the go-to in Elixir. It's extremely powerful, and in my opinion provides the right amount of abstraction without going too far.
Ecto can do a reasonable impression with its DSL & direct use of tables. You don't need to go through the whole stack of schemas and such.
What in particular do you like about Sequel (over what Ecto is providing)?
The mental shift is that "schemas" are not objects in an ORM sense, but instead are just data, or views over data. The functions come from taking in data or a Changeset and manipulating those, versus calling a function on a class.
As far as errors, Elixir 1.4.5 has much better error messages, specifically printing the args to crashes and such, and OTP 20/ Elixir 1.5 should drastically improve Dialyzer error messages. I am not a Ruby guy, so perhaps you can say what you feel is missing from Elixir's messages and how the Ruby gem improves it. Also you can literally inspect running state on the fly with the BEAM tooling, so the need for things like debuggers goes down.
(For context, Dialyzer is a static analysis tool detecting type errors which is part of the core Erlang distribution.)
OTP-14369 Application(s): compiler, dialyzer, stdlib
Related Id(s): PR-1367
The format of debug information that is stored in BEAM
files (when debug_info is used) has been changed. The
purpose of the change is to better support other
BEAM-based languages such as Elixir or LFE.
All tools included in OTP (dialyzer, debugger, cover,
and so on) will handle both the new format and the
previous format. Tools that retrieve the debug
information using beam_lib:chunk(Beam, [abstract_code])
will continue to work with both the new and old format.
Tools that call beam_lib:chunk(Beam, ["Abst"]) will not
work with the new format.
For more information, see the description of debug_info
in the documentation for beam_lib and the description
of the {debug_info,{Backend,Data}} option in the
documentation for compile.
I wonder how Cloud Haskell would fare in such a scenario
click link
[Error 504 Gateway time-out]
only on Hacker News