Hacker News Reader

Comment by SEMW | Hacker News Reader

> In this approach, threads own their data, and communicate with message-passing. This is easier said than done, because the language constructs, primitives, and design patterns for building system software this way are still in their infancy

"Still in their infancy"? That's basically a description of Erlang's concurrency model, almost three decades old now.

Is there a concurrency equivalent of Spencer's law -- something along the lines of "Those who do not understand Erlang are doomed to reinvent it"?

View on HN

occam, as well.

They're not entirely wrong, though. Actor model and CSP are in their infancy w.r.t. mainstream usage, even though they seem to be working really well.

Then Erlang/OTP gives you all sorts of other perks besides concurrency.

OTP will probably really take off once there's a solid, maintained, port for a nicer and faster language.

Solid, maintained and nice is already there. As for speed, one must define their needs. Throughput, latency, asynchronicity and GC strategy (which is per-process and concurrent in Erlang) can all affect perceptions of "speed". Then you're given a multitude of ways to interface with C code, right down to treating a C program like it's an Erlang VM node.

The Erlang team is working on a JIT for the language, and the first release should be out soon. That should help with the "faster" part. "Nicer" is rather subjective. I happen to enjoy how Erlang looks and works, though it took me a couple of tries before I really liked it. I'm not sure why people don't jump on the idea of a bulletproof VM.

Doesn't seem as concise as an ML is one of my factors in "nice".

I guess I don't care about a bulletproof VM because I've not had issues with the JVM, CLR or other systems. And BEAM reeks of instability.

Furthermore the resulting apps seem to bear no relation to Erlang's robustness. Look at RabbitMQ. Lots of stability problems... So what's Erlang saving us from?

Doesn't seem as concise as an ML is one of my factors in "nice".

Sorry, but that disqualifies most programming languages out there. It's not as concise as ML, but it's far more so than your average ALGOL dialect.

(Also, ML isn't as concise as Scheme.)

And BEAM reeks of instability.

[citation needed]

Furthermore the resulting apps seem to bear no relation to Erlang's robustness. Look at RabbitMQ. Lots of stability problems... So what's Erlang saving us from?

A sample size of 1, particularly one that's already known to be an infamous exception (RabbitMQ), isn't helping your case.

Apologies. I was thinking if HIPE, not BEAM. You're right a lot of languages are disqualified. A lot just suck and have strange hacky stuff thrown around helter skelter. Erlang isn't that bad, but it is just questionably verbose and not fast.

And apologies, apart from playing a bit with some custom ejabberd code (which was terrible, but it was written by an inexperienced coder so nothing expected), RabbitMQ is the only high profile Erlang thing I've got experience with. I hear couch or something is also written in Erlang, but I've not heard of amazing HA, more than, say, Redis.

It's just nothing that Erlang has seems that special, apart from the process controller/network stuff. Which is straightforward to write in another language. The forced serialization, eh, I'd like the option to share memory when needed. Hot code patching? From what I understand, it has very strict compatibility requirements, and under those constraints, you can do it in another language. So apart from OTP, I really don't get the point. Erlang's big thing seems to be something that should be a library, not a language.

> Apologies. I was thinking if HIPE, not BEAM.

Other than the fact that it makes your BEAM files "non-portable" [read: can only be run on the arch that they were compiled on], what's unstable about HiPE in Erlang 16.x or later?

> Hot code patching? From what I understand, it has very strict compatibility requirements...

Eh? What requirements might those be?

I've used hot patching, but not in anger. The only requirement that I see is that callers must called the replaced code by its fully qualified (module:fun()) name in order to get the new code. What am I missing?

Riak is another piece of well-regarded code written in Erlang. (If you read Apyhr's writeup about Riak, note that it's from 2013!)

> It's just nothing that Erlang has seems that special, apart from the process controller/network stuff. Which is straightforward to write in another language.

So, like, where are the libs that do all of this in C++, Python, and/or Java? I'm not snarking here. If there exist bulletproof libs to do 99% of what you get with Erlang, then I really need to know about them.

So HiPE is stable now? I haven't used any Erlang for a few years, so I'm glad to hear it's fixed.

I understood that Erlang hot patching was only at a function level, or some other limitation that required much care? No guarantees when the old code will be unloaded? It's been years and I only skimmed things so perhaps I'm wrong and should shut up. But anyways, nothing stops you from doing the same in other languages. Indeed, asp.net does this. But it's just a light perf hack - in general I'm unsure of the usefulness vs more robust approaches. Where do you find it beneficial? (Only time I desire such stuff is to hand off control of a socket to a new server version, and there's nonintrusive ways of doing that.)

I remember seeing someone porting OTP to Java or .Net. Again, what are the language level features that prevents this? Or is it more of a lack of a commercial entity to back the beginning of such a project? Personally, having to serialize everything and getting "transparent" scalability by moving stuff over the network doesn't appeal to me. I'm guessing the rest of the world just runs a cluster of HTTP servers or other middleware and leaves it at that.

Maybe I've just had a terrible exposure to Erlang. Users of it seem to enjoy it, but so do many users. The main point seems to be that Ericsson made a good switch with it... Which seems as fallacious as pointing out that Facebook used PHP.

> So HiPE is stable now?

I mean, it looks like HiPE has been shipped with OTP since 2001 (three years before work started on Dialyzer), and it certainly is enabled by default on all supported platforms (of which x86 and amd64 appear to be two). Now, you do have to pass the "native" option to the code compiler to make it compile to native code, but you don't need to jump through any more hoops than that.

For the software I've written, bytecode-compiled Erlang was fast enough for me, so I haven't much experience with HiPE. I'm pretty sure that none of the documentation on erlang.org indicates that HiPE is experimental or unstable. What was wrong with HiPE when you used it, and when did you use it?

> I understood that Erlang hot patching was only at a function level... No guarantees when the old code will be unloaded?

Yeah, it's hot patching at a whole function level. This level of granularity almost doesn't require any care at all, actually... far less than if you could patch at a statement level [0]. Read the first several paragraphs of [1] to get a high-level overview of how hot code swapping (and code unloading) works. (The prose from "There are ways to bind yourself" onwards is not relevant to your interests, so you can stop there.)

> Where do you find [hot code loading] beneficial?

Whenever I have a service whose code I need to upgrade, and won't need any complicated data migration as a result of the upgrade. Hot code loading is not absolutely critical, but it's another useful tool in Erlang's high-availability toolbox.

> I remember seeing someone porting OTP to Java or .Net.

That's cool! :D What parts of OTP did they not port? Mnesia? The Erlang stdlib? (There's lots more to OTP than gen_server and friends.) Did they also port single-assignment variables, transparent-to-application-code IPC, and distributed code loading-and-execution, or was this just an OTP-the-library port and not a "Let's port some of the nicer Erlang/OTP runtime features, as well as gen_server and friends." project?

> Personally, having to serialize everything and getting "transparent" scalability by moving stuff over the network doesn't appeal to me.

There's no need for scare quotes. Erlang process distribution is transparent to program code. And, like, anyone writing distributed software has to be aware that accessing off-node data is almost always more expensive than accessing on-node data. It's a law of physics. You can't ignore it.

Anyway. As an application writer, Erlang's process distribution is also incredibly nice (until measurements demonstrate that it's too slow for your application, and you have to do a bit of redesign).

For most web app backend services, and a lot of web infrastructure Erlang is more than fast enough, and gives you the tools to trivially scale to meet increasing demand.

> The main point seems to be that Ericsson made a good switch with it...

The point of that example is that over 1.5 million lines of Erlang were used in a piece of telecom hardware that provided 99.9999999% uptime. (That means that the switch was down for no more than 31 milliseconds per year.)

Erlang isn't good for every project. Only people who don't know what they're talking about make that claim. Erlang and OTP do provide you with the tools to relatively easily make fault-tolerant, scalable software. Is it the only toolset that does this? Fuck no. But it is a pretty-well-thought-out, battle-tested, actively maintained one.

What tools do you use when you must write highly-fault-tolerant, scalable software?

[0] If we assume a moderately complex function, there are certainly people alive who could keep all of the interactions between the first half of the currently-running-code and the second half of the to-be-switched-to-code in their head. I'm not one of them.

[1] http://learnyousomeerlang.com/designing-a-concurrent-applica... (You might need to reload the page after it is first loaded. Late image loading scrolled the page away from the intended anchor on my system.)

Thank you for this comment.

The transparent part I put in quotes because going over a network isn't transparent. Even .net remoting can transparently create objects over the network. It's just s bad idea and better to be explicit. Like you say, the performance issues are simply a fact.

I've written some telecom stuff and ran the first VoIP oriented 911 service provider. We missed a single call in a year, and we followed up manually on that one (it was during a hard, scheduled, failover, and we were monitoring for call attempts). It's mostly a matter of testing and just assuming everything will fail. From having higher and higher level exception handlers, to assuming every connection, server, process -- anything-- will fail and making sure there's a failover path available. After that, there's monitoring, to try to prevent system wide cascading failures.

Looking back over VoIP stuff I did more recently, the availability rate is way, way lower. A huge chunk of the problems were just lack of testing or procedure. It's embarrassing, really. After that, lack of limits in order to prevent resource exhaustion was the second biggest problem. Failure of the runtimes/VMs was never an issue, across Windows and Linux, CLR and Mono.

How is Erlang going to help with logic errors more than any managed language that discourages state? I don't see how crash and retry fixes the majority of bugs. I can see how it's better than an unmanaged/unverified language where a single fault trashes the entire process, sure. But against JVM/CLR languages, say?

As far as a million lines with high uptime: the Linux kernel is pretty big and haven't people achieved high uptime with it? But I wouldn't consider that in favor of C, just that it shows it's possible in C. Is this an invalid comparison? I'd be more interested in Ericsson's engineering dept, but I imagine it's gonna be what we expect right? Heavy testing and specs?

And suppose I say OK, and move to Erlang. How is it going to maintain HA while pushing, say, a million packets a second of RTP traffic? Right now I just lb stuff out to various processing servers and call it a day. Even the guys I know that use Erlang, all the heavy lifting is C. Erlang's just the signal plane (even then, I wonder how they'd scale to handling DDoS levels of signaling).

As far as I remember, the process distribution part, it's just shuttling around serialized function calls over TCP, right? Not to demean it, just it's not a secret magic perf sauce, is it?

I'll give it another look, I've most likely missed something.

I notice that you haven't mentioned when you last used HiPE, nor have you mentioned what was unstable about it when you used it. I also notice that you haven't offered any sort of detailed description of or link to the OTP port that you saw some time ago. I'm genuinely interested in all of these things.

Every single time I have heard of "network transparency", whether it was from the Project Athena documentation out of MIT, OpenGL tomes, or CORBA programmer's guides, the author has said something to the effect of:

"Network transparency means that -to client code- access of local resources appears to be identical to access of remote resources; client code doesn't have to care where a resource is. However, access of non-local resources is bound to be slower (often substantially slower) than access of local resources. Be aware when writing performance critical code!"

Everything I've read defines network transparency in this way. Everyone I've talked to knows this definition and knows about its performance implications. Everyone I know who's not a freshly-minted web developer agrees that if you lack a certain level of experience, you have no business designing distributed systems. I'm not sure why you have such trouble with the term.

In regards to your VoIP service: Erlang/OTP provides battle-hardened monitoring, failover, exception handling, and the like. When you use Erlang, you don't have to write any of that, or fish for libraries of unknown quality to provide that functionality. That's one of the big things that's nice about the language and platform.

> Failure of the runtimes/VMs was never an issue...

Neither I, nor most folks who work with languages that run on a properly-developed VM often run into VM failures. If we did, we probably would stop using that particular faulty system. :)

> How is Erlang going to help with logic errors...

It helps by letting you write code only for the happy path. See my next comment.

> I don't see how crash and retry fixes the majority of bugs.

You might be confused about what it means to crash in Erlang. When you write Erlang, you code only for the cases that you must handle. This reduces the amount of code you must write and test. If a component of your software encounters unexpected or invalid input, or gets put into an unanticipated state, it crashes, the invalid state is lost, and the supervisor for that part of the system restarts the component. Folks sometimes talk about writing a crash-free error kernel [0] surrounded by code that dies when it runs into something unexpected.

Do you get this for free? Yes and no. The process supervision code is built in. The modular software design to take advantage of the supervisory stuff you must do for yourself. You would likely end up doing very similar design work regardless of what language or platform you used.

> ...the Linux kernel is pretty big and haven't people achieved high uptime with it? ... Is this an invalid comparison?

It probably is an invalid comparison. And yeah, Ericsson's engineering department is likely full of good, disciplined programmers. However, most tools (like Erlang/OTP) that improve programmer productivity will improve the productivity of all but the very, very weakest of programmers.

> Even the guys I know that use Erlang, all the heavy lifting is C. Erlang's just the signal plane.

Yeah. As I understand it, that's the general pattern for high-performance systems. If you have really serious performance needs for your robust distributed (or simply fault-tolerant) system, do some perf tests, write the performance-critical parts in C or C++ or whatever, then use an Erlang Port Driver (or maybe some IPC mechanism of some sort) to connect them to the more difficult or logically tricky bits that are written in Erlang.

As I understand it, this is how the software running the AXD301 was designed. If you have an hour or so, you might be interested in [1]. It's an Ericsson presentation on the design of the switch in question. Some of it is stuff you undoubtedly already know about, but much of it is probably not. If you're looking for more, any one of the papers from Joe Armstrong on the proper way to go about designing Erlang systems are always good reads.

> As far as I remember, the process distribution part ... [isn't] a secret magic perf sauce, is it?

It was written by programmers much like you and me so -no- it's not secret or magical. The protocol is even partially documented [2]. What's more, AFAICT noone claims that it's the source of serious performance gains. It is -however- reliable, and -I gather- well understood, and backed by an active, talented development team.

[0] http://learnyousomeerlang.com/building-applications-with-otp

[1] www.erlang.se/publications/Ulf_Wiger.pdf

[2] http://erlang.org/doc/apps/erts/erl_dist_protocol.html

Your sense of nice conflicts with Erlang's goal of supporting reliable systems. For example, Erlang is not designed using the paradigm du jour. It may seem like Erlang borrows from functional programming, but Erlang's designers invented all that independently because it supported writing reliable systems. Concise and maintainable are antagonistic qualities, and so Erlang's designers sacrificed all the conciseness they could get away with.

Erlang is not deisgned to be nice, elegant or have any other property people normally advertise their language as having. Erlang is designed so that programmers at Ericsson could write better code for phone switches, with constant feedback from said programmers. That's Erlang's mission statement, originally.

If you want concise actors, Scala with Akka allows you to write pretty unreadable code if you want to.

> Your sense of nice conflicts with Erlang's goal of supporting reliable systems.

Obvious counterexample: Elixir. Plenty of modern niceties that Erlang lacks (rubyesque syntactic sugar, scheme-style hygenic macros, the pipe operator, mix, etc), but shares most of Erlang's semantics and ultimately compiles down to the same bytecode, so AFAICS doesn't sacrifice any of the things that make Erlang good for building reliable systems.

This in no way diminishes the power of your counterexample, but -AIUI- Elixir is implemented as a bunch of macros on top of Erlang.

Syntax is crucial when developing for reliability. Brainfuck trivially proves this. Both semantics and syntax need to support reliability.

I'm not sure if you're implying that Elixir's syntax makes it less good at developing for reliability than Erlang. If so, I'd appreciate if you could explain your thinking further -- if the existence of brainfuck proves that Erlang's syntax is better than Elixir's at developing for reliability, I'm afraid the nature of that proof is eluding me.

> Syntax is crucial when developing for reliability. Brainfuck trivially proves this. Both semantics and syntax need to support reliability.

This is an almost content-free comment. You could have made your comment substantially better by giving some examples of language syntax that makes it easier to write reliable code than unreliable code.

I think that we all agree that Brainfuck and Whitespace are two examples of languages that are very difficult to read. A comment along the lines of "Languages that are difficult to read do not support reliability." is also almost content-free and probably not entirely true. (Some people find C terribly difficult to read -for some reason or other- but very reliable systems have been written in it.[0]) :)

[0] Yes, a quint-squillion unreliable and even dangerous systems have also been written in C. This doesn't invalidate my point.

> Concise and maintainable are antagonistic qualities

Uh? How so?

That's quite a strange argument to rationalize Erlang's verbosity and boiler plate.

I keep thinking about OTP in Rust. Unfortunately, they got rid of M:N threads, which makes threading less of a useful abstraction.

We're still pretty confident that Rust will be able to support a high quality green threading library and other M:N concurrency libraries without needing to be integrated into the language itself. All the type magic that we do to pass data between threads should work quite well for a green threading library. We already have a couple projects experimenting with this [1], [2], [3]. Servo also has a pretty nifty work-stealing queue [4] that is yet another abstraction for easily sharing data between threads.

1: https://github.com/rustcc/coroutine-rs 2: https://github.com/zonyitoo/simplesched 3: https://github.com/dpc/mioco 4: https://github.com/servo/servo/blob/c76720d67c0d04f6d77c138c...

Reply on news.ycombinator.com