tychver

Running T-Mobile's Erlang powered LTE network.

Sure they can. There’s a separate shared heap which uses atomic ref counting.

Did you check stack overflow jobs? What people are currently earning and what people are openly offering now has diverged quite a bit just in 18 months or so. It’s hurting the smaller companies with less budget and it’ll only get worse when Google opens their big office.

Berlin is a bit odd because of the huge changes in the last few years in companies starting offices here and the influx of VC funding.

SoundCloud for example, had a bunch of engineers earning around that. Probably all laid off now, though.

There are companies on StackOverflow jobs openly advertising €80-90k for senior developers now, and occasionally over 100k for specialist roles like ML or performance sensitive development in C++.

Erlang has a really nice solution here by using a separate heap per Erland "process", and because they're owned by that process, can do a copying/compacting GC without having to take any kind of locking. Ruby's Guilds could adopt a similar idea and avoid STW GC there too.

Go merely adopts a different strategy which quietly fucks you in a different set of circumstances: https://news.ycombinator.com/item?id=15823683

Yeah, it mostly happens when you have a server which runs an abnormally large task or payload, causes the heap to grow, and does it in a way that Go can't release the memory back to the OS due to heap fragmentation.

I haven't looked at this for a while but I believe it's still an issue.

https://github.com/cockroachdb/cockroach/issues/2007 https://github.com/golang/go/issues/18896 https://github.com/golang/go/issues/14045

Go trades the complexity of a compacting GC for forcing you to restart the app regularly if you have a workload with plenty of heap allocation.

I would also say the same if you host a Ruby or Python app, or anything using forking really.

Similar to the issues you had with Redis, the kernel change to THP on by default totally destroyed CoW sharing for forked Ruby processes, despite Koichi Sasada's change to make the GC more CoW friendly. Without disabling THP, a single run of GC marking can cause the entire heap to be copied for the child.

Do you mean: "take a few thousand rows and map them to a different data structure"? Because I benchmarked that recently and mapping 16,000 rows of GPS points using haversine distance in pure Ruby takes about 12ms, and about 5ms in Go. It's not that much slower. There are tasks where Ruby can be ~100x slower than Go, but a simple map isn't one of them.

500MB per worker is totally standard. What happens is a job causes a huge array or hash to be allocated, and after it‘s finished the memory can’t be returned to the OS due to heap fragmentation. Java does some crazy stuff with compaction. C programs typically try and internally allocate into arenas to avoid it.

The class definitions will be a tiny fraction of memory usage. A template Rails app memory usage only has about 20% managed by Ruby. The rest is the VM, C libraries, maybe long strings etc. Definitely not class definitions pulled in from gems.

Nah, unfortunately simply moving the GC bits from the object itself to bitmap in the page header made Ruby CoW friendlier but not CoW friendly! Each Ruby page is 4x OS pages on Linux, so marking into the header of each Ruby page still causes 1/4 of the parent heap to copy into the child process.

The bigger issue is heap fragmentation. Lots of inherited pages have a couple of free slots that Ruby will happily consume, effectively causing you to pay a 4kb copy for a 40 byte object.

This means allocating a few hundred objects can cause basically the entire heap to copy. Combine this with Transparent Huge Pages, where flipping one bit causes a 2MB copy rather than 4kb, and a few hundred object allocations can cause the entire parent process memory to be duplicated into the child.

Aaron Patterson is doing some great work to bring GC compaction to MRI, which will help reduce fragmentation, but it's a huge task.

Ruby is actually pretty good in this regard. If you define your module or class anonymously, but give it a name using a constant, Ruby will GC it when possible. The standard way of defining modules and classes obviously means they can never be GCed.

Java doesn't, or at least didn't, collect anonymous classes without an additional GC flag being enabled, which could bite you quite hard using JRuby with gems which made heavy use of anonymous classes.

This was probably true 10 years ago, but these days it's basically FUD. Since then Ruby improved performance something like 5-80x from 1.8 to 2.5 and moved from an interpreted language to a VM nearly as fast as LuaJIT. Go only has 3-5x the throughput for tasks like: grab 100 rows from PostgreSQL, serialize to JSON, respond via HTTP. Lots of the hot code in Ruby around fetching rows from the DB, serializing to JSON, and HTTP parsing have also since been implemented as native functions in C, massively improving performance with zero effort required by the developer.

This is basically my job at ChartMogul and we've pretty much solved this problem. The two biggest issues for us were: Ruby prefers to grow the heap really quickly rather than spend much time in garbage collection. You can turn this growth factor down at runtime using an enviroment variable.

The second problem is importing a huge chunk of rows at once means they have to exist in RAM at the same time. Use batched iterators to reduce peak memory usage. All GCed languages have this problem, Go included.

You'd think Go's GC was somehow revolutionary from the way they talk about it, but it's basically the same as the Ruby GC, plus a little more parallelism. What helps Go is that the compiler can optimise and use stack allocation and re-use temprory variables. If it fails, it causes a nightmare, and the Go standard library is full of tricks to convince the compiler to do stack allocation.

Java, OTOH, has compacting garbace collection, so after high peak memory usage, it can release the memory back to the OS. Aaron Patterson has been working on doing the same for Ruby. If you use JRuby, you'll get this right now, plus it's about 3x faster for shuffling data around.

You should really only need one process per core, plus a few thread per process. Obviously, this is a problem on Heroku where they give you 8 "cores" but only 512mb of RAM per 1x dyno but on a DigitalOcean or AWS server you shouldn't be running out of RAM before maxing all cores.

Yes, but you only need one process per core, just like NodeJS.

Since 1.9 you can use real OS threads to achieve parallel IO, and certain parallel computations which can proceed without holding the Global Interpreter Lock.

JRuby offers completely unrestricted threading with a single process, plus a 3x performance boost, plus the advantage of an incredible amount of work put into their VM and GC. It's a really underrated option these days.

The problem with forking, is MRI Ruby is unaware what memory is actually inherited by forking, so eventually it causes the entire heap inherited from the parent to copy into the child.

I'm actually working on a patch to fix this. The solution is simple, just don't mark, collect, or allocate into inherited pages, but the implementation itself is fiddly.

What's really exacerbated this, is that most Linux distros now have Transparent Huge Pages turned on by default, and flipping a single bit in inherited causes a 2MB copy instead of a 4kb copy!

There's nothing really that special about Goroutines. Ruby also introduced Fibres in 2007. There's been some discussion of adding a more automatic M:N threading model to Ruby 3.

Patrick, on the 2.6+ Linux kernels, is there a significant difference between threads and processes? It seems like both threads and processes are created via clone and the only difference is memory access?

I often hear "context switching between threads is cheaper" but pthreads still have their own PID and everything, so is this really the case?

Is there really much advantage to pthreads over the way PostgreSQL does things with efficient CoW sharing between processes for the binary?

Yeah, that'll do it!

C extensions are a core part of Ruby. A big chunk of the std lib is implemented in C. You shouldn't be afraid of writing a tiny bit of C. Ruby and Python are scripting languages. They're literally designed for scripting another language.

The JIT for Ruby 3 is well underway, so in a bit it'll be a complete non-issue anyway.

Using Haversine distance increases it to only 13ms. I'm not sure why your Python implementation is so slow? In the real world, things like haversine are implemented as C extensions, so I wrote one for Ruby as a comparison and it runs in 3.8ms for 16k points. https://gist.github.com/jamatthews/d910a2b39c87a871264dd31d1...

The Ruby GIS libaries already have C implementations for haversine but I just wanted to show how easy it is to hook out to a C function for math heavy stuff.

The whole Ruby VM only takes ~50ms to start, so that sounds a bit slow. I wrote a quick version in Ruby and it only takes 5ms: https://gist.github.com/jamatthews/d910a2b39c87a871264dd31d1...

Not for almost a decade. Ruby web servers and job processing frameworks have used forking out of the box since the release of Phusion Passanger 2 in 2008 and Resque in 2009.

Forking 10 processes does not use 10x the memory of a single process starting 10 threads. It's actually almost identical. Both are implemented by the kernel using clone(). Many older tools written in "fast" languages like PostgreSQL and Apache also use forking.

It's definitely preemptive rather than cooperative. Ruby/Puma is actually using one OS thread per Ruby thread, so when one hits a DB call and blocks on sync IO, it releases the GVL and another Ruby thread can proceed. There's a timer thread Ruby runs and pre-empts each thread to schedule them.

Yeah, that's pretty much what happens. There's also non-blocking IO you can use with EventMachine but the DB drivers are a bit of an issue AFAIK.

Wow, steveklabnik replied to my comment! Unfortunately, both implementations of the service are CPU bound. I think we're running 25 threads in Iron but I'll have to check.

Simple benchmarks are a useful yardstick. I recently wrote a service in Rust/Iron which only has 4.7x the throughput of the same Ruby/Rails service. That was rather disappointing considering how much more effort is required to do it in a lower level language.

Is Python/Django performance significantly worse than Ruby/Rails? The situations you describe are things I do every day in Ruby. Getting 1000 rows from the DB and performing some operation only takes a couple of milliseconds in Ruby.

Ruby/Python are meant to be glue, and you can most certainly use them to glue together "interesting things", like image processing or audio processing in a web layer.

Memory caching rarely changed, often accessed, but ultimately persisted in a DB things like exchange rates in a global object is exactly what you do in Rails. There's a specific helper for doing it. http://api.rubyonrails.org/classes/ActiveSupport/Cache/Memor...

Twitter's issue was more with concurrent IO than anything. This problem was solved about 10 years ago with the release of Ruby 1.9 and since then only one Ruby process per core is required, just like NodeJS. In that time most CPU intensive parts of the web stack have also been re-written as native extensions.

Recent benchmarks of a properly setup Ruby environment vs Go/Gin are showing Ruby/Sinatra as having 50% of the throughput https://www.techempower.com/benchmarks/#section=data-r14&hw=...

You can also just use JRuby and use a single JVM process. For small CLI apps, MRuby can even be compiled to C, then compiled as an executable.