Two Performance Walls Approaching (2009)

One should also look at Gustafson's Law [0], which is kind of a foil to Amdahl's law. While the serial portion of a program can not necessarily be parallelized, who cares if the parallelizable part is what continues to grow in size year over year?

Take this example: If you are a video rendering house, and do all of your work during the day and render at night (for say, 8 hours) you can get 'x' amount of polygons rendered (or whatever other metric). If for your next movie, say you want to have twice the number of polygons making up your scene. Gustafson's law says that doubling the data set size (or number of polygons, flops, or whatever other metric that has parallelizable computation) means you can actually get '2n' done in the same amount of time if you were to double the amount of computational performance of a system.

You do run into a bit harder problem today with memory bound (either size or bandwidth) applications, but there are many approaches to addressing that.

[0] https://en.wikipedia.org/wiki/Gustafson%27s_law

Also: Here is the original 1988 paper by Gustafson... a fantastic read, and winner of the first Gordon Bell award. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.85.6...

One way to ameliorate this is to write multi-threaded Python extensions -- i.e. with the threads on the C side, not the Python side.

You can use one core for the Python interpreter, and then 63 or 127 cores for the C/C++ code. Python is the control plane, and C is the data plane. This is a really useful pattern, but there is not much built-in support for it.

It would be nice if there were Go-like channels between Python and C (i.e. to communicate between threads, instead of locking and unlocking the GIL and mutating shared data structures). And this means that you need to write/rewrite your own extensions too. You need to factor your program differently, but I find this can make it more modular.

Almdahl’s law is a problem mostly restricted to scientific code and databases nowadays. Almost everybody else can just get more problems to solve in parallel, and split computing power between them. (Eg. you just run more instances of your web-server worker processes.)

And well, databases do not use glue code in high level languages (or at least, do not lock on it). So, outside of the scientific computing community, that's an almost complete non-issue.

Loved this post. So prescient.

I was exposed to Amdahl's law first through this blog post about how the concept can be leveraged for competitive advantage. Always love seeing how the technicals affect the biz side of things.

https://www.extrahop.com/blog/2013/20gbps-realtime-transacti...

So what would be the alternative for the "glue code" language today? From the top of my head I can think of: (1) Javascript - granted, it is single-threaded in nature but it acquired enough parts to deal with asynchronicity and communication with other processes. (2) Clojure - immutable data structure, refs, atoms and all that (3) Erlang (?)

That was a really interesting post, I'd not really considered that as we hit the limits that the glue layer will be the bottleneck.

In hindsight that makes stuff like Go and Rust look doubly obvious, most of the benefits of something like Python with the down and dirty speed of a compiled language.

Interestingly enough, these days I think that most people won't consider most Python or Ruby code as "glue code".

Are there benchmarks that show for simple "glue code" that commonly used interpreted languages are 20x slower.