That aside, I wonder how hard proper software development would be for such a machine. It seems like multi-threaded software is something that we struggle to get right in our current low-core-count systems, so...
That aside, I wonder how hard proper software development would be for such a machine. It seems like multi-threaded software is something that we struggle to get right in our current low-core-count systems, so...
The author has reinvented the architecture used by Cell (PS3) and Intel IXP. Both of these architectures are dead specifically because they are too damn hard to program for compared with a multicore ARM/x86 chip.
GPUs would be the most successful modern implementation of this idea. There are opportunities with FPGAs, but GPU silicon is so far ahead (and still advancing fast) that you're usually better off designing for GPUs.
You could also consider Cavium parts (16-64 way ARM chips) which ship today in high-end network hardware.
The common lessons across all of these are that:
* Memory is slow compared with computation
* Put caches everywhere and have the machine decide what to cache
* Synchronisation is hard and puts tremendous load on your scarce memory resources
* It's much easier to do the same job on different data a million times than to do a million different jobs on the same data. In other words, high throughput is easier to achieve than low latency.
The article talks about a much more "anarchistic" parallelism where thousands of different (in code and data) objects are each doing their thing, sending messages to each other when necessary. I guess that Erlang/Elixir's threads are closest currently, as mentioned in the article.
Every single one of the CPUs was independent and happy to run as many branches and/or memory accesses you want without significant performance penalty, unlike modern GPUs.
So yeah, you could put different objects on different CPUs if you want. Except that that's not where the bottleneck in either energy or computation is. Remember that that local RAM needs to be powered if it's to retain state (ignoring FeRAM), so CPUs are no longer free; you have to commit objects back to main DRAM before switching off the CPU. And so you've just reinvented caching and might as well just run on a fast few-core CPU anyway.
Though you'd probably want to read the old Connection Machine papers too, the CM1 had up to 65kCPU (although individual CM CPUs were much, much simpler than ARM cores).
So long as you are sensible (using mutexes for shared data structures, using queues or whatever for inter-thread messaging, using semaphores for inter-thread signaling, etc) it just isn't hard.
The only thing that briefly got me in trouble was when I tried my hand at lock-free data structures. Once they are working, they are fine, but it takes a few surprises before you really understand the A-B-A problem.
But the "so long as you are sensible" advice is terribly, terribly wrong. You have to rethink how parts of your code are isolated from one another, and you have to avoid entirely new classes of errors—such as deadlocks, which can appear when you combine two well-tested and correct multithreaded components. And the number of threads has nothing to do with it.
Communicating sequential processes[0][1] more or less models that. Golang has some history with CSP (they are based on the concept of pipes and CSP was described with pipes before) while haskel/erlang and similar functional languages model the "sequential" part of processing data.
[0]https://en.wikipedia.org/wiki/Communicating_sequential_proce...