We also got stuck with stopgap SIMD measures like SSE/AltiVec instead of a truly scalable multicore/multiprocessor MIMD architecture. Which made it much more difficult to experiment with stuff like genetic algorithms or any other machine learning algorithm that don't map well to vectors. Because a 10 billion transistor GPU that reports thousands of shading units may only have 64 physical cores. Whereas something like a 1 GHz DEC Alpha with 15 million transistors would give us more on the order of 100-1000 cores on the same die, especially if the cache was removed and it used a distributed content-addressable memory for data locality and a near-linear bandwidth increase for workloads like map-reduce.
Anyway, maybe something like MIMD could be built on GPGPU? I just want to get info on my system and see 1024+ threads and 10 TB/sec memory bandwidth on a chip that costs no more than $1000. The fact that this doesn't exist today after all these years, and that people are perfectly happy to keep repurchasing the same desktop computer the same speed as ones from a decade ago, and on top of that go to the ends of the earth to get the simplest shader code working (rather than just coding directly in stuff like Rust/D/C#/Java/C++ and Docker and/or Lisp) just absolutely blows my mind.
https://en.wikipedia.org/wiki/Multiple_instruction,_multiple...