Lego style supercomputers are still very interesting in my eye though. As the technical complexity involved in scaling the raw compute performance has simplified to a "how many do you want" problem the technical complexity in the interconnects has remained interesting and innovative both for connectivity intra and inter node. You won't really see that in the FLOPS number that makes the headlines but the interconnect can be the difference between a type of workload being feasible or not. The main push here is how large you can make certain levels of shared memory access happen at what latencies to run larger jobs instead of just more jobs.
However the second-fastest (ARM based Fugaku) absolutely wipes the floor with the fastest in certain tasks due to a difference in interconnect topology. Fugaku futhermore has no GPUs unlike many other super computers and instead a CPU with some vector instructions, leading to a different programming model.
If you are more into specialized hardware, Anton3 is amazing.
That's the Cloud, not supercomputing. Supercomputing is all about interconnect.
That said, each of the nodes is itself composed of multiple cores with shared memory. So in cases where you really want to grind out performance, you actually end up using message passing to divvy up chunks of work, and then use classic pthreads to parallelize things further, with lower latency.
I forget the exact terminology used, but the parent is right that the interconnect is the "killer feature." To make that message passing fast, there's a lot of crazy topography to keep the number of hops down. The Q had nodes connected in a "torus" configuration to that end [1].
Debugging is a bit of a nightmare, though, since some bugs inevitably only come up once you have a large number of nodes running the algorithm in parallel. But you'll probably be in a mainframe-style time-sharing setup, so you may have to wait hours or more to rerun things.
This applies less to some of the newer supercomputers, which are more or less clusters of GPUs instead of clusters of CPUs. I imagine there's some commonality, but I haven't worked with any of them so I can't really say.
[1] https://www.scorec.rpi.edu/~shephard/FEP19/notes-2019/Introd...
https://longnow.org/essays/richard-feynman-connection-machin...
Besides which, at that level the goal is not to go "look at this cool thing we built", it's more like "how do we cheaply and effectively build something that can solve these massive weather/nuclear explosion/human brain/etc. simulation problems we have?" and if ganging together lots of off-the-shelf CPUs/GPUs achieves that goal with less time, effort, and cost than building super-custom, boutique-schmoutique hardware, so be it.
Said that, #1 is about twice as fast as #2, which is about three times as fast as number #3. Those gaps are much wider then I would have expected this late in the game.