CEO implies in the FYI podcast that it can handle model sizes of a max of ~4 billion parameters per wafer. Respectable but not as large as I assumed would be possible when I first read of the scale of the chip.
CEO claims model parallelism will actually work well with these devices. Would be intriguing to know the limits of this.
Based on the cooling requirements, at least a GHz clock speed appears likely. If we take the heretical position that 1 parameter is >= 1 synapse, 4 billion parameters is about the size of a bee brain. 1GHz is about 5 million times faster than a bee brain.
It would take about 20000 such chips to simulate a human-sized network. Not economical and presumably the model parallelism would break long before this. But it is interesting to note that under the not implausible 1 parameter is >= 1 synapse assumption, we are only a few orders of magnitude away from human-sized networks training 5 million times real-time.