Yet LLMs can achive miraculous results with so little. It really begs the question, where is the limit? How smart would a artificial human brain be if it would clock in gigahertz and had the somehow orders of magnitudes more efficient reasoning and detail recall of LLMs?
By lowering the clock speed, power and heat drop exponentially.
I'm just hoping we figure out something logistically better than laminating chips together.
However LLMs are not really tiny compared to real brains, brains FLOP/s have been estimated to be around 1e15 to 1e21, while a single H100 SXM can already do 1e15 BF16 FLOP/s.
The brain also has about 1e15 synapses, each holding the equivalent of about 4.7 bits of information, which would be about 590 TB worth of parameters, equivalent to 294T BF16 parameters (would need 7.4k H100s to hold those, at 80GB per H100).
So in my opinion, it's more of a memory bandwidth problem than a computing power problem, as H100s have comparatively very little memory bandwidth per FLOP than the brain, which holds information directly inside the computing units (synapses).
And I also think that detail recall is a double-edged sword. Memorizing details can certainly be useful, but it can also compete with pattern generalization. If you can perfectly remember the training set (overfit) you don't need to learn to generalize.