A Fable 5 model running at 9,000 tokens/s on an ASIC rather than 150 tokens/s on electricity chugging Nvidia GPUs, or even giant SRAM Cerebras or Groq chips could be good enough to meet the majority of demand.
A Fable 5 model running at 9,000 tokens/s on an ASIC rather than 150 tokens/s on electricity chugging Nvidia GPUs, or even giant SRAM Cerebras or Groq chips could be good enough to meet the majority of demand.
The incremental unlock of capability by ever increasing frontier model sizes will eventually reach diminishing returns.
I would argue tnference speed increases would actually unlock a different kind of more meaningful value for a wider audience.
pretty stupid statement lmao
640k was enough ... in 1981 ... almost fifty years later is 50,000 lower than a standard off the shelf PC now
The comment seems like the nerd equivalent of 6 7
The quote became famous because it's a failure to imagine that people would find new uses for computer memory if it became plentiful and cheap. In the quote Gates is not expecting people to come up with more demanding applications for computer memory (RAM) than the ones which were available in 1981.
Computer memory did in fact become plentiful and cheap after this, and computer programs became more complex and memory-hungry and today we wouldn't consider a 640kb an acceptable amount of ram for even the lowest-end device.
OP is repeating the quote about memory to indicate that they think that modern LLMs will be a similar resource. We should not expect that demand for LLM inference will stay flat, and that once everyone has cheap access to Fable level, we won't find new more demanding uses for it.
The question is really whether 640k is enough to last you until your next hardware upgrade.
I get this reference!