I'm expecting someone to come up with an LLM version of the Coral USB Accelerator: https://www.coral.ai/products/accelerator
Just plug in a stick in your USB-C port or add an M.2 or PCIe board and you'll get dramatically faster AI inference.
Just plug in a stick in your USB-C port or add an M.2 or PCIe board and you'll get dramatically faster AI inference.
Put another way, there already exist add-in boards like this, and they’re called GPUs.
An "LLM chip" does not need that and so can be much more efficient.