>But even NorthPole’s 224 megabytes of RAM are not enough for large language models, such as those used by the chatbot ChatGPT, which take up several thousand megabytes of data even in their most stripped-down versions.
Is that right?
Is that right?
> NorthPole's core array includes 192 MB of flexible memory (768KB of unified memory per core). Assigning 2/3rd of this memory to parameters, such as weights and biases, provides 128MB for network storage.
https://www.science.org/doi/suppl/10.1126/science.adh1174/su...
My understanding is that this is a small energy-efficient chip for edge computing, like to stuff in an IoT device. Way too little memory to expect to run any recent language models, but could maybe do some basic object detection in a doorbell camera, for example.
The article's author seems to believe 224 megabytes is a huge amount of memory, and is a few orders of magnitude too low on the ChatGPT estimate too.
Small LLMs that have been quantized to optimize for space still sit at 4-9 gigs
GPT-3 is about 175 billion parameters (though I have no idea how many bits per parameter OpenAI uses at inference-time), and is apparently trained on 45 TB of data[0]
[0] Caution: citation was first hit on google, YMMV — https://www.springboard.com/blog/data-science/machine-learni...
On-site training is… not really solved yet. Not efficiently, at any rate: any task can be trained with sufficient compute and/or examples, but probably more than most companies would care to bother with, and certainly more than we'd get onto one of the chips in the article.
That's not to diss the chips: As I understand it, one of the biggest issues is the power envelope of mobile units, which means making the computations more energy efficient is going to help massively, it's just that "training" and "inference" are currently very distinct tasks with very different hardware requirements.
(Also, I'm not sure if you mean those examples as illustrations or are serious about them: if you're serious, I suspect an old-fashioned robot arm bolted to the ground and following a pre-programmed path will probably cover your needs — GOFAI is great in restricted domains, the more modern AI models are more appropriate when the environment is more chaotic and less predictable, such as collaborating in a kitchen that also has humans or being asked on the fly to do a new recipe it's never encountered before).