https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is?
It's a simple app download and allows you to select from multiple available models. No hacking required.
It's a simple app download and allows you to select from multiple available models. No hacking required.
Guanaco-65B is per https://arxiv.org/abs/2305.14314
CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML
GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF
4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ
The more you can fit on your GPU (in VRAM) the better (for speed), but no GPU is strictly required.
I also remember something more directly analogous to @home, but I'm having a hard time finding it.
Edit: Petals! Thank you sibling commenter!
I think the problem is that it's hard to parallelize this effectively across a network with Internet-scale latency - individual matrix multiplications parallelize very well, but there would need to be coordination of each result, which would be much slower than just doing the computation locally.
In the case of Folding@home, evaluating possible folds could be done completely in parallel and only need to be coordinated on discovery of a plausible fold (which is rare), so distributing over a high-latency network is still beneficial.
While this is a MINST classifier (and can be run in the browser) you can get an idea of the math behind it. https://www.3blue1brown.com/lessons/gradient-descent and https://www.3blue1brown.com/lessons/neural-network-analysis
When dealing with a LLM, it's being run again and again - token by token. Pick the best next token, append it to the input, run it again.
If you want to generate 100 tokens (rather small amount of data when compared to much of the GPT-4 conversations), that means running it 100 times.
The network traffic makes the entire system much slower than running it all in one spot.
Consider also the "are you the input to everyone?" and the privacy implications of that.