Comment by unshavedyak | Hacker News Reader

Comment by unshavedyak | Hacker News Reader

What's the best way to download and get setup with this stuff atm? Ie lets say i want to run currently available variations of LLaMA -- 7B, 13B, and 30B [1] -- is there a current summary of how to acquire them, possibly quantize them, etc? Would i download a quantized version or do it myself? etc

I ran Alpaca 7B Q4 almost instantly because they provided Curl's to download it. Super simple. But it seems most aren't doing that because it's prone to getting Facebook's gaze. So.. what's recommended?

I happened to find this[2], but i think that's the non-quantized raw models? Not sure yet.

[1]: Won't bother with 65B, can't fit in memory i believe? [2]: https://github.com/shawwn/llama-dl/blob/main/llama.sh

edit: I forgot about https://github.com/cocktailpeanut/dalai - i suspect this is best in breed atm? Though a Docker container would be nice to wrangle all the dependencies

The shell script you linked is indeed the raw weights, but if you have the storage space you can download them and use the llama-cpp repo posted here a few days ago (https://github.com/ggerganov/llama.cpp) to compress them, the 'Usage' section of the readme worked for me.

I'm using a not-that-new Macbook Pro (intel, 16GB memory) and was able to run the 7B and 13B models that way, tried 30B but it seemed to hang.

I'm in the process of using a PR'd Dockerfile for Dalai, which i think will handle everything hands free. Ie download the raw files, quantize them, pick which you want (7, 13, etc). We'll see how it goes.

A bit heavy handed perhaps to use NPM/etc, but the Dockerfile really helps me ignore all the dependencies i'm adding hah.

I might be being dense, but why not distribute the quantized files?

With alpaca you get just the quantized file, for llama you need to do it all yourself. Why? No idea

Try this: it installs with a simple python command if you have an NVIDIA GPU: https://github.com/kuleshov/minillm

Reply on news.ycombinator.com