Hacker News Reader

Comment by sixhobbits | Hacker News Reader

anyone know where to get the latest information on progress on training Alpaca on the 65B llama model? I've seen a few reports on Twitter etc of people running the llama CPP script with the 65B model on an M1 macbook, but would be good to see more detailed instructions on how to do it and a writeup of how well it works compared to the smaller models etc.

I'm happy to help out if anyone is working on this, either with running/testing or the write up - I have a 64GB m1 macbook, but not a lot of time or patience to figure out how to do the correct conversions/splitting etc.

View on HN

I’d keep an eye out on huggingface. I’ve yet to see alpaca 65B, but there are copies of 30B (and 7/13B) floating around already. I’m sure someone will eventually pony up for the compute time to do 65B. I’ve found watching these models, https://huggingface.co/decapoda-research and then looking at the “spaces using …” section is a good way to find what’s being worked on (for example, check the comments for ipfs/torrents links). Also just doing a search for “alpaca” could work.

You won’t be able to do the finetuning on your MacBook, but should be able to run inference on a 4 bit quantized model.

> I’m sure someone will eventually pony up for the compute time to do 65B.

How much would it cost? Can you give a breakdown of hardware/paas requirements and the costs?

I was able to finetune the pretrained 13B model (using the same alpaca dataset, just running on my dime) in about 14 hours on an A6000. I think it will work out to between $50 and $60 when the bill hits me.

Be warned everyone is slapping code together so fast that, if your experience is like mine, you'll spend most of your time working around assumptions made by prior authors or hand merging patches between forks to get your setup running well.

Thank you! Very insightful.

Crazy pace.

Another datapoint, first hand report from a random internet person that 7B took 2.5-3hrs on a 8xA100 80Gb setup. If it ends up being a straight up linear thing, that means about ~27 hours for the 65B model. Depending on the host and if it's a preemptable instance or not, that could be about $12-30 per hour.

Have a link? I haven't seen any finetuning scripts in the wild that train a PEFT model on a multrigpu setup yet and would love to play around with one.

The original Alpaca repo has the training script. The readme has the torchrun command and arguments used for train.py. https://github.com/tatsu-lab/stanford_alpaca/blob/main/train...

Awesome, thank you!

I’ve been sidetracked with work but planning on tuning llama 65B to produce alpaca 65B, will distribute via huggingface or torrent..

FWIW running the 30B alpaca-lora model quantized to 4-bit via llama.cpp has given me great results, and while I don’t expect much of an improvement from 65B at FP16, 65B will probably perform better than 30B when quantized

The interesting next steps in my head are more focused around curating a better instruction-tuning dataset using GPT-4, then fine-tuning again, and integrating the LangChain project with the resulting agent

Sounds great! It would be great if you could release your intermediate steps, so Alpaca-65B and then Alpaca-65B-LangChain, etc. I'd like to compare the model sizes directly when fine-tuned using the same method / dataset!

I also just realized that I don't believe there's an "alpaca-native" 30B floating around, just the alpaca-lora one, so 30B would be pretty cool too (and the biggest I can run w/ llama.cpp on my MacBook).

Is the coding itself pretty trivial, cos I've got a decent pc and fancy giving it a go. Got threadripper 3960x with 128GB RAM and 2x quadro 8000s on nvlink. It's not an AI powerhouse by any means but should be better than a Macbook

I got the llama 65B model working on my M1 MacBook Pro, although not alpaca on top of it yet. The models are VERY slow unless you have enough RAM to fit them in. The 65B one is basically unusable with my 16GB of RAM (it produced about 8 words in 45 minutes). However, the 13B one works at a reasonable speed and the 7B is fast.

Note that the 65B model requires around 300GB of free disk space to prepare.

I followed the instructions here: https://github.com/simonw/til/blob/main/llms/llama-7b-m2.md

I found what I was looking for here [0]. Looking forward to seeing Alpaca on top of it - so far it seems pretty underwhelming compared to GPT, and often just regurgitates 4chan-looking stuff (with timestamps and usernames) like earlier generative models did.

[0] https://lawrencecchen.com/llama-65b-m1-max

Reply on news.ycombinator.com