Hacker News Reader

Comment by mk_stjames | Hacker News Reader

192GB of memory on the Mac Studio is enough to run Llama 65B in full FP16.

And at 800GB/s bandwidth, it will do so pretty quickly. I think my M1 Pro memory bandwidth is 200GB/s and I was running quantized 13B Alpaca relatively quickly, I'd say useable for a personal chatbot, and I think it was swapping every now and then causing pauses.

So having 4x the memory bandwidth should allow large models to run pretty damn fast. Maybe not H100 GPGPU speeds but enough for people to do some development on.

View on HN

> I think it was swapping every now and then causing pauses.

What you're seeing is probably "context swapping", not swapping memory to disk. The model can't keep the entire history of its output in context at all times, so LLaMA periodically resets the context and re-prompts it with a portion of its recent output.

https://github.com/ggerganov/llama.cpp/blob/f4c55d3bd7e124b1...

How many tokens per second are you getting from Alpaca?

When llamaCPP came out, I was running 13B at 100ms/token on a base model MacBook Pro 14".

Edit: apparently llama.cpp supports running on GPU, so I imagine it's gonna be a bit faster. Maybe a fun evening project for me to get going.

What do you think the odds are we can get H100s or equivalent in Mac Studio?

Nvidia gpus haven't been supported by Macs for a long time. Apple and Nvidia relations are not good for some reason.

https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-w...

Nvidia had a knack for putting Apple into difficult situations.

I think Apple is afraid of the AI people switching to CUDA and neglecting Apple's own accelerators. Whatever the case is, it's entirely petty - Nvidia still ships FreeBSD drivers and would probably gladly pick up Apple support again (at least for CUDA drivers).

If it was as simple as "Nvidia was an unreliable hardware partner", they wouldn't go out of their way to arbitrarily limit the drivers you can run.

> […] Nvidia still ships FreeBSD drivers and would probably gladly pick up Apple support again (at least for CUDA drivers).

Heck, Nvidia still ships Solaris drivers:

* https://www.nvidia.com/en-us/drivers/unix/

I guess now that they have the infrastructure for these more niches OSes, it's not too hard to just keep going (especially if the kernel APIs are relatively stable).

Apple doesn't arbitrarily limit the drivers you can run.

And up until now there hasn't been a Mac Nvidia could even sell graphics cards for.

Even now Mac Pro users will be such a tiny segment it's probably not worth the effort.

you can connect external graphic cards via thunderbolt and external casings

Apple’s icy relationship with Nvidia started right around when CUDA came into existence and before Metal ever existed. Nvidia created tensions between Apple and Intel and later handed Apple the headache of a class action lawsuit by knowingly shipping faulty GPUs.

I think Apple Silicon and Metal vs CUDA might have been a strategic reason for not repairing that relationship but that came years later.

  > knowingly shipping faulty GPUs.

wow, how did i miss this...

https://blog.greggant.com/posts/2021/10/13/apple-vs-nvidia-w...

Reply on news.ycombinator.com