Hacker News Reader

I don't believe they could, need CUDA and more VRAM...

128 gigs is more than enough to load 13B model into. Pytorch has M1 support for some time now so CUDA isn't required.

Does M1 use system memory as VRAM as well?

Yes, it's all unified memory.

As much as I dislike the loss of socketed RAM, I have to say it's working out well for Apple users so far. I wonder if CXL will change the situation for consumer devices or if it will only be useful at the scale of a server rack.

13B fits on a single 3090 (24GB) in int8.

128 gigs might even be enough for 65B, if slowly.

Reply on news.ycombinator.com