Hacker News Reader

A Mac with a lot of unified RAM can do it, or a dual 3090/4090 setup gets you 48gb of VRAM.

I’ve got a 64gb Mac M2. All of the openllm models seem to hang on startup or on API calls. I got them working through GCP colab. Not sure if it’s a configuration issue or if the hardware just isn’t up to it?

Valiant et al work great on my 64Gb Studio at Q4_K_M. Happy to answer questions.

Try llama.cpp with Metal (critical) and GGUF models from TheBloke

Or wait another month or so for https://ChatOnMac.com

Does this actually work? I had thought that you can't use SLI to increase your net memory for the modal?

It works. I use ollama these days, with litellm for the api compatibility, and it seems to use both 24GB GPUs on the server.

Reply on news.ycombinator.com