Hacker News Reader

Hacker News

Comment by mlsu | Hacker News Reader

Full thread

mlsu

We're now at "free, local, AI friend that you can have conversations with on consumer hardware" territory.

- synthesize an avatar using stablediffusion

- synthesize conversation with llama

- synthesize the voice with this text thing

soon

- VR

- Video

wild times!

View on HN

Seems like a fun afternoon project to get this hooked into one of the Skyrim TTS mods. I previously messed around with elevenlabs, but it had too much latency and would be somewhat expensive long term so I’m excited to try local and free.

I’m sure I have a lot of reading up to do first, but is it a safe assumption that I’d be better served running this on an m2 mbp rather than tax out my desktop’s poor 3070 running it on top of Skyrim VR?

Would be great to have a local home assistant voice interface with this + llama + whisper.

I'm looking forward to this tech being used in video games, as well as generative models in general. Interacting with smart NPCs will make everyone's experience different. The avatars themselves could be dynamically generated, and entire environments for that matter. Truly game changing technology for interactive entertainment.

Which consumer gpu runs llama 70B?

Prosumer gear.

MacBook Pro M3 Max.

A Mac with a lot of unified RAM can do it, or a dual 3090/4090 setup gets you 48gb of VRAM.

Does this actually work? I had thought that you can't use SLI to increase your net memory for the modal?

It works. I use ollama these days, with litellm for the api compatibility, and it seems to use both 24GB GPUs on the server.

I’ve got a 64gb Mac M2. All of the openllm models seem to hang on startup or on API calls. I got them working through GCP colab. Not sure if it’s a configuration issue or if the hardware just isn’t up to it?

Valiant et al work great on my 64Gb Studio at Q4_K_M. Happy to answer questions.

Try llama.cpp with Metal (critical) and GGUF models from TheBloke

Or wait another month or so for https://ChatOnMac.com

A single 3090, or any 24GB GPU. Just barely.

Yi 34B is a much better fit. I can cram 75K context onto 24GB without brutalizing the model with <3bpw quantization, like you have to do with 70B for 4K context.

Can it produce any meaningful outputs with such an extreme quantisation?

Yeah, quite good actually, especially if you quantize it on text close to what you are trying to output.

Llama 70B is a huge compromise at 2.65bpw... This does make the much "dumber." Yi 34B is much better, as you can quantize it at ~4bpw and still have a huge context.

How would you compare mistral-7b-instruct 16fp (or similar 7b/13b model like llama2 etc) to Yi-34b quantized?

34B is better. Quantization hurts some, especially in "pro" non chat use cases like RAG, but the increased parameter count makes models so much smarter in comparison.

The perplexity graph here is a pretty good illustration: https://github.com/ggerganov/llama.cpp/pull/1684

YMMV, as Mistral and Yi are not necessarily comparable like different sizes of llama, and it depends on the task.

Yup, and you can already mix and match both local and cloud AIs with stuff like SillyTavern/RealmPlay if you wanna try what the experience is like, people have been using it to roleplay for a while.

Reply on news.ycombinator.com