Hacker News Reader

Comment by ineedasername | Hacker News Reader

>Most new Python projects are still about AI

Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ <= 4GB GPU Ram in a reasonable amount of time?

Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something.

Trying to search for such a thing-- if it exists-- is nearly impossible right now with so much being done & talked about. Some content talks about running something on only 16GB GPU ram (!!!) which is far beyond many discrete cards. a 3080 has up to 16, some only 8 (I think 16 is the max?) and fairly decent entry level+ cards top out at 4GB.

Any options out there?

View on HN

Yeah, that's possible now. You don't actually need a GPU to run inference against some of the smaller language models - I've heard reports of them running on a RaspberryPi.

The trick for the moment is to skip Python though. lambda.cpp and its many variants are the ones that I've heard working best.

I suggest starting with LLaMA 7B or Alpaca. More notes here: https://simonwillison.net/tags/homebrewllms/

This one is the easiest to get working I think: https://github.com/nomic-ai/gpt4all

Thanks! I think have my weekend free time planned out now. Or a least a Saturday. I don't do AI at work but playing with code on my time off, even for fun, is a little too work like to really relax. I needs me some mindless entertainment too.

None of the models out in the open really have GPT-3-ish capabilities, not even llama-65b.

That aside, https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable amount of system RAM for larger models - 20 Gb for llama-30b, 40 Gb for llama-65b. Swap is an option here, but performance with it is abysmal.

These things are huge, and I don't think there is any way around that. You can play tricks with quantization, and larger models seem to be more tolerant of it, but even if, as some claim, 65b could be quantized to 2-bit while retaining decent performance, that would still need 20 Gb of RAM (CPU or GPU) to load - never mind additional requirements for actual inference.

The breakthrough here is more likely to come from new hardware that is highly optimized for those LLMs - basically just matmul for various bitnesses all the way down to 2-bit plus as much fast memory as can be packed into it for a given price point.

What about something that's maybe more along the lines of a GPT-2.5?

Of the small models, Vicuna and WizardLM seem to be the best of what's available right now. Quantized 7B models are just under 4 Gb of RAM, so they should fit on most GPUs out there; and even if not, they're "fast enough" on the CPU. But if you have enough RAM to run vicuna-13b (12 Gb should do it quite nicely), I'd start there.

If you're really alright with a few minutes delay, you might be able to run =<13b param. models on CPU

Any particular project that's prebuilt to not throw "out of ram" errors for it? I just want to play around with things without the restrictions ChatGPT. The OpenAI API is much less restrictive but can get pricey fast if I want quality above the lowest tier, at least for hobbyist purposes. I ran up a $10 charge in a few hours-- that's what I had budgeted for playing with a new "toy".

It get's very tedious prompt engineering to convince ChatGPT to respond to some perfectly reasonable prompts without and an answer that amounts to "aww shucks, I'm just a simple LLM and couldn't possible generate an answer to that sort of thing". Recently I asked it-- after about 20 prompts back and forth-- tp "analyze the personality of the person submitting these prompts" (They were prompts about the nature of AI, and how did it know that it was not a simplified LLM being run & controlled by a true advanced AGI, etc). It took about a dozen tried to get it to (finally) "Generate a fictional bio in the form of and RPG character sheet based on the prompts".

Even if it takes a few minutes and the quality is lacking a bit lacking and unpolished, I don't want to have to argue with the LLM to get a response.

What do you mean by "quality above the lowest tier"? ChatGPT-3.5 is the cheapest of their models right now, and it's also generally the best of all the 3.x ones, although the older GPT-3 models can sometimes beat it on non-task-oriented stuff (like long compositions).

>"What do you mean by "quality above the lowest tier"?

OpenAI's API allows you to choose different models in the GPT3 series. Cost increases with quality. AFAIK GPT3.5 corresponds to their davinci-03 model, and it's the most expensive one to use.

That's the thing - "cost increases with quality" is not really true anymore. When they released gpt-3.5-turbo in the API, it was massively cheaper than all their older stuff, but it generally produces better output. OTOH text-davinci-003 is the older GPT-3.5 model that is not fine-tuned for chat (which is where much of task solving capability of ChatGPT seems to be coming from).

Reply on news.ycombinator.com