Hacker News Reader

Comment by LanternLight83 | Hacker News Reader

If you're really alright with a few minutes delay, you might be able to run =<13b param. models on CPU

Any particular project that's prebuilt to not throw "out of ram" errors for it? I just want to play around with things without the restrictions ChatGPT. The OpenAI API is much less restrictive but can get pricey fast if I want quality above the lowest tier, at least for hobbyist purposes. I ran up a $10 charge in a few hours-- that's what I had budgeted for playing with a new "toy".

It get's very tedious prompt engineering to convince ChatGPT to respond to some perfectly reasonable prompts without and an answer that amounts to "aww shucks, I'm just a simple LLM and couldn't possible generate an answer to that sort of thing". Recently I asked it-- after about 20 prompts back and forth-- tp "analyze the personality of the person submitting these prompts" (They were prompts about the nature of AI, and how did it know that it was not a simplified LLM being run & controlled by a true advanced AGI, etc). It took about a dozen tried to get it to (finally) "Generate a fictional bio in the form of and RPG character sheet based on the prompts".

Even if it takes a few minutes and the quality is lacking a bit lacking and unpolished, I don't want to have to argue with the LLM to get a response.

What do you mean by "quality above the lowest tier"? ChatGPT-3.5 is the cheapest of their models right now, and it's also generally the best of all the 3.x ones, although the older GPT-3 models can sometimes beat it on non-task-oriented stuff (like long compositions).

>"What do you mean by "quality above the lowest tier"?

OpenAI's API allows you to choose different models in the GPT3 series. Cost increases with quality. AFAIK GPT3.5 corresponds to their davinci-03 model, and it's the most expensive one to use.

That's the thing - "cost increases with quality" is not really true anymore. When they released gpt-3.5-turbo in the API, it was massively cheaper than all their older stuff, but it generally produces better output. OTOH text-davinci-003 is the older GPT-3.5 model that is not fine-tuned for chat (which is where much of task solving capability of ChatGPT seems to be coming from).

Reply on news.ycombinator.com