Comment by gymbeaux | Hacker News Reader

Comment by gymbeaux | Hacker News Reader

Other than feeding sensitive documents? I’m not sure. OpenAI has certainly gone out of its way to eliminate the need for local LLMs- they offer fine-tuning of GPT 3.5 and 4, so it’s harder to argue you might get better results from a local model for a particular task with particular data. Though I don’t have personal experience with the ChatGPT fine-tuning.

It’s fun running models from HuggingFace on my computer. Finally, something that utilizes my computer’s 64GB of RAM and 24GB of VRAM. It’s neat seeing the immense performance difference between CPU (Ryzen 7 5700X) and GPU (RTX 3090) offloading.

I think as with most “Cloud vs on-prem” arguments, it comes down to cost vs convenience. Building an application on Azure or AWS is as easy as it gets, but if money is scarce, you can’t beat on-prem for raw resources.

I’m writing a program right now that will query ChatGPT 4 with… A LOT of tokens. We project it will cost between $5k and $15k and probably run for around 2 days. *OR* I could feed that same data through a local model running on the RTX 3090 and it’ll cost like $20 in electricity, and take maybe 6 days.

>I’m writing a program right now that will query ChatGPT 4 with… A LOT of tokens.

What are you writing a program to do? That's more what I feel OP's question is about, and also what I'm interested in.

Document comprehension/extracting certain fields from the document. There’s some overlap with what can be achieved with regex, but some of the “questions” for ChatGPT require an “intelligent” understanding of the documents. Think “what is this document about?” It’s good at that, but less-good at reliably extracting properties… which is where regex comes in handy I suppose.

Thanks. I'm keen to hear how well it works for you.

So far it’s mixed. Prompt engineering is important and makes all the difference, but for what we pay to hit the OpenAI API, I’m not particularly pleased.

A local LLM wont be rate limited, so if you are an AI company you may prefer to secure (as in purchase outright!) your own servers for your AI workload rather than be at the mercy of a third party who could rate limit you. I guess that is not local per-se.

For local it has to be privacy and having uncensored models (and maybe the combination of those things).

True, I forgot about the uncensored aspect. No “sorry, I can’t advise on that….” answers.

> Other than feeding sensitive documents?

Everything in our developer's portal is sensitive to a degree, so a local LLM is required to allow talking to it

Reply on news.ycombinator.com