Comment by usrbinbash | Hacker News Reader

Comment by usrbinbash | Hacker News Reader

Parent Full thread

> Guess who controls these markets

These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs.

> For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling.

Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use...same btw. as I have to do for cloud-LLM APIs, because the product ideally wants to be able to use different providers, so I have to abstract the API away anyway.

Sure, I have to worry about infrastructure, maintenance, and performance characteristics, but that's hardly different from running our databases or web-backends.

> Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it.

Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace?

>These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs.

The old moats did not stop existing because LLMs came along. e.g. a big use case for LLMs is running over one's own documents. We can guess who very likely already has access (MS and Google), and who is going to get a scary permission prompt at best (everyone else). OS integration is obviously coming, MS announced Copilot and Google can just silently upgrade Assistant. We may not be far from a repeat of the IE saga, where MS declares Copilot as essential part of Windows.

My point is that commercial demand for privacy is not enough, and may not prevent Cloud APIs from dominating.

>>For LLMs, either you supply a cloud API, or deal with a significant deployment burden >Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use...

Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, and many many hardware variations (like the many Intel AVX variants, low memory configurations, etc.). Just take a look at the recent moves to flatpack/snap everything, and that's merely due to software variations on a single OS.

>Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace?

I expect LLMs to be a ubiquitous UI paradigm. We do expect OS providers to bundle a UI kit, and not everyone to roll their own, right?

Besides, you are comparing to server software. But non-cloud LLMs are expected to run locally on clients. That's a different world. Clients are a lot more diverse, weaker in processing power, and most can't handle upgrading postgres.

All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice. We're talking about a lot of data/power in the hands of a few corps.

> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs,

You mean, like a runtime that can be a single, statically linked binary, that requires a c-compiler? With bindings for several other popular languages? https://github.com/ggerganov/llama.cpp/

Yeah, maybe I am an optimist, but somehow I don't see that being a big problem...

> I expect LLMs to be a ubiquitous UI paradigm.

I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software. Software that already bundles large amounts of libraries.

But what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter.

> But non-cloud LLMs are expected to run locally on clients.

They don't have to. I can already see many useful applications for LLMs that run in on-premises servers. I never said that open source LLMs have to be restricted to client-devices.

> All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice.

Legislation is another good argument for open source LLMs taking over. Looking at the EU AI Act, and the recent ideas coming up in the US, it looks like model training data being open for audits is gonna be a thing in the future, especially if the model is used in a context a lot of peoples livelihoods could rely on.

>>> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs,

>llama.cpp

I'm well aware of llama.cpp, and run it myself. I'm also aware of the recent breaking change to the quantization format because 'everyone can just rebuild from the ggml'. And its difficulties with base level AVX or AVX512 (in fairness it focuses on M2 and has support for AVX2). And how you need to work to add additional non-Llama models to it (necessary due to the commercial clause). It's not quite ready yet. Even when it's ready, there's the work for actual deployment, and the mess of diverse user systems beyond what devs tend to run.

I'm not saying the challenge is impossible - but it's an extra hurdle for OSS.

>> I expect LLMs to be a ubiquitous UI paradigm.

> I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software

Look at the recent photoshop generative fill feature for what could be done.

>what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter.

Good ideas. I think for now BigCorp can avoid it by pushing LLMs as an addon UI and keeping the base UI functional. It's possible they'd ship smaller models on device too.

>Legislation is another good argument for open source LLMs taking over.

That I can buy. The legal details are however a bit beyond me right now, and I don't know what the current status is. It could push OSS or stop it.

Reply on news.ycombinator.com