Hacker News Reader

Comment by amazingamazing | Hacker News Reader

I never have seen the point of running locally. Not cost effective, worse model, etc.

Even if it was not cost-effective, or you're just running worse models, it's learning an important skill.

Take for instance, self-hosting your website may have all these considerations, but you're getting information from the LLMs. It would be helpful to know that the LLM is in your control.

Self hosting website as local server running in your room? what’s the point?

Same with LLMs, you can use providers who don’t log requests and SOC2 compliant.

Small models that run locally is a waste of time as they don’t have adequate value compared to larger models.

> Self hosting website as local server running in your room? what’s the point?

It depends on how important the web site is and what the purpose is. Personal blog (Ghost, WordPress)? File sharing (Nextcloud)? Document collaboration (Etherpad)? Media (Jellyfin)? Why _not_ run it from your room with a reverse proxy? You're paying for internet anyway.

> Same with LLMs, you can use providers who don’t log requests and SOC2 compliant.

Sure. Until they change their minds and decide they want to, or until they go belly-up because they didn't monetize you.

> Small models that run locally is a waste of time as they don’t have adequate value compared to larger models.

A small model won't get you GPT-4o value, but for coding, simple image generation, story prompts, "how long should I boil an egg" questions, etc. it'll do just fine, and it's yours. As a bonus, while a lot of energy went into creating the models you'd use, you're saving a lot of energy using them compared to asking the giant models simple questions.

My self hosted website that has a bunch of articles about my self hosted setup and documentation, that i have running from a server in my living room got me my 6 figure devOps job.

Local is private. You are not handing over your data to An AI for training.

Most major providers have EULAs that specify that they don’t keep your data and don’t use it for training.

Considering how most major providers got their data to begin with these are EULAs that are probably impossible to enforce, and none of the major players have given much reason to believe they would stick to it anyway. And why bother when local models don't have nearly the same complications?

This is FUD.

Millions of businesses trust Google and Microsoft with their spreadsheets, emails, documents, etc… The EULAs are very similar.

If any major provider was caught violating the terms of these agreements they’d lose a massive amount of business! It’s just not worth it.

Right. And the former CEO of Google basically explained their operating procedure. You go and steal everything, you break the rules, "then you have your lawyers clean all that up" when you get caught.

https://youtu.be/pvcK9X4XEF8?si=9j2ELkbXxHGoZEnZ&t=1157

> ...if you're a Silicon Valley entrepreneur, which hopefully all of you will be, is if it took off then you'd hire a whole bunch of lawyers to go clean the mess up, right. But if nobody uses your product it doesn't matter that you stole all the content.

> This is FUD.

Yeah. Because I have tons and tons of fear, uncertainty and doubt about how these giga-entities control so many aspects of my life. Maybe I shouldn't be spreading it but on the other hand they are so massively huge that any FUD I spread will have absolutely no effect on how they operate. If you want to smell real FUD go sniff Microsoft[1].

[1] https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt#M...

And my car doesn't keep full precision logs of it's location according to the terms that I agree to every time I drive somewhere.

But it turns out they did /do.

Privacy?

Can you elaborate on the not cost effective part? That seems surprising unless the API providers are running at a loss.

A 4090 is $1000 at least. That’s years of subscription for the latest models, which don’t run locally anyway.

unless you have one already for playing games

I run LLMs locally on a 2080TI I bought used years ago for another deep learning project. It's not the fastest thing in the world, but adequate for running 8B models. 70B models technically work but are too slow to realistically use them.

Reply on news.ycombinator.com