captainregex

one of my day to day responsibilities involves using a portal tied to MSFT dynamics on the back end and it is the laggiest and most terrible experience ever. we used to have java apps that ran locally and then moved to this in the name of cloud migration and it feels like it was designed by someone whose product knowledge was limited to the first 2/5 lessons in a free Coursera (RIP) module

this is such a clean and articulate way of putting it. The discussion around here the last few days about local and the role it is going to play has been phenomenal and really genuine

more likely than not I think this ends with a vague promise, a loudly declared victory, and a quiet defanging of the promise or just outright ignoring it in the future

money is great! I like money! but if this is their version of buy me a coffee I think there’s room to run elsewhere for their skillset/area of expertise

I am so so so confused as to why Ollama of all companies did this other than an emblematic stab at making money-perhaps to appease someone putting pressure on them to do so. Their stuff does a wonderful job of enabling local for those who want it. So many things to explore there but instead they stand up yet another cloud thing? Love Ollama and hope it stays awesome

I do think Devs are one of the genuine users of local into the future. No price hikes or random caps dropped in the middle of the night and in many instances I think local agentic coding is going to be faster than the cloud. It’s a great use case

love the first and am sad you’re going to be right about the second

Aren’t there HIPPA compliant clouds? I thought Azure had an offer to that effect and I imagine that’s the type of place they’re doing a lot of things now. I’ve landed roughly where you have though- text stuff is fine but don’t ask it to interact with files/data you can’t copy paste into the box. If a user doesn’t care to go through the trouble to preserve privacy, and I think it’s fair to say a lot of people claim to care but their behavior doesn’t change, then I just don’t see it being a thing people bother with. Maybe something to use offline while on a plane? but even then I guess United will have Starlink soon so plane connectivity is gonna get better

I am all for the privacy angle and while I think there’s certainly a group of us, myself included, who care deeply about it I don’t think most people or enterprises will. I think most of those will go for the easy button and then wring their hands about privacy and security as they have always done while continuing to let the big companies do pretty much whatever they want. I would be so happy to be wrong but aren’t we already seeing it? Middle of the night price changes, leaks of data, private things that turned out to not be…and yet!

I’m still trying to understand what is the biggest group of people that uses local AI (or will)? Students who don’t want to pay but somehow have the hardware? Devs who are price conscious and want free agentic coding?

Local, in my experience, can’t even pull data from an image without hallucinating (Qwen 2.5 VI in that example). Hopefully local/small models keep getting better and devices get better at running bigger ones

It feels like we do it because we can more than because it makes sense- which I am all for! I just wonder if i’m missing some kind of major use case all around me that justifies chaining together a bunch of mac studios or buying a really great graphics card. Tools like exo are cool and the idea of distributed compute is neat but what edge cases truly need it so badly that it’s worth all the effort?

What is the biggest group of people that uses local AI? Students who don’t want to pay but somehow have hardware? Devs who are price conscious?

I kinda think about it like Linux- there’s an audience for it but most people will just use Windows. Are AI capabilities in a local device necessary even in whatever future self serving state some companies are pushing?

It feels like we do it because we can more than because it makes sense- which I am all for! I just wonder if i’m missing some kind of major use case all around me

yes but think of all those poor shareholders with unmaximized value you heartless man!

you’re not the enterprise they’re going after. buying a used fleet is going to be a different exercise in risk mitigation

ah yes. “park far away and simply walk 300 pounds of groceries to be stocked at the only corner store for miles” is a very clean and simple solution only you were smart enough to think of. There definitely aren’t hills or ruts making this impossible because everyone lives in perfectly paved suburbia and I’ve always heard tell of this obvious laziness of checks notes the men and women who actually deliver your prime packages at 4AM.

spoken like a person who’s never had to load/unload a truck before

Anyone else just refreshed and happy to be reading about something not AI?

Why not just do it locally? Or were the RAM consumption and the cloud cost comments distinct?

AnythingLLM uses NPU

literally all of them

he…isn’t?

Easy for AH- they charge fees on the capital they deploy. Adam and his BS are ultimately the problem of your poor grandmother’s teaching pension getting invested in a noted charlatan.