Apple has new Mac app coming to make running local AI models easy
9to5mac.com
9to5mac.com
Distillation works: we can get models leaner, have better performance with lesser and lesser parameters, more efficient, running locally. Apple gets this and we'll see the result with M7 Max/ultra chips and I reckon this will be one of the titular apps when M7 line of chips are introduced.
I guess it explains the haste Anthropic and OpenAI are in to IPO, cash out and leave the the retail investor holding the bag.
Apple, to me, is the winner in the end of this AI chapter.
I don't think it shows yet but I think the dominance of the App Store will end. I think a ton of iOS apps will be replaced by ChatGPT/Claude. I think there is a big opening for a company to redesign the phone from the ground up purely around an AI assistant instead of apps.
I think AI will severely weaken iOS, App Store, and the iPhone in the next few years. It was nearly impossible to start a 3rd competitor to iOS and Android until recently because no one is going to redesign their apps for your new phone. Today, that's different. Many productivity apps are just being replaced by GenAI/ChatGPT/etc and services are making it easy for AI agents to access directly without an app.
Not to mention higher chip prices will eat into hardware margins.
I do not think local LLM inference will make up for lesser iPhone dominance.
Commodities like todo apps and calculators rarely compete on features, they compete on product design. This includes look and feel, relation between elements, design choices on what exact common features are available.
All products are, at their essence, just a set of design decisions stitched and shipped together.
Now, making these design decisions is hard. It requires knowledge, consistency, and insights into how real people operate real products. Even as a sole user of the app you build you rarely know what you actually need.
LLMs are 'design-less' in their essence. Building your own todo app takes literal seconds and it all looks simple because LLMs will default most of these design decisions for you - until you try to use your new app daily and notice it somehow sucks. Building a good todo app or good calculator requires tweaking, publishing, polishing, and solving bugs, testing, rinse and repeat.
Another tricky side of this is that you might know logically what features you want, but you are not selecting one product over another logically. You need to try and see if that works for you, all the vibes and good parts and bad parts and design choices and colors, till you find something that "clicks" for you. Using pure Claude as a todo app only works as long as you want to have more Claude in your daily life.
In the meantime app store provides dozens of well-reviewed alternative solutions which look good and require zero maintenance.
And that's just a calculator or todo app. Non-commodity apps are even harder than that.
I don't see any future where programs follow the static nature of old paradigm programming. A nice intuitive UI? What about just telling the app what I need when I need it? The gold standard of consumer UI is...another person.
If I have a team of god tier light speed SWE's at my disposal, why the fuck am I fumbling around in an actions panel? The program can just be dynamically modified in near real time to be whatever I need it to be in that moment.
If you extrapolate LLMs out to their extreme, you get a shape-shifting liquid array of bits in memory that manifest anything at the behest of a completely natural language interface.
I dont think llms are going to replace banking apps anytime soon. But I no longer look for an app to pull data from my Peloton app into another exercise app. LLMs can build an app or a script that will do that and I wont have to pay some annual or monthly fee. It feels like these types of apps are going to die off.
I recently used a coding agent to build a TV-like YouTube player for my mother, who is quite elderly. It's inspired by channelsurfer.tv, but runs on the Google TV Streamer in my living room. Unlike any set top box I've seen, it has on-screen diagrams that tell her which button to press on the remote to change the "channel." It doesn't have ads. The UI will never change unless I ask the AI to change it.
You can't say that about most apps in an app store. They will do a UI redesign and you're out of luck if you preferred the old version.
Do you really think Google or Apple or anyone else will build this app? Maybe some self-hosted media box out there would be similar, but I can't be bothered to search for one and I'm doubtful that any of them were designed for an elderly parent who isn't good with technology. It's still work in progress, but at least she can use a single remote to turn the TV on and watch TV without being bombarded with things she doesn't understand and doesn't care about.
There are downsides that make this app less compelling for other people, but I didn't build the app for them.
There's a recent benchmark indicating Hermes Agent and, I think, Qwen's harness are far and away the most effective.
Pi isn't bad but it doesn't seem as talented as Hermes. Nor does it currently array of self-learning/adaptive plugins seem as capable. That said, it's difficult to keep track of the SotA with pi; it grows like the node ecosystem in its first year.
However, you could expect LLMs to train on existing successful apps and hardened design patterns that compose well into good UIs. In that case, you might expect the last mile polish to be the differentiating factor. However, the more marginal this last mile becomes, the more other social, political, business, etc. forces will determine which app rises and which falls. But this has always been the case, certainly in software.
I think folks who live in spec-driven worlds just don’t appreciate how poorly understood the capabilities and processes of computing and networks are, to the point where even if you gave people hallucination free LLMs you’d probably find people just frustrating themselves because they can’t figure out what they want, what it should do, and how it should be used.
Quite apart from the huge chunk of the app ecosystem that relies on common protocols, centralised services and so on.
people don’t know what they want, and when they do know they can’t explain it.
* Based on our music discussions, make a playlist for me in Spotify. No songs older than 15 years.
* Look through my email, find all my receipts, show me a graph of my spend and categorize them.
* My business needs warehouse space. Find warehouse services within 15 miles of San Francisco. Ask them for their pricing and capacity. Here are pics of my items. Send these pics to them for quote.
* Find a hotel for me in Los Angeles on October 1st to Oct 5th. 4 starts. Max $200/night. Gym is a must. Don't care about pool. Close to airport. I have Platinum membership with Accor so try to find Accor hotels but not a must.
People do know what they want.
It is easy to express yes, but is this how people think about what they want to listen to?
Someone can already do this in a chat bot, see the list of things and then find it on Spotify or whatever their favorite music service is.
What the chat bot won't know is what they have listened to, which is probably a much higher predictor of what you'd like than the discussions you've had about music. You've probably had those discussions because you are already listening to that music!
I thought the parent's point is that the App Store will loose dominance (because you won't need apps).
I have seen this with yet "AI-unexperienced" users - for simple questions or textreformating, they are absolutely fine with the Chatbots. But thats all they can get out of it: Above that level their experience is very often frustrating due to too less understanding while having high expectancy towards the model in reverse (like "its AI so it should know...")
And this people end up then not using any of the AI tools beyond that because they do not "believe" / trust that this thingy can get work done.
Essentially every single non-builtin app on my phone fundamentally requires the internet to work, and is backed by some sort of SaaS with a proprietary API. And even if/when the APIs are open, I doubt any normal person is going to decide that from now on they want to somehow use YouTube or Tiktok or Feeld or Spotify by talking to ChatGPT.
Spotify is an interesting one. I could see myself wanting to control Spotify with my AI agent.
For instance; I don't think LLMs will be replacing Spotify, Facebook, Insta, or Youtube for my parents. They absolutely do not want to purely interact with a blackbox. They want to scroll through songs/shows/etc. they want to tap on it, and have the app do the thing.
Even as a SWE, I have absolutely zero desire to interact with an LLM for everything, but I'm not the litmus. I have found use with these tools for taking on some nested set of tasks, but at no point do I want to give up the app entirely.
For people who like shopping (not me!), most of it is actually browsing and there is a joy in sifting through options. The OpenAI designers seem to understand this.
That's already happened. The curated feed you see is a machine-learning driven output of an AI model that Spotify calls a "Large Taste Model", supplimented by a host of models trained using Spotify's NEO framework. It might look pretty but it absolutely is the output of AI. I'd bet a fair sum that FB, Insta, and YouTube are also showing you the output of a model now rather than a traditional application.
Sure, but I don't think it's usually the apps that people pay much money for. And on top of that, app store revenue is (from a rough estimation) 6-7% of total revenue (although it's high-margin). Apple would be entirely fine.
And if you're right that the phone of the future will be reimagined as centred around being an AI assistant: innovative device design, user experience, hiding technology, integrating software/hardware/cloud are all things that Apple is historically pretty good at (with hiccups and bad periods for sure, but other companies certainly aren't better in general).
~15-20% of Apple's total profits at ~75% margin.
Investors are not going to like if that suddenly starts shrinking.
An LLM can probably whip up a music player app, but won't have the licenses to provide you the music.
An LLM can probably whip up someone akin to a free and single player Genshin Impact, but the whales want attention from other players.
An LLM can probably whip up some sort of SFTP from phone <-> Desktop for pseudo-cloud storage, but you still need to provide disk space
The other avenue that AI is hitting their services revenue is the Google Search deal, it's a revenue sharing agreement and AI is eroding that revenue on Google's side.
all things that Apple is historically pretty good at
Not to mention, Apple is going to want to not disrupt itself so they're going to deliberately slow down which opens up the opportunity for someone else.
Do I want to set up my scale to sync its data to my vibe coded app? Not really. Do I want to set up the dozens of visualizations and hundreds of different actions and thousands of different items for calorie tracking, where even a 5% error is extremely significant? No way. Am I going to convince any of my friends to download my vibe coded chat application? Absolutely not. Do I want to store my photos in my own vibe coded solution and trust it to run in 20 years? Not a chance.
While a segment of apps will surely become obsolete, I don’t think the majority of users time will be spent on their own vibe coded replacements. Even as a developer I have no interest in turning all my conveniences into maintenance burdens, with risk of catastrophic failure
The issue is there will be less demand for apps. I don’t need to pay for a dulingo subscription when Claude can write a 5 minute language quiz for me. I don’t need to pay for Turbo Tax when Claude can prepare my tax documents for me.
Sure, it won’t replace all apps, but people will spend less time in apps and more time interacting with llms which hurts App Store revenue (Apples golden goose)
And it’s not like Apple is fixed. If this really did become a threat, Apple could surely integrate into Siri some ability to generate apps from your iPhone and put it behind their iCloud subscription.
They’ve had XCode for years, managed their entire UI library basically since conception, and have their own language for their apps. Fine tune DeepSeek for a while on swift iOS apps and I’m sure you could generate apps rivaling frontier models.
I really don’t see any existential threats for Apple here.
Top paid apps outside of F2P gambling apps(all the "games" remaining on phones are gambling apps because they've repeatedly lured in and punished all traditional games, definitively burning bridges), making hundreds of dollars per paying users, had been like $3.50 ad blockers. iOS app development had been a lot less approachable than Android apps due to Mac host requirement, and with LLMs generating Android apps and HTMLs with ease, the fact that iOS can't execute .apk will make it successively less appealing.
Personally I don't think things like agent-first OS that don't rely on third party apps will come soon - that will have to be built from ground up as proper OS kernels, and I think it will run on desktops first.
The third OS always existed, the problem with those is that telcos won't carry it and people won't use such a phone with the nation's pick of social media - Fb Messenger for US, WhatsApp for EU, Telegram for Eastern Europe or whatever, MAX for Russia(presumably?), WeChat for China, LINE for Japan, KakaoTalk for Korea, ...
... thinking out loud, the third platform has to be the one endorsed by a social media. Like the old HTC ChaCha, or like, idk, "Startile" slab device running TeslaOS with Starlink DTC SIM built in. I wouldn't touch such a thing, but something spiritually like that. Or like a "Rakuten Star" with AST SpaceMobile eSIM baked in. Things like that.
If Nvidia wanted they could end it overnight with a 60GB 6090.
We've already seen several attempts to make devices purely around assistants like Humane Pin or Rabbit. In 99% of cases users need a reliable screen with clear buttons not a conversational barrier between them and action
I personally didn't like the voice-approach of Humane Pin. Rabbit R1 was more interesting but execution was poor, screen was not big enough, models weren't smart enough, apps and services weren't as open via MCP as they are now and in the future.
I want to see a company like OpenAI or a very well funded veteran company give it a try.
To get something like this: https://www.youtube.com/watch?v=b9hxwgH2tN0
Apple will adapt, if they see (and they will) that consumer senitment changes. Right now, 90% of regular "consumers" seem to despise AI and anything made by it.
From what I have seen you belong to the vast majority. Though not here on HN.
Is this master stroke in the room with us now?
What about doing something like REA on a whole app store collection, rewriting the app's reversed description set into code according the new devices' requirements? Perhaps we can't see the future as there are so many variations yet imagined.
This is what we all want, this is what kicked off the open web back when the internet first started. This is what made everybody excited for user agents. Now we have user agents once more and we are getting excited about the new agentic web, but temper your expectations. This time, rent-seeking corporations are in it from the start, see the exact same promises that excited you about the open web, and plan to take full advantage right away. This is why everyone is trying to sell their own agent instead of giving access to your own.
This always puzzled me. Here's how that would go:
1. A new company, NeoPhone, launches that is redesigning a new "AI forward" phone.
2. The customers of that phone start noticing that they do many repetitive things with their phone, that they want the ability to design some repeat experiences with their chatbots. Call these repeat experiences "applets." Neophone ships some Applets on their phone that they created, a default set.
3. Customer A looks at customer B's phone, and notices their different applets that they designed. They work slightly better than customer A's applets, because customer B is slightly better at prompting.
4. NeoPhone notices this too. They create a new thing on their phone, the Applet store, where customers can share their applets and rate them and it surfaces the best ones to all customers.
5. Customers start complaining that they wish they had more creative control over Applets. Maybe the API is a bit more restrictive than they need, or the Applet needs to do special things with the phone's hardware, outside the context of the AI agent. Or they find that Claude Opus Fable 6.2 is better at making Applets even than NeoPhone's available chatbots.
6. NeoPhone allows 3rd party developers to develop on its phone, launching an "App Store" and a "Chat API" that is totally independent of the AI agent stuff.
???
Of course, Apple eats their lunch at step 2 or 3.
Sometimes even DoorDash gets it wrong, but at least I trust DoorDash to deterministically improve their systems so it doesn't happen frequently. With LLMs I would have to think that sometimes it just gets it wrong due to nondeterminism and insufficient reinforcement learning in the area of "reliable food ordering".
Apple is absurdly late to the game. The MLX community looks anemic compared to Linux.
So far, I prefer tailnet to my home Hermes agent inference system and giving it ssh into my Mac.
It's easier having a single powerful inference system.
A Claude Pro account and Openeouter credits are my backups.
They’ll be fine
You’re so DELUSIONAL. It’s hilarious.
Show us your portfolio of work as an innovator - just to ensure you have substance to support thes ridiculous claims.
If a new mobile device form-factor proves to be usable, they'll come out with one eventually like they always do- like the iPhone duo being years and years behind?
When the bubble pops they could still snap up an AI lab on the cheap later.
Would you trust access to your bank to a vibecoded phone of unknown origin? Backs will likely limit access as best they can for customer security. We already have banks who block GrapheneOS.
Look at the hardware wallet company that got sold and a spy chip to steal the contents was added.
I do believe AI is going to destroy companies like Adobe however.
This is the new reality that we must prepare to accept. You and I may not trust the companies, but the circle is closing in, and people around you are already on the "hype train".
Please, no. Just TELL me; that's why I'm using voice to talk to you in the first place. Summarize and tell me, don't show me a list of links from a web search.....
Please.
I want to see the projected numbers.
However will they survive if they cede ground in commoditized consumer spaces to explore market fit in luxury goods?!
Huh? I just took a look at my phone -- I have about 100 apps on there, though probably only use 20 of them with any regularity. I don't see more than a small handful that could be replace with Claude (if it had access to the APIs that the app does),
And even then, a chat interface isn't the optimal way to interact with many services. Random example: A simple metronome / tuner. Could I ask Claude to play a tick at a certain frequency for X amount of time, or a E-flat? Sure. Would it be easier than using the app? No. Could Claude design a metronome app for me? Yes, but I don't want to have to deal with that process -- I just want a metronome app that works and I don't have to spend time designing or fiddling with, and that works offline, and all of that is well worth the $5 I might spend to buy it.
Literally nobody in the world wants more AI slop in their lives.
This is coming from my own experience as a software developer. I do not want to leave Claude Code. I'm hooking up everything into Claude Code and have it do what I want for me. I do not want to log into my cloud provider. I do not want to log into my Github. I don't want to log into any 3rd party services. I just want to stay in Claude Code.
I think developers tend to be ahead of the general public by 1-2 years in AI use. I imagine a future where most people simply want to stay within their ChatGPT/Claude/Muse/Grok/Gemini agent to do just about everything.
If my prediction is correct, I think Apple is going to be in trouble because it weakens the iPhone ecosystem severely and they are so far behind in AI everything that it's hard for me to see them ever catching up. Not to mention they do not have access to compute since they've not invested in compute.
Failing to evaluate AI beyond "slop" is down to a lack of imagination.
Right now 128GB MacBook is the top of the line and you could do Qwen 3.8 Flash or similar.
There is a rumor that the M7 chip can use 1.5TB of memory which would imply Apple is aiming for more memory on devices.[1]
Would that imply the entry-level MacBook will have 128GB and top of the line models could run larger reasoning models? Then the question is price and if any of this will translate to iPhone for on-phone models.
With how expensive memory is, IDK if they can pull this off but I hope they do
[1] https://www.tomshardware.com/tech-industry/semiconductors/ap...
Most people neither need nor want to run large LLMs locally.
The entry level MacBook Neo costs $599. Even if RAM prices revert to the pre-AI mean, the entry level models are not going to be that stacked with RAM.
Eventually we'll get to a doubling where entry level means 16GB of RAM, but most people won't be running anything more than simple models for light local work.
Have you ever tried running large LLMs on a MacBook Pro? It's hot, loud, and makes your battery level disappear at rates you didn't know were possible if you're not plugged in. It's a very niche hobby.
Whatever local LLM device comes out of this, it certainly won't be a laptop.
(I envision a black box like a router you plug into a socket and let sit in a corner of some room.)
I'm old enough to remember the time when most people neither needed nor wanted a phone with a large screen and no buttons
If Apple, in 2030 has a compute product that a worker can BYO(X) to work that has 2TB RAM, with an encrypted personalized context that can float between agents, then they will be in a very interesting place.
holy shit
But the family shouldn’t be limited by that. We’re not going to see M6 Max hardware, because the M7 line is being fast-tracked and pulled forward. And I wouldn’t be at all surprised to see M7 capable of 1-2TB or more.
It is, at the very least, a practical limit. There are unlikely to be enough customers willing to pay $50k+ for a hypothetical 1TB RAM version to make it worth manufacturing.
i suspect that they will at least do offloading for casual usage as well. if they can ship 24GB in a homeview or appletv style product, it's game over.
why would a company who locks iPad Pro entry level literally out of ram that exists on its own chips.
I get their high end 10k-20k laptops will be able to reach these specs and gatekept behind also upgrading the cpu to the ultra but no way will it be an option in entry level.
Even so, DDR5 is so high-demand that it's likely that Apple can't outbid customers like Nvidia or OpenAI when they drive the margins they do.
I don’t think Apple can be in a position where all of the Chinese device-making companies can get memory at a significantly cheaper price from CMXT, and Apple is stuck with buying memory from the three hyenas of memory at a significantly higher price.
That’s not going to help Apple sell their devices at a competitive price worldwide over time against the Chinese computer makers (Lenovo, Asus, and Motorola) and make no mistake about it: their share of the market worldwide will get bigger because of this change in the memory market; their focus will be on the long term, while the lawyers/MBA/finance types in the West will have their heads stuck in the short term.
It’s the (Chinese companies’) opportunity, and they’re not going to let it pass; I wouldn’t. In addition, Apple is in competition with Nvidia, Microsoft, Intel, AMD, Qualcomm, Google, Meta, and indirectly with Anthropic and OpenAI (whose CEO, Sam Altman, recently declared that computer ram is wasted on low-level regular computer users), along with a whole legion of other 3D model makers on the procurement of memory.
The combination of all these elements going into the future means that if Apple wants to stay ahead, I believe Apple has no choice but to take memory in house, they have the design and engineering capability in-house, and they have the money because they didn’t burn it on AI models and data centers.
This scenario is no different than replacing Intel or Qualcomm; you can see the writing/destination and the ultimate train wreck up ahead if Apple doesn’t roll up its sleeves and get busy, the market has changed and again Apple has to change and adapt to the new conditions.
I wouldn't be surprised if that was happening already -- they plan five years ahead, after all, and they've been slowly taking other systems in-house (e.g. baseband/modem) already.
Probably sitting under a router, so that it can hand off bigger jobs to an Apple DC somewhere, probably with a certain number of credits based on your iCloud sub.
And with a device like this, you pay one single time to buy the product and then never have to think about paying for tokens ever again. For ordinary users that's silly, but for users or businesses chewing through lots of tokens, this means they can create their own turnkey replacement for "the cloud" where the only cost after purchasing the machine is the electricity to run it (which with these machines is very little). Mix that with the privacy benefits and suddenly it makes a lot of sense.
But on the other side, equipping data centers was never Apples main business. Their approach to markets is to design products for the individuals. They also usually have very long time plans. Look at where the Apple Silicon today is and how long a path it took. What about the day, where at least a medium sized Mac becomes a great platform for running large models locally? Very easy to use, no setup, no privacy concerns. Also never worry about "tokens". You can then use AI as you use other local resources of the hardware you own today.
Could be the winner, in your opinion. They are extremely well positioned with hardware, devices, ecosystem, privacy, brand value & trust, and a significant active base. So far, they have not yet delivered on any useful AI application. There are so many opportunities in both MacOS and iOS, I guess they just need a new software VP.
So does Apple even need to create applications (that nowadays will get copied and opensourced in a day), or the hardware is the winning strategy in the end?
Who has? Google, Anthropic, OpenAI and Microsoft have spent hundreds of billions and produced nothing but chat apps that Apple could build in a few days, they’re all just interfaces to models that are available to Apple via an API. Hard to imagine Apple as anything but the winner of this era unless OpenAI or Anthropic pull something out of the bag.
AI doesn’t have to be full-blown conversational AI.
In Photos, text recognition in Photos works well, object recognition and face recognition are somewhat decent.
In Mail, I find message summarization useful.
Small steps? Yes, but I think these are useful AI applications.
I do not use them, but in accessibility, they have AI applications to describe scenes which, I think, are useful to some.
But this is all based on marketing not facts. Apple is as privacy invasive as the next guy. Obviously this shtick will work as long as marketing continues do to good job.
I don't think so. LLM inference enjoys huge economies of scale. At no point in the foreseeable future will a local model be able to compete with a cloud model in terms of price. Local models will have the privacy angle, but very few people will be willing to pay top dollar for that. There will be some people, for sure, but the vast majority will just use cloud models.
Does it actually? Sure, it is operating at huge scale and this design choice allows enormous models (presumably enormous MoE models). But is it achieving economies of scale?
Economies of scale are rooted more in the idea of a profitable thing becoming broadly sustainably cheaper as the market grows, allowing cheaper manufacturing and further growing the market.
The big two or three LLMs are cheap because they are substantially subsidised by investor cash, and they are not profitable at all in any sustainable sense: the current pricing can only be sustained for as long as they can raise money, and the market will (and does!) shrink when it is asked to pay a less-subsidised price.
There appears to me to be no route to true profitability for this design strategy at all. Even the open weights router companies are downstream of largesse from state-affiliated companies.
No, it's exactly the opposite.
The big money is in small(ish) but finely tuned for specific tasks models.
Whoever invents an app store for LoRA adapters will win this AI race.
What new bike should I get? How should I rephrase this email? What to do about my persistent headaches? Recipes based on these ingredients?
In the mainframe's heyday, it was difficult to grasp a world in which compute became so dense and so cheap that it made sense for each individual to have all processing and memory on their desk, talk less of on their lap and in their hand.
Of course this time could be different. But, when we look at previous relentless progress in compute and efficiency, then also couple that with the breakthroughs AI itself unlocks, it's hard for me to imagine that centralized AI will meet both the scale and flexibility desired for all use cases.
Just as centralized compute never fully disappeared -- it took the form of client-server computing and the cloud, there's no question that centralized inference will continue to exist.
But it wouldn't surprise me if, ten years from now, 90% of global inference happens at the client.
While simultaneously, the 10% of inference that takes place in the datacenter is wildly lucrative and orders of magnitude greater than today.
Put simply: the inference pie may become so large that both client and datacenter win.
It'll become easier and more affordable to distill/train/optimize smaller local models to meet your specific work needs. Multi-agent efforts will orchestrate the balance between using on-prem hardware for most tasks, and then defer to cloud models for more demanding/edge-case tasks. Especially if an organization can manage what data is shared with which model, and most of their user base are everyday casual users.
I also expect cloud providers to eventually start charging different rates for on-peak and off-peak usage, which would incentivize running what you can with the local compute you have otherwise sitting idle across dozens to hundreds to thousands of machines in your fleet.
Anthropic just changed their policy to invalidate any unused credits without refund if you buy your API credits annually. I see that as a sign of unpredictable budgets for cloud models for customers.
I could be wrong and all your future hardware purchases for end-users will be thin clients, without the local processing capabilities of today's laptops and towers. I don't see the hardware manufacturers profiting much from that, but maybe we'll see more companies like AMD declaring they prioritize cloud compute customers over everything else.
With a hardware refresh for cloud computing every 3-5 years, we'll likely see today's compute hardware driving Opus and Sol available on the secondary market for companies willing to build more on-prem capabilities.
Qwen flash-next is at least as good as Sonnet for my uses, and the speed is perfectly acceptable.
I really think the big labs are going to soon start feeling pricing pressure for everything smaller than Opus-tier models, and it will come for those next.
At some point, the real cost of inference/intelligence is going to come due, and the existing open source models will be good enough for 95% of tasks users want (or training their own model(s) will be inexpensive for Apple).
Apple will be there, as usual, to provide secure, on-device, faster inference that the vast majority of their customers will be fine with. Let everyone incur the R&D costs.
Look at what it costs to serve Kimi K3.1 Pro on a B300. It suggests margins of 98-99% at API prices for OpenAI and Anthropic.
Should the shortages for memory and GPUs ever improve, this will only become much cheaper, won't it?
That said, R&D is another story.
Where I am, it's cheaper to purchase multiple claude subscriptions than it is for the power bill from running a local model. Factor in the hardware costs, and you're looking at a break over a decade in the future, for comparable cloud models. Yes, there's privacy, etc, but now you're down into fractions of a percent of users who care about the government reading the memory of their computer.
Apple and Microsoft and other companies will own that front-end experience with the customer, and it’s not going to matter to most customers whose backend gets used for what.
Afterwards we’ll have 128gb apples for under 10k.
It's not as if they are short of money.
Or are you saying that Apple should become a memory manufacturer, rather than working with TSMC?
Also a lot of people work from laptops and running inference on data centres for 10th of the price is a no-brainer. Local models are more of a niche and always will be.
MacBook Air + Claude max sub for 20 years would still cost less before a maxed-out Mac Studio makes sense (not taking into account running costs), even then it’s not a Frontier or as fast.
It's not comparable to huge models of course, but given the power/size ratio here it shows that serious gains are on the table for models that can run fast on a high-end laptop.
A few years back we never thought open source llm would get as good and even the small and cheap model
Its good to have a race and such competition
NVidia aggressively segments the market on VRAM. Apple’s shared memory model is a competitive advantage. Well, it will be when they hand the raw PFLOPS, which is coming.
Yes, I promise you, there are many, many people who are hesitant to build anything real on the assumption that either Anthropic or OpenAI will continue maintaining their services as they are today in perpetuity.
But the hardware. The hardware is fantastic for local LLM's, generative ai, etc. And they seem to maybe finally be waking up to that, in true slow sleepy Apple fashion.
If they'd been a bit quicker about it we could have a higher tier iCloud plan that included a chatgpt plus style setup with the "its private" marketing tag that cant be provided by anyone else. Instead we get Siri being slightly less useless than before.
Soon enough: "Claude, please build me a MacOS clone. Hell, copy the entire Apple ecosystem the way I want it."
Not if they can't sell anything because of ridiculous DRAM prices.
I'm not sure which chip it'll be, but I now for sure I'm not buying new devices until the AI boom has somewhat settled and we kinda understand what we'll be using it for. Until then, every manufacturer will just be guessing what device to build and will be outdated within a few months.
Factoring in the increased cost of memory, Apple's threatened service revenue and the stagnation of ecosystem products like the Mac and iPad, I think they have a really challenging future to navigate. The whole "sell people a B2C ecosystem" business model is more threatened than ever.
I don’t think there will be a choice long-term for Apple the conditions of the market have changed, and you have to change with it or get left by the wayside. They have shown that they will adapt and they certainly haven’t wasted money like their tech competition over the years.
Local models literally cost more in electricity than using far more powerful cloud models cost in inference.
When your options are "dramatically drain your iPhone battery for substandard AI, or have instantaneous cloud AI with no battery drain" what do you think the average user will choose?
And that's before we discuss how ridiculous the pricing is for home hardware that has a chance of running a halfway decent model at a halfway decent speed. Residential electricity + overpriced underutilized hardware means you're going to pay 100X or more per token.
I think the future prospects for local AI are looking worse than ever. It will be far more expensive for a far worse experience for the foreseeable future. It will fill a privacy/control niche, but beyond that, it's not a serious option
Anthropic and OpenAI want to exit because they know they have no moat... and personally, I believe that the sociopaths running the US at the moment want to provoke a great economic crash. The Evangelical ones because they want the armageddon [1], the rich fucks because they took the "you shall own nothing" part for real and want to be the new neo-feudal overlords. And the best way to achieve that is by tying everyone's retirement savings up into their bubbly assets and then crashing down the economy, leading them to buy up everyone's asset at fire-sale prices with the liquid cash they made during the exit.
[1] https://www.theguardian.com/world/2023/oct/30/us-evangelical...
And Apple really wants to sell people who have no cred in the space their stuff because THEY have no cred in the space.
Its basically a club for stupid f**s
There's a name I haven't heard in a long time! Claris (always an Apple subsidiary, since being founded in 1987) renamed themselves to FileMaker in 1998, but apparently they rebranded back to Claris in 2019.
Looks like they've mostly just released annual upgrades to FileMaker for the past decade, but they did add MCP to it last year: https://www.claris.com/blog/2025/why-mcp-matters-for-claris-...
Some of that software has been in use since the 90s and still works. Adding AI to the equation will empower a lot of people.
However, FileMaker is not really targeted at software engineers. In fact using it as a developer will likely drive you up a wall. It's things that feel totally fine to non-devs but completely insane if you're used to something else. E.g. I remember having to click a thousand GUI buttons just to edit a few lines in a script once (regular users just "record" them through the UI). It did all work fine though, very reliable.
They added some more developer-specific features later, but it took a long time. E.g. editing scripts as text(-ish) only came in 2015 (FileMaker 14) and an actual programming language in 2020 (JS, FileMaker 19).
Jev allows for numerous small judgments: Is this email important? Which program should handle this request? Is this operation risky? There's no need for large LLMs to slowly generate long paragraphs of text; all you need is a fast, structured result that returns the message.
Having a local Jev-like model in the operating system for quick decision-making, and then calling the larger model when deeper thinking is needed, would be very useful.
Do you know how big jev is? Decision models are small but the "smart" (aka useable) ones aren't that small.
dabbling with it in browsers but i expect it would answer the mail for you, if you find it accurate enough
Note that it is under the ggml-org. So that's at least not a third-party taking advantage of llama.cpp
With the clean landing page URLs, you just don't know. So GitHub is probably the best source to link
https://github.com/ggml-org/llama.cpp/discussions/23875
I have the same reflex as you, especially something hosted on a .app TLD, I instinctively think it's a copy of something else.
You can always check here on HN as well to get a sense of its authenticity: https://news.ycombinator.com/item?id=49267928
That you can't configure the models independently is a glaring oversight. The alt-space to enter the chat window (at least on Windows) is awful and with no way to disable it I've resorted to just using the app as an easier interface to their downloader.
It does have a nice, clean UI and it doesn't hide the models behind some opaque identifier.
I am super interested to see what the Apple silicon team can deliver when their full attention is turned to local LLM acceleration.
But the problem is a $20 ChatGPT/Claude subscription utterly trounces it in usability, user experience, and performance. By the time we get Opus 5.5 at home, we'll be on like Opus 7 or 8, and I'd bet Sonnet 7 or 8 will dust whatever can be run on the base MacBook of that time. It's fine if people think local 5.5 will be good enough, but it just seems like as AI keeps getting better, you'd want to use the better tooling - nobody has decided a 2010 MacBook was good enough when the M1 came out.
- As AI continues to improve, the gains will become increasingly incremental. This means that your local model will be almost identical in performance as the latest cloud version, for all but edge case problems.
- If your local model is good enough for your current needs, the only "tokens" you are chewing through are utility costs and asset depreciation. Relatively cheap. It is easy to overrun even 200 /month cloud models for serious AI use.
I ran OpenClaw in a VM on my mac a few months back, only using local models. It was painfully slow (no MLX) but what it created was on-par with Claude at that time too.
The models are certainly powerful enough, but the frontier models are more than just an LLM model in the back.
in 5yrs time running an M12 with a Bionic LLM Engine (or whatever they call it) may be good enough for a very very good AI experience. Enterprise / professionals will likely still pay for frontier models.
When comparing API prices and privacy concerns, local might win.
Not good for PII/PPI
For starters, they need to let me get rid of their AI stuff and free up the space. Apple aggressively pushing their AI down the throat of their users means that they suck at it but trying to make it happen using their muscles in other areas.
Just like people figured out how to make their Android TV not suck, it suggests that some time it the future people can make their OS not suck and this will make macOS irrelevant if Apple doesn't fix it first. Unfortunately for Apple, they operate in a system where they must keep growing their numbers.
Such a great company, its a shame that they couldn't figure out this AI stuff. Fingers crossed that this tool actually fixes that by just giving up and let people do whatever they want to do as efficiently as they can on the brilliant hardware Apple creates.
Especially in the modern era, the firmware/software side of macOS definitely doesn't seem like a priority.
If/when Anthropic changes to pure API pricing, or for API-heavy projects, that's when I'd switch to more local models. Of course there are tradeoffs for speed, quality, upgradeability, etc.
I currently run a Mac with 128GB and qwen3.8:27b-mlx. It hits the heat limit of the Mac when working on a 200k LOC codebase for security auditing and hardening for many hours at a time. This isn't as strong/fast as Sonnet, to be sure, yet the cost is only $6K one-time plus the usual electricity, maintenance time, etc.
$6k is not the only cost - electricity adds up over time. And hardware can fail, redundancy is priced into the token prices and subscription plans. Maintenance is mostly negligible, but some updates/upgrades might regress. There’s a lot to SRE to keep it running at that scale.
I run "npx ccusage@latest weekly" and also spot-check Claude Code sessions via "/usage".
> No consumer hardware will give you that level of latency at comparable cost.
You're right. My belief is the frontier AI companies are currently subsidizing real costs for the purpose of rapid growth customer sign ups, and that in a couple years or so local costs will become reasonable, at least for full-throttle work that's queueable.
in 10 years your 10 year old Mac will almost certainly be a relic for running cutting edge local models
What am I missing
For example one of the projects is a web app that's processing user input, making AI suggestions, generating reports, doing ongoing observability, and running batch processing overnight.
Claude currently does these via API pricing, not Claude Ultra personal.
yes, but you pay the Ultra plan, that's the whole point, isn't it? Let's be honest also - qwen3.8:27b-mlx is nowhere near Sonnet, both in size and cap, and if you do $10k a week of usage, you should know that.
I've tried it out on a 512GB dram server with 2*3090s and it works really quite well. Draws enough power to cost something similar to Luna, though.
Another bonus: Here in Germany a huge chunk of the hardware cost can be deducted from the tax bill.
Let the market decide.
Seems like both enterprises and individual users are switching to local models.
To my surprise, people are willing to pay for privacy.
I know a total of zero and individuals or companies using local inference for their workhorse inference.
Local models are just not up to most tasks for now at least.
This seems like the main way it would differentiate itself over existing solutions like LMStudio etc.
Although this might be the shortest timeline as-yet that Apple has "Sherlock'd" 3rd party apps
If all of this were easy, I would happily invest in new dedicated hardware to run it. This could be a full-blown Mac or it could be a souped up HomePod-type device, loaded with RAM. Better yet, make it a revitalized Matter-enabled Airport!
They are going to sell sure 10,000$ and you won’t need anything else for the rest of your life (if you are over 90).
Btw, llama.cpp with llama-swap ain’t that hard to configure if you have access to a potential LLM and have 5h of your life available to that task.
I also want something that integrates super easily with my iPhone and other devices, and hopefully Apple will make it possible to use my server as an intermediate fallback between my iPhone and their cloud models.
In my case it reached the recursive self improvement stage a few weeks ago. It feels like the early days of personal computing in the era of DOS and Apple II.
The hardware is already there. The only limit is product imagination.
Running local models is already extremely easy.
I have a suspicion this app will be a way to add a too many gigabytes to the mac and of course like the Chess.app there will be no way to get rid of it. There are excellent cli apps/GUI apps like this https://github.com/ddalcu/mlx-serve - they are both cli and gui – and even its gui surprisingly nimble (I use this, but I am not related to the project). I am yet to try this https://github.com/ggml-org/Llama-macOS but I've seen only good reviews of it.
I also do not think Apple is doing it for the users who won't explore around. I can't wait to get off this operating system. Next time a friend travels to the continent I will explore getting a Framework laptop via them.
Their customer base includes many non-tech people who don’t want open a terminal window or even know of what a “GitHub” is. And they probably would not trust it as much as an app build by Apple.
I agree it’s easy to run locally (although I do so on Linux), but what’s easy for the HN bubble and easy for the rest of population is not the same.
I don’t find it reasonable at all. Subsidized Subscriptions or api for cheaper models, is the best value.
But the future of AI is so uncertain, regarding model sizes. I don’t think making huge hardware investments is reasonable for individuals.
If you buy your own hardware, then after those four years you still have that hardware, and it likely still holds significant value. Plus you know the local model you're running today won't change or suddenly degrade in four months, let alone four years. Whereas after four years of paying subscriptions and API costs, you have literally nothing.
There is no ZDR at that price.
I would not let a non-local AI have access to my financial or health data.
I think it is a great joy to run most things locally, and still occasionally use the best commercial models when local will not cut it.
LLMs with <30B parameter models seem like the sweet spot. I would rather focus on multiple tiny LLMs that are tuned to specific tasks (SQL queries, finance/accounting, backend/frontend dev, etc.).
For larger models, a $10-40 ish Neuralwatt/Opencode Go or any similar subscription goes so far for the average user that it is ridiculous. We are still in the "youth" era of agents. There are many patterns that work well with smaller LLMs. Or with a big-small LLM combination.
My context:
I own the tiny 16GB M4 Mac Mini and I use LLMs (coding agents) to build software and a few of my products use LLMs directly. Full-time user of LLMs, do not read or write code manually. I run the tiny Qwen series (<10B params) sometimes but a couple $20 subscriptions (Claude, Neuralwatt) take care of my full-time agent usage.
How far away that is, or whether it will ever happen, I don't know. I agree though that the tooling is still so young it's hard to say what shape this type of computing will look like and therefore what kinds of resources will be in-demand.
Same context as yourself. I run 8 - 12Bish models locally on a 24GB M4 Macbook Air and also have a couple subscriptions (OpenCode Go, Mistral Pro) for access to DeepSeek etc. to power through yucky big bloated codebases at work.
There was a task manager which was given context of the tech stack so it could split the incoming requests. Then branch out the tasks to appropriate agents. Combine the generated code - this was a file/module/import manager.
The other way is let tiny models (again I would use agents for different use cases) do the busy work - tool calling, finding evidence, etc. I usually setup agents to re-write tool call history and create fresh API requests without using multi-turn chat with tool call history.
Pi has a subagents package to enable in its harness
They desperately need to fix that.
It SHOULD be totally possible. But it fights you.
Lots of things pop up and can’t be dismissed any other way than though mouse/keyboard on the display.
OS updated and wants to tell you it worked? No command line way to get rid of that. Program wants to do something and one of those system security prompts pops up (“BBEdit wants to access your Documents folder”)? Same problem.
You can set it to auto login and auto update and grant permissions and everything.
And then, one day, it will stop responding and you’ll have to hook it up and log in to see some pointless dialog that just wanted someone to click “OK”. And until you do no work gets done.
Of course Apple will not offer this.
I use oMLX btw, on 32GB URAM, running Hermes with Qwen 3.6 @ 128k context window.
I realize I am a smalltimer but for general day to day sensitive agentic stuff, this is the entry point.
its built just for qwen, healthy competition on tokens/sec
Claris?? Ok, this sounds like it's just someone's personal project. Something like this that was actually going to ship would be in a different org.
https://www.pacifict.com/Story/
If true, it is an astounding story. That said, I'm happy to see that Apple ships a graphing calculator with macOS 27 (albeit, a different one). I may not use it much these days, but I have fond memories of it when I was in university.
It’s the “year of the Linux desktop” meme all over again. Any day now.
- From a happy iPhone Duo user
so you can control and monitor what the whole setup is doing inside and outside the box.
Even better would be a peer to peer model hosting hardware pooling system but that’s probably too much to ask and would expose Apple to liability issues in some jurisdictions.
If Apple were willing to eat some margin on RAM they could use the present RAM bubble to gain ground in the hardware market too. Financially you could call it buying market share.
... Looks like I picked the wrong week to buy parts for a linux-based inference box.