Thanks, I’ve changed the submission title from the author’s original.
4,399 karma · joined December 12, 2007
https://unsloth.ai/docs/new/studio#unsloth-as-an-api-endpoin...
Llama.app/.cpp and Unsloth Studio on Mac already provide guidance to pick the optimal model for the available memory, all model wheights hosted by huggingface. Both apps can use the local huggingface model cache locally.
If you make it configurable, the user could also run a separate machine with an NVidia card and Linux to serve tokens on the LAN. Or use OpenRouter or any other cloud hosted model offering (DeepSeek, Z.ai, etc.).
One concern I have is about sending private info to Anthropic. So it would be great to have the option to use local AI.
Local options should support the OpenAI API that seemingly is the standard for these connections, as provided by Llama.app and Unsloth Studio for example. That way one could run against a local model and keep personal information private.
That of course begs the question if a local model is smart enough for the tasks at hand.
I'm also trying out OOMU, a similar but closed source personal assistant app for MacOS. It emphasizes privacy with local models, and even has an 'Airplane' mode. The app has a built in llama.cpp server and uses Gemma-4-4B, their own router model plus a vision model.
Crazy how my gut feeling was more than vindicated.
I do like the sound of Verscheissification.
As Picard said: Enshitify! (Apologies to my Trekkie friends.)
You can chat with any model in the built-in WebUI, connect other apps (coding agents, chat UIs, editors), or use the API directly. Models load when requested and unload when idle, so they don't take up memory when not in use.
> Features
- 100% local — Models run on your Mac; no data ever leaves it
- Small footprint — 4 MB native macOS app
- Zero configuration — models are auto-configured with optimal settings for your Mac
- Model recommendations — a built-in list of models your Mac can run, installable in one click
- Standard storage — models live in the Hugging Face cache, shared with llama.cpp and other tools
- Built on llama.cpp — from the GGML org, developed alongside llama.cpp
> Installation
To install llama.cpp , run:
brew install llama.cpp
brew install --cask llama-appOfficial repo, also has documentation how to configure server parameters:
https://github.com/ggml-org/Llama-macOS
Small tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date with llama.cpp, since llama.app is on a slower release cadence.
Also, models installed with the hugging face CLI (hf) are picked up by llama.app automatically. The CLI will keep the model cache updated, e.g. when models get updated.
Llama.cpp became part of Huggingface recently.
https://luxurylaunches.com/transport/gabe-newell-explorer-ve...
https://www.forbes.com/sites/deajusufi/2026/06/13/gabe-newel...
High-Performance AI on a Budget: Optimizing llama.cpp for Qwen3.5 Inference on a Dual-GPU HP Z440
But this is of huge interest to carriers, since it allows them to skip the PSTN/peering cost when the callee endpoint is an IP phone.
There is private ENUM for carrier use I recall, not sure what the current status is, with LTE/VoLTE, RCS etc.pp.
http://dam3d3.free.fr/PFE/Pathfinder/GSMA_PathFinder_WebSite...
Here the list of countries that have ENUM delegated for their country code.
https://www.itu.int/en/ITU-T/inr/enum/Pages/delegations.aspx
Decentralized and under user control, no shitty silos like FaceTime, WhatsApp.
ENUM stands for “Telephone Number Mapping.” It is essentially a bridge between the world of telecommunications and the Internet. With a single ENUM domain, you can combine all your contact options under your familiar phone number:
It would have provided geographical information based on a domain encoded grid, not for human but machine consumption (e.g. acme.2e5n.10e30n.geo).
https://en.wikipedia.org/wiki/.geo
In a similar vein there is the 'e164.arpa' domain for mapping telephone numbers.
https://huggingface.co/collections/unsloth/gemma-4
Edit: Sorry, I'm not sure if this is a quant, but it says 'finetuned' from the Google Gemma 4 parent snapshot. It's the same size as the UD 8-bit quant though.
https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
For the best quality reply, I used the Gemma-4 31B UD-Q8_K_XL quant with Unsloth Studio to summarize the URL with web search. It produced 4.9 tok/s (including web search) on an MacBook Pro M1 Max with 64GB.
Here an excerpt of it's own words:
Unsloth Dynamic 2.0 Quantization
Dynamic 2.0 is not just a "bit-reduction" but an intelligent, per-layer optimization strategy.
- Selective Layer Quantization: Instead of making every layer 4-bit, Dynamic 2.0 analyzes every single layer and selectively adjusts the quantization type. Some critical layers may be kept at higher precision, while less critical layers are compressed more.
- Model-Specific Tailoring: The quantization scheme is custom-built for each model. For example, the layers selected for quantization in Gemma 3 are completely different from those in Llama 4.
- High-Quality Calibration: They use a hand-curated calibration dataset of >1.5M tokens specifically designed to enhance conversational chat performance, rather than just optimizing for Wikipedia-style text.
- Architecture Agnostic: While previous versions were mostly effective for MoE (Mixture of Experts) models, Dynamic 2.0 works for all architectures (both MoE and non-MoE).