hasperdi

I have several my own little success stories:

- For polishing Whisper speech to text output, so I can dictate things to my computer and get coherent sentences, or for shaping the dictation to specific format eg. "generate ffmpeg to convert mp4 video to flac with fade in and out, input file is myvideo.mp4 output is myaudio flac with pascal case" -> Whisper -> "generate ff mpeg to convert mp4 video to flak with fade in and out input file is my video mp4 output is my audio flak with pascal case" -> Local LLM -> "ffmpeg ..."

- Doing classification / selection type of work eg. classifying business leads based on the profile

Basically the win for local llm is that the running cost (in my case, second hand M1 Ultra) is so low, I can run large quantity of calls that don't need frontier models.

Correction, my GLM-4.6 models are not Q4, I can only run lower ones eg:

- https://huggingface.co/unsloth/GLM-4.6-GGUF/blob/main/GLM-4.... - 84GB, Q1 - https://huggingface.co/unsloth/GLM-4.6-REAP-268B-A32B-GGUF/t... - 92GB, Q2

I ensure that there are enough RAM leftover ie limited context window setting, so no swapping.

As for GLM-4.5-Air, I run that daily, switching between noctrex/GLM-4.5-Air-REAP-82B-A12B-MXFP4_MOE-GGUF and kldzj/gpt-oss-120b-heretic

I bought a second‑hand Mac Studio Ultra M1 with 128 GB of RAM, intending to run an LLM locally for coding. Unfortunately, it's just way too slow.

For instance, an 4‑bit quantized model of GLM 4.6 runs very slowly on my Mac. It's not only about tokens per second speed but also input processing, tokenization, and prompt loading; it takes so much time that it's testing my patience. People often mention about the TPS numbers, but they neglect to mention the input loading times.

Even LinkedIn is now down. Opening linkedin.com gives me a 500 server error and Cloudflare at the bottom. Quite embarassing.

Sure Bun has its benefits, but I don't see the strategic reasons why Anthropic is doing this

That's not the case. Some are dense and some are hybrid.

MOE is not the holy grail, as there are drawbacks eg. less consistency, expert under/over-use

As pointed out by a sibling comment. MOE consists of a router and a number of experts (eg 8). These experts can be imagined as parts of the brain with specialization, although in reality they probably don't work exactly like that. These aren't separate models, they are components of a single large model.

Typically, input gets routed to a number of of experts eg. top 2, leaving the others inactive. This reduces number of activation / processing requirements.

Mistral is an example of a model that's designed like this. Clever people created converters to transform dense models to MOE models. These days many popular models are also available in MOE configuration

and can be faster if you can get an MOE model of that

With quantization, converting it to an MOE model... it can be a fast walk

One thing that the Chinese are really good at is cost innovation, reducing costs as many ways possible to make their products affordable for the majority. Their aim for good enough quality.

I bet the sales ratio of the Chinese vs the German sharpeners exceed 20:1

If they did adopt JSON then it wouldn't be XMPP anymore.

As a user, I don't care much. But my experience with XMPP is that was not as solid as other solutions, including closed source ones. I could've been issues in clients' implementation, but overall it wasn't great

Fun fact, you can use Ghostty and vibecode the shader you want. In fact, the other day I used Claude Code to create me a custom CRT shader.

> .001% higher than the flu

That isn't true. Just from this paper https://pmc.ncbi.nlm.nih.gov/articles/PMC9115089 ... COVID-19 killed roughly five to seven times more hospitalized older adults than influenza.

Anecdotal, my uncle, several friends' relatives died from COVID during those lockdowns. I don't know / heard anyone died because of flu (in my extended circle of people I know)

I just tried that on their playground:

7B:Hi! I'm Olmo 3, an AI assistant created by the non-profit organization Ai2. I'm here to help with questions, ideas, or tasks you have—just let me know what you need! How can I assist you today? Rawr!

32B: Hi! I'm Olmo, a helpful AI assistant built by the Allen Institute for AI (Ai2). My knowledge is up to December 2024, and I'm designed to assist with a wide range of tasks. How can I help you today?

If you share that API key here...

- we all get to use free LLM

- they'll learn to do it properly next time

So it's a win-win situation

If you don't mind closed source paid app, I can recommend MacWhisper. You can select different models of Whisper & Parakeet for dictation and transcription. My favorite feature is that it allows sending the transcription output to an LLM for clean-up, or anything you want basically eg. professional polish, translate, write poems etc.

I have enough RAM on my Mac that I can run smaller LLMs locally. So for me the whole thing stays local

It's not fine. Some apps particularly banking apps have developer mode detection and refuse to work if developer mode is enabled.

It a projection of what they could do. ie. logical step

The whole SafetyNet and "secure chain" things are PITA, eg. ChatGPT app wouldn't work if the phone bootloader isn't signed by Google. Lots of banking app wouldn't work, HSBC banking app for instance wouldn't allow login if Android developer mode is enabled.

Dave2D has additional info. User upgradable RAM and SSD, but not CPU.

https://www.youtube.com/watch?v=356rZ8IBCps

Does anyone know the price?

AMD 395+ is more bang for the buck IMO.

GMKtec EVO-X2 vs GX10 vs MacBook Pro M4 Max

  Price:  $2,199.99 / $3,000 / $5,000
  CPU:  Ryzen AI Max 395+ (Strix Halo, 16C/32T) / NVIDIA Grace Blackwell GB200 Superchip (20-core ARM v9.2) / Apple M4 Max (12C)
  GPU:  Radeon 890M (RDNA3 iGPU) / Integrated Blackwell GPU (up to 1 PFLOP FP4) / 40-core integrated GPU
  Memory:  128GB LPDDR5X / 128GB LPDDR5X unified / 128GB unified
  Memory bandwidth:  ???GB/s / ~500GB/s / ~546GB/s
  SSD:  1TB PCIe 4.0 / 4TB PCIe 5.0 / 1TB NVMe
  GPU advantage:  Similar (EVO-X2 trades blows with GB10 depending on model and framework)
  Network:  2.5GbE / 10GbE / 10GbE (via TB)
  Direct cluster:  40Gb (USB4/TB4) / 200Gb / 80Gb
  Portable:  Semi (compact desktop) / No / Yes
  Free Mac included:  No / No / Yes
  Free monitor:  No / No / Yes
  Linux out of the box:  Yes / Yes / No
  CUDA dev environment:  No (ROCm) / Yes / No

Regular viewers of Asianometry channel will probably know that this is unlikely

I think a lot of people would disagree with that statement that he is a patriot. My own opinion is that he's more of a profiteer than a patriot.

Reading the article, I get the feeling that the author is not in the manufacturing (automation) business. Just fantasizing.

I bought Pebble time many years ago, it arrived with the front panel gaping open, it wasn't properly attached to the watch body. I emailed them about this and it took weeks for them to reply... giving me a boilerplate answer. I had to spam as many people in the company as I could including the CEO, to get a replacement.

Never again.

yes, all Mx processors have unified ram

Does anyone know anything similar for Android?

I want to be able to choose the number. You can charge me more, that's fine!

Also... commuting is much less eco friendly

Deno and latest versions of Nodejs run TS code without transpilation