notnullorvoid

I hope this highlights that many (most?) non teach people don't consider or know that their use of AI services is not private.

Which quantization are you using to reach those numbers?

Building a LEGO set is akin to building a model car. Adult collectors and builders of model cars, planes, trains, etc. were a thing before millennials.

> Anti-cheat systems have become controversial recently

Not recently, they have a long history of being controversial.

Ideally we'd have consumer protection laws that prevent companies from doing things like using HWIDs to ban accounts.

I'll stick with Fedora and RHEL.

From what I can tell NixOS still has no support for AppArmor or SELinux.

Changing code without justification is still very bad.

LLMs aggressively rewriting parts of the system break the mental models people have built. The less people have a accurate mental model of the system the more quality will suffer.

Most emotional attachment to code I have seen is a byproduct of understanding the code and being averse to breaking that mental model.

Doesn't mean sometimes people need to look past that and accept change. Reducing attachment though implies less people are understanding the code.

> I am very pro-AI

Now a good preface to add true or not to avoid being targeted.

On a personal level it would be demoralizing to see some billionaires live multiple lifetimes, but only a tad more demoralizing than seeing the wealth disparity we already have and the changing line of idiot faces at the top.

Perhaps if you assume we will continually be offered highly valuable medical advances into the future without humans who understand it.

That's a big assumption to make, so I hope you at least have some proof to back it up.

If one can build a MCP with proper protections, they can certainly do the same for their API/CLI/SDK.

> People and their AI agents are making the act of being on the internet extremely annoying, and the problem is about to get much worse.

As much as I have a distaste for what passes as "AI agents" as a technology, I think it's a mistake to put any of the blame on the tech. People are the problem!

We've been making a societal mistake for a long time, accepting and even celebrating obnoxious people who do obnoxious things. Letting your AI agents loose on others is one such obnoxious thing.

They are more than a accessory. You wouldn't say someone who funds a contract killing is merely an accessory to murder. VCs should likewise be considered as principal offenders, unless there is proof that the company was not transparent with the VCs.

Yes they are quite good, but are not able to run on a 16GB RX 9070.

Quantized Qwen 3.8 Flash Next could maybe run eventually on that card with a highly optimized inference engine that dynamically caches the hottest layer experts. Even then you run into some hard limits.

Traffic jams can be a feature, if it's the game Prototype where you could barrel through cars like a freight train.

Qwen 3.8 27B is so far the closest I have run, though it suffers on speed compared to Gemma4 26B MoE model which I still use. Neither are going to match Opus or Sol though, but they can be as fast or faster depending on what you are using them for.

I don't use a harness, all the mainstream ones I have tried have tanked my productivity. I know that's not a common sentiment, but it's been my experience. None seem built for the way I work. I program mostly in my head first away from keyboard, then go type it out (faster than it would take to describe the solution to an LLM). Also a perfectionist who likes to learn, and tends to work on out of distribution problems. All I need is a simple chat interface for light research, quick small scoped prototypes, and generating simple scripts.

Not saying a harness is out of the question for me, just all I have seen and tested so far are not for me. Maybe if someone builds a more deterministic harness that doesn't rely on plain English skill files that bloat context and only sometimes do what you want.

I haven't yet, though plan to when this model is released. The models that I've been daily driving (Gemma 4 26B, Qwen 3.8 27B) have fit nicely on my 3090. I think FreeToken only offers a perf increase for MoE models that you can't feasibly fit in VRAM.

Yeah I'm sometimes unsure how to get best perf out of llama.cpp, and honestly thought it already did what the FreeToken paper discusses, but from everything I've been able to find since llama.cpp has no dynamic expert cache for GPU. An RFC discusses adding such capability and there's impressive results some are claiming from a fork, but I had to stop reading the thread, reading all the LLM generated comments and summaries from people was making me dizzy.

RFC here https://github.com/ggml-org/llama.cpp/discussions/24528 which also links to some experimental implementations throughout the thread.

> It is still slow, a lot slower than what you are used to with claude and co.

That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.

In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost.

> Then when it runs for 30 minutes for something claude needs 5, your device will get hot.

I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.

> And even a used 3090 is apparently now between 1-2k.

Yeah I guess the price went up significantly in the last couple months, used to be hovering around 1k. 3090 isn't the only option though.

True, but you can link them up over thunderbolt or Ethernet. If your goal is to run local LLMs, not all weights need to live on the same computer. You can segment the workload by layers and pass the activations along the lower bandwidth interconnect with a small perf penalty. Also you get double the CPU/GPU cores allowing for better multi user/agent performance.

It will be interesting to see the intersection of this with inference engines like FreeToken which improve distribution of work for MoE models across CPU/RAM and GPU/VRAM.

If all it takes for a competitive model to run locally at good speeds is a used 3090 and some DDR4, then we might be in for the year of local AI.

https://github.com/FlashML-org/FreeToken

It needs to be a little less than double the overall price for 256GB option, otherwise it's better to get 2 256GB and link them.

Try it out with your own prompt, I suspect you'll be surprised.

I've done a few variations, I've been impressed with all of them. My favourite so far has been "Generate an SVG of a turtle flying a kite", result: https://imgur.com/a/bdKJPV4.

Some will say conflating flying and flying a kite is bad, but I like it.

Congrats on the release! Nice to see transitions API go away, though I am not entirely convinced of the async model. It's capability is impressive, yet I do not like hiding what values are async, and throwing to await like React has never seemed a good solution.

Yes latency, and the usual preference of ownership over rentership. Similarly their are benefits to running local AI too, like data privacy and control.

The gains wouldn't be "free lunch", it's the result of time and effort researching optimal design and architecture.

Even if the idea of "no free lunch" was taken liberally discounting the cost of research, it would only be limiting to systems built from a foundation of optimization, but that's not the case. The foundation so far has been one of brute force scaling. Usually meaning there is lots of room for optimization.

If the brain does rely on quantum effects, it's still possible the quantum effects in use are able to be simulated efficiently on a classical computer. For example if it's a matter of signal transfer rather than quantum computation, that could be simulated rather easily.

You can find it today for gaming. Even despite the outlandish rise in hardware costs, there is very little demand for cloud gaming.

It may not make financial sense for someone retired, not into tech, and/or data privacy to host their own LLMs. However if usage of AI in day to day lives continues to increase, I think it will eventually make sense for the majority.

Many tasks suited for AI assistants are background asynchronous tasks. They can run in the downtime where immediate demand is low, keeping overall utilization high enough.

Your argument is similar to those who argue that owning a GPU for gaming doesn't make sense when you can stream from something like GeForce Now. However like with gaming locally (improved latency) there are also benefits to local AI (data privacy).

We've barely even started on optimizations like advanced language aware grammars, and specialization routing (dynamically loading fine tunes or seperate weights for specific tasks or languages).

They should've done $0.01 off.

Effectively the same discount, while being more outrageous.