sosodev

In my experience, even with basic project concepts the small models struggle to spin up greenfield stuff. There's just too many decisions to be made and they're not good at that.

Modifying existing code is way easier if you don't expect it to be smart about it. Don't say "add X feature" and let it explore the codebase and build its own understanding. Point it at the relevant files and say "the goal is to add X feature to this code, follow Y guidelines". Now you've done the hardest part of making the decisions and it just has to follow instructions while coloring within the lines.

The license scheme makes me sad. It reads like a subscription pretending to be a one-time purchase.

I think it really just depends on your goals. Slow tokens per second is fine by some people if they cost a fraction of a single node setup that can run a trillion param model. If you’re actually running a small business and want to have multiple users getting a good experience in parallel then yeah I think you need a single node. At that point you can afford it I suppose.

I don’t know what the scaling for multiple strix halo boards looks like in practice. From what I understand each server has to process the model in serial. Meaning server A has 1/4 the weights and sends server B the results to process and so on. So you don’t get compute scaling just memory scaling.

I read the comment, thanks. I just disagree with your cost estimate. Even for a small business that needs high throughput they could probably do it for far less than $300k if they aren’t just blindly buying the first big nvidia setup they can.

I don’t have any particular model in mind, sorry. My data is just rough estimates based on my experience with a single node setup. You might need to opt for a 2 or 3 bit model to get the full context window. The KV cache memory consumption as well overall performance will be heavily dependent on the model’s architecture. The performance too will depend a lot on the inference server chosen and its configuration. I suspect a sub-agent running a much smaller model would be the ideal way to get the latest knowledge via web search and summarization.

I’m not trying to say that this would be a great experience or really compete with just buying a subscription to the top models. Rather I just wanted to point out that $300k is an absurd estimate for a trillion param model meant for personal use.

It depends entirely on what you want to do and think is feasible. Small models can almost certainly run on the computer that you already have. They can do good tool calling.

You can run a trillion parameter model with decent quality for far less than $300k. A cluster of 4 AMD AI Max 395+ boards with 128GB unified memory each can be had for around $15k. That would run the 4-bit quant of a trillion param model well enough for personal use. At full use the cluster would only be consuming around 400-500W of power too. That's about the same as one high end graphics card.

That's still a lot of money, but most people don't really need a trillion parameter model. If privacy is more valuable than the frontier capabilities then they could almost certainly get by with much less.

They have it, we just haven’t enabled them. The smart model with a chat box is the wrong abstraction for local. Ideally we would have it built into applications as a clear and easy to use opt-in feature. Like allowing a user to index a folder on their hard drive and then search it semantically via embeddings. You could do that on fairly low end hardware these days. Like 2GB of RAM with any processor made within the last 10 years.

Note that AA's coding index is only made up of two benchmarks: Terminal-Bench Hard and SciCode. I'm skeptical that it makes a good coding index. It ranks Gemma 4 31B above Deepseek V4 Flash. Having used both of those models for a broad variety of coding tasks I would choose Deepseek every day.

Petsitter's default tricks doesn't seem to do much for Qwen3.6, right? JSON mode could be useful I suppose, but that's not really going to make it better at writing code. Do you have any other example tricks? I'm having a hard time understanding how I would apply them.

Meh. My server can run these models for neglible power draw (like ~130W fully maxed out). That's with ~30 tok/s which isn't that bad. I do agree that they're still nowhere near as good as the frontier models though. I do lean on those when I need to get something done with better quality or at a faster speed.

I've also been using Deepseek V4 pro/flash for some work stuff and I do find them to be much closer to frontier capability. I may try running flash at home soon for very patient edits. :)

Any artifacts or blogs I can check out? I'm curious how you manage to make them all useful in parallel. I have a hard enough time getting one instance of Qwen3.6-27B being useful full time haha.

I think this is overselling their capabilities. I've used Gemma 4 and Qwen 3.6 quite a bit on my strix halo home server. They're great models and the dense variants are significantly better, but they're still very far behind the frontier. If you boot up Gemma 4 MoE and OpenCode/Pi and expect to perform anything like Claude Code or Codex you're going to be very disappointed.

My strix halo board is feeling more useful and less toylike with the recent performance gains combined from MTP, better quantization, and generalized performance improvements across the stack. For example, I can run Unsloth's Gemma4-31B 4-bit QAT model with around 30tg and 200pp. I don't find that to be too slow at all. Particularly because it's nearly full accuracy and good enough for a lot of different stuff I throw at it.

I think it also helps that I'm using my machine to do home server stuff. It excels at all of the traditional workloads. Then I can lean on the AI to help with automation here and there. I find it deeply satisfying.

The problem with this question is that it encompasses a huge spectrum of capabilities and expectations. If you can only run an 8B model and expect it to be good at vibe coding / one shotting things you're going to have a bad time.

If you're able to run a model on the scale of ~30B, you can find that with a reasonably scoped and well defined task they do very well. I've found both Gemma4-31B and Qwen3.6-27B to be the best in this range at the moment. You can swap in the MoE models for faster inference, but they are noticeably worse at most tasks. They can one-shot / vibe code tasks with small scope, but still do much better with guidance.

If you really want frontier-like capabilities, you'll probably need at least 128GB of memory and either huge compute or a lot of patience. Most people just don't have either the money or the patience to make these local models work.

The patience required for local model usage goes far beyond just waiting for tokens though. It takes a lot of effort to get things configured and working properly for your workflow and hardware.

I think it heavily depends on what you're asking the model to do. Qwen3.6, both 27B and 35B-A3B, do agentic tool use very well. Their decision making is sus, but the dense model is decent in that way. A 4-bit quant for either of those can run on many home systems with a bit of configuration.

The biggest issue I've noticed is that the chat templates for open models are really hit or miss. The default Qwen3.6 chat template mostly works these days, but depending on your workload it may cause major issues. There are plenty of "fixed" chat templates on hugging face, but people report mixed success. It really seems to depend a lot on what the tool you're using expects.

I don’t know why you’re getting downvoted. It’s true. Averaged across a wide variety of benchmarks Fable is the only Anthropic model that performs better than GPT 5.5 xhigh.

I wonder if model distillation will continue to work as well as it has. Given hidden reasoning, the ever expanding number of expected capabilities, a serious compute shortage, the looming possibility of model collapse, and dramatically higher API costs I would guess that it's getting much harder to do.

Do you have any resources to share regarding independent expert training? I was under the impression that it's not feasible.

Support requests have always been the weakest link in the security chain for big corps. I've had accounts of mine turned over with 2FA disabled by humans before. I guess we shouldn't be surprised that the LLMs are doing the same thing.

The simple fact that 2FA can be removed by low level support staff drives me mad. It defeats the whole purpose of the process.

Most of the examples they've chosen seem.. not good? What an odd mix of bad game engine and AI slop. I can't imagine that this stuff makes good training data for real-world applications.

I’m not sure I understand your question. Every interaction you have with a model in a web page does the same thing in the backend. It feeds the whole conversation history, perhaps with a bit of processing, into the model so it can process the next generation. Filling the context window is how these models retain coherence.

Isn’t this contradictory to your point? They dropped it, collected data, and then reverted when the evidence suggested they made the wrong choice.

I also cut off JetBrains recently after a long relationship with their tools. I agree with the points made by the author. The tools are clunky resource hogs for seemingly no reason. I was really excited when JetBrains announced Fleet and promised a lightweight UI with the old analysis engines as lighter background processes. It seemed like it would solve a lot of the problems I had with their IDEs. That never materialized though. They say that Fleet integrated into Air, but Air is not an IDE. So now we're just left with the diminishing value of their traditional IDE offering and some floundering attempts to get into the AI market. What a shame.

Why is nobody on HN talking about this? Unless this is a very advanced fake, it seems like the first proof that humanoid robots are actually capable of real labor.

If Wall Street was so wise they would only reward meaningful layoffs. Laying off 10% of a company by stack ranking every team accomplishes nothing. Particularly if the company just hires the same number of cut people next quarter.

If a tree has a dead branch, you cut it off. Cutting off 10% of the leaves evenly distributed among branches will remove some dead leaves, but it leaves the source of the problems unaddressed.

Well put. I too am optimistic that, in the long term, good will prevail and we'll be stronger because of the suffering. I also agree that there's happiness and meaning to be found in presence and local life. However, it feels quite hard to let it wash over me when I spend so much time at work. The hippie lifestyle is very tempting, but I want stability and a family.

Is the US one of the best places for career growth and income? I'm 30. I've been in the tech industry for several years. During the COVID tech boom I would have agreed. I made insane amounts of money for a new grad. Then I was laid off, cut, or just downright fired for unethical reasons a few times. The combination of reduced demand for software folk, the further loss of autonomy and meaning thanks to LLMs, and that blight on my resume has made it very difficult to believe this is a great place for a career. I know I'm not alone in thinking this. Many of my techy friends, and strangers that I've met, share a bleak sentiment about the future of our careers. It seems that negativity stretches far beyond tech lately. White collar work seems to be more hopeless than ever before.

I've retreated to a public servant tech role. I was drawn to the theoretical stability of this position and the idea that might effort might do some genuine good for my local community. After several months of being here I'm skeptical that I'll be able to do any good because I'm no good at the internal politics. The stability is somewhat comforting, but only in the sense that I will not starve to death. Inflation seems like it will continue to outpace my potential earnings.

Does anybody have more insight into the demand for electricians during data center construction? This article is really light on the details. I was researching it recently and got the impression that the majority of electricians hired during DC construction are much more specialized than the average residential electrician. It also seems like large data center construction typically demands a magnitude of several hundred electricians during the peak of construction. Which to me sounds like a lot less demand for the average electrician than some of the news outlets have been claiming.

I sense that the frustration you feel is that professors are able to make choices based on their values, but the average person is not. That is broadly speaking, of course.

I think it is a great shame that we live in a modern world where we do we must to survive regardless of how it makes us feel. I suspect it is the root of much suffering.