11,905 karma · joined May 21, 2009
I personally prefer Pi as I like the fact that it's minimalist and extensible. But some people just use Claude Code, some OpenCode, there are a ton of options out there and most of them can be used with local models.
You can get open models that are competitive with Sonnet 4.6 on benchmarks (though some people say that they focus a bit too heavily on benchmarks, so maybe slightly weaker on real-world tasks than the benchmarks indicate), but you need >500 GiB of VRAM to run even pretty aggressive quantizations (4 bits or less), and to run them at any reasonable speed they need to be on multi-GPU setups rather than the now discontinued Mac Studio 512 GiB.
The big advantage is that you have full control, and you're not paying a $200/month subscription and still being throttled on tokens, you are guaranteed that your data is not being used to train models, and you're not financially supporting an industry that many people find questionable. Also, if you want to, you can use "abliterated" versions which strip away the censoring that labs do to cause their models to refuse to answer certain questions, or you can use fine-tunes that adapt it for various other purposes, like improving certain coding abilities, making it better for roleplay, etc.
In fact, I use the pi agent, which doesn't have command sandboxing, it's always in yolo mode, I just run it in a container and then I get the benefit of not having to confirm every command, while strictly controlling what I share with it from the beginning of the session.
How can people be so naive as to run something like Claude anywhere other than in a strictly locked down sandbox that has no access to anything but the single git repo they are working on (and certainly no creds to push code)?
This is absolutely insane behavior that you would give Claude access to your GitHub creds. What happens when it sees a prompt injection attack somewhere and exfiltrates all of your creds or wipes out all of your repos?
I can't believe how far people have fallen for this "AI" mania. You are giving a stochastic model that is easily misdirected the keys to all of your productive work.
I can understand the appeal to a degree, that it can seem to do useful work sometimes.
But even so, you can't trust it with anything, not running it in a locked down container that has no access to anything but a Git repo which has all important history stored elsewhere seems crazy.
Shouting harder and harder at the statistical model might give you a higher probability of avoiding the bad behavior, but no guarantee; actually lock down your random text generator properly if you want to avoid it causing you problems.
And of course, given that you've seen how hard it is to get it follow these instructions properly, you are reviewing every line of output code thoroughly, right? Because you can't trust that either.
Nemotron 3 Super doesn't perform quite as well on benchmarks as the similarly sized Qwen3.5 122B A10B model, but it goes faster and is cheaper to run.
https://artificialanalysis.ai/?models=gpt-oss-120b%2Cmistral...
Now, you're not exactly comparing apples to apples there, since the training process (mix of data for pre-training, and the fine tuning stages of instruction turning, RLVR, etc) could have as much or more impact on how well it does as the architecture itself. Nemotron 3 Super does get better scores on performance than GPT-OSS 120B and Mistral Small 4, both also similarly sized open weights models.
Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of performance, as the linear context mechanism can work for the tasks its well suited for while allowing attention to play to its strengths.
The recent Nemotron 3 Nano and Super models from NVIDIA are hybrid architectures this way, with most of their context layers as Mamba while retaining enough attention to continue to be competitive on the more complex tasks that require the quadratic attention.
See https://magazine.sebastianraschka.com/i/168650848/18-nemotro... for some discussion on this architecture
Why do you think that "ethical vegans" like the "taste of plants" any more than anyone else? The whole point of being an ethical vegan/vegetarian is to not consume animals, not because you don't like the taste.
Health conscious folks would definitely choose these over hamburgers. Sure, they're not perfect from a health food point of view, but they're lower in sodium and saturated fat than your average hamburger patty. So from a health conscious point of view, it's a decent substitute.
Then there are the people who just want to reduce their meat consumption overall. Maybe they're not vegan or vegetarian, but they're trying to watch their saturated fat intake, or reduce their carbon impact, or they suffer from gout and are trying to reduce the amount of meat they eat to ease that.
Sometimes you just want to go out with your friends for a burger, and the Beyond patty can make a better substitute than a black bean or mushroom patty that used to be common.
And at most restaurants, I've never noticed a "premium" for it, it usually costs the same as a beef patty; it just provides another option, for the days I want to skip meat. I have, for a long time, done a low meat diet; I don't avoid it entirely, but I try not to eat it at every meal. It provides a nice alternative for that.
Is it a bit of a niche market? Sure. But, not every product needs to be for everyone.
And there are still plenty of hybrid architectures. Nemotron 3 Super 120B A12B just came out, it's mostly Mamba with a few attention layers, and it's pretty competitive for its size class.
But yeah, these different architectures seem to be relatively small micro-optimizations for how it performs on different hardware or difference in tradeoffs for how it scales with the context window, but most of the actual differentiation seems to be in training pipeline.
We are seeing substantial increases in performance without continuing to scale up further, we've hit 1T parameters in open models but are still having smaller models outperform that with better and better training pipelines.
Yes, the New York proposed law is far worse, and we absolutely should be pushing back against that. And Facebook doesn't care, because they only care about moving the liability onto the OS vendor, not on actual privacy.
But still, just because this was supported by Facebook doesn't make it bad. Sure, Facebook doesn't care about privacy, but they do care about not being liable for this, and in this case, they're right, it is actually much more efficient to centralize this function in the OS, and it happens that that way it can be done in a privacy preserving way as California's law shows.
We shouldn't object to a reasonable law just becasue it might, theoretically, pave the way to an unreasonable law.
In fact, this is put in place as an alternative to the kind of law being enacted elsewhere, right now, which is much worse; the ones requiring ID based verification for accessing many online services. This one provides an alternative solution, which is far more privacy preserving, and leaves all of the actual power in the hands of the owner of the computer.
It requires that operating systems provide a way, at account setup, to specify the age or birthdate of a user, and provides an API for indicating which age range the user falls in (under 13, 13 to 16, 16 to 18, or over 18) to an application, so the application can use that information to comply with any laws or regulations relating to the age of the user.
It doesn't make any requirement that the parent actually truthfully put that information in. It doesn't require that anyone verify the information. It doesnt provide for any requirement that a child not set up a user themselves. It explicitly calls out that there is no liability on any of the parties if one user uses a computer under another user's account.
So all it's doing is saying that there must be a reasonably accessible mechanism for a parent to indicate a child's age so that rough information about which age range the child is in can be provided.
Now, is it perfect? No.
It does seem a bit over broad as there are lots of things which be classified as computers uner this, like routers, smart TVs, graphing calculators, cars, etc. Having to provide account setup with age and an API to accesss it in all of these environments could be a bit of a lift in the time frame given. And it doesn't leave a lot of time for something like standardization of Unix APIs between operatings systems, so for systems not running graphical environments I'm sure we're going to get a bunch of different solutions from different OSes as everyone sticks it in a different place and provides a different way to access it. And this would need to be a new feature added into long-term supported maintenance releases operating systems.
So yeah, could it have been done better? Yes. Is it likely that they are actually going to fine OpenWRT developers if they don't implement this? I doubt it; it's pretty clear that the legislative intent is desktop and phone OSes, and other mass market consumer oriented devices that might offer app stores.
So yeah, I see some issues, but overall this seems like the right way to do things; just provide a way for parents to set an age on their children's account, and then provide that to any apps that might need to do age verification. That's it.
I work in aviation, a highly regulated field. And that's a good thing. It does take some work to regulate well; there has been a migration in aviation to more prescriptive regulation about how things need to be, to less prescriptive like what the ultimate performance needs to be. But yeah, the aviation regulations aren't that you have to implement something a specific way, but that you have to be able to show that your aircraft has no more than a certain probability of catastrophic failure (where the probability varies base on certain things like the size and type of aircraft).
For this age verification law, all that is required is that there is an API provided for this purpose, and there is a way for the owner of the machine to set up user accounts with age information indicated, and that the APIs need to provide several rough age ranges, not specific birthdays.
I am so puzzled by everyone who objects so strongly to these operating system based opt in systems; all it does is provide for a way for a parent to indicate the age of a child's account, and an API for apps and browsers to get that information. If you're the owner/admin of a system, you get to set that information however you want, and it's required that it only provides ranges and not specific birthdays in order to be privacy preserving.
Also, which sites that are impacted by the age verification laws are involved in grooming in any way?
Please be specific.
On Windows, I think you're right, it's max 96 GiB to the GPU and it requires a reboot to change it.
What do you mean? On Linux I can dynamically allocate memory between CPU and GPU. Just have to set a few kernel parameters to set the max allowable allocation to the GPU, and set the BIOS to the minimum amount of dedicated graphics memory.
Trying to run LLM models somehow makes 128 GiB of memory feel incredibly tight. I'm frequently getting OOMs when I'm running models that are pushing the limits of what this can fit, I need to leave more memory free for system memory than I was expecting. I was expecting to be able to run models of up to ~100 GiB quantized, leaving 28 GiB for system memory, but it turns out I need to leave more room for context and overhead. ~80 GiB quantized seems like a better max limit when trying not running on a headless system so I'm running a desktop environment, browser, IDE, compilers, etc in addition to the model.
And memory bandwidth limitations for running the models is real! 10B active parameters at 4-6 bit quants feels usable but slow, much more than that and it really starts to feel sluggish.
So this can fit models like Qwen3.5-122B-A10B but it's not the speediest and I had to use a smaller quant than expected. Qwen3-Coder-Next (80B/3B active) feels quite on speed, though not quite as smart. Still trying out models, Nemotron-3-Super-120B-A12B just came out, but looks like it'll be a bit slower than Qwen3.5 while not offering up any more performance, though I do really like that they have been transparent in releasing most of its training data.
So by doing so, this calculator is telling you that you should be running entirely dense models, and sparse MoE models that maybe both faster and perform better are not recommended.
For MoE models, it should be using the active parameters in memory bandwidth computation, not the total parameters.
But none of them that I've tried have ever come close to the workflow.
I can stage and unstage individual hunks, do complex interactive rebases, squash commits, break apart commits, etc. much faster in Magit than I can in other Git GUIs.
Maybe you're hung up on the "G" part; perhaps I should have just said "UI" rather than "GUI".
So no, I haven't tried that one because it's Mac only, but I'm not really seeing from the screen recordings the kind of workflow that I find so powerful in Magit.
Unfortunately, for most people the fact that it's part of Emacs is a blocker.
And because most people use worse Git tools, they tend to use workflows that are easier with more cumbersome tools; generally just committing all kinds of junk commits to a branch, and using the "squash and merge" feature of GitHub or GitLab to clean everything in a PR/MR up into a single commit.
So yeah, it's sad that people don't use Git to its full potential because almost no other Git interface works as well, and most people aren't going to learn Emacs just for a Git UI.
I cannot imagine how you can properly supervise an LLM agent if you can't effectively do the work yourself, maybe slightly slower. If the agent is going a significant amount faster than you could do it, you're probably not actually supervising it, and all kinds of weird crap could sneak in.
Like, I can see how it can be a bit quicker for generating some boilerplate, or iterating on some uninteresting API weirdness that's tedious to do by hand. But if you're fundamentally going so much faster with the agent than by hand, you're not properly supervising it.
So yeah, just go back to coding by hand. You should be doing tha probably ~20% of the time anyhow just to keep in practice.
You need to be careful of the amount of reinforcement learning vs continued pretraining you do, but they already do plenty of other forms of reinforcement learning, I'm sure they have it dialed in.
Also, hollandaise is pretty integral to eggs benedict, I've had lots of variations but the traditional with poached egg, canadian bacon, english muffin, and hollandaise is really by far the best.