20,873 karma · joined April 14, 2010
https://mas.to/@comex
https://github.com/comex
[ my public key: https://keybase.io/comex; my proof: https://keybase.io/comex/sigs/zqyU4M2sWbnebjbK6rSSPEKZTFCW-LzNNW4ODMbaOOM ]
For me it's only been in the last month with Opus 5.5 that I've been seeing code quality I actually respect (though this is with substantial human effort in the loop). Though this experience is from work where I don't have access to non-Anthropic models, nor Fable.
In the long term, AI writing - mostly done solo, not directed by humans - will likely have higher quality than human writing, while being better suited to people's tastes since it can be personalized for every reader. If so, it's not hard to imagine it crowding out human writing.
You might ask, though, is that a bad thing? If it's legitimately higher quality, then what's wrong with consuming it? Well, I don't know. It certainly seems dangerous for everyone to be in their own little bubble of content. Social media of course tries to do that today, but there are limits. You can't pick any topic, any set of attitudes you want and get an infinite supply of content tailored exactly for you. Even if you find some super niche community to set up shop in, it's still a community; you're still exposed to the outside world at least a little. What happens when that changes?
(This used to come at the cost of code quality, but increasingly does not.)
The main issue with this is that the programs are usually still single-threaded. For some key utilities like find and grep, this has allowed more modern alternatives to leapfrog them in performance. (To be fair, it’s hard to add parallelism without breaking compatibility somewhat.)
But in most other respects, there’s nothing wrong with being a little old-fashioned. Using global variables is an example of that. It’s a problem in larger programs but at this scale it’s fine.
Actually, there’s another exception to my previous statement that I should have mentioned. For mitigations that are meant to prevent an attacker from turning memory corruption into code execution or code-execution-like control – such as restrictions on mapping memory executable (at least that’s one purpose for such restrictions), or memory map lockdown – the mitigations’ effectiveness doesn’t depend on whether they’re applied to children. Those mitigations are sometimes treated as part of the sandbox system, sometimes separate.
If asr can be used to bypass FDA then that is just a security vulnerability.
Though, there is also a third type of Documents directory, the kind located at paths such as ~/Library/Mobile Documents/com~apple~TextEdit/Documents. This is part of iCloud Drive and is the same directory that you get in Finder by going to iCloud Drive -> TextEdit. Very confusing.
But yes, it's unfortunate that macOS sandboxes cannot be nested.
Syncthing-For
k
is one of the first pieces of text in the first screenshot. With the k on its own line like that. Not usually something I’d comment on, but if you’re showing off a redesign…I’ve been using Work for all my queries, since it seems to just be the same interface as Chat but with more features. I don’t understand why they’re two separate things.
Tiny models are sometimes used out of necessity, but the results often don't hold up at scale, or in different scenarios.
If you train a tiny transformer to use a fraction of its capacity to perform an algorithm simple enough that you could design the transformer weights by hand, the results might generalize to natural-language reasoning, but they easily might just not.
So it is hard to generalize even to something like GPT-3. But once you get to a modern LLM, the nature of the chain of thought is also very different.
First of all, the CoT can be very long and encode reasoning with long chains of serial dependencies. Filler tokens don’t increase the limit on serial computation, particularly in the dots example where the output is a single token (as opposed to real language model which can somewhat substitute for missing CoT by reasoning within the answer itself).
Also, a modern LLM gains much of its reasoning capability in post-training, where it learns which CoT tokens to emit rather than just relying on pre-existing chains of thought in its training data. The post-training involves reinforcement learning where the grader ignores the CoT and only grades the final result. The only things tying the CoT to natural language are (1) the starting point (pre-training data) containing natural-language CoT, and (2) the CoT being produced using the same weights as the final output. On one hand this could potentially make the CoT less faithful to the model's reasoning, since nothing directly punishes the model for, say, coming up with its own private meanings for words within the CoT. But on the other hand, this incentivizes the model to make good use of the CoT. To the extent that the CoT is a "parallel construction" (i.e. a chain of reasoning that has to be coherent-looking but differs from how the model is actually coming up with the answer), coming up with this duplicate reasoning is both a waste of computation and a big opportunity cost. The model should learn to instead use the CoT tokens for information that will be useful to it later. But in practice models do keep producing readable CoT, suggesting that the CoT does reflect the real thinking process to a significant extent (probably more so after post-training than before).
The dots paper has no post-training, so it doesn’t encounter this dynamic at all.
For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.
(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
So, not native in the sense of using or even resembling the native UI toolkit.
For large tasks like a web browser or a compiler, even expensive swarms of frontier LLMs have not been shown capable of producing codebases that actually work. (Anthropic built a C compiler with Opus 4.6 but it lacked optimizations and apparently hit a complexity wall.)
I also want to use LLMs for reverse engineering, but apparently it's pretty hit-or-miss, especially if you're forced to use open-source models to avoid restrictions.