It’d be absurd to believe nobody tested these, and that no individuals involved in relevant standards bodies were aware of this weakness.
3,921 karma · joined March 12, 2012
What's neat about "safety evaluations" in LLM parlance is apparently encountering any novel information constitutes a "safety event" that can result in new reinforcement learning data... This anthropic 2022 paper that basically describes how it's a perpetual information siphoning machine with a cute little graphic https://arxiv.org/html/2212.08073 - they publish their "constitution". OpenAI has a "model spec" they somewhat regularly update that I suspect is their equivalent process. I suspect we've all been unwittingly advancing their models capabilities..
This Dec 2025 Google paper "A Practical Guide to Generating Synthetic Data With Differential Privacy" spells it out pretty clearly - the focus is on "privacy" - nothing about protecting the user's IP or unique knowledge/ insights.. https://arxiv.org/html/2512.03238v1
In OpenAI's case it seems to essentially generating synthetic training tuples from {prompt, chain-of-thought, answer} or scores on the chain of thought for reinforcement learning.
It would appear opting out of "Improve the model for everyone" didn't actually mean what we thought it meant.
I suspect anything you make in these, the thinking traces or "safety evaluations" of the content allows whatever you build to become RL or eventual pre-training data.
A .github/CODEOWNERS file seems to help when it’s going down that path, but I don’t like to indulge it..
An open weight tool call auto-reviewer, has all sorts of achievable scaling curve milestones.
I maintain my own fork of Codex for "fun". Whenever I look at the sandboxing churn they're doing every release, as someone who used to work at Microsoft on Windows, my reaction is usually: https://c.tenor.com/vTzzhTiypwQAAAAC/tenor.gif
But I’m reminded of ~2008 and the rise of “the cloud” as a marketing term that seemed to me to be a cover for dropping an expectation of rich clients, increasing a companies margins around subscriptions that would chip away at local ownership.
Then I got offput by the zealotry and absolutism around “true FoSS”, told myself I was young and moved on.
And really, a lot of subscription models I kind of can appreciate/ tolerate. Might be irksome but whatever, I get that software is expensive to make and it’s not fair in 2026 to value a yearly upgrade of Photoshop at $200. The capricious UI changes to things that’ve worked for 20 years and they take away say the classic color swatches altogether - silly and dumb.
I can use another professionally necessary tool I pay $200/ mo for, Codex, to whip up a classic swatch plugin.
Is that $200 a fair price for my token usage? I think an extremely heavy month I might’ve used a billion tokens?
But that right there is the problem. They have no idea what, specifically, profitability looks like and are going to be pulling endless levers for … I genuinely have no idea how long - at least through 2030/2032 if we tea leaves their debt obligations?
I don’t want to think about any of that. At all. I don’t want to spend time evaluating model preference and degradation and updating the nuances of how I “speak” to an AI because there’s some mystery backend experiment running on the output I use to produce functional outputs — ie the actual products I get paid to build/ maintain.
AI’s something between a tool and coworking companion, and the capricious “personality” changes due to playing with poorly understood and knobs and levers at the inference level - is maddening. To that end, I want a box in the corner I can point to and know exactly the quality of outputs that no one but myself modifies.
This is already a thing
What specifically about their release strategy “reminded” you of that conjecture?
The premise that they only open source the models … because it somehow helps them leapfrog American labs, and once they actually can leapfrog them, they’d close source them, doesn’t really track for me. Am I missing something?
I mean I think we need our own domestic open weight labs. I just don’t particularly understand the point you’re making
That, the 5 different secret levers you have to pull to make it not stupid, the fact you hs e to go to the guy’s twitter account to find all the un-dumbing features and flags that aren’t documented anywhere else. That they decrease thinking budgets silently when they run out of compute instead of announcing the rationing, and gaslighting users at every step of discovery. The fact that internally they have their own coding harness and don’t use Claude Code primarily. The lack of formal evals and consideration for millions of users collective hundreds of millions of hours of investment in their workflows — that’s all off the top of my head, let me tell you how I really feel about what they did to Claude Code..
I adore gpt5.5 and maintain my own codex fork - but I have no idea how long I’ll get this performance / cost - I know it won’t be forever. I’d like to know precisely how much it’ll cost in hardware to run a gpt5.5 open source model locally. Hell a lifetime license to a model I can run locally is also be open to.
But I like building my own tools, from software to physical shop tools. I like being able to rely on my tools.
More responding here to the assertion that this is blowing up due to Fable.
If the author of the blog post ever comes across this thread/ comment, bravo and I hope you feel pride in your work and I’d go so far to say discovery.
“No, it never does. I mean, these people somehow delude themselves into thinking it might, but……
…But it might work for us!”
As someone that used to work on Windows, I kind of had a vision of a similar in scope e2e testing harness, similar to Windows Vista/ 7 (knowing about bugs/ issues doesn't mean you can necessarily fix them ... hence Vista then 7) - and that Anthropic must provide some Enterprise guarantee backed by this testing matrix I imagined must exist - long way of saying, I think they might just YOLO regressions by constantly updating their testing/ acceptance criteria.
Why not provide pinable versions or something? This episode and wasted 2 months of suboptimal productivity hits on the absurdity of constantly changing the user/ system prompt and doing so much of the R&D and feature development at two brittle prompts with unclear interplay. And so until there’s like a compostable system/user prompt framework they reliably develop tests against, I personally would prefer pegged selectable versions. But each version probably has like known critical bugs they’re dancing around so there is no version they’d feel comfortable making a pegged stable release..
There was a lawsuit in 2020 or 2021 where some evidence was unsealed showing ARM gets 1% of the CPU cost. I can’t recall how that CPU cost was calculated - but I believe that was a part of their deal through the early 30’s. That’s less than a dollar per iPhone.