11,821 karma · joined April 26, 2009
I do think it ended up being after a power outage though and it was some kind of config corruption.
I have reason to believe OpenAI doesn't respect training opt outs, but there is a good chance they let something get through without an opt-out too.
Not all.
Ehrhart’s volume conjecture
Quantum parallel repetition for general two-player quantum games
Erdős Problem #183 on multicolor Ramsey numbers
Erdős–Sárközy Problem #12(i)/(ii)
Erdős Problem #125
Log-concavity of codimension-3, type-2 pure O-sequences
Optimal O(1/t) last-iterate convergence for Anchored Gradient Descent-Ascent
Jane Street is apparently one of Anthropic's biggest customers. Probably engineering, finance, and some math.
So far around one in 350,000 PhD math grads solve a millenium prize problem (Perelman).
> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”
You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.
These people don't give a shit and aren't taking things seriously at all.
Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.
One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.
The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1.
A balanced trade deficit at each connection between each nation worldwide would be very inefficient.
From a recent Dwarkesh interview with Dylan Patel it sounded like Jane Street might be double digits of Anthropic's revenue, but I'm not sure where they sourced that.
They add a note that:
> with the caveat that it doesn’t include massive players like Microsoft or major banks, and customers can opt out of being included in research.
I'd prefer things be opt in, and especially not start opt out, then try to trick you opt in with a popup defaulting to opt-in, like Anthropic did on consumer plans, but if they submitted anything on an opted-in plan it's not reasonable to be mad it trained on it.
Even still, I also believe for significant reasons that OpenAI would ignore the opt-out in selective cases and could be in the wrong here.
And the threats and terms they offered seem wrong either way, pending more context.
Why do we push at all on this?
For knowledge type questions is that necessarily true? In the past we've seen things like Google's models degrading on general knowledge after the preview releases while improving on code/tool use, presumably due to catastrophic forgetting from the additional training. Their preview would be free, get lots of agentic use from users, then additional training on that and probably additional automated RL.
However Astra is on an entirely new base model so I also wouldn't expect it to be worse.