GPT-Rosalind-6 System Card Addendum
deploymentsafety.openai.com
deploymentsafety.openai.com
I feel like 6 Astra just turned a lot lazier during planning and strangely enough claude has become a lot more assertive with Opus 5.5 while it used to be very lazy.
It’s really obvious during planning, it used to be that claude would just give a best effort guess and handwave things but now it will actually write small tests, launch playwright to test a hypothetical and even look things up in github during a question. This was something codex’s models were really good at.
I’ve read so many stories about people claiming models getting nerfed after a new release and now it seems I can finally join them.
From memory, prompts like “this file is a bit messy” would trigger Sol and Claude to start editing the file immediately, while Astra wouldn’t. But if you give Astra a clear plan and a definition of what you want it to do, then it will keep working until that ask is fulfilled.
Especially compared with Claude that’s a huge difference, Claude is so proactive, I have to ESC it multiple times during the day and tell it to wait because I only asked a simple question.
* It doesn't do things you ask it to
* It adds things you didn't ask it to
It still does both and therefore I still consider it failing. If you don't want to call it "lazy"; fine, but it seems as though the underlying issue is still there.
Claude does this too, FWIW.
If you use Codex, Claude Code, etc, those are being constantly updated. That sort of behavior would be highly harness dependent.
Performance seems to vary like the weather.
I think you also likely have three letter agencies involved heavily now. The NSA is, without a doubt, policing what can be offered to the public and ensuring what the NSA has access to is a step or two beyond what is public.
5.5 Sol would anticipate what I wanted and get it right more than often than not. 5.6 Sol kept anticipating, but would do things that made no sense in context. 6.1 Sol seems to like that but even more so. It spends a lot of time thinking, but the quality of the decisions isn't outstanding.
Didn’t Sol-6.1 top Astra? Or is it maybe that Astra is still a larger model and therefore can comprehend deeper complexity better in the long run?
On specific coding skewed benchmarks yes.
But Astra is in a different league when it comes to doing research and investigative tasks that require exploration.
For my usecases, Astra is pareto frontier - it dominates Fable 5.1 on cost. Fable 5.1 is a phenomenal model but Astra is cheaper and uses way less tokens and steps to produce similar output.
No, Astra is still smarter (https://artificialanalysis.ai/models/?models=gpt-6-astra%2Cg...).
My assumption is Astra is the biggest model in terms of param size and has the most intellectual horsepower.
But Sol-6.1 is more instruction tuned to perform better on the average use cases for most users, so it benchmarks better and feels better to use.
My feel is it's the opposite of a "detune".
Sol 6.1 is likely a distillation that's highly instruction tuned and designed to perform better on benchmarks.
How are you getting these numbers?
Did they essentially package a system prompt?
You just want to make all mathematicians and white collar employees unemployed and want to distract the press and the public with your extinction nonsense.
Though you may care about bioweapons because they could affect you and your trafficked children.
How is open source going to work when governments start declaring models as munitions?
There are a lot of unanswered questions the open communities are going to have to deal with.
>How is open source going to work when governments start declaring models as munitions?
I mean open source cryptography did win that battle in the end. Even if it's illegal you could host the model weights on i2psnark and crowdfund with monero.
And it took a long time and it's something still at risk.
>Even if it's illegal
If your pissing around with munitions laws you might as well go commit all the felonies cause you'll end up in the same prison.
Makes sense.
I dont think I want researchers offloading their thinking to some specialty model after seeing what its done to me and my colleagues in software, it certainly isn't making anyone more competent.
It really only enables people to be able to do less novel and lower quality work faster.
OpenAI and Anthropic creating these platforms seems mostly just to create narratives around AI that its going to lead to cures for disease, which it wont, but plenty of normies assume it will cure cancer now. Also a bunch tech oligarchs who are equally convinced or 'one-shot' they're going to live forever because of language models lol.
Promises of the Fountain of Youth continue to be a time tested tools for snake oil salesmen.
The cool thing is since we don't change the structure/activity of an antibody it doesn't require the usual 1B + 13 years of clinical trials. Instead we cut down on the amount of times it takes and also increase production yields.
Do add me me on linkedin (Taiga Clarke) if there might be some synergy there :)
For me its not a solved problem and AI/ML would still help to solve plenty Drugdiscovery in itself is such a broad area
There are still normal labs out there doing research for years on one drug alone
And biology of humans is huge and complex, not only is it not solved, its also expensive.
Even if you just not do ML drug discovery, AI can hellp with making the research faster: writing grants, finding issues, creating plans/ideas/suggestions, hepling with finding the newest research, writing code for supporting or analysis, controling microscopes and machines.
Researchers in bioinfomatics that are doing the ML work are already using llms where its helpful, they dont need a specialized model. I'll bet my entire net worth and future lifetime earnings that this model will not continue to get updates a year from now.
This idea that LLMs are going to be able to help with drug discovery better than existing ML approaches is a just a lie, I dont think there's a single biotech/bioinfomatics PhD who agrees with you.
They are language models!!! Nobody said researchers wont leverage llms to do the boring stuff, but that doesnt speed up the fundamental bottleneck which are clinical studies. We already have great drug candidates, backlogs of them and having llms is not going to enable you to suddenly be able to do more clinical trials!!!
These things happen in the real world, with real constraints There is no terminal, there is no compiler, there are no proofs, money is required (loads of it).
Regardless of what Trump says, everything is not computer.
Drugdiscovery has also a lot of physics involved btw
And you said ML yourself. You do understand, that the current LLM explositon also means that ML in itself gets more funding and now infrastructure for ML itself got easier to get, cheaper and faster right?
And yes for sure bigger and better LLMs can help: Alone for finding potential better or more optimized experiments, there is value in an LLM having all the current research in context.
For-profit corporations shouldn't have this power, this is not democracy.
What you are asking for is regulations, but as we have seen regulations are pointed at giving the government and corporations more control.
Maybe elect better leaders and not corrupt old businessmen.