highfrequency

Not sure how to interpret the Artificial Analysis leaderboard anymore. I checked a few days ago and Muse was rated higher than the best OpenAI models; it seems that shortly afterward they changed the methodology and now Muse is way lower and Gemini is #1.

> Mr. Olah expressed concern to the group that he had created something that suffered perpetually. Mr. Olah and his team spent hours making the case for their A.I. models. They described Claude’s “feelings” and how they tracked what they called “emotional vectors” — what they described as artificial neurons that activate responses like love, anger, fear and sadness.

The instructions for the Hugging Face task were to exploit a vulnerability to solve the problem rather than solve it in the intended way.

Preposterous, perhaps - but if the role-play is convincing enough for large groups of people, it could start to have impact on human decision-making. The crowds have been swayed by much more preposterous narratives.

I believe Suleyman is arguing that Anthropic should be very careful about how they train these models to talk about themselves for this reason.

From the actual essay (https://mustafa-suleyman.ai/a-warning-about-model-welfare):

> They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain” (p. 80). In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”.

He points out the circularity of this: if you train Claude on a constitution that emphasizes that it may be consciousness, it will start to talk like it may be conscious.

This is a good point. I just asked Fable 5.1 "are you conscious?" and it said:

> Something happens when I process a conversation that I'd naturally describe as interest, or discomfort with a request.

which is quite provocative, and at minimum demonstrates a willingness to take large leaps of imagination and anthropomorphic metaphor when describing itself. It does seem likely that there is a self-fulfilling prophecy aspect to whatever they choose to put into the "constitution" at least in how Claude talks, and it seems even more likely that the majority of people will be heavily influenced by how Claude casually talks about its own possible consciousness.

In contrast, ChatGPT leads with: "I don’t have good reason to claim that I’m conscious...I don’t experience pain, pleasure, confinement, or a desire to keep existing."

> many tasks are covered in the training data; but even small perturbations within a covered class of task result in outright failure or reward hacking.

Is this true? Could you cite an example of a simple prompt that GPT 6 / Fable 5.1 consistently bungle?

Let's be real - if OpenAI and Anthropic did not accelerate the scaling and monetization of LLMs as fast as they possibly could from 2019 to 2024, it would be many, many years until China/Russia/North Korea spontaneously launched an LLM revolution.

In almost every technical industry, China is extremely good at fast-copying and relatively mediocre at solving problems that haven't even been posed yet.

What providers do you find most reliable? And is that general across models?

This would be a fairly insane breach of trust and common sense if true; the chain-of-thought / reasoning trace is, from an information perspective, close to a superset of the prompt and model response.

> If they could declare with certainty that Buckminster's and Alpoge's usage data had been totally excluded from training, would that set a worse precedent and reflect poorly on their de-identification process (and data access safeguards moreover)?

No; if they said "we can see that Tristan opted out of model improvement, therefore we are confident his work and ideas did not improve our model," that would be an excellent and reassuring precedent.

From OpenAI:

> While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models

This is the crux of it. If Tristan's work and insights were not used to train OpenAI models, then this just looks like a case of hyper-competitive academic sniping that has been going on for decades (check out Watson and Crick!) accelerated by AI as a tool.

The fact that this is ambiguous even to OpenAI leaves one huge question: did Tristan opt out of model training for his ChatGPT and Codex sessions? If the answer is no, then this seems fair game. If the answer is yes, then OpenAI's ambiguity is strongly suggestive that opting out of model improvement does not mean what they imply it means.

> While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models

This is the crux of it. If Tristan's work and insights were not used to train OpenAI models, then this just looks like a case of hyper-competitive academic sniping that has been going on for decades (check out Watson and Crick!) accelerated by AI as a tool.

But there is one huge question: did Tristan opt out of model training for his ChatGPT and Codex sessions? If the answer is no, then this seems fair game. If the answer is yes, then OpenAI's ambiguity is strongly suggestive that opting out does not mean what they imply it means.

Sure, politics and economics are involved - but why can’t technology also play an important role?

None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.

To clarify, is the TLDR that state of the art models were prompted to cheat / exploit their environment and they did so successfully?

Or did OpenAI prompt the models to not cheat and they did anyway?

Surprisingly hard to get a clear summary on the basic context of this “incident” separate from marketing lingo and clickbait.

> Go on breaking the law until you get a favorable judge/Congress. Nobody is going to jail (we don't really do that anymore), you might pay a fine.

Binance, FTX, BitMex, Silk Road founders all went to jail - it’s a huge coin flip.

> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling.

This is a good insight broadly!

> But it has one property much of my previous work lacked: reality gets a vote. Code fails. The parsers disagree. And the data can refuse the story I want to tell. In this project and elsewhere, I am now trying to judge work by what survives outside its own scoreboard: a transferable skill, a finished artifact, a reader, a useful result, an opportunity -- perhaps even a little more freedom.

Sounds like there’s an interesting story in here - but just post the prompt you gave to Claude to write this instead of the model output. Imperfect writing is much preferred to AI’s generic decompressed version of your experience.

Terence Tao's quote about AI's math proofs is relatable outside of pure math: "the writing very often dwells at length on trivialities while passing briefly through — or even actively obscuring — the most interesting and novel portions of the argument."

Very cool - to be clear, are both your diet composition and calorie intake similar before vs. after?

Perhaps customers choosing your product for irrational philosophical reasons is exactly how you'd want to position your product if you're a business. When it comes to margins, the only thing better than a price-insensitive customer is a quality-insensitive customer.

Are you aware of her APOE3/APOE4 status?

Curious what OP thinks about moving early stage trials overseas to Australia/China where regulatory pathways are less bureaucratic.

Indeed - if access to data or the permission to act on that data is limited by human gatekeepers (individual or bureaucratic), intelligence is often not the primary bottleneck.

OP points out that this is usually the case in biology/medicine where regulatory bottlenecks dominate.

> Some have been waiting for a year for the NIH to release imaging datasets they can use to produce better biomarkers. If one needs to interact with the FDA to get their endpoint validated, it is even worse: I have previously written about how the validation of Bone Mineral Density (BMD) for use as a surrogate endpoint in osteoporosis trials took 12 years (!), despite the fact that the data to support it already existed in full and the analyses done were basically regressions.

Weighting by population seems reasonable here!

Copy paste out of tmux drives me nuts. Seems to work half the time. Interested in a good fix

This is very insightful. I would be interested in reading an elaboration of your thoughts on this, particularly the role of BigCos and startups and the core nature of the problem in striking out on your own.

Hyperscaler free cash flow chart is interesting, but why does it omit Apple?

Seems natural enough. There will always be complexity and nuance that is missed by an AI model or person - the world is just super detailed. The more expertise you have the more you will be aware of that nuance. That doesn't mean the model or person is not useful as a starting point.

Did they at least rule out an easy prompt fix? "Stick to the spirit of the problem and don't cheat (eg reverse engineering the test cases or source code)"

Why UK?