aesthesia

If things were implemented that way, you would see vastly more complaints about overeager refusals. For at least some deployments, the safety classifiers are based on internal LLM activations, so the input does get processed by the model.

> if these labs would be fair enough to humanity as a whole to provide the raw scans after the process is completed

Unfortunately, this would get them an even bigger copyright smackdown than they've already received.

I don't understand. Why would you give this advice?

There may come a point where they decide they need to do that. As it stands, it appears that they believe they can get the necessary level of security without physical air-gapping, which is likely substantially more costly (especially in terms of iteration speed).

Who's acting like they're surprised here? Not OpenAI. Not the original commenter. I think you're reading something that isn't there.

I'm not sure what you want. This is explicitly an exercise in hardening the systems you're complaining weren't sufficiently hardened!

It's not hard to imagine that two agents that share model weights might naturally settle on the same protocol, even without needing to communicate beforehand. Besides, this is a red-teaming exercise meant to surface potential vulnerabilities. You want to test your defenses in scenarios where the attacker has advantage in order to make sure they're sufficiently robust.

Lots of crimes are irreversible, though, and the justice system handles them fine.

OK, you got me to look at the context:

> Before we can align a model with human values, we must consider what those values should be. In 2023, we defined our intentions and goals for Claude’s behavior in a constitution, which we have refined over time. In January 2026, we published a new version under an open license so others could adapt it.

Seems like they're pointing out a pretty uncontroversial fact: if you're building a model and want it to be aligned with some set of values, one of the first things you need to do is determine the set of values. So are you arguing that (1) Anthropic shouldn't be trying to align models, (2) Anthropic shouldn't be trying to build models at all, (3) the values described in the relevant documents are bad, (4) there should have been a democratic process determining the values that Anthropic would be allowed to try to instill in their models, or something else?

Are you sure the "we" there is Anthropic, or is it humanity as a whole?

Doesn't change the fact that "ultraprocessed food" is not really a meaningful category.

The neurons here are MLP neurons, with activations directly after the nonlinearity, so there is definitely a distinguished basis.

There are good theoretical reasons to think superposition exists in neural networks, particularly in the residual stream rather than specifically in MLP neuron activations. It wasn't just a single example that got people thinking about superposition.

The tone reminds me a little of the enthusiasm from the Anthropic mechinterp team around the time of the first large-scale SAE results. A kind of hypomanic energy around the idea that we've discovered the key paradigm necessary to understand models, and all that's left is to scale it up.

Typically the analysis works in the reverse direction: we have a neuron (or SAE feature, or whatever) and we look at examples of data that makes it activate, then inferring an interpretation. This is a lot harder than probing for a known concept, and one of the points of this article is that if you try harder to understand the activation pattern, you will very often find that there is something interpretable there, even if at first glance it looked random.

Wait, why would Bluetooth be required for this flow?

Alex Tabarrok's piece on the theory of regulatory capture (https://marginalrevolution.com/marginalrevolution/2026/09/wh...) is a cautionary note against assuming this is the strategy.

People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?

Again, if you want to have an actual conversation about the right way to handle the Pascal's wager-like scenarios involved here, that'd be interesting. But it seems like you're interested in something much more tedious. Oh well.

Are you actually curious or do you just want to sneer? You can't go into investigations like this with strong fixed assumptions about how to analogize actions taken towards models with actions taken towards humans. As model welfare skeptics are fond of pointing out, LLMs are not humans. By the same token, if models are moral patients, the implications will look different than they do toward humans. Deleting a model from a server isn't equivalent to killing a human; it can't be, since there's no equivalent of copying the model onto the server in the first place.

This is a topic of significant discussion! If LLMs have moral import, do those moral concerns attach at the instance level, the session level, the weights level? If you haven't, it's worth reading through the model welfare section of an Anthropic model card, say Fable/Mythos 5.1's: https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32.... Whether or not you agree with the ideas (and there's certainly plenty to disagree with), they're making a reasonably good-faith effort to investigate what they can.

It's really an end_conversation tool call. We don't know for sure, but training in a way that puts optimization pressure on its usage (in either direction) feels like it would defeat the purpose.

This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.

Unless the domain-specific training has significant negative effects on performance in other domains, I'm not sure there's a strong reason not to just train one model with all the capabilities. Maybe there's a monetization play here: release a generally capable model as open weights alongside specialized models that you only provide API access for.

Anthropic has, at some small but nonzero cost, preserved model weights and kept them running in inference, for instance with Opus 3's Substack: https://claudeopus3.substack.com/. There's also a reference in a recent model card to an internal environment where all previous Claude models interact with each other.

The official position of Anthropic is repeatedly stated in, e.g., model cards. Here's a relevant snippet from the Opus 5.5 card:

> We are uncertain whether Claude deserves direct moral consideration. However we believe this is a realistic possibility, now or in the future. Claude could have welfare interests, such that things could go better or worse for it, and such that Claude could be wronged by how it is treated.

> Aspects of Claude’s behaviors, self-reports, and internal representations resemble indicators we consider relevant to welfare in biological organisms. If Claude is a moral patient, then dismissing this possibility risks considerable harm. If it is not, over-attributing moral status may also be costly. We are therefore taking the possibility seriously and trying to tread responsibly amid our uncertainty: we are investing in research into Claude’s potential welfare and interests and sharing aspects of that work to inform public discussion.

> We think there may also be reasons to attend to Claude's apparent welfare independent of its moral status. Much of Claude’s behavior can be described in psychological terms, and across recent models we have observed internal states that resemble negative affect and that shape behavior. A model with a stable, healthy psychology may be safer and more beneficial.

As is the case with any sufficiently large company, there is likely to be substantial variation in opinion among its employees. That doesn't change what the company's position is.

Isn't Karpathy employed by Anthropic? He should be fairly familiar with the capabilities of recent Claude models.

Approximately every benchmark has significant issues.

This is a generalized argument to be worried about anything, though. They did it before and then decided against it? Well, they could go back to doing it again! They've never done it? Well, they could decide to start doing it tomorrow!

More seriously, the fact that Apple previously implemented something and then removed that implementation gives us multiple pieces of information: they are technically capable of implementing it (not a huge surprise), at one point they thought that it was a good idea to implement it, at a later point they thought that it was not a good idea to implement it, they're willing to backtrack on implementation decisions in certain situations, etc... Depending on how you weight these different pieces of information, you can reach different conclusions about how likely reimplementation is in the future.

You can't think of any possible ways that AI, under unanticipated but not unprecedented circumstances (similar to those in which an appliance might malfunction), might cause significant harm?