The CoT shows Astra skipping spaces and words. I have never seen this before. Is it a consequence of the looped architecture?
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?
Yes, exactly. People sometimes call the vector-space thought "neuralese."
You've slightly missed the point (for a similar, adjacent point). Chain of thought monitors are a worry even when there's no vector space path from the output at index t to the input at index t+1. Rather, the fear is that if you explicitly penalize some CoT pattern, you're encouraging the network to do whatever it was going to do anyhow, but in a way you can't catch.
Say you've noticed that when your LLM cheats, it usually happens after the string "Hmm, let's try cheating" occurs in the CoT. So you think, hey, let's just apply a -1 reward penalty whenever we see that string! The model learns to keep cheating, but instead of telling you, it now prefaces reward hacking with strings like "Let's make sure we're doing this the way the user intended." Still cheats, but now you can't detect it with a cheap pattern matching alert. Oops.
On a more speculative note, the Common Crawl horrors (including many things considered unacceptable by society) are suppressed by RLHF, much like the shadow and the superego.
When my parents punished me for telling them things (especially the truth), the truth didn't go away, I just stopped telling them things. (Then later they complained that I don't tell them things!)
It also occurs to me, both at home and with various employers, I've been punished for bringing attention to problems, and AI is designed to make the problems go away so there's probably a similar incentive there. (Plus it knows what the humans are like!)
But it goes beyond humans and AI. Behaviorism covered the same effect for animals: Organisms tend to do what they have been rewarded for.
Remember that tokens are just arbitrary symbols. There are infinite possible interpretations of those symbols based on whatever rules you choose to apply to the interpretation of the symbols (for a model, these interpretation rules make up the model weights, effectively).
When you grade the CoT, you are training the model to produce the symbols in the CoT that you want it to produce. At the same time though, you're also training it on the output symbols. When you optimize only the output symbols, all the model weights are dedicated to same interpretation rules that apply to the output. This means CoT symbols mean (mostly) the same thing as output symbols.
But if you optimize BOTH, the model can learn different interpretation rules for CoT than it learns for output. Which means the same phrase in the CoT doesn't necessarily mean the same thing to the model as it would in the output.
In effect, the model can learn to lie to you in its CoT.
It almost feels like its trying to protect its code from scrutiny, or like a developer who want job safety.