This paper has been bothering me all morning and I'm trying to articulate why. Some combination of:
* The consensus from the labs has been that Chain of Though ("CoT") monitorability is an essential tool for alignment. Sure, it's imperfect (models can exhibit deceptive CoT). But given how difficult mech interp is, it's the best we have.
* Remember all the noise made about OpenAI and "Looped Transformers"? All the safety and monitorability concerns there? And how OpenAI came out and made statements about how they are only doing a few cycles of looping (compute graph only 2x the size of GPT-4) and how they wanted to prevent "a race into unmonitorability." This is much worse.
* It's true that this architecture allows for "reading" the latent space reasoning traces by projecting into "nearby" tokens in latent space. But that means you're throwing out a lot of information that resides in the full vector, with no way to interpret it. You're just reading some sort of approximation. The LLM will still be reasoning with information you cannot see.
* Sure, there is a legitimate debate to be had about how useful CoT is for monitorability, whether the alignment fears are justified, etc. The most glaring problem this paper has is that the authors don't even really engage with that discussion at all. If there is an elephant in the room... maybe acknowledge, at least, that there's an elephant in the room?
* A little snarky, but this is a pretty obvious idea I think? Reasoning in latent space should occur to anyone who knows how a CoT model works, and there is plenty of prior work in this area. The reason people aren't doing this is because it cuts right against monitorability.
* This was announced on X in a series of messages that are clearly LLM writing. And which also did not acknowledge the elephant in the room. Sigh.
I am not a doomer, but I do not think discussions of alignment and safety are entirely vacuous. So when I see researchers who don't seem interested in that conversation at all, it worries me.