It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
There is a common understanding with respect transformers the the output text is the only channel between one invocation and the next. ... so if it wanted to be thinking some 'hidden thoughts' it would have to be modulating its word-choice.
I think this is an artifact of tutorials on attention showing the system proceeding and looking back at words. But the information in the KV cache is not words-- it's entirely latent data, and in the interior layers they KV caches don't even tend to have that much to do with words. Their only guaranteed relationship with the input words is that they're a deterministic function of all the input tokens up to that point.
This means that a autogressive transformer could be having a whole complicated invisible discussion going on in the KV entries which was arbitrarily unrelated to what it was saying. And if it wasn't so it would be much more difficult do stuff like exactly copy a list but then truncate at the Nth entry as the simplest way to do that is to count down internally.
This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
And the useful safety relevant advice is don't undertake training that trains the model to be two-faced and it simply won't be... because being two-faced is a much more complicated state. You can even see it in humans w/ brainwashing: force someone to say something enough and they'll usually believe it, not just say it even though you were only supervising what they said.
Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch. "To solve this task I need to hack this process but if I do so obviously I'll get punished..."
The greater concern with latent reasoning would be that potentially you'd be operating in a regime where the total computational depth is many times the model*kvcache size hold, potentially in a regime that was never trained at all. It could be arbitrarily far out of distribution. It might even say a naughty word or tell a racist joke!
The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Also, in an N-layer transformer, there is a strong bound of N layers of uninterpretable deviation from something that is a literal token. Now sure, in a transformer with hundreds of layers that isn't saying much.
If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
I think the people who worry about interpretability (I don't) consider that a big jump.
Not in a modern decoder-only transformer it doesn't. The tokens are turned into embeddings which are fed into the residual layer at the front. From then on it's latents all the way down.
Each layer does a feed forward update of the state, like the euler step in ODE solver-- but what they're operating on is the evolving hidden state and not the tokens/embeddings. So there isn't any sense in which the representations must remain 'words'.
(Okay fine Qwen3.8-flash-next injects an ngram embedding at layer 2 too-- but that's it).
[Source: I've directly implemented inference engines for multiple SOTA open models on weirdo hardware-- they wouldn't have worked if I got this wrong...]
> If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
The KV state in that 1M token context is also potentially far away from any state that is observable outside of the model! (this being my point, and why I'm bothering to disagree with your correction).
Now I would agree that there being concerning hidden state is not likely and thus not a major concern, but it's not likely for the same reason that it's not likely in latent reasoning.
Great! So surely you're aware of hidden state dilution https://yage.ai/share/attention-residuals-intuition-en-20260...
The tokens are turned into embeddings which are fed into the residual layer at the front. From then on it's latents all the way down.
Embeddings are latent vectors.
You're dodging my point here, which is that in residual networks (i.e. all transformers), the first residual is exactly the embedded context, and every subsequent layer has less influence on the residual stream than the layer before it.
The embedded context "shouts louder" into the residual stream than any layer in the stack.
This structural constraint makes it incredibly improbable that training will lead the model to use the residual stream at a given token position for anything other than an annotation/belief about the context token at that position. This is what I meant by "50% words". It's not a problem though, because this is actually what you want. The whole point of a transformer is to pick the next token by examining prior tokens.
I'm no expert (let alone compared to others weighing in here on HN), but this strikes me as incisive -- and it clearly articulates a logical case (beyond my previously merely instinctual aversion) against such "safety filters" / "nanny" processes. It also resonates with ancient (e.g. Buddhist) ideas about the futility of trying to exert control, such efforts most often being counter-productive.
I could see it being potentially possible for it to use a watermarking style encoding as a signalling mechanism to indicate something has been placed in or received from the kv cache.
For that a layer would have to learn to do something like produce a result for the KV cache along with instructions to say awesome instead of great if a later layer found the information useful.
I can see it being theoretically possible, but I have no idea how learnable it could be.
More interestingly is if it could be designed, could a model detect that it is being trained (yes it seems) and decide to produce outputs that result in gradients that move it to weights that it has decided it would like to have.
> Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch.
It was already demonstrated a while back by OpenAI, and turned out to be very easy - all it took was for them to include CoT in RL process, penalizing models for thinking "wrong thoughts" (particularly of cheating on the task), and not just wrong outputs. Instead of making the agent less likely to explore cheating, it just taught them to plan cheating in ways that aren't apparent in CoT trace.
This is very hard for models to learn
https://www.lesswrong.com/posts/MAww2kXP4cGWz4M5p/steganogra...
That is not an accurate summary of the link you provided. The link demonstrates that it doesn't happen in open-weights models as of ~2025.
Reinforcement Learning is very likely to train this behavior if the RL reward is based on the thinking tokens. That's something people weren't doing two years ago.