There are many obvious avenues to try. This paper proposes one. I think it's likely we'll end up with some kind of auxiliary model that monitors and/or decodes the latent space of the thinking model. Certainly it is a non-trivial problem, but I don't need to know exactly how it will work in detail to believe that it is possible.
well best throw away our tools in this uphill battle of a problem while we get that working
Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
Why people don't treat categorizing model activations as a science is beyond me.
Please, go on. How would you do it, specifically?
Mechanistic interpretability is a field that studies exactly this.
Are you saying no one is trying to do exactly that?
They do, there was a paper that claimed a major breakthrough in that field just a few days ago. See the "Neural networks are clean" discussion: