I believe there are 3 sides to this:

- yes, you can definitely do that

- but you'll never be sure the two systems aren't colluding to hide things

- yet, even for layers that don't output text directly, you can find meaning in weights - there's an old example where you provide the LLM samples like 123, 321 and sampling from the middle layers lets you find the word "reverse" which is not in the output

Anthropic has published about how to do this - basic plan: train an interpreter that can go back and forth from English to neuralese on the round trip loss. Pretty smart - they’ve published for a bunch of open weights models.