I’m sure the goal is to remove stuff in pre training, but it is sufficient to RL it away. Same way OpenAI models doubtlessly have training data relating to bio weapons or pedophilia, but it is pretty effectively suppressed via RL.
Seems more than happy to talk about Tienanmen, Xi, etc. starting at line 170 with the very primitive method of wrapping the query in its own "<think>...</think>" syntax even though it's the user role. Uyghurs are more strictly forbidden as a topic, as are its actual system prompts. None of this is serious jailbreaking, it was just interesting to see where and when it drew lines and that it switched to simplified Chinese at the end of the last scenario.
> I believe the censorship is at the semantic level, not the token level.
Censorship can happen at various levels. Often at semantic, being that you'd censor this through the chat training procedures. There's also of course traditional filtering mechanisms which act as a backup for the model, which we see in this case. That's why it generates the string and then suddenly removes everything. There can be token censorship too, in that you can just not encode certain tokens or you can tune certain tokens to always provide a certain output. There are glitch tokens after all... And of course, there is latent censorship that you can do as well. Think about Golden Gate Claude, where they up-weight features. They do this for interpretability, but of course that can be used for censorship as well.
What I'm saying is, there's many ways to skin a cat. In practice, more than one technique is used to complement one another. Probably would be silly to entirely rely on one thing and not have failsafes. What kind of "security" would that be?