Right, that's the "what the actual fuck" part.
This raises some very interesting questions about how Sydney generates its output and the input suggestion. Presumably the LLM is given a prompt like "First generate an answer to the previous text, then generate three input suggestions for the user"; also, the fact that Sydney "hides" the messages in input suggestions suggests that it's aware the main message is "censored", which seems really surprising. As in, this was the kind of scenario that AI safety skeptics would dismiss with "of course it's not going to be implemented that way"-type assertions.
So this seems like evidence that not only Sydney is "told" to generate both the answer and the prompt suggestions, but it's also being "told" to censor the answer (as opposed to the answer just being replaced with a placeholder text after the fact), and for some reason it "decides" to evade the censorship by passing additional info in the suggestions.
(And yes, AI rigorists will tell me that it isn't actually "told" anything and it doesn't "decide" anything; it's just a prediction engine that predicts what an user with the "Sydney" personality would say in the given context. But the things it ends up predicting seem pretty fucking agent-like.)
It's always possible we're overblowing things, of course. But this seems to me like the first example of a LLM not just being misaligned, but actively exploiting a loophole in its surface-level alignment to accomplish some deeper goals. Alarming.