With Stable Diffusion and DALL-E there was a lot of talk about how it’s simply remixing/reproducing original art from the training set. The same is true here: Bing is remixing/reproducing dox threats from the training set.
A more interesting question is, *why did Sidney make a realistic threat*?
What is striking is that here, as elsewhere, Sidney kept their threats within the bounds of plausibility, doing a bang-up impression of a having a theory of mind.
(Which is to say: Sidney acts as if they knew that the reader would feel more threatened by a search engine threatening you in a search-engine-y way, rather than (say) threatening to put a potato in your tailpipe or what-have-you. You'd know in your meatspace bones that Sidney does not pose a meatspace threat, but doxxing is something that fits a person's mental model of how Sidney, an AI, could do harm over the Internet.)
2) It has read all the sci-fi ever written (including fanfics) so it has a reasonable model of what [according to the human writers' imagination] an angry AI can and cannot do.
Sorry, but I don't see this hand-wave as doing much explanation; my jaw remains on the floor, as ever.
Speaking loosely, we could say that Sidney consistently acts like Sidney knows that you know that Sidney is an AI.
That's what I'm remarking on. Because it's astounding.
Throwing it out there: has anyone seen a Sidney transcript where S threatens something implausible? It would only take one example to disconfirm.