> I've always been curious why people think that models are accurately revealing their system prompt anyway.

I have a few reasons for assuming that these are normally accurate:

1. Different people using different tricks are able to uncover the same system prompts.

2. LLMs are really, really good at repeating text they have just seen.

3. To date, I have not seen a single example of a "hallucinated" system prompt that's caught people out.

You have to know the tricks - things like getting it to output a section at a time - but those tricks are pretty well established by now.