>So I don't think your tinystories example qualifies for the PRH, since it's not enough data and it's not representative of the whole Internet. And RNA data is (I would guess) something very different altogether.

My thought there was that you'd be comparing tinystories to a model that trained on the entire internet. The RNA related information would be a subset of the second representation that has no comparable encoding in the tinystories space. Can you detect that? If both models have to be of sufficient scale to work the question becomes "what is the scale, is it sliding or a threshold? "

>> Assuming that is true. If you can detect when you are trying to put a square peg into a round hole, does this mean you have the ability to remove square holes from a system?

>Not sure I follow this part.

Perhaps the metaphor doesn't work so well. If you can detect if something is encodable in one embedding model but not another. Can you then leverage that detection ability in order to modify an embedding model so that it cannot represent an idea.