Encoding as a prime doesn’t impose any special requirements on the data. All you have to do is append a bit of junk to make the whole thing prime. The rest of it can be plain text or whatever.
In that case, the knowledge that information is encoded in that specific number and the drawing attention to that fact is the illegal metadata. In other words, numbers are just numbers - until you draw attention to specific numbers with metadata, that's when you cross the line.
If you make a website with a prominently displayed huge prime number with flashing lights and arrows, you've added illegal metadata that says "try decoding me" or "I am probably a cryptographic key to something"
Someone who knowingly hosts a base-256 encoded secret should be held as accountable as if they had hosted the same information in picture form, morse code form, or plaintext.