I thought entropy worked the other way round - high probability means low entropy and low probability means high entropy. Which is it?
Random information = high entropy. A common word is therefore low entropy.
Here’s a good analogy I found:
> Informally, the amount of information in an email is proportional to the amount of “surprise” its reading causes. For example, if an email is simply a repeat of an earlier email, then it is not informative at all. On the other hand, if say the email reveals the outcome of a cliff-hanger election, then it is highly informative. Similarly, the information in a variable is tied to the amount of surprise that value of the variable causes when revealed. Shannon’s entropy quantifies the amount of information in a variable, thus providing the foundation for a theory around the notion of information.