How Claude Shannon Re-Invented Information (2017) | Hacker News Reader

How Claude Shannon Re-Invented Information (2017)

nautil.us

160 pointsdnetesn21 comments

This is the first thing I have read about information theory that really made sense to me. I think I now understand it about ten times as well as I did before.

>What does information really measure? It measures the uncertainty we overcome

My thought here is that matters because human beings have to make decisions all the time, and reducing the uncertainty regarding the situation makes it more likely you will make the correct decision.

There's a pithy slogan that sums it up nicely: information is a correlation between the state of two systems.

For example: if you measure a voltage using a volt-meter then, if the volt-meter is working properly, there will be a correlation between the readout on the meter and the actual voltage, and so the meter contains information about the voltage.

Note that information is symmetric: the voltage in this case also contains information about the meter.

Your comment and the one above it are awesome summaries of some of Shannon's concepts. Re:your statement on information being symmetrical, how does zero knowledge encryption, Homomorphic encryption and the like fit into this when there is asymmetrical knowledge and validation without full access to the information?

The best way to approach that question is to start with the one-time pad (OTP). The OTP has a random key K which is the same size as the message M. The ciphertext C is M XOR K.

So... if information resides in correlations, where does the information reside in this system? Because K is random, it has zero correlation to M (that's what "random" means). It also has zero correlation to C, though this is not so obvious. (Proof is left as an exercise.) In fact it turns out that C and K are both completely random, and so it's not even possible to tell by looking at C and K which is which. They are interchangeable.

So we have the following situation:

1. Information about M obviously has to reside somewhere in the composite system comprising C and K because we can completely recover M simply by computing C XOR K.

2. The information is not in C and it is not in K.

So where is it?

[Spoiler alert. It's worth thinking about this for a while to see if you can figure it out on your own. Hint: think about the case where C, M and K are all just a single bit, because the multi-bit case is just a straightforward extension of the single-bit case.]

Answer: the information about M is contained in the correlations (or anti-correlations) between C and K. In the one-bit case, if C and K correspond (i.e. are both 0 or are both 1) then M is 1. If they don't correspond, then M is 0.

The more complicated cryptographic cases are very similar, but of course, more complicated. What cryptography is really about in some sense is trying to get the same results but with a smaller key, i.e. it's about trying to take a fixed-size K that is (much!) smaller than M and "distribute" the entropy in such a way that the information is hidden in correspondences between bits in such a way that an attacker can't figure out where those correspondences are.

This is hands down the best Hacker news interaction I've had! I took a screenshot of your explanation and did not scroll past the spoiler. Will give this some thought before proceeding... Thanks!

BTW, I just realized that I got this part backwards:

> In the one-bit case, if C and K correspond (i.e. are both 0 or are both 1) then M is 1.

That should have read "... then M is 0." (Obviously.)

Thanks for the kind words. I'm glad it was helpful.

Here is a great primer on entropy that you might find helpful - http://colah.github.io/posts/2015-09-Visual-Information/

> The signicant aspect is that the actual message is one selected from a set of possible messages

> If the number of messages in the set is finite then this number [...] can be regarded as a measure of the information produced when one message is chosen from the set, all choices being equally likely.

From the abstract of his famous paper http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

Discussed at the time: https://news.ycombinator.com/item?id=14978399

It is submitted by the very same user.

He keeps us informed.

Maybe he's adding redundancy, because he's worried that Hacker news is a noisy channel.

Or, maybe he's adding redundancy, because he's worried that Hacker news is a noisy channel.

And I am so happy he did, since I missed it the first time.

If anyone from Nautilus is here: Would you consider having different prices for different countries? The current pricing is too high for India. A reduced price would bring customers from various countries.

A lifetime subscription is $99. If you factor in life expectancy, they are actually charging more to people in India than they are to Americans.

Yes, if you look at raw dollars per year... but you also have to factor in expected lifetime earnings. What fraction of your expected income is $99?

I would wager that it's a much higher fraction in India than in the US.

I recently read the Shannon biography 'A Mind at Play' [0] which I guess is the full version of the article (same authors). It does (IMHO) a very good job at explaining entropy / information.

[0] https://www.simonandschuster.com/books/A-Mind-at-Play/Jimmy-...

I enjoyed the story but found the author(s) practically peddling Shannon's importance as if any of that was needed. All in all the writing in form and substance seemed to me of low quality and in search of dramatic revision and editing. Great history job and lazy writing,

>I enjoyed the story but found the author(s) practically peddling Shannon's importance as if any of that was needed.

The vast majority of the public have never heard of Shannon, or have but don't really understand how important he was, so yes, it was needed.

Reply on news.ycombinator.com