Hacker News Reader

On the topic of finding Bayes' theorem intuitively, I can never remember it on its own, but starting from the joint probability P(A,B) for any arbitrary A and B always helps me:

    // A and B are arbitrary.
    P(A,B) = P(B,A)

    // These two make sense if I sound them out in words.
    P(A,B) = P(A) P(B|A)
    P(B,A) = P(B) P(A|B)

    // Combine to give Bayes' theorem.
    P(A) P(B|A) = P(B) P(A|B)

Exactly! The mathematical aspect of Bayes' Theorem is a simple algebraic manipulation of conditional probabilities.

Another way to intuitively justify your middle two equations is to visualise a Venn diagram with overlapping circles representing A and B, such that areas correspond to probabilities. Then P(B|A) is the area of the overlap - i.e. P(A,B) - divided by the area of A - i.e. P(A).

The advantage of the odds formulation is that you can compute posterior odds in your head, whereas computing the posterior probabilities via the standard form of the theorem (that you wrote down here) involves an unpleasant normalization factor.

E.g. for the example in https://en.wikipedia.org/wiki/Base_rate_fallacy#Low-incidenc...:

    prior odds of infection =   2 :  98
  × likelihood ratio        = 100 :   5
  = posterior odds          ≈ 200 : 500

and thus the probabilities (if you actually need them at this point) are ≈ (2/7, 5/7).

See also 3Blue1Brown’s exposition: https://youtu.be/watch?v=lG4VkPoG3ko.

For me, taking the logarithm makes it even more intuitive somehow - the test becomes something like a modifier in an RPG-style game. In your example, the test gives +2 to the possibility of having a disease, a more powerful test could be +3, etc. Adding all the different +/- modifiers works just as well, and all this math is easy to do in one’s head.

Edit: also, obligatory 3blue1brown video https://youtu.be/lG4VkPoG3ko

Jaynes was a big proponent of taking the logarithm as well (as you probably know), referring to it as “measuring in decibels”. Unfortunately, I don’t actually understand what this log p/(1-p) gadget (“logit”?) actually does, mathematically speaking: it looks tantalizingly similar to an entropy of something, but I don’t think it is? Relatedly, I don’t really see how this would work for more than two outcomes—or why it shouldn’t.

The logit function is just a mapping from non-log prob space to log odds space. It’s the odds formula wrapped in a natural logarithm. In one way, it’s not “doing” much of anything. It’s not the journey, it’s the destination.

Why hang out in log odds space? Well, it’s a bigger space. It’s [-inf, inf], which is bigger than the [0, inf] of odds and the [0, 1] of prob. Odds space means you can multiply without normalizing. Log space means you multiply by adding. And being a natural log means you’re good to go when you start doing calculus and the like.

You can also cover a huge range of probabilities in a small range of numbers. -21 to 21 covers 1 in a billion to 1 - (1 in a billion).

I usually prefer to write this as

  [P(H|E) : P(¬H|E)] = [P(H) : P(¬H)] × [P(E|H) : P(E|¬H)],

which is admittedly not notation you can find in a high-school textbook, but you still understand what I mean.

One advantage is that it makes it abundantly clear that the restriction to two hypotheses (one of which is H, the other is then inevitably ¬H) is completely incidental and you absolutely can use the analogous expression for more (e.g. the three doors in the Monty Hall paradox). Another, funnier one is that it suggests that when talking about “odds” you’re using projective coordinates instead of the usual L1-normalized ones (and you are! your “o” is then the affine coordinate of course).

> [P(H|E) : P(¬H|E)] = [P(H) : P(¬H)] × [P(E|H) : P(E|¬H)]

Hmm, you can do even better by formatting it as:

  p( H|E)   p( H)   p(E| H)
  ─────── = ───── * ───────
  p(¬H|E)   p(¬H)   p(E|¬H)

which is vertically symmetric about H/¬H, or possibly:

  ⎡p( H|E)⎤ = ⎡p( H)⎤ [*] ⎡p(E| H)⎤
  ⎣p(¬H|E)⎦   ⎣p(¬H)⎦     ⎣p(E|¬H)⎦

(where `[*]` is elementwise mutiplication over a vector).

What I wanted to say is, the :-separated blocks are not fractions, they are two-component vectors up to a multiple (aka coordinates on [a part of] the projective real line, for those who like that sort of thing), and the “two” here is completely incidental.

For example, suppose that, in the Monty Hall paradox, you chose door one and observed Monty open door three. Then the posterior odds for the car being behind each door are calculated as

    prior        =  1  : 1 : 1
  × likelihoods  = 1/2 : 1 : 0
  = posterior    = 1/2 : 1 : 0
  =                 1  : 2 : 0

(Exercise: prove this formulation of the Bayes theorem for >2 hypotheses.)

> the :-separated blocks are not fractions, they are two-component vectors up to a multiple

Projective coordinates are fractions, or more precisely the ℤ⁺²-projective vector space[1] (over the positive integers) is ℚ⁺, the positive rational numbers[0]. You generally want to use ℝ instead of ℤ for obvious reasons, but that's not really "not fractions" in any useful sense.

But writing it as vectors does have the benefit of making the generalization from ℝ² to ℝⁿ (n≥2) somewhere between obvious and trivial, whereas fractions are at-least-implicitly specific to the ℝ² case.

0: give or take some issues with 0 and ∞ that we don't care about because 0 and 1 are not probabilities

1: I'm sure someone can out-pedant me on the terminology here, but the point is it's two-dimensional, which (2) is the fewest dimensions this works non-degenerately for (leaving one degree of freedom, versus zero for the ℝ¹ case, and negative one (aka doesn't work at all) for ℝ⁰)

> the ℤ⁺²-projective vector space

Semi-erratum: that's "(ℤ⁺)²" (aka "ℤ⁺×ℤ⁺"), not "ℤ⁽⁺²⁾".