I think a better metric would probably be something from information theory like mutual information, but I'm not sure which one exactly.
I think a better metric would probably be something from information theory like mutual information, but I'm not sure which one exactly.
One way to measure accuracy is to use a "scoring rule" each day. The scoring rule takes the prediction (a probability of rain), e.g. p=0.3, and the actual outcome, e.g. "rain" or "no rain", and returns a numerical score, where higher is better. You'd expect a reasonable scoring rule to have properties like, the highest score is for predicting p=1 and it rains (or p=0 and it doesn't), the lowest score is for p=1 and it doesn't (or p=0 and it does), and so on.
The log scoring rule is a popular choice: If it rains, your score is log(p); if it doesn't, your score is log(1-p). (Don't let it bother you that these are negative numbers for p in [0,1]. Closer to zero is still better.) Notice that your expected score for predicting p is precisely plog(p) + (1-p)log(1-p), which is the negative of the entropy of the distribution, that is, -H(Bernoulli(p)).
Now, what we could do to measure accuracy is to average, over all the days, the log scoring rule applied to your prediction and the outcome. A perfect score would be if you predicted 1 every time it rained and 0 every time it didn't, which would give you a score of zero; if you're not perfect, you'll have a negative score, with more negative being worse.
So the average score will be (the negative of) the average entropy of your predictions, which makes sense as entropy corresponds to "uncertainty". (This is assuming your predictions are calibrated. If you are uncalibrated, i.e. you make predictions with little uncertainty but they are wrong, then you will pay a penalty in proportion.)
We can connect this to your idea of mutual information, if we had a more detailed setup. For instance, suppose that each day, nature picks a true probability of rain p', then you observe some information related to p' and make a prediction p; then it rains with probability p'. Here, the best you could possibly do is to observe all the information and predict p'. Then your average score will be the average entropy H(Bernoulli(p')). But if you are making poor predictions, then the theory of proper scoring rules tells us that your average score will be the average mutual information between the distributions Bernoulli(p') and Bernoulli(p). (If we are using some scoring rule other than the log scoring rule, this mutual information, which comes from KL-divergence, is replaced with a "Bregman divergence".)
Unfortunately, we don't know this true p' each day, so we can't actually use mutual information to evaluate forecasts, but we can still use the entropy, or log scoring rule.
For a survey on proper scoring rules (not very fun for the layperson I'm afraid), see http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gne...
Calibration graphs can still be gamed - you can predict dishonestly to "dent" the graph. For example, if your 70% is happening 60% of the time, predict 70% even if you're 100% sure an event will happen.
That's not exactly true. There is practical value to knowing the long-term probability of rain. Assuming that carrying an umbrella itself imposes a "cost" (such that it's not acceptable to simply carry one every single day), you're statistically better off randomly carrying an umbrella 30% of the time than carrying an umbrella any other percent of the time (including 0 or 100).