How to assign partial credit on an exam of true-false questions?

The Brier score -- you score minus the average squared error between your prediction and [1 for the right outcome, 0 for the others] -- is also a proper scoring rule (i.e., incentivizes you to report your probabilities accurately) and doesn't penalize maximally-wrong answers infinitely. For some purposes it's a better choice than the logarithmic score.

First of all it seems like everyone had exactly the same probability of getting any random question right. Some people got every question right, and some got none right. But in exactly the proportion you would expect by random chance - that some were just particularly lucky or unlucky.

The second is that probability estimates did not vary much based on questions gotten right. Everyone expected to get about 44% of questions right, regardless how many questions they actually got right. People who only got 1 right assigned the same probability as people who got 5 right.

Likewise people who estimated higher probabilities of getting more questions right, got the same amount of questions right. And a decent percent of people were underconfident too, and assigned probabilities too low (but got the same amount of questions right.)

Lastly, people are really uncalibrated. Some people are just bad at estimating probability. When they say "80% chance of something" they mean that thing will only happen 58% of the time. You can be trained, in a relatively short time, to become calibrated. By estimating probabilities and seeing how many you actually got right. But most people aren't trained, so it would be a bit unfair to put this on a real test.

Overall it seems like the biggest flaws in this system are that

1: Scores still get mapped to discrete letter grades.

2: A student's goal is not to get the highest score possible, rather it is to ensure that he or she is most likely to get an "A".

For example:

Given a 10 question quiz, a student who knows (in truth) she is 80% likely to answer each question correctly, and who needs to achieve a score of 0.2 to pass. The student is led to believe that by accurately estimating her confidence for each question at 80% she is giving herself the biggest advantage. If she does this and she gets 4 of the questions wrong her final score will be -1.219 and she will fail. However, if she had instead underestimated her confidence and given each question a confidence of 0.6 her final score would instead have been 0.290 and she would have passed. She could of course go even further and determine the likelihood of her getting N questions wrong and use that to determine the optimal confidence level to select which will maximize her expected score while ensuring that she is most likely to pass the class.

The task itself of working out how 'confident' you feel about an answer also feels like an impossible task (almost like a non-technical project manager asking a developer how long a task will take). I guess it would be fun to see the results of tests taken using this approach and then compare the scores with the new system, the old system and ultimately what they get as their final grade in the subject.

Intuitively, if there are a bunch of questions about the same "feature" and you get them all wrong, all those mistakes stem from the same misunderstanding. I guess in a well-written test where questions are conceptually spaced out this is not as much of an issue...

So to give partial credit, we try to find a simple model consistent with the T/F answers and award credit based on how wrong the model is.

In particular, this would catch the problem where you have a single misunderstanding that cascades into a whole bunch of wrong answers even if you actually understood the rest of the system fine. That's one of the main uses for partial credit in longhand answers, isn't it?

How would we do this systematically? Well, we can't, really. But there are places where we could. I worked with a professor who did research in program synthesis, and I remember he had an interesting idea for education: when a student submits an incorrect program in, say, Scheme, we could try to synthesize an interpreter that would make the program correct and then extract what the students misunderstanding was. (For example: they used dynamic scoping instead of static scoping.) If you seeded your synthesis system with the various kinds of things students actually get wrong, this could be both useful and practical.

You can apply the same idea to grading a test about Scheme. Award partial credit if somebody has a consistent mental model that just happens to be wrong. If they got a whole bunch of questions wrong just because they mixed up scoping rules—but understood everything else—partial credit seems fair.

I guess this is pretty explicitly rewarding consistency with partial credit, but that also seems fair in a lot of classes like CS.

To be clear, I don't actually think this approach is really practical: it's more of a thought experiment on what could be interesting. Even if it was possible, doing it on T/F questions would likely require a lot of questions since each one only provides one bit of input. If you had questions along the lines of "what does this Scheme program produce", you could get away with significantly fewer if you were clever about choosing them—but you'd still want some redundancy to be at least a bit robust to the student making typos as well as conceptual mistakes.

Consider assigning an additional value of '1' to each question initially as a representation of the author's confidence that the question is not flawed in any way. Then, if a significant segment of the test taking population indicates that they believe that the question is flawed in some way, then, the "author's confidence" for that question would be reduced by an amount that wouldn't penalize me for identifying that flaw while still allowing for less 'partial credit' to those who answer incorrectly without being aware of the flaw, or more specifically, not giving as much credit to those who answered 'correctly' in spite of the flaw.

It's also possible that the question I think is a 'flaw' is really a 'trick' question and that in order to answer 'correctly' you must discover the trick -- this would still allow for a better result in that it more accurately assesses whether I really understand the question or not.

It also occurs to me that if the test administrator wants to give a test where they can assign some manner of partial credit to an answer, then they shouldn't give a true/false test in the first place.

The main flaw of the scheme is that it is a purely mathematical analysis. Answers are not based solely on confidence but often relies of reading comprehension skills. (Even at the pure mathematical level. For example missing a square factor and a minus sign.) So you can have 100% confident incorrect answer just because you mis-read or mis-interpreted the question. Then you get punished hard. Given one's incapcity of self introspection and detection of such mis-reading, and the harsh punishment for such undetectable failure on the part of the student, his scheme is wrong headed.

But then goes on to use a very technical, statistical version of confidence where 0% confidence is somehow equal to 100% confidence you picked the wrong answer.

All of this leads me to assert that despite the math being internally consistent, it does not apply well to the situation. A rational outside observer without reading such an article would assume that 0% confidence in your answer is equal to a 50% chance of being right.

The phrase "confidence that the answer is 'true'" can easily be interpreted as "confidence that the answer I marked is correct".

If you assigned probability p to the correct choice, your score for that question was 1 + ln(p)/ln(4). You were not allowed to assign a probability of 0 or 1 to a choice.

The handout points out the importance of thinking about how to approach these tests ahead of time, and also explains the benefits of using this scoring system:

---- begin quote ----

I cannot stress strongly enough the need for each of you to sit down and think about different strategies for answering the questions. This grading technique completely removes any benefit of random guessing. Such a guess could be disastrous. You're much better off admitting that you don't know the answer to a question. (Placing a 0.25 probability by every option indicates that you have no idea which answer is correct, and your score will be 0 for that question). Assessments of probability 0 (0%) or 1 (100%) are not allowed. These answers will be interpreted as probability 0.001 (0.1%) and 0.997 (99.7%) respectively. Your probability assessments must sum to 1 (100%). A probability of 0.001 by the correct answer will result in a score of -4. In contrast, a probability of 0.997 on the correct answer only earns a score of 1. Think about the implications of this before the day of the test.

I strongly recommend that you analyze the grading problem from a decision analysis perspective. Calculate expected values (or expected utilities) for various levels of personal uncertainty. Notice what happens if you are overconfident or underconfident.

This grading scheme makes the midterm harder then a standard multiple choice test, but this is the point. It has many benefits from a teaching/learning perspective.

1) It teaches you to apply the techniques that have been discussed in class. You have to assess your own personal probabilities and apply them to problems that have very real (and potentially) important payoffs. It is impossible to get these points across with the few simple lotteries demonstrated during class.

2) It helps to removes the element of chance from the test. Because of the severe penalty for guessing, the test will more accurately measure your knowledge.

3) The test will also measure what you know about your knowledge.

4) By analyzing how you answer the questions, I will be able to determine which questions are hard and which questions you believe are hard. (A "flatter" distribution for a question indicates a lower confidence in the question, and therefore, a belief that it is hard.)

5) It will allow you to appreciate how hard it is to assess probabilities.

---- end quote ----