Two Kinds of Judgement
paulgraham.com
paulgraham.com
The Nobel Prize committee, for instance, has to pick a handful prizewinners out of hundreds of first-class scientists. The ones that don't get picked are still first-class, but if the "Nobel Prize Effect" holds true, the ones that do receive the prize are the ones who don't sustain first-class work.
Of course this works differently for a sports team, because the feedback of your efforts are more or less immediate (from game winnings). But for things like startups then, if we really digest both these articles, I'd say the Nobel Prize analogy holds better. The ones who experience failure, or a lack of success, and obstinately jab at their problems, will still make a big kill. And the rejects are always far more numerous. For every Nobelist, there are probably a thousand others doing comparable work.
I once met a scientist who was considered Nobel-worthy. He came off mostly as a grumpy old man though, because he kept lamenting how his classmate got one but he didn't. If I was half as intelligent as him I would be doing cutting edge research as a hobby! Perhaps the world is better now that he didn't get the prize.
With any application the you write take a step back, leave it for a few days and then look at it as if it was writen by someone else. You'll quickly see where your assumptions lie and what you've missed.
Jason13 is saying that there are plenty of cases where the "stars" often appear mediocre until they hit stardom, at which point it's too late to bid for their talent. It certainly makes a difference whether your team includes the stars or not, and since they often won't be obvious at the initial selection...
I disagree that you can or should only protest those judgements about you specifically, but I do agree with the entire second half of your essay.
I agree with the idea behind the essay, that "If it doesn't matter to someone else whether they choose you, then the other party has the leverage. You want to put yourself in positions where you have the leverage." For example, if you apply to a college, and thousands of applicants are just like you, then you don't have leverage.
Your examples just ignore the fact that people get rejected from about 4 colleges a year, while they have a chance to correct their tests and homeworks about 1,000 times a semester.
This means that the numbers simply don't match up. If you appealed 1,000 college decisions, you would have a great chance to get into a lot of them. If you appealed 1,000 questions on a homework to one busy and egotistical professor, they would likely start avoiding you after the 10th question.
Therefore, people SHOULD appeal everything they want to, whether the judgement is specifically about them or not.
I saw that this was written originally as a post to a question about YC decisions. As far as advice about YC, I think the best explanation of why some people were rejected is that there is a hidden bias (though very well earned) to hang around with people you think you'll find interesting or ideas you think you'd love or have a lot to contribute to over the summer.
If your partner breaks up with you, for example, appealing to 1000 different committees is only going to make things 1000 times worse for you.
You've got to decide whether the reward is worth the effort. Sometimes the reward doesn't make sense if you had to fight to get it. (eg, if you're not picked for a team then bitch your way in, there might be resentment that brings the whole team down such that you'd have been better off not bothering)
I also did NOT say you should appeal to 1,000 different places. It was an example to point out that Paul was comparing apples to oranges.
And, by appeal, I don't think either I or Paul meant just words, but whatever it would be (retaking the physical, sending videos to a coach of your performance, sending a college a picture of a national award you just received in March after you were just rejected, etc.)
Your mentioning about hidden bias is somewhat true, except that it's not really hidden, but just implicit---in most of the judgement there are always some implicit criteria the choosers have, and good players know that. Assuming otherwise is too naive.
About appealing: As far as you're not in the few of the obvious top group or in the obvious fall-behind groups in terms of the primary factors, any arbitrary secondary factors can put you in either side. Those factors are so random and it's waste of time to appeal---sure, you may have some chance of the judgements reversed, but it is much wiser to spend that time to make yourself excel in the primary factors, so that next time such randomness will affect less to the outcome, because if you like to live a life taking challenges, you will be constantly judged, and you won't have time to deal with those randomness.
In fact, the incentive to be accurate seems much larger for investors than for teachers. Thus, I would be much more bummed about being ignored by a smart investor, than about being given an F by a 9-to-5 teacher.
I thought that the coach was responsible for minimizing the measurement error. Then, the best he could do is to pick the best team. How do you define measurement?
But the scores themselves mean nothing, on absolute terms. What really matters is the article's relative rankings - which (roughly) determine the article's placement on the main page.
And that's what makes News.YC so appealing. Even though the community's smaller, we don't have to wade through pages of higher-ranked flickr photos to find the quality content.
>"There will be a few stars who clearly should make the team, and many players who clearly shouldn't. "
Rather, there are a few who clearly should, a few who clearly shouldn't, and many in-between.
Moreover, this assumes there is a quantity called "ability" and that the evaluator is accurately measuring it. I note the later downplaying of accuracy, but before that, it says:
>"Probably the difference between them will be less than the measurement error."
How do you arrive at an estimate of "measurement error" in order to claim that? Did you first measure the evaluator's "measurement ability"? With what?
Even if we naively assume that we can order a group by "ability" with regard to start-ups, and that the difference between adjacent slots is small, it does not follow that your evaluation comes anywhere close to the presumed omniscient set-ordering.
The data you have is your self-selected subset, and even there you find that choice #30 is actually in the top few, which is a rather large measurement error; you might argue this doesn't matter in aggregate, but it still argues against any notion of small measurement error. If you can be off by almost your entire selection size then who knows what you missed in the numbers up to twice the chosen cut-off?
The statement "it matters least to judge accurately in precisely the cases where judgement has the most effect" seems self-contradictory. What I would guess you mean is that if someone is clearly good, they make the cut, and you don't need to scrutinize to find out how good; but it still matters that you put them on the right side of the cut!
It matters the most to judge accurately where judgement has the most effect -- given the nature of start-ups, your missed borderline case could be the one that would have been larger than all the rest put together. Remember your perceived borderline is not the same as the actual borderline. See Bessemer's anti-portfolio: http://www.bvp.com/port/anti.asp
The article seems to focus on accuracy initially, then semi-mea-culpa its way out of it later.
Reading hundreds of applications, your eyes glaze over, and you may only notice someone when they happen to push one of your buttons by chance.
It's fine to say you think you selected an "optimal subset" of people according to your checklist, biases, or opinions; but don't confuse that with "ability". You have an idea of what you want and pick according to resemblance to it: small teams that remind you of Viaweb's structure.
Using the game theory definitions, I don't find it discouraging that you chose an "optimal" set, which is clearly different in this case from "maximal" ;)
Just don't confuse maximal individual ability with optimal set selection, which is what you appear to do with your example of selecting players for a national team.
Tanenbaum would've failed Torvalds due to his own bias, which is clearly separate from Linus's ability.
Suppose you're buying stocks; you choose a portfolio of stocks to maximize your return. Of course you'll make mistakes and pick bad stocks, but as long as you have enough good stocks in your portfolio, you'll still turn a profit (and companies won't come and argue "why didn't you buy my stock?"). The difference between YC and the stock market is that being chosen by YC can potentially increase your value; they are more like the coaches that train athletes. So whatever portfolio they pick will actually end up with a higher value than if it were just left alone. Also, it makes sense to evaluate people by things not so one-dimensional as "ability" (which by gyro_robo's argument we should maximize), otherwise we ignore other possible synergies between coach and trainee, such as similar personalities, that can cause the trainee to train more productively, independent of ability.
The problem with the stock market analogy is that all you need to do is to beat the market, and to do that you just need to perform better than average. Actually YC could do that if all they wanted was to make some money. Perhaps similarly, a sports coach could be lax about picking athletes if the second-best team was way behind. This is where I see the game theory coming in. If there was another team, similar to YC, that was similar in performance, then YC would have more of an incentive to minimize the measurement error. I'm not saying they don't already, I'm just saying it would make sense to do so.
Anyway, I don't know if this makes any sense but the essay is fun and probably a good piece to show to some younger people.