For full double-blindness you could even do interviews with the EXPERIMENT group, then use a program to automatically either use the interview result or not. This might be aggravating for reviewers, though, seeing their work wasted by random choices.
You get one or two huge winners out of hundreds, and they are so huge that they are bigger than all the others combined. The investors care most about getting these rare winners, and if YC does any kind of controlled study on who becomes these winners, I don't believe there will be any statistical significance to speak of.
Since the winners are what they are after they can omit the interview process.
I have a feeling that eventually there will be a Vangaurd of startups that will massively outperform traditional VC/incubators due to reduced management fees similar to index funds vs. regular funds.
So PG's claim could very well be supported by proprietary information.
Never been accepted to YC, but I gather that the value of access to YC's demo day and a strong network of investors and alumni is far from trivial.
So the question with luck is how far down the causal ladder you start considering things inputs (i.e. things which may or may not influence the outcome) instead of outputs (i.e. things which might be the result of those inputs or of luck). So with YC, is that getting or not getting into YC is an input variable that we could check against success, or is getting a YC spot one of those proximal factors, and the luck was just in getting that, at least in part?
So the "is it pure luck?" question isn't very well defined, but the GP's argument might still be useful in answering it. Moreover--larger questions of luck aside--it would be helpful in determining whether the YC selection process is actually useful. But probably only if they didn't tell anyone about it.
It's well-known that the most successful startup in a group of startups is likely to earn more in profit than all the rest combined. We're here on HN talking about PG because YC has been the most successful of all incubators. However, there's always going to be one incubator that significantly outperforms all the others as a result of the distribution of startup success.
I don't know enough about how well PG's investments have done to make informed judgments, but only a very select few people are likely to have that info. I suspect those people have also done their due diligence to figure out if YC is a value-add compared to other incubators (it should be possible to do this without a convoluted double blind experiment), but anyone who's not them really can't say for sure that that's the case.
The key is to perform proper Bayesian updates in the face of evidence in either direction; if you do, as long as your prior wasn't totally insane it doesn't really matter where you started.
It would be an interesting experiment if YC lower the acceptance threshold to say twice the original batch, and randomly pick from that bigger pool, and see if there's any difference between the ones that didn't make the original cut and those did. (A/B in its finest :)
The idea that the null hypothesis is the only source of information needs to die.
Figuring out what makes people successful at starting companies is extremely complex. And without objective evidence, no level of expertise, be it PG or other VCs, will convince me to take someone's word for it.
Your job is to select people who were lucky enough to meet the right people, meet the market, had luck to have enough traction, lived in a country where he could spend time learning and so on...
But when you are entrepreneur, you have to maximize your luck by maximizing number of opportunities.
Some skilled people succeed, therefore all skilled people succeed eventually?
Alternatively, it's a tautological statement that defines "genuinely skilled" people as those who succeeded, therefore unsuccessful people must not be skilled.
For a DropBox-like product, feeling safe you won't lose your files is the baseline. If you cannot reliably ensure the files won't get randomly deleted, you don't have a product. Once you have a product, the value you add gets increasingly more subtle - better metadata capture, nicer ways to organize your files, an outstanding slideshow or music app - and what determines your ultimate success may feel completely random and totally unrelated to the original way you defined your product.
It would be interesting if we could measure luck based on previous YC batches.
90% failure rates, but the remaining 10% (5 startups) is all Dropboxes and AirBNBs --> Average YC startup valuation up 10-fold.
"Your best? Losers always whine about their best. Winners go home and fuck the prom queen."
a) each pick his/her top 5 yc companies per batch that they think will be most successful (and main drivers why) b) each pick his/her bottom 5 that they think will be least successful (and main drivers why) c) each assign the value of each of the 5 most successful companies 5 years from now. (and build a bridge as to how that value will be achieved).
and then run some fun numbers on your predictive capabilities.
when i ran "c" for my private equity firm on all investments made over 20 years in various vintages and industries, i found my answer as to why my job of building complex financial models was pretty much worthless number painting. very low predictive ability in value drivers and financial projections being met (these are later stage businesses, too). also, i realized that i should get out when every newer vintage had a larger % of the driver of value stemming from leverage (i.e. financial engineering) than ebitda outperformance.