Some hackers just want to be Karl Rove. The problem with that is unless somebody tells you about Karl Rove, you'll think all the pawns he puts into place are the ones who are irreplaceable. They aren't, though you might mistake them as such because they are the public face of operations.
It's not common, but also not rare for teams to have the invisible Wizard of Oz who is really pulling the strings behind the scenes. Neither Ferrari nor Michael Schumacher have won much since Ross Brawn parted ways with them as technical director. Yet he managed to take Brawn GP (formerly Honda F1), which was a joke of a team, to both the Drivers and Constructors championships in its second year with him at the helm (first as Brawn GP). While even more variables are at play, take a look at the LA Lakers with and without Jackson as head coach. Then tell me how many people attribute Kobe to their success and don't really ever mention Jackson's name. Would Apple have been the same company (wait for it) had it not had Jonathan Ive?
I'm not quite sure how YCombinator would accept this business based on one co-founder and then dismiss them when he was the only one left. Not being upfront about things would definitely be a warning sign. But again, in his mind, you accepted the business based only on him. Why would it be such a big deal if the two other co-workers left? I've read some other comments about it being a vote of non-confidence in the business. Talk to 95% of employees in companies and probably 2/3s of partners. Secretly, they think the company is going to hell in a hand basket and will be dead or in serious trouble in 2 years time. It's human nature to be skeptical to the point of fearing a business' prospects; Maslow's Hierarchy and all that jazz. A startup doesn't satisfy very many of those levels... people get antsy.