There's a huge difference between "here's something that should take you a day or two, but take your time" and "come work for us for $25/hour, with no benefits; but as a perk, you don't have to quit your current job"
There's a huge difference between "here's something that should take you a day or two, but take your time" and "come work for us for $25/hour, with no benefits; but as a perk, you don't have to quit your current job"
I think the other key part is that they're not just evaluating you on the bit of work that you're doing. It's largely about your communication during the project. If you just took the project and banged out a perfect feature without talking to anyone or asking questions, I'm guessing you wouldn't get an offer. In order to make an all-remote workforce work, communication is incredibly important. "Communication is oxygen" is the mantra at Automattic. And the channels we use for it are IRC and our yammer-like system. We rarely talk to each other and almost never use email. That's really non-standard, and you can't easily get a feel for that in an interview or with pair-programmed fizzbuzz.
In many ways, I think working at Automattic is somewhat similar to contributing to an open source project (to be fair, I haven't done a ton of this). And that makes sense, given its roots in and continued investment in open source. In fact when one developer asked if we could open source a particular component, Matt's response was that he was pretty much ok with open sourcing everything except our passwords. Given how open source works -- your voice in a project basically equals your investment and contribution to it -- I think the paid-project interview makes some sense in that light.
Anyway, I'm kinda responding to a few posts here, but I hope that gives a little more background into the process.