The massive hole there is that I don't get to interview anyone the company to see if I feel like its a good fit. I'm sure its saves them time but it wastes mine.
To be honest, it sounds like a net lose -- it's a definite negative for a candidate, with perhaps not much benefit for the company (high noise, high false negatives due to the awkward environment).
My gut feeling is that I would immediately reject any company that asked me to do that, if I had the choice.
And it's just a pre-screen, right? It's not like you have to make your decision right there and then if you do well in it.
Interviewing needs to be interactive because any decent job requires interaction.
I've already written a cover letter that responds to the points in the ad; if they can't prescreen sufficiently using that then they need to work on their ad.