Google vs. Yahoo vs. Bing. Most results are virtually identical.

I can never think of any "fair" searches to try. So often I just use google to get to a specific webpage from the Firefox search bar faster than I could by typing in the full address. If I search for something with a specific result in mind, all 3 engines will nearly always give me that as the first result (for example: woot, eztv, twit).

I never use search engines to look for a generic term the way search engines are always marketed, for example searching for "buy used textbooks" or "cheap prescription drugs" or "book a flight" since those results are always so commercialized that they're generally useless anyway.

Yesterday, I had two searches for which Google was failing me, "prius chipmunks" and "ie6 image reload", so on a whim I tried bing, and bing had the answer in the top three for both. I was really surprised. I'll keep using Google, but I'll probably try bing when I'm stuck in the future.

You can often tell which one is Google from prominent Wikipedia results.

Here's a thought.. build a list of questions. Give them a search box to find the answer, using their own queries, and let them choose which result set was most helpful.

"What is" "Who is" "How was" type of questions.

Would be much more work, but could be worth it if you're trying to answer this question accurately.

Tho personally, I've begun using Bing in addition to Google. I think this is what most people will do. The idea that we only have room in our lives for one search engine seems silly. Doesn't anyone remember what life was like before Google? I'd often use multiple engines for a given query -- and each engine had links to 'Try this search on ____' at the bottom of each result page.

Google has been so far better than the competition that such behavior has died off. But I think it's going to make a comeback now. I guess we'll see.

The best way I can think of would be to have a long term test where a number of users did all of their searching through a site that would randomly show the results of one search engine, and then have the users rate their satisfaction with the results. It wouldn't be quite as good at cross comparing, but it would have a much more realistic selection of searches to draw from, and it would reduce the sort of biases that the gp post was talking about.

Maybe kick it off with the first search pre-populated from, say, a dictionary word. (Though maybe that's not fair either, if it isn't the type of search you're likely to perform in real life)