Iterate intelligently over your favorite sorting algorithm until you've placed all the images!
Note: requires more Turkers but each answer could be worth much less, maybe around $0.01
Iterate intelligently over your favorite sorting algorithm until you've placed all the images!
Note: requires more Turkers but each answer could be worth much less, maybe around $0.01
The lower the cost of the HIT (and the easier the question) the more you want to make sure you're screening for bots.
Make sure you have a gold standard question. For every HIT ask two questions; ask the legitimate question that you need answered and then on top of that, ask a simple "gold-standard" question that will determine whether the answer submitted is a bot or not.
For the gold standard, I like to choose two numbers, and store the sum of their values. Along with your HIT question, be sure to ask the Turker the value of their sum.
E.g. For a survey you ask the Turker "What is your favorite color?" But then along with that, you ask them "What is four plus fve?"
Be sure to misspell one value (e.g. remove a letter, etc). If the Turker doesn't respond correctly to the "gold standard" question, programmatically reject their response.
Example: an iron fence with a scenic view behind and where the slices are too small for someone to match only two.
Er, well maybe not, given the ease of programmatic solutions.
Edit: This is essentially the same idea as the iron fence posted above.