Mechanical Turk Lessons Learned
engineering.curalate.com
engineering.curalate.com
Sorry!
Still doesn't help the gifs though.
I'm an idiot.
EDIT: Seems like I'm years behind https://bugzilla.mozilla.org/show_bug.cgi?id=825486
It's stop() now.
OP don't get discouraged by one off comments like this on HN. They rarely add any real value and most of the time just personal preferences.
Your content was very informative and valuable.
This is more like judging a book by its poor type-setting.
http://automan-lang.com https://github.com/plasma-umass/AutoMan
Paper here on AutoMan, round one: * http://cacm.acm.org/magazines/2016/6/202648-automan/abstract (CACM Research Highlight, 2015)
Original paper, not behind a paywall: * https://people.cs.umass.edu/~emery/pubs/res0007-barowy.pdf (OOPSLA '12)
New features described here have been rolled into AutoMan: "VoxPL: Programming with the Wisdom of the Crowd" (CHI '17, to appear): https://people.cs.umass.edu/~emery/pubs/voxpl-chi.pdf
We're working on a lot of the same things at Scale API (www.scaleapi.com). Starting with a higher quality set of task-completers, and building in similar statistical guarantees for our tasks.
One of the things we work on is building quality for responses that are little more complex (bounding boxes and audio transcription, for example). I'd be interested to see if we can apply some of your learnings to those task types!
It makes me sad that so much of the work available on Mechanical Turk is poorly designed, and that workers have little recourse to bad Requesters.
If min wage is say $7.25 an hour, which is what you aim to pay turks... but then you end up throwing away 2/3 of your work (say have 3 turks do each piece to validate they did it right).... why not hire 1 local person for $15 an hour? Decent wage, less total money (15 vs $22 an hour), and likely better results.
I didn't see an answer to this ... so I'll take a stab at it.
One advantage of using a turk is it's on an as needed basis. Hire someone for 1 hour, 1 week, or even 1 month of what used to be called "piecework".
That's quite different than hiring a person locally. Where can you find local people to do piecework? It's quite likely the task or tasks envisioned don't constitute full time or even part time employment. They're just one-time tasks.
Maybe temp agencies still supply people for jobs like this? But you're limiting yourself to a small pool of people unless you're in a big metropolitan area or unless there's a University with a large group of hungry students nearby.
I'm not sure I would do it again though. A lot of the businesses we were targeting don't have a web presence and therefore "No URL/No Email" was a viable answer. However, when I went through the list to see 150 "No URL/No Email" answers I didn't know for sure whether that is true or whether the Turker realized they could just copy/paste and make a quick buck. Amazon does provide the amount of time they spent on the task so I rejected any that were less than 10 seconds as I felt like they didn't give it a good enough try. Over that, I just accepted the answer realize that it may be false.
In the end I think I spent more time going through results and correcting them then it actually saved me. I'm excited to use MTurk in the future again, but only for appropriate projects.
All that said, Scale API seems like a nice alternative to Mechanical Turk for some kinds of tasks.
The pricing page isn't totally helpful. I'm not sure if I would fall under `Comparison $0.10` or `Data Collection $0.25/minute`... or if many of those prices apply to a single job.
Either way, I'll keep your product in mind.
It was a bad enough experience that I'd never base my mission critical product on a 3rd party startup. Amazon could go bad too but a small startup it's almost expected.
seems like the situation is ripe for such a tool
---------
start with a subset of questions you have a predetermined answer to, only keep feed questions if the person responding has met a certain quality threshold on those question
every so often feed them a QA question
every so often send the same question to someone else to check it for redundancy
seems like there is a lot you could do to adjust theses based on Bayesian confidence intervals, and exactly how mission critical you need certain data to be
maybe something like that already exists, idk
------------- (edit: is this what the Scale API does?)
Exactly what I was looking for: the cold brutal logic of captialism. It's all good if it's low-cost.
Even after all of this you will still get bad answers.
...yes, of course, you're not paying for quality, you're paying for quantity and to reduce your costs. If you were paying for quality you would put up a few posters on college campuses and pay more.
It's almost like somebody was trying hard to make this article unreadable.
I really respect your ability to work with MTurk and have it work for you guys. In our experience, it often takes significant effort to get anything remotely functional and reliable on MTurk. That's why we're building Scale :)
It's on iTunes - title is: "Getting to ground truth with Amazon web services mechanical Turk"
Video also available on YouTube: https://m.youtube.com/watch?v=vRtLdeNl7Tg
I still have some credits on the system.
I am not even sure if I could come up with a good use. For those with current usage experience, could I create in theory a task where people would look up the best sights to see in the top 10 travel destinations?
Would this be a valid use case, and how would you deal with duplicates?
Want another opinion on something you're thinking about?
Make a task asking people to write in their opinion.
Ask people for domain name ideas. Take two pictures of yourself, ask people which outfit looks better.
Having trouble writing something? Feeling uncreative? Ask people to come up with 3 ideas on some topic. Ask people to re-word a paragraph for you.
On thing to keep in mind is that you can ask for the same task to be done 10 times by 10 different people, and you should reward people when they put in at least a good faith effort. It's a good idea to get redundancy intentionally if the task is somewhat open-ended, as you may not get exactly what you want and want the best of the effort of a variety of people.
Drink and Turk is a drinking game where you try to turk enough to pay for your alcohol. I've seen some non-cooperative behavior of trying to find the funniest turk so the group laughs or give the most ridiculous answer still providing payment. Most turking is pretty boring but you'd be surprised what alcohol and youth and a slightly warped mind can do. There is supposedly a weed variation on this game, you can guess its obvious single rule.
For now, non US-based people can't sign up for AMT as workers (I think they made this change 2 years ago).
Someone somewhere must've written a guide about sourcing SEO articles from MTurk because I used to write 150 word article summaries (more for fun than anything, liked to try to go as fast as possible, they provide link, you summarize, even if the source article is less than 150 words, so its more like rephrasing) for between $0.25-$0.50/summary. There was usually only about 30 summaries on any given day which makes me think it was involved as a part of a guide or someone's startup maybe because the requesters were almost always different.
Somedays there would be several hundred. I made over $15 in an hour on a few occasions, but it wouldn't be a consistent source of income.
Also can you include javascript in the HIT HTML file?