Hacker News Reader

Comment by giarc | Hacker News Reader

I have an anecdotal story on the other side. I had a small startup and we had a list of URLs. We loaded them into mTurk and asked turkers to visit site and find an email address (probably could have automated this ourselves, but we didn't). Most of the responses were just info@<<websitename>>.com and we could see they completed the task in <10 seconds. So although info@ is a legit answer, after checking a few we found out the turker just guessed it would be a suitable answer and get paid for it. We basically couldn't trust any of the answers and just walked away from that service.

View on HN

You set up the HIT the wrong way. The correct way to do this is to assign the same URL to multiple Turkers and look for cases where people disagree in order to verify the response quality. Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.

You can also use qualification tests (https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanical...) to require Turkers to demonstrate competence before doing tasks. I'd imagine you could inject a certain percentage of known-answer tasks into the queue as well, as ongoing quality control.

> Granted it's still possible that both Turkers assigned to the HIT could commit fraud in the same way, but surely just as a matter of chance this bounds your error rate pretty substantially.

I wonder if there's been research into the effort in time or persons to perpetuate this fraud. Knowing Amazon, I'm certain they've analyzed this internally if not constantly at least a few times over the years. I wonder if there's a public attempt at the same analysis.

I tried MTurk as well out of curiosity. I only made a few dollars before giving up, but my observation was that like anything else in life, you get what you pay for. High paying jobs on MTurk require a impeccable record of a high number of accepted jobs and a near perfect acceptance ratio. You slowly (I can't emphasize how slow this is) build up your reputation as you complete jobs for pennies until you get to the category where you can accept jobs that pay $100 dollars or more.

Turkers who work hard to get the reputation to do high paying jobs won't risk losing the reputation by cheating. So if you want good results you need to pay up to get those Turkers on your job.

Seems like there should be other ways to boost your reputation. Maybe putting a deposit down which you can lose if your acceptance ratio gets too bad.

Wtf is with people suggesting contractors getting paid less than minimum wage should be required to put deposits that can be stolen from them if they don't work?

It's this just schilling or...?

No it's the standard solution for quickly establishing some reputation, prove that you have something to lose.

What's even worse is the people who do that simple guessing vs. actually doing the task ruins it for others -- eventually the Amazon mTurk AI thinks that your correct answers are 'wrong' due to a mass of others garbage responses. Thus your accuracy rate goes down and you can't qualify for many or most of the new mTurk tasks.

Amusing. Couldn't someone on the other end automate this, then cash in on it? I wonder if there's some potential in this thought: paying for automation as a service, based on the volume of output...

So... Fiverr or Upwork?

Those are paid on a per job basis. Different incentive structure.

Here you would basically be defining how much you'd be willing to pay per work item, regardless of how it's done; you wouldn't get the secret sauce, just the black box output.

It's not really a great idea, but in fairness I don't think it's much worse than what's out there either.

There are quite a few "scraping" companies out there that do this. They generally focus on getting you the data in an automated fashion, but it is black box and could be manual if they wanted.

I sense an opportunity to arbitrage mTurk -> Automation -> mTurk. Repeat as many times as necessary for a successful exit (IPO vs. Acquisition doesn't matter, either are acceptable)

Seems to me the answer to this is a sliding scale of compensation that rises for consistently valid responses and decays to zero for these sorts. Of course, this can be gamed as well, but it seems like everything's a Red Queen's Race these days so the only way to avoid^H^H^H^H^Hminimize one's losses is to race.

Did you mean "... is to [not] race"?

No, I meant exactly what I wrote. For example, if you were to try to duplicate Google today, you would drown in content farms and other sorts of BlackHat SEO because Google has been engaged in just such a Red Queen's Race with the BlackHats for over a decade.

So even if you come up with a scheme that rewards WhiteHat Turks, the BlackHats will inevitably crack it unless you insure it's a moving target.

Have seen similar things, as it sounds like a great way to use Mturk on the surface. Perhaps the answer is more direction and qualifiers, such as "must be a name, cannot be info@ or editor@"

Side note, if anyone has a way to automate the above in a way better than Hunter.io, there's big money in that.

Had we known about Hunter.io we would have used it. We had a small list of URL (~500). Would have been the small price to simply have it automated by a specific program rather than humans.

Reply on news.ycombinator.com