Indeed, I've seen at least one paper where they used Mecanical Turk and aggregated results (by showing the same examples to multiple people) to check for quality. It sounds like they did almost exactly the same thing here.

Paper: 'Antisocial Behaviour In Online Discussion Communities' (Cheng et al., 2015)

Link: http://arxiv.org/pdf/1504.00680v1.pdf

(Search for 'Mechanical Turk' / section Data Preparation -> Measuring text quality.)