Facebook moderators in America
theverge.com
theverge.com
I've also wondered how they deal with long videos that may seem correct for the first minutes, but the real inappropriate content is interlaced for a few frames in-between or later in the video.
I used to work there, and I built tools for these workers. Every one of the other engineers that I knew had nothing but respect for them, and everybody recognized that it was important work.
Facebook makes literally billions of dollars, they could easily pay more and provide better support to these people. They simply do not wish to do so.
The use of "contractors" in general is a shell game of abject responsibility.
Facebook takes a hands on approach to vendor management: many of these policies are agreed with vendors at contract negotiation stage, but even beyond that, many are set and enforced day-to-day by Facebook employees working as vendor managers.
The other thing to note (and this is true of most outsourcing arrangements) is that, while engineers may respect these people and wish to provide good tooling, they're kept at arms length by management in terms of communication channels. Outsourced workers don't have access to internal ticketing tools, aren't party to any internal meetings on tooling (even those at management level in the contracted company). Engineers can mean well but that means little if they don't have any means to understand the needs of these indirect employees.
If not, I'm sorry for the mixup - I'm just guessing based on your username.
See, for example, Apple's supplier code of conduct: https://www.apple.com/supplier-responsibility/
Consider for yourself: if you hired a contractor to remodel your house, and you found out that the contractor wasn't giving their employees bathroom breaks, would you simply shrug your shoulders?
Even in Phoenix, yes, I think $28k is being underpaid for this job, but you can't tell me they wouldn't also call them underpaid at twice the money, and take the opportunity to compare that salary to a Menlo Park salary.
I don't think OP meant to disparage the good honest work, but really, if FB (as a company) did value the work, it'd be in-house, wouldn't it? There's too much opportunity for mangement to 'tone down' issues when metrics are shared between companies, whereas in-house there'd be no hiding from the problems.
That said, it's great to hear that you devs were very mindful of the problems that face these people. I certainly didn't not realise how bad it was, so kudos to all involved really.
Janitorial work always has been under-appreciated. But cleaning up shit tends not to be high skill work.
> Janitorial work always has been under-appreciated. But cleaning up shit tends not to be high skill work.
the latter is a non-sequitur, IMO. work that causes that sort of personal injury should be well compensated, regardless of the skill level. it takes a lot of a person. we should try our best to make that person whole again, or to not take so much from them.
It may even be possible to selectively blur only parts of images for some classes of content. For instance, a filter that blurs portions of the image that are vivid red. If the red blur is on a dinner plate, then it's probably just a plate of pasta with tomato sauce. If the red blur is where somebody's head should be, that's probably gore and witnessing even that could be disturbing. But I know I'd rather see the blurred version.
Not it seems clear that if you open yourself to host something for your users you also need to be prepared to accept that you're going to be hosting the worst of humanity, exposing others to it, including people who work for you.
It seems like such an extreme contrast from what it seemed like user content could be and what it is.
Moderation isn't something that one could really escape from even about 20 years ago, just now we're more cognizant of the effects of poor moderation and users are more aware of how people behave when moderators/admins are pushed to the boundaries of the set rules.
It is a big contrast as far as I think what folks thought in the early days of user content.
Now, if even the most milquetoast comment which could possible be offensive if I assume the absolute worst about the poster, isn't moderated post-hate, you'll be reported to the media who will almost certainly archive it and use it to fill the next 1 - 24 hours of their outrage media cycle. I honestly see a future where people file lawsuits against web sites for not moderating enough, a failure to keep safe or sort of negligence. "Your web site gave me PTSD or enabled cyber-bulling". What's worst, they'll win.
I agree with the second part of that. Not necessarily the first.
Viewing the past through the lens of the present is always going to be a different exercise than viewing the present through the lens of the present.
Unfortunately, we live in a culture of intolerance. A few months ago there was a race car driver who lost his sponsorship because of something his father said before the driver was even born.
This is quite the theory - do you have more backing other than the single instance you've just quoted? And what is "intolerance" and why is this "bad"?
Would you believe me if I told you "we live in the greatest age of free speech in human history - our societies are more tolerant than ever before?" No? Well, I've given as much argument as you, so I suppose we are at an impasse.
Example: The 1985 Dire Straits song "Money for Nothing" is shortened when played on radio in Canada, and on non-rock stations on Sirius Satellite Radio because it has a verse that is now considered unacceptable.
Example: In the 1990's sit-com Frasier, it was mentioned at least once that people were being referred to psychological counseling for homosexuality.
Now, look at facebook. Billions of users. No constrained vision or purpose to being there; if you're human, you're a customer, and human behavior can be pretty gross at the fringes. And maybe most importantly; they can't just Ban whatever people or content they want, due to both a profit motive and public outrage due to their preeminent position.
This isn't a problem that has already existed, because the problem is with the Scale, not with the Behavior.
No. If you're human, you're a product. Can't have the product spoiled, now, can we?
> They are pressured not to discuss the emotional toll that their job takes on them, even with loved ones, leading to increased feelings of isolation and anxiety.
Just doesn't make any sense. Not only does it not protect the moderators, I can't even figure out what the cynical corporate interest would be.
And how do you even talk with your loved ones about this kind of work?
Usually the first corporate response is to sweep things under the rug and hope they will go away. Only if that doesn't work they will take real steps.
"It couldn't have been that bad, you didn't even mention it to your mother or your husband."
I think it is generally understood that if you outsource you can get another company to enforce policies you'd otherwise be embarrassed by / legally responsible for if they worked direct for the company. So if something goes wrong "oh that wasn't us, we told that crazy outsourced company not to do it". The other company doesn't care as they're not selling a product and it doesn't hurt them.
I worked for an outsourced customer service company when I was in college. The policies for their in house service and outsourced were explicitly different but customers were exactly the same. This was no mystery to anyone in the company. Policies for the outsourced companies were much much less generous to the customers. So much so at one point they retroactively changed their warranty policy that was enforced by outsourced customer service. In house stuck to the original rules, outsourced declared X, Y, Z wasn't covered. When they got caught after a year of saving money they declared that the company they outsourced service to did it wrong. Then after a time they'd switch back....
I'm guessing the complexity of dealing with political stuff very much requires a different kind of moderator than say just gore.
I'm not sure the employee's perks (or lack thereof) are the biggest story there.
Still your question is valid.
Facebook is just doing what they have to do to avoid falling foul of those laws.
As someone else said, Facebook is blocked in China so it's not really about China at all.
It would be interesting to see whether the same policy exists for work such as 911 dispatchers.
I suspect that as a billionaire and the CXO of one of the largest companies of the world, you can get disconnected from the details.
It would be valuable for them to see a different reality of their platform.
And the secret is that there is no current solution.
There is no automation to stop the sheer amount of dumb, sick, sad, horrifying, malignant and monstrous stuff being put up every minute from around the world.
Frankly it could probably break social media the same way cancer could kill the tobacco industry.
And this is simple moderation - ignoring newsworthiness, censorship, speech, propaganda and other issues this throws up.
The best they probably hope is that the users forget about it.
I mean seriously - what are publicly owned firms supposed to do?
Verification checkmarks allow one to screen out all content posted by unverified users, verification is available in return for a street address where FB mails a letter for a low-bandwidth 2FA. That's not so hard to fake, but also not hard to detect as fake when people deciding whether or not to filter out your content using shared blocklists.
But, that's the odd experience. Having to deal with new levels of depravity every day is different.
Anyway, what do you believe would change if Zuck did it for a day? They need moderators, as long as the technology for detecting it automatically isn't good enough yet. He knows it's a problem.
Well for starters he might stop outsourcing the work through a contractor who pays its employees 1/8 what the average Facebook employee is paid. Maybe the powers that be accept and acknowledge these low/underpaid contractors are developing PTSD like symptoms and are clearly not getting services internally much less the financial compensation to get such services externally...not to mention the contractual arrangement which seems to keep the contractors from seeking help externally.
People that work at FB in London also don’t average a $240k average Silicon Valley compensation.
After all what is the local market rate for a job that has shown the tendency to trigger PTSD without sufficient benefits to treat said PTSD? You would hope at least enough to cover PTSD treatments...Would you continue to use Facebook or allow your kids to use Facebook if there was a good chance of them developing PTSD symptoms? Would you take a job where there was a good chance you would develop PTSD and the job wouldn’t cover it and didn’t pay enough for you to cover it?
Considering that Mark Zuckerbeg enjoys personally using a stun gun on a goat then slicing it's throat[1], and spent a year similarly personally killing any animals he ate. After his day of undercover b may enjoy the work and declare the complaints baseless.
[1] https://www.newsweek.com/twitter-ceo-says-mark-zuckerberg-on...
I can't even imagine doing this job. This should be 100% automated with only 0.01% error rate for the first-pass filter (e.g. before it reaches humans).
They are willing to spare their users the horrors but not their own employees/contractors...
They returned to non Facebook employed humans doing the moderation
For some things liken nudity ML based solution work fine.
But other things such as hate speech are more nuanced and require human oversight.
That said ML is used everywhere possible to flag content to moderators.
For some categories the work of human moderators is used to train the ML models.
Facebook and other companies like YouTube are well insentivized to do the right thing here and automate as much as possible for all the reasons outlined in OP
You'd still have to investigate/review partial matches, but something like that could cut down a lot on duplicate effort and could auto-identify a lot of things that would need manual review.
One advice from a legal counsel that I received at that time still stands in my memory; she said: "Never look at the images, nevermind how curious you are. It's not just illegal and can land you in prison, but it's permanent. You cannot un-see something you've seen."
These mods may randomly see gang videos (go look for them if you want to see how tough you are) on any day.
Viewing them would make many people I know retch, worse than if they saw a gory horror movie.
And then some ideas, if presented without an antidote (creationism, flat earth, conspiracies) will subvert the people watching it.
And then comes the worst part - the damage to society itself.
If violent images are removed, depending on what people in Facebook and their friends, consider safe- then they are also deciding what is newsworthy.
And Facebook mods are not in a position to decide if an image of burning child, will stop a war or awaken the conscience of a people.
All they know is that it breaks “rule 4- death: graphic wounding, no gore.”
> “If we weren’t there doing that job, Facebook would be so ugly,” Li says. “We’re seeing all that stuff on their behalf. And hell yeah, we make some wrong calls. But people don’t know that there’s actually human beings behind those seats.”
Some respect and acknowledgement, and more than $15/hour, seems like it would be a good start.
It also occurs to me that even if this work arguably needs to be done, it doesn't need to be done by anyone 8 hours/day or 40/week. (Let alone in conditions where your job security depends on maximizing your "efficiency"). Anyone needing to do this should do it as a part of a job with a balanced set of work duties, with this being something they do only some of their work time.
These things would hurt Facebook's bottom line of course. These people are being chewed up and spit out for Facebook's profit and reputation.
We could just as easily make the argument that it's not safe for garbage men to hang off the back of the garbage truck, but at the end of the day a) nobody cares and b) I've got my own job that nobody wants to do that I'd love for someone to come along and make less shitty.
Yes, that's the problem. There is no law of nature that says we need to pay people with the jobs that are worst for their health the worst and give them no respect or acknowledgement.
If there are other jobs that is true for, yes, those too.
I wouldn't tell you what to care about, but I would say that we all would benefit by taking care of those of us whose jobs harm them. Your job should be less shitty too, it doesn't take away from that to point out other people who's jobs are harming them and aren't getting what they need. It is reasonable to begin focus on the jobs that are the _worst_, mentally or physically. Maybe yours is one of those too.
It's also reasonable in the HN venue to focus on the jobs that are the worst that make possible the economy that many of us reading this benefit from by being paid a lot more than $15/hour to contribute to, and probably aren't as harmful to our health as those for many of us. If your situation is especially dire, then, yes, you too.
It's not a law of nature, no; but it is a law of human psychology/economics.
The jobs that nobody wants to do aren't competed for—there are always plenty of such jobs to go around. Therefore, there's nobody kept out of the field. Therefore, the field doesn't ever build any social cachet of exclusivity. And that's a problem, because exclusivity is a prerequisite for the general halo-effect of respect that a field's members can earn.
Example: doctors must know a lot to do their jobs—but we wouldn't default to thinking of doctors as "knowing a lot" if anyone could be one without proving that they know a lot. We'd distinguish "well-trained doctors" from "badly-trained doctors" and we'd certainly give some props to the well-trained ones, but "being a doctor" would no longer command respect on its own. It'd be like... being a programmer, for instance.
That general halo-effect of respect is essentially what drives the ability of a class of workers to collectively bargain—it's what makes the other side of the negotiation treat them seriously. They know that people that are respected by society have a voice in that society, that they can use to denounce bad employers. So they actually sit down and bargain, rather than just firing all their old employees and getting a new batch.
Microecon says that jobs with more employer-demand than employee-supply should pay more, because employers must compete for employees. But because of this failure-of-negotiating-power, this doesn't happen; all the employers effectively are in implicit collusion to keep the wages of such employees down, by just not even taking seriously the idea that they should be competing for such employees. (It's kind of the same effect as employers "implicitly colluding" to fail to compete for talented female employees—but instead of being driven by employers' biases about demographics, it's driven by employers' biases about the assumed social-power that an employee must have if they are willing to take a particular job.)
But to get the members of the field to be respected—rather than just sympathized with (see e.g. sanitation workers—they have "hard jobs", everyone knows this, and people try to not make their lives harder than it is—but being one still doesn't make you any more likely to be listened to at a town council meeting), the field must first become more exclusive. That can work for fields where most of the work occurs in one place—see Hollywood and their actors' and writers' guilds. But if the work is distributed (like with community moderators)—how is the exclusivity going to happen?
Fundamentally all an employer cares about, and all that will determine if you are paid more, is if there aren't many other people applying for the same job. Even high skilled jobs requiring extensive qualifications - an opening for a CS PHD will always pay way more than a History PHD because there is so much less competition for the later pool of candidates, even if there is a greater performance disparity between a "bad" and "good" CS PHD than a history one in a similar capacity.
Appraising candidates definitely factors in, but even in highly skilled disciplines like game development, as long as the employer can assume a strong likelihood of getting good qualified candidates to replace you they will exploit you. Its when the prospect for replacing you is in any way questionable that suddenly your wages and respect will rise.
IMHO, construing that this happens because of a conspiracy between the employers isn't going to help.
There is a law of capitalism for it though, which we have, as a species, put well above nature.
These jobs are only so shitty because Facebook can get people willing to put up with them.
We sit in ivory towers of decades learning to program to be qualified as software engineers making... what I would really call just a reasonable wage for where we, as humanity, are in terms of wealth creation, technology, and productivity. But those are skills built over extreme lengths of time most people don't have, and they will suffer their entire life for it in subsistence wages.
There are so few fields today where there is both demand for labor and a requirement of skill to gatekeep the "unwashed masses" from driving the wages to the wage floor. Probably the most damning information I know of is that the Bureau of Labor Statistics shows that, one, the largest employment sectors are all unskilled, and two, they are the ones that will grow in the next decade [1]. US society is built on a foundation of despondent poor workers with no future and that foundation is intent to expand faster than fulfilling careers will... all of which make up only about 20-40% depending on how you interpret the data of actual employment.
[1] https://www.bls.gov/emp/tables/emp-by-detailed-occupation.ht...
Like... sentient beings putting themselves in harm's way to pick up garbage? Or just spending huge amounts of time taking orders for hamburgers?
Yikes!
I assume the major motivator for most of these cases is money- essentially, you may just kind of have to work all day, possibly at significant risk to yourself, to get paid minimal amounts so you can eat and have a roof over your head.
Ultimately, humanity is deeply undervalued. There are jobs which would absolutely be a net benefit to society if eliminated- the catch is you'd put those currently employed out of work.
Of course, that only happened after years of organizing and strikes. These are actions that I'm sure Cognizant could quell in a pinch - if Phoenix makes demands, they temporarily outsource to India and set up a new center in Albuquerque or Tuscon.
This may sound out of touch, but if Facebook wants to maintain this workforce and avoid more bad PR, they may need to restructure this as a flexible or incentive-driven environment. I'm trying really hard not to suggest "uberifying" or "gigging" the work, but allowing moderators to choose their hours or content type for the day could turn this into a desirable role.
And paying them more and improving their care, without a doubt. Subjecting people to that kind of trauma for next to minimum wage sounds like a class action suit in the making.
If it isn't already the case in the US, psychologically hazardous jobs should come with wage premiums.
It's not. It's typically in the interests of shareholders that those costs be externalized onto employees, especially if those employees lack the economic leverage to demand better treatment.
Don't get me wrong: the article is incredibly hard to read. When humans become untethered from social mores, they are capable of shocking and evil behavior.
But...can't people mostly police this themselves? If I had a "friend" posting video of a dog being stabbed, I wouldn't be their friend for much longer. It seems that the result would be relatively small cesspools of highly antisocial behavior. Cesspools which will continue to exist regardless of any moderating effort.
Alternative to moderation: introduce "public content ratings" for the content users post publicly. Machines automatically assess content and rate the user based on the maturity level of their public discussion. I would be "teen-mature" because I sometimes say "shit" online and perhaps another user would be "highly mature - disturbing violence and sexual content". Their ratings would weight groups.
Then, it's up to me to decide who I associate with online.
Caveat: nothing is a panacea. We're all trying to figure out how to handle the fact that we gave every maniac a megaphone to the masses.
But Facebook has a more user-curated thread, whereas HN is public. In a sense, I am my own mod on FB.
FWIW...I definitely don't use FB on a regular basis. Haven't since Oct 2017.
It's less about it being for you, and more about that it would be hard to sell ad space when it sits next to content of that type. Your friend might not post those materials, but what about Pages and Groups, which is where I'd say most FB users spend their time these days.
Not that I'm aware of, but there's a commercial one because advertisers don't want to be associated with certain kinds of content and FB is a public company. Usenet worked great as a distributed social communication protocol, but fell apart under commercial pressure, and it's not obvious how to compete with the huge infrastructural advantages of commercial platforms using voluntary protocols.
It seems that the result would be relatively small cesspools of highly antisocial behavior. Cesspools which will continue to exist regardless of any moderating effort.
One approach is to monitor the cesspools and discriminate against content matchable to those originating points.
The workload is massive. There’s not enough people, and the firms don’t want to pay as much as they need to.
Stepdad is a retired firefighter, has kind of that old-timey tough guy thing going on (although deep down he's a real softy) - he rarely likes talking about the hardest parts of the job, but the few times it has come up he mentions seeing some things that haunt him.
I remember one time him mentioning seeing something so grotesque when he pulled up to the scene that he started laughing at the absurdity. Sort of in a way that he could do nothing but throw his hands up and say "well fuck this."
He jokes about some of that stuff, but I can sense pain beneath the humor.
Someone with a strong constitution/stomach (i.e. the people who can browse 4chan while eating) might find this work relatively leisurely/tame for decent pay and flexible hours if adapted to a gig-economy model. I mean, there's a whole class of sick fucks who watch the stuff on reddit's /r/watchpeopledie for no compensation.
Might end up with less mental health problems if this type of desensitized personality was leveraged for this nasty-but-necessary job instead of recruiting from naive 19-year-old kids or whoever in between retail jobs and exploiting them in atrocious work conditions until they get PTSD.
Uhh, this doesn't sound at all like the stories of driving Uber that I've heard.
I don't think any of the people in the article would be doing this job if they had other options that paid as well, so I don't exactly see how your idea would improve their lives.
Wow. Wow. Is this really a thing, excusing forced trauma by claiming it makes people better?
Of course that's no excuse to downplay the occupational hazard of PTSD. But it's a real thing; facebook didn't make it up.
For example, some people take traumatic stress at really poorly (PTSD). Some people develop a world outlook to the tune of "well no matter what happens life is looking up from here" (post traumatic growth).
The counselor is looking at the glass half full. You kind of need to be a glass half full person to persist in that line of work.
The actual benefit of what you describe as growth is quite low to offset clinical PTSD which affects life quite a lot.
Furthermore there are many factors in play. The nature of the trauma and the predisposition of the person experiencing the trauma both seem to play a big role in whether or not post-traumatic growth is likely to occur. People with social support networks or spirituality are more likely to experience post-traumatic growth. Perhaps for related reasons, trauma that is systematic or collective (such as being a prisoner of war) is more likely to induce post-traumatic growth than trauma which is personal or individual (e.g. sexual assault.)
(Furthermore, no matter how distasteful the possibility may seem, it is possible that post-traumatic growth can take people past whatever their baseline was prior to the inducement of PTSD.)
Other people experiencing post-traumatic growth does not offset all that. It does not make for losses and answer to "is it overall beneficial for people to go through that" is still no.
Counselor answer suggests yes and that is what people take issue with. Whether one can have PTSD and post traumatic growth at the same time is different question.
I think it's readily apparent that "just show everything" doesn't work if you want to attract a mainstream audience, but I'm reluctant to just give up on the global public platform that FB was originally idealized as.
[1] I think I'd still find it unacceptable if the moderators were being paid 6 figures, had extensive 1:1 counseling, or any other perks - selling mental health for money is something I'm happy saying a utopian society wouldn't include.
We currently live in a dystopia where your Twitter or Facebook could be banned at their whim leaving you a digital outcast.
I suspect reddit is the best example of blending "something for everybody" with "default users don't see offensive things", while also being able to remove things like the content in the story, but obviously they still have human moderators.
I think the ultra-open / libertarian model is fundamentally incompatible with broad acceptance, personally [1], but I'm happy to be proven wrong. Early IRC/BBSes aren't very convincing to me because even if they were unmoderated the barrier to entry (knowledge, hardware) was high enough to limit adoption.
[1]: I think sites like Gab indicate that even if you clone a successful product and market it as "<x> but with free speech", the free speech part winds up being a negative factor, concentrating elements that will scare off mainstream users.
It’s a very difficult problem to solve and I don’t think saying “make it decentralized! Make it live on a protocol!” is useful without the extensive “how” that everyone seems to ignore.
Source: I'm a moderator on a Mastodon instance with about 1000 users (connected to the larger Fediverse of about 2 million users). We've got 5 moderators, and we respond to reports (either by or about our users) promptly and well. We don't have to police the behavior of the whole Fediverse (just our users), and we don't have to protect the whole Fediverse (just our users).
Do Mastodon admins share common blocklists or anything? If a bad actor decided to start posting offensive content to random instances, I assume you can ban that {username | IP} from the instance used by you and your users, but would they then be able to just iterate through the other n Mastodon instances? Is there anything in place to prevent them from creating a new account and repeating ad nauseam? (not that there is on Facebook, necessarily)
(I don't know anything about Mastodon, which I'm sure is obvious from some of my questions - if they're incoherent in the context of Mastodon that's totally fine)
Mastodon instance administrators have the ability to block whole instances, and this is usually done because of bad/nonexistent moderation policies. There is some sharing of instance blocklists, more as a matter of convenience than of policy.
(As for your footnote - people will rightly point out that any media will be used for some awful content, but that doesn't mean every system has an acceptable rate. "Can we get the rate low enough to tolerate?" is still a legitimate question.)
Every time Youtube makes the news for having unsavory content, their responses seem to have the same underlying tone: "we really are trying, but this is impossible." Every day, 24,000 days of video are uploaded to Youtube. Live human review for every new upload would take >100,000 full-time viewers, plus whatever is needed for comments, review appeals, and copyright notices. I can't even find good estimates on how much existing content there is. Some analysts have suggested Youtube might produce $15B/year in revenue. That review team would cost $3B a year at $15/hour, before payroll tax, benefits, counseling, etc; so we're talking about numbers in the ballpark of determining whether Youtube can be profitable. And yeah, you can do playback at triple speed, automate some basic content-filter removals, prioritize content posted under dubious keywords, de-prioritize major channels unlikely to upload something awful. That all lowers costs and makes it more plausible that troubling content will actually get caught.
But you're right: even if we wanted to run this as a public good, all those cost-savings only serve to concentrate the human toll. Law enforcement has been dealing with this issue since photography became widespread, and hasn't found a way past traumatizing people who have to sort through abhorrent content. Now, the same digitalization that makes distributing media nearly free makes it possible to create that experience a thousand times as often. (Shock sites, after all, are just a version of the same pattern consciously centered on the unwanted views.)
So what to do?
Robust tagging systems can help reduce unwanted exposure to content without requiring moderator involvement, the same way it's helped fanfiction communities reach a detente over protecting readers while allowing mature content. But it's telling that the shouting matches over insufficient tagging and robust age controls continue, and of course this only works for content that's acceptable on the site - a "take this down immediately" flag doesn't get used.
Federated approaches like Mastodon and WhatsApp groups might reduce legal liability and help people find spaces they're comfortable in and diminish unwelcome surprises. But that sacrifices both oversight and unity - even if we accept that some instances will be used to trade illegal content, plot violence, etc., we still lose the network effects of "searching Youtube" or "being on Facebook".
At the other end, I suppose ever-more-aggressive centralization is possible; the ability to multiply harm depends on being able to act repeatedly. If Google demanded your SSN to leave a Youtube upload or comment and banned bad actors for years, or the government ran Facebook with warrantless access to every message, the frequency of this sort of content would decline and the moderation burden might shrink greatly. (After all, anyone can tack a grotesque picture to a physical bulletin board, but there's not much of an issue with obscene UGC there.) Of course, the issues with that are screamingly obvious. Identity and the risk of consequences have a chilling effect, legitimate behavior can still be embarrassing when such a store is predictably breached, usage creep for this sort information is not just a risk but an invariant, and governments are not only inclined but often required by law to act on all sorts of not-actually-bad content like "promoting drug legalization".
Beyond that... I don't actually know. Allowing even minimal privacy and UGC while trying to maintain a palatable space seems like a genuinely unsolved problem. There might be a lot of unexplored value in trying to reduce psychological cost without taking humans out of the loop. Most 'digital natives' have seen some horrible shock content without lasting harm, so perhaps the winning approach is to reduce the time/frequency/vividness of exposure to a manageable level. At the very least, it seems like a less exhausted space than the technical side of things.
It's a little discouraging (considering how influential it is in the evolution of tech) how quickly HN reaches for "rah rah decentralization" as a panacea for anything social media.
They don't even try anymore.
Now that pit is billions of people not thousands. I have little hope it’s going to ever get better. Is this the destiny of all internet discussions? I worry about this a lot.
I have not seen this documentary, but compared to the verge article, I think a major difference is that the filmmakers did not get cooperation from Facebook or the third-party content moderation company. In fact their photos were posted and employees were told to stay away from them.
https://www.vice.com/en_us/article/ywe7gb/the-companies-clea...
>dark humor and camaraderie
Yes. No one can know what happens if you start moderating, except unusually inquisitive users or other mods.
> believing conspiracy theories
Not all ideas are equal - and not all brains/people are equipped to deal with them equally.
Your mod team will soon get polarized or even subverted if faced with the kind of crazy work these facebooks mods have to do.
If you are actual community mod, life is slightly better, since there’s more context.
https://www.amazon.com/Behind-Screen-Content-Moderation-Shad...
Here's a presentation Dr. Roberts' gave at conference last week on the topic too:
https://www.youtube.com/watch?v=7mdMtukvtxc#t=15m50s
(Until recently at least, most of this content moderation work was done by contractors overseas, largely in the Phillipines. Can you imagine what the working conditions are like there?)
At least pay these people more than a dishwasher, please.
Seriously, who thought blocking this is a good idea? Do you really need to see an entire video to see it is against the rules? It will also save (a lot of) time
"Oh but then they can just do a shallow evaluation of the video" yeah I don't think that is the case with most of the videos.
It would be 4chan, minus the anonymity.
The moderators told me it’s a place where the conspiracy videos and memes that they see each day gradually lead them to embrace fringe views. One auditor walks the floor promoting the idea that the Earth is flat. A former employee told me he has begun to question certain aspects of the Holocaust. Another former employee, who told me he has mapped every escape route out of his house and sleeps with a gun at his side, said: “I no longer believe 9/11 was a terrorist attack.”
I have a hope that people will eventually develop immunity to conspiracy theories after falling for enough of them and eventually seeing them disproved. But with things like pizzagate/Qanon it seems some people just keep going deeper.
It doesn't surprise me that it can lead to similar outcomes, as being in a cult.
- vaccines, which build immunity by providing an inert or weak threat that the body builds resistance to.
- herd immunity: restricts the spread of diseases if a sufficiently high proportion of the population is immune to the disease.
Well, many of these things essentially have a starting state as 'disproved', ya?
Although what I'm curious about in my own example is, how much of a conspiracy theory was this before it was confirmed, as in, were there many people saying that the CIA is doing mind control experiments?
Less important but related, how much do people distinguish between that reality and sort of 'corollary' / related conspiracy theories that are not proven (as an example, that RFK's assassinator was a subject of CIA mind control through MK-ULTRA -- this is unproven but people might assume it's true when they find out that MK-ULTRA was a real program).
It's not quite a good analogy, but it also reminds me of how I noticed that building/construction contractors I know almost invariably believe that the risks of asbestos and lead paint are highly overrated -- I've thought it's kind of for their own cognitive self-preservation, since they unavoidably get a lot of exposure, and don't want to believe their job is killing them. But maybe it's also just got something to do with a sort of "exposure therapy" effect where whatever you are exposed to becomes normalized.
Please realize that brains are hackable, and that there is always a set of content that will be absorbed by a brain which appears plausible to the brain, but in reality is false.
Ideas can be configured to get past an audience’s mental defenses. It will happen to you, and you will see it happen to your children, your educated parents, your well mannered neighbors and more.
Maybe there is a mental type which is immune to conspiracies, but that kind of immunity will come with some other mental cost.
I’m seeing gentle people in my family, repeat sentences that always predicted a user was being radicalized/polarized online.
The only safe move is to get off of social media.
I always figured the Facebooks of the world needed more moderation. But this makes me think that maybe the only effective move is to target further upstream and somehow constrain submission or account creation.
I can't see how this ends any way other than with network authentication of computers with signed bootloaders that can stipulate the operator's identity, paired with criminal statutes prohibiting operating a computer under someone else's identity or defeating the authentication/encryption that makes it possible.
Conspiracy theories don't appeal to the rational. A conspiracy is built on an emotional foundation and the 'logic' is just scaffolding to hold the emotional payload. A lot of people (but (i don't know what proportion) are susceptible to emotional addiction and contagion.
It's not wise or safe to assume that people can/will/should just develop cognitive resistance (though I totally understand your hope that that would happen). Sadly we have abundant and growing evidence that it is relatively easy for bad actors to cultivate antisocial behavior up to and including murderous rampages.
There is a portion of population that would be generally unfazed by the content, it would be best for everyone if they worked these jobs.
I'm sure some of them are jobless and would love to be paid $28k a year to do something they already do as a bit of a hobby.
You're also entirely missing my original point. The people currently doing the job don't like it. They don't want to be doing it. And yet they are. Why do you suppose that is? Perhaps... because there are no other jobs available? Saying "you should be looking for a new job" to these people is like telling a homeless man he really ought to be looking for a rental property in his budget range.
It's like having someone with poor sense of smell take out the garbage.
No, it's like having someone a poor sense of smell detect which things smell and which things don't.
Not to mention, having them do this work is taking advantage of a dysfunction rather than helping to address it. In fact it would be likely to make the dysfunction worse.
I'd guess most can identify murder and gore films even if they don't mind them. They're pretty obvious.
I don't understand why taking advantage of this "dysfunction" is a bad thing.
What you see as dysfunction, their apathy when viewing bad content, is their normal state of being.
Why do you think some folk are drawn to habitually view such content?
What other things do you think satisfy that impulse?
Curiosity, memento mori maybe.
It's paradoxically unusual but at the same time incredibly relatable. Everybody dies, so there will be broad spectrum curiosity for the subject, inhibited only by natural squeamishness.
Years ago when highschools still had shop classes, the teacher showed my class a binder full of color photographs of what a lathe accident looks like. The lesson was that lathes aren't toys. I'd never seen anything like those pictures before. They were disgusting, and fascinating. Probably psychologically damaging, but not as damaging as getting caught in a lathe. Did I mention the pictures were fascinating? For most people, seeing that sort of thing is rare. Some people are drawn to novelty, particularly when they can relate to it. Very little in life is as relatable as death; death is even more universally relatable than eating.
Is the suppression of squeamishness a form of mental illness, or a form of psychological damage? I think it definitely can be. But I'm far from convinced it necessarily is.
Because their squeamishness is suppressed to an extent that is unusual among the general public, and what remains is the now uninhibited fascination.
There's all sorts of shit you might have to put up with in a shitty job, the work, the environment the customers. This job is shitty but some people are bothered less by this particular kind of shitty than they are by by angry customers yelling at you or laying bricks in the Arizona weather. Sure you might have to see some really screwed up shit among the sea of nip-slips and racial slurs but you get to do it in an air conditioned office without killing your body like the guys in the Amazon warehouse. Some people prefer that.
Different people have different degrees to which they'll tolerate the various ways shitty jobs are shitty. At least one person the author interviewed said "well, it sucks less than Walmart". Back at the point in my life when I was doing shitty jobs I would have preferred brick laying or custodial work but I wouldn't have turned down this job if I needed it. Of course it's terrible for you and turnover is high but all the equally shitty jobs are this way. There really is no winning in the "minimum wage or thereabouts" income bracket.
Edit: Since apparently this opinion is unpopular can anyone tell me why this particular implementation of shitty job is worse than all the other implementations?
A lot of manual labour jobs are protected by unions, which establish safety standards, limits to shift length and healthcare to help employees when they need it. Sounds like these employees could do with the same.
I'm not comparing to those jobs. The union dishwasher in some public university cafeteria gets time off, healthcare, etc, etc. that makes his or her job much less shitty than the infinite-term temp that works beside them. I'm comparing to the temp.
You can make a job better/worse by adding/subtracting pay, benefits or working conditions. If I could get tech pay and benefits for construction work I'd be doing construction.
If we're gonna compare this job to others then we should compare to other jobs that are "equally shitty" not jobs that are similar but actually compensated more highly if you roll the benefits into total comp.
"Nobody ever is," is also an absolute dripping with sadness. Just my 2 cents on the reactions, I doubt it's because they think the working poor aren't systemically disadvantaged.
I feel like that doesn't do justice to people who have to work shitty jobs to make their living but I can't quite articulate why.
Reason being that, from a practical/cynical perspective, these proposals can too easily be co-opted into red-herring cudgels intended to end discussions before they delve into the realm of the more granular, nuanced, uncomfortable details.
People go to war zones and put their lives on their lines to have their minds be harmed in the same way - in contrast these moderators don’t even get hazard pay.
Not because of the exposure to "bad things" like the article pushes, but because I've always felt totally fine (even bored) looking at that stuff, and I don't agree with censoring it.
It's just "meh", another day of reality on Earth; the Internet is just a mirror held up next to it.
Will we ever collectively realize this? It seems younger crowds are more normalized to this stuff because they grew up on the Internet. But add in the hypersensitivity in today's public social network sphere, and we're all freaking out over anything even potentially flammable.
In the article, the example of a stabbing video is given. I would strongly feel the need for retribution or justice for that victim. Same for all content that shows someone being hurt.
I guess what I mean is that it's wrong to try and purge all this stuff as if it just doesn't exist. This is real content, real people, being hurt in reality. I'd feel disgusting trying to censor it, not disgusted by the content.
In other words - I sympathize with the hard-ACLU/EFF stance on free speech. But there's a difference between government censorship and societal moderation - the latter has always existed, and the scale and automation of modern platforms in publicizing content that normally wouldn't spread so far is what's new.
Should people be aware of bad things that happen in the world? Absolutely. Is broadening the audience for disturbing videos the right way to raise awareness? Maybe not.
And if folks really feel such content needs to be published, they still have more options and reach than they did in years past even if "mainstream" places like Facebook moderate them. Granted this last point is getting a bit trickier, as people go after registrars and web hosts themselves for political reasons occasionally (i.e. if this was Cloudflare we were talking about I'd be in full agreement with you - then again Cloudflare doesn't run a recommendation service that automatically causes new unexpected content to appear in front of billions of people).
I’m not sure I really understand why big companies feel it’s their job to censor the world. Just let the floodgates open: allow violence, allow porn, allow whatever else. Who cares? And maybe this might actually good for people to see. I feel like we are leaving in this outrage culture that wants to censor everything. It’s not healthy for individuals or society.
That's not how social networks work.
Just blocking it yourself seems like a better solution. You dont have to be fair in what you block yourself and (surprise surprise) you dont need to watch it in the first place.
How does that work if nobody watches it, how do you know what to block?
You only have to watch it through if you have to make sure its actually gore. Which an individual different to an employee doesnt have to.
Or simply automatically collect reports and if a high enough percentage of views vs reports comes in it put it nsfw.
I'm not sure that is any different than what the article describes.
>It’s a place where employees can be fired for making just a few errors a week
As I understand it, an employee has to make sure its extremely likely that it is a murder. She is told she can pause it by the psychiatrist, not to close it. You as an individual dont need to find out. Just close it immediately if you assume its gore.
I don't know what you mean by that. If you visit facebook, you don't get to pick what you see. Certainly a few seconds of a murder is going to be less than desirable.
> Certainly a few seconds of a murder is going to be less than desirable.
While not desirable, outsourcing your second every few days to some poor guy who has to watch it all day long doesnt sound like a good solution to me.
I dont think its a simple problem. It is one without an optimal solution. I just think the downsides of having the stuff around are preferable to having people ruining their mental health out of financial desperation.
I also believe that the reliability would go up with time, people are able to learn quite a lot.
Not a popular opinion, but its not reasonable to let kids have access to the current version of the internet and childproofing it is not a viable option. A lot of the issues we currently have with the internet can be boiled down to the contradiction of kids using it and the inability to child proof it. If we think that kids shouldnt have access to everything the internet allows access to, it should be feasible to get a second child proofed one online. With the focus on it being child proofed in mind during design, the maintenance wouldnt be that bad if you focus on access control. Kids are not allowed to walk into a sex shop or a bar, they get the internet right on their phone.
All societies have regulated speech. I don’t know anyone who doesn’t believe this should be so. A desire to censor some things should not be spoken of as a desire to censor everything. Such hyperbole undermines your position.
With Facebook and Google speech is easier to disseminate. It’s also way easier to target said speech. No one can be vigilant at all times in terms of checking sources and accuracy of what we see/read/watch. People are easily duped. People are easily led to believe things that are obviously false. The anti-vax movement is an example of this.
We have entered an information age unlike any other in the past. This ability to sway, target, and reach so many at scale for so little cost has implications that may cause society to re-examine the nature of free speech. If Facebook let’s the floodgates open as you say then this re-examination will come much sooner and likely be the result of outrage and thus end up hurting Facebook in the long run. Facebook would like to prevent this from happening. Hence they take steps toward mitigating the public perception of Facebook enabling bad actors.
Unfortunately, yes in various ways. But the devil is in the details. Who decides what specific ideas should be outlawed?
The politicians who write laws. Lawmakers decide what should be outlawed.
I'm not sure why people paint this process as complicated. We live in a society, that society decides what is acceptable and what is not. You are welcome to disagree with the conclusion, but it's not some shrouded mystery how we make the choices we do.
Reading between your lines here it seems like you have never tried to define what is allowed behavior and what is not and more importantly, tried to applied that to specific issues.
It's easy to all agree that pornography, hate speech etc. is bad. But what is pornography, and what is hate speech? Where do you draw the line...
If I had things my way, no piece of information would be illegal. Child porn, nuclear weapon plans, whatever, I don’t care, it should all be legal.
But I accept that it’s a fringe position, though a completely tenable and consistent one. It’s all information and I think it’s ridiculous to try and censor something that only exists in the abstract.
Moreover, once you start censoring you will never be able to stop. Even though I’m sure people will do bad things with information, I believe that history has shown that the good will outweigh the bad.
And, as history shows, censorship waxes and wanes in every society; you certainly can stop censoring things that once were censored.
I don't think everyone having access to nuclear plans and biological weapons is tenable. Those things aren't abstract.
Moreover, once you start censoring you will never be able to stop.
Your position become less credible. You need to argue that the censoring that is currently being proposed or being done is indeed bad for society. Or that we’d be better off without said censoring. Engaging in slippery slope style reasoning is not helpful.
Acceptance of child porn isn't tenable, that literally creates a market for rape. Sure, you may be believe markets are an unstoppable force in human nature and that suffering is inevitable, but those are just the axiomatic beliefs you've chosen to subscribe to. If you don't care about the imposition of suffering on others that these entail, are you willing to accept the same degree of insecurity for your own person?
I wont even attempt to define "porn".
Because that's what the users and the customers both want. Facebook is popular and profitable BECAUSE it provides a mostly-safe (compared to eg 4chan) place to communicate, not despite that.
If people see offensive content on Facebook, but not on a competitor, then both the users and the advertisers are going to abandon Facebook for the competitor. Every company tries to give their customers what they think the customers want.
now the moderators are just subjected to constant child porn or potential child porn, which, let's be honest, is still essentially the same issue re: their mental health.