Ask HN: Scientific Study of HN flag dynamics | Hacker News Reader

Ask HN: Scientific Study of HN flag dynamics

7 pointswalterbell15 comments

Would there be any interest in a scientific study of the effectiveness of HN flagging, including:

  - false positives
  - false negatives
  - brigading
  - flagged topics vs off-HN contexts
  - flagged topic trends over extended time
  - content origin (domain, geo, org, author)

Pointers to prior research on distributed flagging and moderation would be welcome.

View on HN

Response to aaron695: https://news.ycombinator.com/item?id=30496522

> The open data would tell an incomplete picture. It would still be an interesting to see what you can pull out, but you'd be guessing why a lot of the time.

The current implementation is like the DMCA. Anything can be censored for any reason without input from HN moderators. If the submitter is aware of the process, they can appeal to HN moderators. If HN readers saw the submission before it was flagged, they can also appeal to HN moderators.

If there were organized efforts to censor specific topics, one mitigation could be to deny the benefits of censorship. This could be done from public data by extracting a queue of flagged submissions and feeding them into another level of non-public analysis, automated + human. The surviving stories could then be posted to a dedicated site, e.g. using https://github.com/lobsters/lobsters.

If there were organized efforts to censor specific topics, a secondary public log of valid-but-censored articles could trigger the Streisand effect. Not only could this deny the benefit of censorship, it could cause censorship to have the opposite effect, reducing demand for organized flagging. This would have no impact on organic, non-coordinated flagging.

Wait - are you proposing to study flagging or “censorship” on HN?

Flagging certainly exists. What is your evidence in favor of censorship existing here?

If memory serves, there are website(s) that track which submission get flagged. If you’re concerned about a favored view or topic being flagged frequently, you could start there.

Otherwise you risk exerting lots of effort to simply find out that a favored view or topic isn’t shared by lots of people here. And, well, not all of them are equally important, or equally important to the HN audience.

The study would be on flagging, which is typically done by independent HN users.

HN has non-public mechanisms to detect some coordinated activity (e.g. upvoting submissions) across registered usernames.

> If there were organized efforts to censor specific topics

Note the "if" at the beginning of the sentence. It's not about transient topics, but long-term patterns. Broadly speaking, "censorship" is only one possible motive for coordinated activity, flagging or otherwise.

> If memory serves, there are website(s) that track which submission get flagged.

Thanks for the pointer, found https://news.ycombinator.com/item?id=29144989 & https://news.ycombinator.com/item?id=29113079 & https://github.com/MoserMichael/flagged-hn.

Definitely.

Add to the facets: the submitted domain. Maybe posts by the armchair experts and quarterbacks on Substack get flagged more often than the WSJ, for example.

Thanks, added "content origin".

https://youtu.be/l2H99Pp9U6o

An larger and noisier category for study: recurring patterns of low-quality (downvoted, flagged) spam comments on some topics.

There are no objective criteria for classifying flags as false positive/negative.

  What does [flagged] mean?

  Users flagged the post as breaking the guidelines or
  otherwise not belonging on HN.

  Moderators sometimes also add [flagged] (though not usually 
  on submissions), and sometimes turn flags off when they are 
  unfair.

A false positive is a user flag that was reversed on appeal, after manual review by HN human moderators.

In a study, a list of false positive flags could be derived from public data.

In a study, additional false positive candidates could be identified after manual review.

Flags add gravity. Sometimes, but only rarely, to the point of flagged-to-dead. Mostly flags just move submissions away from the top of the front page.

It is very rare for those to be “deflagged.” Rare enough that there’s a moderation comment about turning flags off.

And an appeal that the community be on its better behavior. They are editorial exceptions not rule following corrections.

> Flags add gravity.

For the purpose of a study, gravity can be inferred from story rank.

> Sometimes, but only rarely, to the point of flagged-to-dead.

A study could examine the three categories of:

  - gravity
  - dead
  - unfair

It would need to be long enough that all of those categories were non-zero.

There’s no unfair…or fair.

HN’s not a court of law.

This sub-thread is about false positives, which has already been stated as a user-flagged event that was subsequently reversed by an HN moderator. That is a physically observable event that is visible in the HN public record, therefore it is an event that is subject to scientific study. It either exists or does not, so it is objective.

In this sub-thread, "unfair" is quoted directly from the HN guidelines as the term for the objective event of a reversed flag, i.e. a "false positive", which we are discussing. It's not being used here as an adjective, it's being used to connect this discussion to the official HN FAQ.

Is this data publicly available?

Raw flag data is not public. Some studies are possible on the public data.

Any interesting results could be investigated further, privately, by HN moderators.

Reply on news.ycombinator.com