Hacker News Reader

I used to work at Twitter, so not sure I can get into specific numbers, but they remove hundreds of thousands to millions of accounts every day (literally every day). It's particularly tricky because bots aren't inherently against the ToS (e.g. the earthquake bot cited in the article), so you can't just ban based on "Is this account behaving like a human?".

Many "bots" though are actually giant networks of humans being orchestrated and they share things that are compelling to a subset of real users (e.g. Hillary did this bad thing... lock her up), so now real users are retweeting and muddying any signal that was there to detect a bot network. It's an incredibly challenging problem at scale and you basically only see the < 1% that Twitter missed.

They've got a pretty sizable team that works on it so hopefully in a few years they'll have solved it much like Gmail has mostly solved spam.

I think it was the 99% Invisible podcast that did an interesting report about whole networks of people who were paid in Mexico to spread election messages; and these networks are often reused by gangs or other organizations to promote stuff or harass opponents. So in this case, they're not bots. They're actual humans, because it's cheaper to do that than pay people to write scripts.

I don't remember that 99pi. Was it Reply All?

https://gimletmedia.com/episode/112-the-prophet/

After Andrea is attacked by a stranger in Mexico City, she just wants to figure out who the guy was. Investigating this question drops her right into the middle of one of Mexico’s biggest conspiracies.

Ah yes, sorry you're right. It was Reply All.

Additionally, you can train a human to become better at astroturfing over time, and train your network as a whole to be resilient to partial takedowns.

These things are much easier to orchestrate than bots, especially if your specialization is not CS but scamming / manipulating opinions, and in many cases, especially with scale, human labor can actually be more expensive.

Of course, subsidizing the labor to impoverished countries definitely helps; these networks tend to lack the sophistication used in more sensitive political or industrial topics.

Sometimes for national or global campaigns you'll find a mix of both cheap labor, a small team of experienced astroturfers with a good grip on their persona acquisition and management, and bot networks, each targeting a different set of demographics. If your target demo is sufficiently ignorant or radicalized in their beliefs, it doesn't take much to convince them and bots will do just fine.

> much like Gmail has mostly solved spam.

Gmail has not solved spam. Gmail and Microsoft dump so much legitimate e-mail that it's basically made e-mail totally unusable:

http://penguindreams.org/blog/how-google-and-microsoft-made-...

No they don't. They, like most abuse sensitive recipients, don't accept mail from small indie servers, so people know not to use small indie servers to send mail.

It's not just small indie servers. They do "reputation scoring" of senders that penalize them for things like sending plaintext mail. See https://groups.google.com/forum/#!msg/mozilla.governance/WWK... for more details of one example.

Right! We can't trust those "small indie servers" on the world wide web. Who knows where they've been?! Make sure you use mail servers furnished by Google, Microsoft, and other NSA-approved facilitators. They only cost $10 / mo / user, that is easily affordable for anyone who is not a Russian agent tricking the unwashed masses into preventing Hillary Clinton from ascending to her rightful place as ----- errrm, uh, from "corrupting democracy".

Any non-Google, non-Microsoft mailers will be blocked, lest users be asked to renew their VIAGRA prescription from a Canadian pharmacy, or a non-compliant political message be received. It's for your own good, citizen!

Thank you, wise and great Google! Thank you, wise and great Microsoft! I wouldn't know how to use this free, non-controlled computer network without your benevolent work!

unsnark: Seriously, the way that Google et al have captured "anti-spam" mechanisms and converted them into "pay a big company for your mail" mechanisms is extremely sad, and speaks to the true forces at work behind all the warm and fuzzies about an "open web".

I’ve read this a lot, often as a counter argument to running your own mail servers, and I believe it to be mostly FUD.

I run a small SaaS and it sends out quite a lot of notification emails to customers (they are paying me for these emails, they aren’t spam or newsletters). I was originally using SendGrid but their deliverability rates were pretty bad and I was about to hit a ceiling for my plan, which would meant the monthly cost would more than double, which was unaffordable for me.

I decided to try setting up my own mail server. Everyone said don’t do it, you emails won’t be delivered, it’s too much work, etc. I setup a VPS with OVH as I’ve heard their policies regarding email spam are pretty strict (I contacted them before to clarify - if they find out you are spamming they cut your off ASAP to protect their other customers). The IP I got was actually on one blacklist, but I got that resolved in a few days.

The deliverability rate was better than SendGrid and it was costing me 1/10th of the price. One thing that I would assume helped is the domain had already been in use for a few years, and I setup proper DKIM and SPF records.

The server has been running for a year and a half now, and the only bounces I get are due to invalid or suspended email accounts. Most of my customers are small to medium sized businesses, a lot of which run their own Exchange servers - with SendGrid those were usually the ones who were bouncing my mail due to being on a blacklist. Now admittedly just being accepted doesn’t mean it is ending up in a users inbox, but I haven’t had any complaints from paying customers.

I'm not convinced about this generalization. Case in point is the mail relay server of our company which definitely counts as a small indie server and whose email is accepted and not marked as spam.

Of course we have configured SPF, DKIM and a proper reverse dns lookup on both ipv4 and IPv6, but that’s just being a good email-citizen anyways.

Even if that were true, then your parent's statement still holds: "basically made e-mail totally unusable"

If it's down to a few big senders and the rest is rejected, they might as well get rid of SMTP, close it off, and just talk a proprietary protocol among each other.

Maybe it's the "one click to share" behavior - an extremely low-resolution action - that is the problem? Perhaps things would be easier to fix if the human engagement component were more than "can you press a button?"

It seems to me the simplest answer is that they're fully aware and not doing anything about it. It doesn't take the best and brightest to figure it out. I've purchased thousands of twitter and facebook followers before for $5 on fiverr just for fun. There's no reason twitter or facebook can't do the same and identify the fake accounts. The logical conclusion is that they don't want to.

I agree. Just buy them. If they spend $100K a year on buying fake accounts and then banning those accounts, that will be more effective than messing about paying a team of 10 engineers to do pattern analysis. If they spent $2M a year on buying and then banning fake accounts, they would squash it completely.

They're not doing it because they like the big follower counts themselves. They're in on the con.

How does paying the botters market rate raise their cost of entry?

The price of a single bot is amortized across thousands upon thousands of follows. If it costs a penny to have one bot issue one follow, and a single bot performs 5,000 follows, then that bot has earned $50. If Twitter were to buy follows for the market rate (one penny) then ban those accounts, it would drive the market rate for bots up to (using our hypothetical example) $50 per follow. (This is an oversimplified example, of course, but reducing the average number of follows that a given bot can perform would indeed have the effect of raising the market rate, though we can continue to argue over the magnitude; it's self-evident that if generating bots didn't have fixed cost overheads, then we'd see fewer bots with incredulously inhuman numbers of followed accounts.)

This seems liable to cause unintended side effects...

https://en.wikipedia.org/wiki/Cobra_effect

I disagree. The cobra effect occurred because the English provided a market for cobras where there was no market before. There's already a large market for followers and twitter adding itself to that market in such a limited way does not add significant new demand.

The Reagan administration tried this in the Iran-Contra affair. They only succeeded in incentivizing more terrorism and abduction of US citizens.

Twitter has a much higher cost for false positives. If 1% of the accounts this article thought were bots were actually legitimate, no big deal. If Twitter banned that 1%, that's ten thousand pissed off people.

Also, bot makers don't care about fooling the New York Times, they care about fooling Twitter. If Twitter applied this analysis, the bot makers would adapt to it. It only looks like it works until you start to use it.

> If Twitter applied this analysis, the bot makers would adapt to it.

The problem with the arms-race argument here is that it implies that just because an arms race is inevitable that the only acceptable course of action is total capitulation. Twitter is bigger than any of the botters, and if it wanted it could raise the cost of entry of botting high enough that nobody would be able to afford the bots in any quantity large enough to make a difference. They just simply don't want to.

(As for false positives, I've seen plenty of people whose accounts were suspended for reasons unclear to them; Twitter seems to care very little whether its captive userbase is pissed off or not.)

I don't think it implies that unless you've already accepted they aren't doing anything. I think it's entirely possible that a lower class of bots are routinely blocked, but a higher class are always mutating to get past the latest filtering, so has a large fairly stable population.

> Twitter is bigger than any of the botters, and if it wanted it could raise the cost of entry of botting high enough that nobody would be able to afford the bots in any quantity large enough to make a difference.

How would they do this without triggering enough false positives to destroy their product?

I mean seriously, if you are capable of doing this that well, you could probably walk into a job paying >>$750k/yr at any of the large tech companies, because they'd be falling all over themselves trying to get at your magic.

Seems like, if you detect a bot, you should impose a captcha on every x actions. Not enough to ruin your customer's experience, but enough to make botting harder.

I would be absolutely stunned if Twitter et al. hadn't already tried this...

I wouldn't be suprised at all. Go find an openly racist account and report them. Two weeks later Twitter will agree with you...but not ban the account

I’ve come to the conclusion after reporting hundreds of obvious spam/racism/hate/etc accounts that twitter just doesn’t care. It increases their metrics and that’s all they want. Makes the share holders happy to see numbers going up and sadly in turn it does increase ad views.

I would be curious to know how this failed, if they did try it. Are captcha automatable? You could make it a bit better by challenging every X actions, and have X decline with a failed captcha.

Captcha has issues in that:

- captcha that isn't hard for humans is often easy to automate

- captcha that is hard to automate frustrates a lot of humans too

- and to add insult to injury you can "automate" the hard ones too by paying like $1 per 1000 captchas or something (think Amazon Turk like stuff)

> How would they do this without triggering enough false positives to destroy their product?

This implies that Twitter has demonstrated that they care about practicing discretion when suspending accounts, and that they care what 99% of their (human) userbase thinks, and that Twitter's product is Twitter accounts. But they haven't; they don't; and it's not. The state of being pissed off at Twitter is the perpetual existence of all Twitter power users. Unless you're a celebrity, Twitter doesn't care what you say about Twitter. And the only thing important to Twitter's business model is keeping celebrities on Twitter. The reason that they don't give a damn about cracking down on bots isn't because they don't have enough $750k engineers, it's because dealing a blow to the egos of their Influencers by banning their botted followers would threaten their bottom line, both in terms of outrage from the 1% of Twitter accounts that matter and in terms of depressing their wildly inflated metrics.

TWitter trashed my very real account that was tied to a disused email address after I tried to log in from my phone and misremembered my password and refused to lift a finger to help me get it back despite me offering to provide all sorts of other verification information.

That’s nonsense. It’s trivially easy, and as noted elsewhere they can just add a verification check for suspect accounts.

It’s easy to see how much harder Facebook and Google accounts are to create bots for than Twitter. Twitter doesn’t care.

> They just simply don't want to.

This really does seem to be the case. And I find it incredibly distasteful that they were used by DJT in his campaign and continues to be used. It was rarely a place where politicians posted anything, and now it seems to be the first place where any of them do.

> It was rarely a place where politicians posted anything

Twitter was founded in 2006 and Barack Obama joined in 2007. Today he has 15.4k tweets. Seems like a lot to me.

There are plenty of ways besides suspending accounts to make the cost of botting too high to be profitable. A captcha for example is fairly frictionless for a typical user.

It seems to me that there may be a more benign explanation: Facebook/Twitter/etc are in a large-scale, iterative, adversarial game with many opponents.

It might be relatively easy for humans to catch clusters of fake followers, but that doesn’t scale. If you try to create heuristics to catch the fake folllowers, the adeversaries will try to adapt. If you try to learn rules, you can update your heuristics faster, but you need trading data, your adversaries could learn too, and adversaries could try to do data poisoning attacks.

It seems like a really tricky situation for Facebook and Twitter.

I think that you are right, but only in the qualitative sense, not the quantitative. Spam networks have shown us that taking out a few big players - the largest few sub-graphs of the network - can have dramatic effects on spam rates, reducing them by e.g. a factor of 20. I’m confident that Twitter could identify and eliminate a large percentage of the problem overnight. So far they not even acknowledged that there is a problem.

O.P. asked for alternative explanations and people are trying to give these companies the benefit of the doubt. There are actually some much less generous explanations, such as that they are knowingly in the pay of forces adversarial to US interests.

How do you "take out" an attacker? They will just create more accounts. The accounts themselves don't have to cluster with each other. And let's say you do find a large set of accounts and ban them. New fake accounts are being constantly created and sold and used.

How did we tackle spam?

SPF, DKIM and DMARC?

I jumped down the rabbit hole and set up my own mail server about a year ago. With postfix configured to look at the above I get zero spam. Looking at logs, I do get lots of connection attempts from random IPs in China and such, but my postfix unilaterally rejects anyone that doesn't have good DNS settings.

spammers can and do trivially setup SPF, DKIM and DMARC

That's where various fingerprints (the sort of thing, incidentally, AI/ML should be pretty good at sorting out) should come in to play.

Spammers, ultimately, act like spammers, and bots like bots. Unlike legitimate accounts, which typically either a) follow a small number of related accounts or b) are fairly-well-known high-profile accounts (and yes, pseudonymous accounts may be high-profile), bots either need to act within specific networks or be very indiscriminate in whom they follow / link with. And they're tremendously active.

Whilst Google+ is not Facebook, Stone Temple Consulting ran an analysis of 500,000 profiles in 2015 to see what posting activity looked like.[1] I'm fairly familiar with the methods used as I'd pioneered most of them on a smaller, 50k sample.

Looking at public posts only (not private posts, and not comments), there are roughly 53,000 G+ users posting 100+ times monthly, and 106,000 posting 50+ times monthly.

From an independent analysis, I've estimated that Facebook traffic is, very roughly, 10x greater than G+ (as of August, 2015). So you might bump these values up by an order of magnitude to get Facebook-level scales of operation. Twitter is actually far closer to G+ in scale.[2]

That is, the search space for bots is roughly 500k - 1m accounts of major active players. And it should be possible, even with only computer-assisted human intervention, to make a pretty good cut through that space.

So the problem is not all that intractable.

________________________________

Notes:

1. https://www.stonetemple.com/real-numbers-for-the-activity-on... STC's Eric Enge credits me on that page.

2. https://www.reddit.com/r/dredmorbius/comments/3hp41w/trackin...

My belief is that they are still profiling their spammers, and are not acknowledging anything specific as a sort of 'poker face.'

>people are trying to give these companies the benefit of the doubt.

I think that's a good thing; we're all too susceptible to cynicism. Being charitable is perfectly fair (and ideal) when dealing with people who haven't demonstrated a propensity to deceive (or operating with malicious intent).

Perhaps they should try shadow-ban-plus. Banned accounts are only visible to, and can see a random subset of users, at random times. If a cluster member looks for other bots he may or may not find them because either he is banned or the target is banned, or both, or there is a glitch, and there is no way to tell.

It would be awesome if the New York Times shared the tools they've developed for this article to identify and analyze bots!

I usually hate those scroll-responsive animated web pages, but the scrolling illustrations and data visualizations in this article were particularly well done and pretty amazing. Not just pretty (the initial face sequence) and clever (the scrolling iPhone) but also actually useful and relevant (like the subscriber graph data visualizations expanding over time). I would love to know more about the tools they used to make those too.

A plea to NYT from a subscriber: Please share some of that great software you've developed, and publish it on your github account!!!

https://github.com/NYTimes

Investigations graphics editor here — as it happens a lot of the tools we use are open source! The interactive components were all built with Svelte (https://svelte.technology) and Rollup (https://rollupjs.org/guide/en).

We'd like to eventually open source some of the stuff we built to do the analysis as well, though it depends on time and priorities.

Thank you for those links, and your great work. If Twitter refuses to do the kind of investigation and analysis that you performed because they're making too much money from bots, then the free press and open source community needs to take up the slack!

Hey Rich! The graphics are on point for this article. I really liked the unfading faces at the start. Also, the times should put a tip jar on these kinds of articles, I'm not an avid reader to subscribe, but would want to appreciate any good articles I find.

On the graphics, it looks like the density changes for the rest of the dates whenever the identified bot families show up (creating the differently colored columns). When the identified bot families stop, the density changes back. Is that an artifact of how you're displaying it, or is it an indication that the followers before/after are also bots, but they've been mixed with bots created at other dates, where the bot families identified are just the low quality (non-mixed) bot families?

NYT has a history of hiring engineering rockstars. I believe creators of libraries like BackboneJS, UnderscoreJS, and D3 were at NYT when they had invented those amazing pieces of work.

In fact, the creator of Backbone and Underscore — Jeremy Ashkenas — worked on this article until he left the NYT last year (to join the creator of D3, Mike Bostock, at https://observablehq.com)

I think Bostock had already created D3 by the time the NYT hired him: https://www.reddit.com/r/dataisbeautiful/comments/3k3if4/hi_...

Your argument that NYT is better at finding bots only makes sense if you assume that NYT found all the bots that exist. A more likely explanation is that everyone who goes looking for bots uses a different methodology and finds a different subset of the bots that exist.

Thus NYT can find some bots that Twitter missed, especially by using labor intensive methods that don't scale to all of Twitter, but there are plenty that NYT missed too. Their methods aren't necessarily better than Twitter's, just different (and likely much more expensive). Note also that the NYT people don't have to care about their false positive rate and don't have to adapt their methods to adversaries since this is a one time analysis.

Think of it like cancer. Identifying cancer, saying “You have a growth”, is one problem is medicine. There are ways of doing this and the cancer is just fine being identified, it is none the wiser.

The moment you start to treat cancer is the moment you realize that identifying and treating the problem are leagues apart. Cancer mutates, cancer squirms, cancer goes incognito until the storm has blown over. By attempting to treat cancer, we’ve made it our adversary and it now will fight against our treatment. UV treatments cause mutations that the cancer might use to avoid further treatment, some drugs target specific causes of cancer which the cancer will then change to, and some treatments just kill the patient (Just as Twitter can’t just ban everyone). There is a balance that these large tech companies need to find just as a physician needs to determine the correct path.

I know this is a very optimistic view of the problem, but it aligns with my experience working, generally, in this space at similar scale.

For the longest time, spammers and fake accounts on Twitter were just a name with a long ass random number on the end. I guess that particular cancer hit a local optimum where Twitter was pumping its numbers aka misleading the shareholders.

4) It's not hard to identify bots with 95% accuracy, which is good enough for reporting, but you need 99.9% accuracy in order to delete their accounts or else too many genuine accounts will be wrongly terminated.

Super easy to solve: identify likely bots, suspend account, and let owner do something trivial that only a human can do. False positives solved but the cost of maintaining large amounts of bots are prohibitive. Also, high-profile non-bot acounts should be trivial to identify with near certainty.

They could join the rest of the internet and serve a Captcha to the suspicious accounts!

They already do this but instead of a CAPTCHA they require a SMS. I don’t know if they had the option to enter a CAPTCHA instead?

How would they know on scale what removed accounts were controlled by a human? Some people may just not want to return after the ban. Bots can probably create new accounts and complain too and a few complaining high profile twitter users that would filter out to media to be verified by journalists don't really make a good study.

Interventions programmes of almost any sort face this problem, and it's part of an ongoing monitoring and evaluation effort.

Ultimately: you monitor direct and indirect effects, offer an appeals and communications process, and try to sort out who is and is not participating in good faith.

It shouldn’t be too hard to get at networks. If you identify an account with suspect followers and then look at the followers common follows and follow dates. They can spread out the follows but still to get it done it any reasonable time the follow times are going to be pretty close.

There are actions other than deletion which can be undertaken.

You might target suspect accounts for increased validation -- Captcha or other elements, or simply logging them out more frequently. Changing programmatic elements such that they cannot be scripted as easily (this is one of the primary credible arguments I've seen against APIs for large public services).

(Craigslist, and several other services, have applied what's effectively a service-degredation level to accounts suspected of malicious activity. It's not a hard fail, and it's possible for a real human to get past the hassle, but it slows down attackers considerably.)

I mean prolly somewhere near 40/50% of Twitter's 330 Million MAU are fake, the company will drop 50% or more in value if they ever decide to delete those accounts, incentives are aligned for Twitter to encourage bots by making it easy as possible (not adding captchas, etc.) and not enforce policy

All relevant parties in FB, twitter, Yelp, TA are very much aware of fake accounts, fake review, and any other bot-related traffic. Yes, they do have teams dedicated only for that. So the real question should be - what actions are being taken to counteract social bots. This is not something any company would want to go public about. So basically - we don't know.

Your premise is centered around tech companies wanting to shut these things down. Is that actually true?

And that they want it to a degree that exceeds the cost of doing it, and of keeping up in that arms race. Chances are they don't.

> The New York Times can graph and identify bots, and other publications / bloggers have been able to identify networks of them.

Because it takes a lot of manpower to do so on a wide scale and almost always raises the question of fairness - the NYT (and others) focus on the egregious examples, but if Twitter would do the same (and kill the followers) their customers (the big celebs) would complain, and when they say "okay, bye Twitter", the normal users who want to follow them would disappear... thus reducing the popularity of Twitter for advertisers. The bots don't count as advertisement recipients, so Twitter doesn't have advertisers complaining about fraud from these fake followers.

In addition, the existing (moderation) manpower is focused on keeping the system being overrun by actual, human-powered Nazi and other troll accounts - fake followers are waaaay on the bottom of the priority list. They don't generate bad press headlines about shitstorms or harassment.

For (2), developers aren't the ones making the decision to deprioritize bot detection. The decision makers don't care because the ROI on detecting/removing bots is likely negative, whereas using those ML people to improve ad targeting and engagement is much higher.

Yeah exactly. All the devs are working on business prioritized projects. Removing bots is not a business priority. Nothing to do with how smart the devs are.

(2) except the blame falls on company leadership - developers are not in charge of funding teams.

Leadership funds teams that generate positive ROI. A successful bot-hunting team will likely decrease revenue... although kicking out bots makes the platform more sustainable.

It's not just social media. The same question kicks around retail sites like Amazon - why are external researchers better at identifying fraudulent reviews? Because review factories propping up shoddy Chinese imports are very profitable for Amazon. These days customer obsession is just a "nice to have".

This is not a technological challenge, but a management one. Why would a product built to expand social graphs delete nodes that further it's size? Not gonna happen. Ever.

> I find it hard to believe that people outside of the ecosystem of these companies have more skill, knowledge, or capabilities

The methods described in this article would be corporate espionage. I don't know what the legal implications would be, but ethically, I'm personally a bit frightened by the idea of a company as large and in possession of so much personal information as Twitter getting into the espionage business.

The object of a political botting campaign is to promote views that are complementary to ones' own political agenda and remove those that are not. With an automated system that removes discourse that corresponds to patterns used by bots, a social adversary would only have to adjust their bots to resemble the voices they want to silence, for a perfectly platform-sanctioned way to silence people they disagree with.

It's not in the company's interest to ban these bots but if it really is that easy to track, we should do it from a third party.

A Twitter/Facebook/whatever client plugin that flags or hides bots seems feasible.

It's even in a certain business interest to track bot traffic for more accurate click through tracking so perhaps there's even a market need for this.

The work of finding exploits to these algorithms probably requires less talent than the work of researching and implementing them.

> There's only a couple possible scenarios I can come up with for why this continues to occur:

Or....this has nothing to do with talent and more to do with market dynamics - they lack any monetary incentive to change.

Q: What do you get when you cross the Weekly World News with a Baleen whale? A: The American electorate.

4) The tech is yet to achieve acceptable level of accuracy.