Company withdrawing from Facebook as analytics show 80% of ad clicks from bots | Hacker News Reader

Company withdrawing from Facebook as analytics show 80% of ad clicks from bots

facebook.com

1,347 pointspetercooper356 comments

Copying the text just incase FB decides they don't like it

---

Hey everyone, we're going to be deleting our Facebook page in the next couple of weeks, but we wanted to explain why before we do. A couple months ago, when we were preparing to launch the new Limited Run, we started to experiment with Facebook ads. Unfortunately, while testing their ad system, we noticed some very strange things. Facebook was charging us for clicks, yet we could only verify about 20% of them actually showing up on our site. At first, we thought it was our analytics service. We tried signing up for a handful of other big name companies, and still, we couldn't verify more than 15-20% of clicks. So we did what any good developers would do. We built our own analytic software. Here's what we found: on about 80% of the clicks Facebook was charging us for, JavaScript wasn't on. And if the person clicking the ad doesn't have JavaScript, it's very difficult for an analytics service to verify the click. What's important here is that in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook. So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs. So we tried contacting Facebook about this. Unfortunately, they wouldn't reply. Do we know who the bots belong too? No. Are we accusing Facebook of using bots to drive up advertising revenue. No. Is it strange? Yes. But let's move on, because who the bots belong to isn't provable.

While we were testing Facebook ads, we were also trying to get Facebook to let us change our name, because we're not Limited Pressing anymore. We contacted them on many occasions about this. Finally, we got a call from someone at Facebook. They said they would allow us to change our name. NICE! But only if we agreed to spend $2000 or more in advertising a month. That's correct. Facebook was holding our name hostage. So we did what any good hardcore kids would do. We cursed that piece of shit out! Damn we were so pissed. We still are. This is why we need to delete this page and move away from Facebook. They're scumbags and we just don't have the patience for scumbags.

Thanks to everyone who has supported this page and liked our posts. We really appreciate it. If you'd like to follow us on Twitter, where we don't get shaken down, you can do so here: http://twitter.com/limitedrun

Happend to us too!! we first thought kissmetrics was wrong, so we built a logger that logged server side requests and a client side code that executes on load of the landing page, and we found that 70% of the server side hits had javascript disabled(or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%).

What user agent was reported by those hits? This is fascinating.

It was a few months ago and we've already deleted the db table, we are soon relaunching with a new campaign, so once I get the data again I'll post it here.

> (or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%).

That's actually an interesting proposal-- 100% of the time when I click on an ad, I didn't mean to and I try to close it before it has a chance to annoy me.

Probably doesn't make sense on a large scale, but it's something to consider.

I agree that some users might close the page or hit the "back" button before the $(document).ready() happens, but I can't believe that 70% of the people did that(taking into account that our ad was really targeted and highly relevant to the landing page).

and then we find out drunk or stupid people accidentally click stuff.

drunk or stupid people probably won't click 'back' that quick.

If that is true and 70-80 percent hit the back button before the js files load, it still says a lot about the quality of the clicks from the perspective of someone considering advertising on FB...

To be fair, that pretty much matches what I've heard about Facebook's click quality.

so the net result would be the same.

Someone's grandmother miss-clicking and then mashing back is about as useful to the advertiser as a bot. conversions are where it's at. Poor quality traffic / bots / it's all the same. the advertiser quits paying

70% is a ratio ! The effective number may be small at the same time.

And here is a screenshot: http://i.imgur.com/4hu66.png

Here's ascii of the screenshot in case they censor your picture of text.

http://pastebin.com/raw.php?i=HHeY5nP1

Here's the MD5 of your ascii version of the screenshot of the blog post just in case: 14e34998a7bc46174bccd981ad5f41f6

I acknowledge receipt of the MD5 of the ASCII of the screenshot of the blog. Please acknowledge receipt of acknowledgement.

I give you a hamburger.

Alert! The MD5 hash of the ASCII screenshot I computed does not match yours:

$ curl -s http://pastebin.com/raw.php?i=HHeY5nP1 | md5sum 7b69269bd0e25daa8e8b171e5d513011

waiting, when screenshot will say "Mark Zuckerberg" likes this :-)

What happened? Is reddit offline again or something?

There is (subjectively) a marked deterioration of comment quality on front-page discussions in the past few days, I feel. I have also noticed that HN articles are getting linked on reddit more frequently (I use both networks, but for highly disparate purposes - gotta get those funny .gif's somewhere). No data supporting this but I fell it's a trend and I'm not liking it.

Few days? Try since forever. As people increase the SNR gets larger. Even on reddit when it started, it was heavily tech focused, you'll find a comment that says it's going downhill. Hard to say what the numbers on HN are like growth wise in comparison to reddit (which has their subscriber numbers on display).

I don't understand why you would post that here. Can you explain?

I didn't write an own analytic program but can confirm that 80% of clicks from my Facebook ads don't show up on my GoSquared dashboard.

you're not alone. I've heard similar stories from other businesses that have tried to advertise on FB. At first I thought they just didn't track properly, but it seems it's a widespread problem.

We built a page logger. Any time a page was loaded, we'd keep track of it.

They reinvented the access log? By hand? Are people really that detached from how servers work?

This is besides the point.

Is it though? It seems they spent a lot of time throwing more and more javascript at the problem until they realized they couldn't figure out what was happening. If they had real logging from the beginning, it would have unconfused them much sooner.

They were taking orthogonal measurements to see if the pattern would appear over and over. Also it sounds like each successive measurement technique was at a lower level than the previous technique. Sounds pretty thorough. I'd love to see some actual data though.

How would you use the access log to determine if client-side javascript was being executed?

Check if it's even being fetched for a start. Assuming it's not inline.

Browser-cache my friend. You'll need to poll the server on page load (or even better in this case, after you load the ad)

For the page in question, count total hits in the server access log and subtract total hits reported by the Javascript analytics software.

Use a javascript function to poll the server for something.

thanks for coining the word "unconfused"

This could be a businessperson's interpretation of what happened.

Or they might have indeed reinvented it. Kids these days and their Heroku! Back in my day we'd process each HTTP request by hand. And we'd do it uphill both ways in the snow.

using awk.

Hey now, awk is simply lovely. It's ridiculous how popular it is to slight it these days.

How does 'using awk' count as a slight?

It was the proper add-on to the previous comment kids today don't learn awk. Becuase they have Perl, Ruby, and Python which are supersets of the unix command line tool set. Awk was the king in the years before Perl.

I never really learned to use cut, so I sitll use awk for one-liners the deal with columns in delimited-data text-files.

Sorry, I read it as a slight because it came right after "uphill both ways in the snow".

Which usually precedes "Get off my lawn!"

Anyway, if I want to take gratuitous pokes at a language, I usually choose XML; sometimes java.

Which, of course, I've just been informed is the "preferred" integration language for the project I'm presently working on. Sigh.

Indeed.

Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon.

Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures).

It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.

You had awk? Ha! Luxuries! Why when I was a young programmer we had to write the code in the snow with our pee, and a compiler was just a word for the pilot of the hovering dirigible that read the instructions and passed them to the ALU, which was another fellow with an abacus. They would wrap the re- sults around a rock, and drop it on my house when the program would exit. We had to walk uphill...

Using butterflies: http://xkcd.com/378/

With VB6 you can check IP Addresses.

Please guys. This is not Reddit.

And that is how to do a c-c-c-c-c-combo breaker on Hacker News.

Damn straight. Cheap jokes are not allowed.

I assumed this just meant they built something on top of access logs that told them what percentage of page loads also loaded JavaScript. I don't know about your webserver, but mine doesn't tell me this kind of thing out of the box.

Anyway, it's pretty irrelevant whether they added logging in their app (one line of code?) or enabled their webserver's built in logging.

Surprised no ones mentioned the noscript plugin.

Since the user is on facebook they'd have facebook unblocked, but noscript still blocking everything else.

How would you discern a noscript user from bot?

You wouldn't, but if the vast majority of your ad click visitors have javascript disabled, and only a tiny percentage of your other visitors do, what conclusion would you draw?

You don't, but it's very unlikely that there were enough noscript users to match the stats. From the post:

"...in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook."

It's a different marketing block.

For example, target something for the Reddit.com users and you'll see 90%+ block ads.

With facebook granular targetting you could very specifically end up with a target segment with a large proportion of noscript type plugin users. Without more details on their campaign I would not be so quick to place blame purely on bots.

But this is about users who click on ads on Facebook. The sort of users who have javascript disabled are going to self-select out of that group pretty strongly.

And even among the most technically literate groups, I'd be amazed if you saw 80% use of noscript. So many sites require javascript that it's easier to browse with it enabled.

Unless you're targeting noscript users there's no way they account for a significant portion of those 80% of clicks.

> How would you discern a noscript user from bot?

There are a few tricks to do that, described on the old ha.ckers.org weblog, or perhaps on the related sla.ckers.org forum. No idea if they still work, though ..

You could always see if you could trigger NoScript's built-in XSS or frame hijacking protection :) Bots will not notice, and users get a scary warning message and believe your ad is trying to hack them. Not good for your conversion rates, but a pretty sure way to tell them apart.

What you say is valid, however no-script users are a small minority of internet users. I doubt their numbers would tip the results in either direction.

Access logs that write continually from different processes or threads (like PHP) can cause System IO to reach critical proportions. In any non-trivial enterprise application it's completely reasonable to have a stat-logic-layer for recording events... The access log can still do it's own sorta thing - this is more true if you have more than 1 server.

Access logs can't record if javascript is enabled. There are other things I would be interested in if I were doing this too, I'd check the browser's TZ, check if flash was enabled, etc. and do it across sessions.

Depends how far they went profiling their visitors, it clearly says in the article they were tracking if js was enabled, so yes, you'd need to roll something or another by hand. GA relies solely on js for example.

easy: < script > (new Image()).src = document.location.href + "Hey, i have javascript < / script >

And then compare in the log.

genius

The way I understand this is that they want to log who accesses their pages on FaceBook's servers.

I am not familiar with how things work on FaceBook, but I would think that FaceBook users do not have access to the logs that FaceBook's servers write.

That escalated quickly!

And so it seems that Facebook could be proven to be the new groupon.

I would absolutely not be suprised if the hits were FB bots - however I'd expect they wouldn't be that directly stupid so as to have an Ad-bot net in house. Surely this is some service they outsource for plausible deniability.

Something they would have learned from their relationship handing all user data over to the NSA/CIA.

EDIT:

I'd like to make a prediction on how this could potentially play out:

1. FB ignores them. They have nothing to do with this issue and they know the bots are tainting traffic - but they can't/wont do anything about it because it boosts their revenues.

2. FB may or may not be involved in the ad-bots, but, they will drop their $ contingency on ad-rev for the name and payout these guys to shut them up. Their stock was improperly priced and taken a beating. They don't want this coming to light in a large way as it will have a negative affect on the perception of their business model. They hav to be kind of careful of the Barbara Streisand effect here too...

3. Many more people will reveal the same results with empirical testing and it will be revealed that FB earnings are 80% over inflated. Their stock will drop to the predicted real value of <$10... maybe hit that actual projected number of $7.00 - Zynga will BEG FB to purchase them as they are now a penny stock and they can't sustain.

So without ANY facts your conclusion is that Facebook is fraudulently deceiving advertisers.

And that they are doing this using experience from working with the NSA/CIA in order to cover up the fact that $800 million dollars of Facebook's yearly profits will disappear overnight once people realise the truth. And as such they are willing to pay out the ad-bot companies in order to hide this conspiracy.

Right.

>...without ANY facts...

What about the farking article? Where they proved that 80% of their clicks were from bots. Then several others confirm they ahve the same experience?

What about those facts?

I also speculated on what FB is doing. Them giving user data to the NSA/CIA - that is a supposed fact, sure, but I am not the only person who believes this.

Am I extremely distrustful of Facebook? Of course! Maybe you have not followed their track record or the character of their founder.

If you are into conspiracy theories then how about maybe this is Google directing a massive bot net to make facebook ads useless?

> I also speculated on what FB is doing. Them giving user data to the NSA/CIA - that is a supposed fact, sure, but I am not the only person who believes this.

Great argument! A lot of people believe it, so it must be true! Are you for real?!

It is not my responsibility to educate you on modern internet history. This has been argued here on HN prior; Do you recall the backdoors that AT&T put at 600 Folsom, in SF?

And the backdoors in Cisco's equipment that are a requirement by the federal government?

Maybe you thin this shit is "conspiracy theory" and if you do, you're simply a naive fool.

Look around you at what governments are doing. The NSA has records on everything you do online. They had a system in 2005 which could trace communications between users to 6 degrees, automatically.

It's odd to me that your posts aren't more downvoted than they are.

You think I make this shit up? There are some really naive and stupid people on HN that have zero grasp on the history of intelligence agencies.

So, this is why people don't take you seriously.

I know a couple of guys like you in my local activist community, who take a very hostile "I know the truth and you're all fools!" attitude, despite the fact that their audience is mostly very sympathetic to most of their assertions. We know spying on the Internet takes place; HN is full of cantankerous old Internet geeks who've seen, first-hand, plenty of examples of the state (whatever state you may choose, as it happens all over the world) behaving unethically on the Internet, spying on people and punishing people for things that shouldn't be crimes.

But, your paranoid approach is counter-productive. You might as well be working for the people you claim to be afraid of, for all the good you do (negative good; you're convincing people that the folks who believe the government is spying are all paranoid nutjobs who scream at anyone who has the gall to mention other possible explanations).

So, let's review:

1. Just because spying has taken place, and is currently taking place, and may even have the complicity of facebook in that spying, it does not mean that facebook is running a botnet to steal advertiser dollars. The simplest explanation is that facebook looks the other way while others run the botnets. facebook wins (a lot, as long as most advertisers don't know it's happening), botnet owner wins (a little), and the advertiser loses. But, there are other plausible explanations, including incompetence.

2. When you paint things in a "Either you accept my theory in its entirety, or you're all idiots", you force people to choose a side. Nobody wants to be on the same side as an asshole, so you force them to choose the other side. You make people who may even agree with you (to a greater or lesser degree) to begin to formulate plausible reasons for why you're wrong about the crazier stuff you're spouting...further convincing themselves that you're entirely wrong. The best you can hope for is people ignore you and don't have the chance to be inoculated against your ideas; having you as their first exposure to these concepts guarantees they will be less likely to believe them in the future, even if they come from a more credible source. Humans are funny creatures.

Thus, I would point out that there are some really naive and stupid people on HN that have zero grasp of effective argument, persuasion, and even basic logic.

You might be well-served by reading about non-violent communication: http://en.wikipedia.org/wiki/Nonviolent_Communication

Edit: Removed the word "schizophrenic" as it was an insensitive use of the term, and was counter-productive to making my point.

>But, your paranoid schizophrenic approach is counter-productive.

Really, now I am a paranoid schizophrenic?

Just because I make claims which are readily confirmed and were completely available in the media - even the EFF filed suit on the AT&T events...

Yet, for some reason, it is my responsibility to educate everyone every single time someone new comes along who hasn't been following these things closely.

Now I am a paranoid schizophrenic?

Also, its a strawman to focus on my tone, rather than content. You're trying really hard to be overly pedantic and, frankly, an asshole.

Thanks for the link on communication.

He's not arguing with you, he can't be using a strawman. He's just pointing out that other people will be distracted by your tone, making them ignore your (mostly correct) message.

Thanks. In that case, I will try to be less emotional about this issue.

When I was talking about China hacking lockheed as far back as 2005, everyone said I was nuts! (I freaking worked at lockheed!)

There needs to be a better way to log and track this stuff so that we can point people to some sort of Tyranny Wiki.

What I am witnessing is someone trying to inform people, while effectively censoring himself by delivering in such a way as impedes its delivery.

You obviously have a vested interest in seeing your "message" be received, so stop ignoring what people are telling you about your tone and change it. Figure out what it is that people do listen to, or else you have no one but yourself to blame when they take issue with your tone.

To wit: Your tone is defensive. Pejoratives and cursing undermine your expression. What you are effectively telling people is that your thoughts are not important enough to merit self-restraint. Detailed explanations following assertions are also helpful.

FWIW, I work in the performance marketing industry and, IMO, the most complicity facebook could be said to have in this issue is in not placing a sufficiently high priority on preventing bots from clicking on their ads. Even if it is a sizable project there, they represent such a large target that the difficulty of the job becomes much greater. I see no salacious story here, other than a company found itself unable to optimize a campaign into profitability, which I think says more about them than facebook. Ho hum, find a different traffic source and move on.

Thanks - Ill take the constructive criticism on my tone.

However, I will point out that with respect to your comment on FB's complicit actions due to the daunting nature of the problem, this does not take into account their other actions of a 24K ransom on the domain.

Everyone can argue in any direction they want - but neither me nor anyone else is really going to know until we get further down this path...

Hey, calling people "schizophrenic" is not cool. Just sayin'.

You're right. It was an unproductive method of describing the behavior I was seeing (and was too easy to interpret as saying the person is schizophrenic rather than exhibiting behavior I associate with schizophrenia). It was also insensitive to schizophrenics.

In my defense, schizophrenia runs in my family, and I'm very familiar with it...I don't think of it as an insult. But that's a local custom in my family that I shouldn't think follows in the rest of the world.

When describing behavior and not as a slur I don't see any problem.

And there are some that actually know what they're talking about in terms if intelligence agencies but choose not to indulge in idle conspiracy hypothesizing. The real operations aren't something that the people with actual knowledge would ever discuss. Those that do claim conspiracies are often ill informed or just have read too many spy novels.

I think problem is more deep, I do not want to make other Conspiracy Theory but here some story from other huge Russian social network (vk 150m. users). There is one small, not that big group (like fb groups), users made agreement, that no one will post next day (something like honeypot), and those who post would be considered as bot and baned forever. So next day couple or so bots generated comments encouraging others do not go on strike against some political party [0].

I do not know is it applies to FB, but those who have like 100k bots have some power too.

[0] http://imgur.com/Vvkbx screenshot in russian

Edit: spelling Edit2: proper link

All this happening as employees in FB are itching to cash out their options in a few months.

We need to see some real numbers before making these kind of predictions. What was the volume of clicks that OP couldn't detect? 10, 1K, 1M? It makes a big difference.

Also given FB users' aversion to ads this whole issue could be explained by users just clicking the back button. Ad supported sites have trained users to do three things very well: install adblock, habitually ignore ads, quickly click away from any interstitial or unexpected ads like on Hulu.

Another note on the "back button": I recently discovered the awesome power of "three-finger swipe back button" on Macbook trackpad. So it's possible users can "click" back even faster than before. I can and before the swipe I was well-trained in the art of "CMD back arrow" :-)

Totally agree. However, I wanted to put my thoughts on this down... Surely we need more info and time to see what the actual truth is - but I just do not believe that FB is in any way some altruistic innocent/neutral entity.

What user data stored on FB servers could be useful to intelligence agencies of that level? People brag about crimes they committed, IP addresses are logged and people may "like" jihad but criminals targeted by the NSA are surely not at that level of stupidity.

Also, FB may not have much of a choice to comply with those agencies, or find themselves pissing off a lot of high level people by refusing to comply with government agencies. But outsourcing / tolerating a network of bots to vastly boost revenue is probably considered fraud, no?

> What user data stored on FB servers could be useful to intelligence agencies of that level? People brag about crimes they committed, IP addresses are logged and people may "like" jihad but criminals targeted by the NSA are surely not at that level of stupidity.

It's not what people brag about, it's the social graph itself that's useful. All the other things like photo-tagging are extra candy.

More info at http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/al... and http://nplusonemag.com/leave-your-cellphone-at-home . Sorry it's late out here, or I'd copy/paste the relevant quotes for you--look for the bits about data mining and machine learning, especially in the 2nd article, everything is a signal and they do have the computing power to process it all, apply Bayesian belief nets, algorithms can pull intelligence information about you from the local shape of your social graph that you don't even realize can be inferred from it (humans are not very good at reading complex info from big graphs much further than simple "friend-of-a-friend" relationships).

>in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled //

I use noscript. When I enable it for visits it's usually by enabling the domain itself and leaving third-party scripts disabled. Depending on your area this could be significant.

Also the possibility of blocking of the scripts by other elements not loading - for example if the browser can't parallelise the requests for some reason and a preceding request can't be handled then the page might be displaying well before the script is loaded, especially if it's loaded at the bottom of the markup.

I think unless you run a forum for javascript virus injection techniques, you're not going to see 80% visitors with disabled javascript.

Very unlikely to have been bots from FB, they're not yet quite that desperate. They would be 90% sure of getting busted, and it would be a huge deal breaker for a lot of their clients. A suicide move.

The more plausible reality is that there are just a lot of bots crawling FB fan pages, poorly programmed bots that will just follow every single link they come across. That's very much in line with my personal experience of FB fan pages.

But FB is certainly to blame for not detecting these as bots, and putting ads on the pages they serve them. Hopefully they'll take care of this issue soon.

In my mind having an in-house bot network and being totally ignorant that you're charging for bot traffic via ads amount to the same thing.

This is very, very bad.

It does not amount to the same thing at all. Keep some perspective.

It does amount to the same thing.

If I am writing a mechanism to charge for ads based on traffic, one of the first things to QA/write test cases for is fraudulent traffic.

The leap here is from facebook being remotely competent (which is likely), and willfully ignoring checks on certain types of traffic because it benefits them financially.

My assertion is that this is the same class of bad behavior as running your own bots. Note the distinction in 'willful' ignorance.

How? In house bot network implies active fraudulent charging, while being unaware of bot clicks is simply neglect or ignorance.

Negligence

The distinction between ignorance and intent does not compute.

Yes, we dont need a conspiracy to explain stupidity. The stupidity is the conspiracy.

People arent ignorant, because they are handcapped or something. They are ignorant, because thats the behaviour and excuse we reward. Just be an idiot, dont put in the intellectual effort, and you will be allowed to speak lies and screw every one and every thing.

From a corporate point of view, its the golden ticket. Plausible deniability. We didnt know that our insurance helped finance weapon exports to dictators. We didnt know. We didnt know. Sorry this, sorry that. We're keeping the money though.

I dont think that provable intent should have legal meaning. Because all neglect is intentional, as is ignorance in a world, where every one has the choice to be well informed.

>"Hopefully they'll take care of this issue soon."

Much like the multiple account issue, it's beneficial for Facebook to turn a blind eye. That is, unless it escalates into a PR issue.

First page in HN, for some people this is considered a PR issue :)

HN's opinion of anything is not remotely significant to FB's PR, really.

These bots, if they do exist, wouldn't be operated by Facebook. It's a market-based ad bidding system. If 80% of clicks are worthless, clicks will be worth 80% less, and advertisers will bid accordingly. Facebook won't make anything in the long run by doing this, and there's clearly risk of getting caught doing something illegal.

If there is a bot, it's likely one of their competitors, though even that seems unlikely. One must be logged into FB to see ads, and I'm positive Facebook has some click fraud detection. They'd almost have to notice a number of accounts clicking the same ads over and over. I guess maybe if Facebook's CAPTCHA is weak enough and a bot was creating an account, setting up the proper interests to see the ad, clicking it, then repeating... Seems unlikely though.

Also none of this explains why they're taking down their Facebook page. One does not need to buy ads to have a Facebook page. You can even have it automatically update from Twitter. They could leave it up there and get free publicity from it and simply cancel their ads.

This whole thing seems like a sham for attention to me.

> Also none of this explains why they're taking down their Facebook page.

I think it pretty clearly explains why they're taking down their Facebook page. Just because you don't agree with them doesn't mean it's not explained. They're shutting down their Facebook page because (according to them) it's sitting around with outdated branding but they can't change the name unless they spend $2000/mo. Also because their user experience with Facebook has been crappy, and they don't want to have anything to do with them anymore. You know. Principle.

Could be a sham for attention, but the article pretty clearly explains why they're shutting the page down.

Couldn't they just make another page with the correct name?

They could, but a) that's a pain in the ass, they'd have to build all the content out from scratch and b) they'd lose a ton of their Likes in the transition.

The name on their page is wrong, and Facebook refuses to change it for them unless they buy $2000 worth of advertising. That's pretty scummy, and I think perfectly justifies their decision to remove their FB page entirely.

My startup is essentially an advertising aggregator (pooling traffic from a variety of publishers and routing it to advertisers) and dealing with things like bot detection is a HUGE chunk of what we work on, technology-wise. Let me try and give you an idea of how deep the rabbit hole can go.

- Okay, you want to detect bots. Well, "good" bots usually have a user agent string like, "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)." So, let's just block those.

- Wait, what are "those?" There is no normalized way of a user agent saying, "if this flag is set, I'm a bot." You literally would have to substring match on user agent.

- Okay, let's just substring match on the word "bot." Wait, then you miss user agents like, "FeedBurner/1.0 (http://www.FeedBurner.com)." Obviously some sort of FeedBurner bot, but it doesn't have "bot" or "crawler" or "spider" or any other term in there.

- How about we just make a "blacklist" of these known bots, look up every user agent, and compare against the blacklist? So now every single request to your site has to do a substring match against every single term in this list. Depending on your site's implementation, this is probably not trivial to do without taking some sort of performance hit.

- Also, you haven't even addressed the fact that user agents are specified by the clients, so its trivial to make a bot that identifies itself as "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1." No blacklist is going to catch that guy.

- Okay, let's use something else to flag bot vs. non-bot. Say, let's see if the client can execute Javascript. If so, let's log information about those that can't execute Javascript, and then built some sort of system that analyzes those clients and finds trends (for example, if they originate from a certain IP range).

- This is smarter than just matching substrings, but this means you may not catch bots until after the fact. So if you have any sort of business where people pay you per click, and they expect those clicks not to be bots, then you need some way to say, "okay, I think I sent you 100 clicks, but let me check if they were all legit, so don't take this number as holy until 24 hours have passed." This is one of the reasons why products like Google AdWords don't have real-time reporting.

- And then when you get successful enough, someone is going to target your site with a a very advanced bot that CAN seem like a legit user in most cases (ie. it can run Javascript, answer CAPTCHAs), and spam-click the shit out of your site, and you're going to have a customer that's on the hook to you for thousands of dollars even though you didn't send them a single legit user. This will cause them to TOTALLY FREAK THE FUCK OUT about this and if you aren't used to handling customers FREAKING THE FUCK OUT, you are going to have a business and technical mess on your hands. You will have a business mess because it will be very easy to conclude you did this maliciously, and you're now one Hacker News post away from having a customer run your name through the mud and for the next several months, 7 out of the top 10 results on any Google search for your company's name will be that post and related ones. And you'll have a technical mess because your system is probably based on, you know, people actually paying you what you think they should, and if you have no concept of "issuing a credit" or "reverting what happened," then get ready for some late nights.

I'm seriously only scratching the surface here. That being said, I'm not saying, "this is a hard problem, cut Facebook some slack." If they're indeed letting in this volume of non-legit traffic, for a company with their resources, there is pretty much no excuse.

Even if you don't have the talent to preemptively flag and invalidate bot traffic, you can still invest in the resources to have a good customer experience and someone that can pick up a phone and say, "yeah, please don't worry about those 50,000 clicks it looks like we sent you, it's going to take us awhile but we'll make sure you don't have to pay that and we'll do everything we can to prevent this from happening again." In my opinion this is Facebook's critical mistake. You can have infallible technology, or you can have a decent customer service experience. Not having either, unfortunately, leads to experiences exactly like what the OP had.

I'm in another field then advertising, but less then 5% of transactions require special handling, but they represent a significant investment in error handling, procedures, accounting and reversal mechanisms. Even then we are still manually dealing with things that represent 0.00001% of activity. Accounting correctly is always a deep rabbit hole, it comes down to how much you plan to right off inside your margins. :) Thanks for writing up this inside perspective of running this type of business.

And then there are the "wet bots" - Amazon Turks and overseas humans (lowly) paid to use a regular browser and PC and manually click on stuff for whatever purpose (rig internet voting, drive fake advertising charges, steal data, skew hit counters - whatever).

Is an impression from a mechanical turk not a true impression? Its a real person, after all. If the premise of marketing and advertising work due to mere exposure, then intent of the view shouldn't be measured.

What you are getting at however is very important to ad-driven internet business models. You make the empathetical deduction that if you are there for one purpose, you are impervious to advertising.

Now, if you are still with me after that statement, then there are larger impications of online advertising than are actually being measured. For instance, if the intent of the view mattered, would it be worth more? I think Google has answered that question rather clearly.

Measuring "why someone is visiting a URL" (and thus generating an ad-view) is a larger technical problem than could be accomplished with some fancy algorithms. That is, unless the service knew all of your browsing history...

In any case, you raise a good point about the intent of the page-view and how that should be "credited" as advertising. To do this however is to challenge existing 'fundamental' theories.

The problem with mechanical turk impressions/clicks is typically the country of origin, since they usually involve paying people in third-world countries a fraction of a cent which is still worth some nominal amount to them. Say you are an advertiser who runs an ecommerce site that only ships within the US. You don't want to pay for impressions/clicks from people in India because you can't actually sell to them, even if they are "real" people and not a script.

Also, there is "brand marketing" and "performance marketing." I'm not sure they have classic definitions, but generally brand marketing is based on "mindshare" and tough to quantify. This is basically any TV commercial -- Budweiser doesn't expect you to jump off your couch the second you see a Bud Light commercial, but maybe a week later when you're in the supermarket, you spot that new brand of beer and recall the commercial with Jay-Z rapping and grinding with some models, and think, "eh, maybe I'll try Bud Light Platinum, I will purchase this and consume it later because I like the idea of rapping with models," even at a subconscious level.

Performance marketing is all about ROI. You spend $100 marketing dollars, and you want to make at least $X back. Typically $X = 100 in this example, although in some cases you may want to adjust profit margins or even run at negative profit (for example, if you want to maximize revenue even at an unprofitable rate). Almost any advertiser on the internet has a performance marketing mentality, mainly because it's much easier to actually measure performance. This is why advertisers like the OP are frustrated with Facebook, because paying for bot traffic makes it extremely hard to hit their performance marketing goals.

You are right though, in that this is a huge gray area. To give two extremes, let's say you run an ecommerce site but you'd also like to monetize with ads. So you reach a deal with a shopping comparison site to display their products on your pages in your footer. If you buy AdWords to your site, and those visitors from AdWords scroll down and click on the shopping comparison ads, they are probably fine with paying you for that traffic, although you are probably paying something like $2/click. If you make a page with JUST the shopping comparison ads, and buy traffic from someone who basically infects computers with trojans and overwrites their Google search results to be your site (say at $0.01 per click), then you'll get clicks on the shopping comparison ads and make lots of money, except the shopping comparison site will realize these clicks don't lead to any actual sales for their merchants (since this is extremely poor quality traffic since all the users that end up there have basically no intent) and will kick you off. Somewhere in between adveritsing for "first page Google AdWords" traffic and "trojan/botnet" traffic is basically every other form of internet advertising, and how gray that area is, is typically completely defined by your business partners.

It's at best a gamed impression.

Advertising and marketing are generally very intentionally tailored. You're not trying to reach any human, you're trying to reach a human in your target demographic.

Skewing results (and wasting resources) by paying people who are entirely outside your marketing effort to click through links (or take other similar resource-costing / data-skewing actions) is not the desired result.

All that is really no excuse for not developing a foolproof system. And if you know that you don't have a great system to catch fraud, then you should simply leave some money on the table.

See, the essence here is not to come up with a great algorithm. This is not a programming contest. But to create a system where everyone (publisher, advertiser, ad-network and customer) wins.

In this very early stage, the success of Facebook's Ad network should be measured only by one metric - ROI of Advertisers. And that number has to be consistently better than Google's.

> And that number has to be consistently better than Google's.

Why's that? This isn't a zero sum game. You don't advertise only on Google or only on Facebook; as much as your budget allows, you advertise on every channel you can achieve a positive ROI on.

For any advertiser looking for a direct response, like a signup or purchase, I doubt anything Facebook can do would make the ROI better than advertising on search. No amount of demographic and interest targeting will make advertising to people as effective as advertising to intent. You can pay Google to send you people while they're in the process of looking for your product; Facebook doesn't do that.

Who is the "you" that your post above mentions?

The advertiser.

ROI to the advertiser is a key metric. Smart marketers compare the LTV or Life time value of customers that come from specific campaigns. ROI metric does not have to be better than Google for an ad campaign on Facebook or Microsoft Adcenter to be profitable. I have worked with clients to use web analytics and cohort analysis to optimize both quality (higher conversion, better LTV ) and quantity of leads. Sometimes Microsoft Adcenter traffic converts better but does have sufficient quantity compared to Google Adwords. Also different ad platforms have different levels or competition as well as different levels of fraud detection and click quality. I am surprised that Facebook does not detect and suspend obviously spammy user accounts whether that account is bot driven or outsourced to humans overseas.Google and Facebook have financial incentives to not filter out all low quality ad traffic because many advertisers don't have expertise or time to closely examine traffic sources as the company mentioned in this news story.

A combination of tech expertise and marketing savvy are needed to reduce this problem. More analytic tools are available for Google Adwords than are available for Facebook ads.

Thank you for introducing me to the term "wet bots"!

this is a great use case for bayesean filtering, and I bet you could even hook up an existing bayes engine to this problem space, if not just write one up using common recipes - I've done great things with the one described in "Programming Collective Intelligence" (http://shop.oreilly.com/product/9780596529321.do). Things like "googlebot", "certain IP range", "was observed clicking 100 links in < 1 minute", and "didn't have a referrer" all go into the hopper of observable "features", which are all aggregated to produce a corpus of bot/non bot server hits. run it over a gig of log files to train it and you'll have a decent system which you can tune to favor false negatives vs. false positives.

I'm sure facebook engineers could come up with something way more sophisticated.

Well yeah, getting a training data set, and then having your algo not be poisoned, seems like it is a challenge all around!

The algo can't be poisoned, if I understand the term correctly, if you only train it manually against datasets you trust.

Each iteration of training that produces improved behavior can then be used to generate a new, more comprehensive training set. When I wrote a bayes engine to classify content, my first training set was based on a corpus that I produced entirely heuristically (not using bayes). I manually tuned it (about 500 pieces of content) and from that produced a new training set of about 5000 content items.

Eyeballing it for a couple of months initially, manually retraining periodically, spot checking daily as well as watching overall stats daily for any anomalous changes in observed bot/non bot ratios will work wonders.

These are good info. In my past life, I oversaw the web monetization pipeline as part of my job. Click fault detection was part of it. It is a hard problem and there's no 100% foolproof solution.

As in your case, it's nearly impossible to detect click fault real time as the clicks streaming in. We batch-processed the clicks throughout the day and do daily billing settlement at the end of the day. It's easier to detect click fault over a period of time. Obvious bots are filtered out, like the search engine crawlers.

For click fault, it's impossible to distinguish one-off click as a user click or fault click. We tried to catch repeated offenders. we looked for duplicate clicks (same ip, same agent, other client attributes) against certain targets (same product, product categories, merchants, etc) over a period of times (1-minute, 5-minute, 30-minute, hours, etc). We also built historic click stats on the targets so we can check the average click pattern against historic stats (yesterday, week ago, month ago, year ago).

We ran quick checks every 1 minute and 5 minutes to detect unusual click patterns. These were mainly for detecting DOS attack against our site or against certain products. We actually had a realtime reaction mechanism to deal with it: slow down the requests, throw up CAPTCHA, redirect to a generic page, or block ip at routers.

Every couple hours, we batch processed the clicks so far and ran the exhausting checks to flag suspicious clicks. At the end of the day, definite fault clicks were thrown out, moderate suspicious clicks were grouped and reviewed by human. It's a fine balance to flag less suspicious clicks since there's money involved.

The sophisticate click fault I've seen involved a distributed botnet clicking from hundreds of IP all over the world doing slow clicks during the day. We found out because it's targeting a product category of a merchant and causing unusual click stats compared to historic pattern.

We've found that identifying sources of traffic and patterns of usage is superior to simple user-agent detection, particularly as, since you've noted, user-agent spoofing is trivial, and/or some bot traffic is driven through user-based tools (say, script-driven MSIE bots).

Instead of that, we'll watch for patterns of use in which high volumes of traffic come from unrecognized non-end-user network space.

"Unrecognized" means that it's not one of your usual search vendors: Google, Yahoo, Bing, Baidu, Yandex, Bender, AddThis, Facebook, etc.

"Non-end-user network space" means a netblock which is clearly not a residential or commercial/business networking provider.

For this you'll mostly want to eyeball, though when we saw very high volumes of search coming at us from ovh.net (a French low-cost hosting provider), it was pretty obvious what the problem was.

Using an IP to ASN/CIDR lookup (such as the Routeviews project: http://www.routeviews.org/) and a local ASN -> name lookup. One source of this is http://bgp.potaroo.net/cidr/autnums.html

One as-yet-unmentioned technique is to adjust a bot score by taking an OS fingerprint, and comparing that to the listed user agent.

It's not perfect for a variety of reasons, but I found it to be a useful input for a similar bot detection problem. That said, this was a long time ago so I'd need to re-run some experiments to see if the hypothesis remained valid.

For those not familiar with OS fingerprinting, it's a method of looking at the details of a connection and determining which operating system is most likely to have created packets with those options and flags. In BSD, it's built into pf.

Interesting. Any Linux equivalents or Java classes which offer similar capabilities (we're using a Java-based application server / webserver).

The linux equivalent is p0f (http://freecode.com/projects/p0f)

I can't speak to Java classes, but they'd need to be dealing with raw packets which isn't an option in a typical stack.

It works off of checking the SYN packet [0], which happens at the transport layer [1]. Once it gets to the application layer the information is not available unless the transport layer stores the information and provides a method to query a given connection.

[0] http://www.openbsd.org/faq/pf/filter.html#osfp

[1] http://en.wikipedia.org/wiki/OSI_model

I suspected something like this, but was hoping enough of the signature might survive for a daemon (or Java class acting as one) to take a peek.

On the packet level ... hm that's pretty clever, definitely not trivial to fake, either.

Just that if your bot runs from, say, Windows 7 and it spoofs an IE8 user-agent header, or even runs by directly automating IE itself, how do you detect it then? Both of those scenarios are not unlikely at all.

If the bot spoofs a user-agent that is viable for the OS, then you need to rely on other signals. It only works as a component of a defense in depth strategy.

could you just add a mouse-over event to each ad that sets an isHuman flag?

IIRC, years ago, Google changed one parameter of it's AdSense click url based on the number of times the user hovered the ad.

That would break screenreaders, some touchscreen mobile devices, and probably other things I'm not thinking of.

The bot could just fire the mouseover event.

Would it be useful to have a kind of open source database of "trusted" bots, just to get past that hurdle? I do realize there are many more issues apart from declared user agents.

I understand that it is challenging for a third party like you to identify the bots. But Facebook has more than enough signal on each user clicking on an ad (friends, posts, location, 'non ad-click activity' to 'ad-click activity' ratio) to determine if that user is a bot. Looks like they are choosing to not do anything with this information. "FB is turning a blind eye to keep their revenue" seems to stand up to Occam's Razor.

Couldn't agree more. It should be almost trivial for them to track down bots and flag bad accounts. I definitely wouldn't classify it as one of their harder problems.

Not saying that's not the case, but you have no data or knowledge to back any of that up. I wouldn't assume a problem is easy or hard until I've got sufficient info on it.

Too often I've heard "how could XYZ not have done this, fixed that?" Then have that same party sign on board to fix this "easy" problem and get themselves in a world of hurt.

data would be * IP addresses, one can assume a bot would only have a set of addresses they could use, barring botnets. * request patterns, ie: did the bot request css/js, etc * request timeframes * UA strings

Sure, its a big data problem, but I can imagine that Facebook has solved these types of scenarios many times over.

What if you start a new Amazon EC2 spot instance (netting you a new IP address), start up Chromium in headless mode (say, using Xvfb), navigate to the website of choice, use mouse automation to start clicking around, click the ad, spend 5 minutes clicking around in a semi-choreographed pattern on the advertisee's website, and then shut down the instance -- only to repeat?

Moreover, Amazon is always buying new IP subnets.

The user from that new IP won't have any real human like history - photos shared and commented on over time etc.

I dont count clicks from any amazonaws or ec2 hostnames on my site.

It sounds like you don't need to go to that much hassle currently, but even that rigmarole is simple enough to combat. The user account should be real, the usage real (comments, photos, messages back and forth) and the friends also real. False positive spam Ids are OK, that will lower your revenue but won't constitute fraud with your customers. Put up a test for uses you think are spamming, the test they already do of identifying photos of your friends would be a good one.

Large numbers of real looking fake accounts should be hard to keep up.

But why would you go through all that to click on ads? If I click on an ad for "Some Record Company" how does that make me money?

It doesn't always need to make you money, sometimes you might just want it to cost your competitor money.

Then pay Amazon for a list of their EC2 IPs, or obtain that information from a public source (i.e. RIPE, ARIN).

Well I do have at least some data to back that up. I know that they are a walled garden that keeps a ton of user data for every account (even the subset represented in Open Graph is significant). And it's with this knowledge that I make my conclusion.

* Accounts subsisting of an unusual proportion of ad clicking activity compared to their other activities are likely bots.

* New accounts with no history but high ad clicks are likely bots.

Before a click is counted, check the source account for it's human rank (activity-to-clicks). If it's below a certain threshold (tuned over time) than don't bill your customer for it.

The algo's to determine human rank can start fairly simple and become more complicated and accurate over time.

In terms of their current sophistication I would definitely call this an easy problem from a tech standpoint. Scaling to millions of concurrent users? Hard. Attracting and keeping millions of users? Hard.

Compare it to google's problem filtering out automatically generated splogs, though - it took them years to make progress against that.

Real human social activity is a lot more random than real informational content in blogs - so fb's problem seems harder.

Also, facebook banning accounts is a much stronger action than google just lowering the pagerank of suspected splogs.

So overall I think fb's problem is harder than a currently google-unsolved problem.

Facebook has total control of the users very action since joining Facebook. Much easier than coming across a random HTMl page and deciding on if it's real (whatever real means, YouTube comments are real but are very low signal but a database that runs itself could provide lots of signal).

He never said to ban the suspicious accounts, just don't charge companies for their clicks.

The detection part aside, it's a simple fix. I just wonder why FB hasn't already. (Is it because it'd hurt their revenue when they need it the most?)

But in the same respect, the fact that real human social activity is much more random is a reason why this should be easier. Bots are inherently pattern based, and while there are bots that could work to maintain fake interactions in order to pass a wide-sweep analysis of low-socializing accounts with a high ad response rate, it would do quite a bit to take a first attack on some of the more obvious bots, which are the ones that many of the people who had these problems noticed.

Now of course I'm simplifying this to a nearly absurd degree, but I do think there are reasons for Facebook to want to ignore these issues and that even a minimal step up in bot-prevention could go a long way when it comes to their advertisers.

I suppose a naive bot would not be "random" enough, and would obviously not be human.

But on the other hand, since valid human behavior includes much more senseless, random, and repetitive actions (compared to blogging), a well-done bot would be harder to detect.

Oh, absolutely! I have no doubts that Facebook will always be a step behind the bots, and I don't think anyone feels or even expects otherwise. That's just part of the social networking climate right now, and all major (and even many minor) social networking sites have to deal with it. However, in many cases that were pointed out in both the article and the comments here, these people weren't having their pages liked and ads clicked through by advanced bots, but instead by obviously shill accounts. Now of course there is always the option that these are legitimate accounts with weird use patterns, but regardless there are methods of doing at least a significant sweep and remove some of the more obvious bots or at least offering them a chance of legitimacy confirmation (even something similar to CAPTCHAs for non-standard account usage that messes with advertisers campaigns).

Like I said, I absolutely believe there will be bots that are incredibly sophisticated that might be difficult to detect, but I still think more could be done to stop the less sophisticated ones.

I come down on the side that it shouldn't be so difficult for Facebook to at least spot some of the 4 bots per human.

However, even if you assume it is a hard problem - thats more reason that they should have listened (and responded!!) when a company identified this problem with some hard data to back it up.

The bot masters then just have to figure out roughly what your activity/ad clicks rate is. They can do this over time by looking at statistics.

Then they make some bundled crapware browser plugin that gets installed by legit facebook users that will click ads at that interval + some random time. Possibly do the ad clicks in background so that the user does not know.

How exactly are they going to know they're being flagged as bots? When all you're doing is discounting those bot clicks to the advertiser it becomes far more difficult to test your limits as a bot (you would need to be a paying advertiser, just to analyze the human to bot threshold). It's similar to hell-banning, in that the user's generally unaware they've been hell-banned.

Take a minute to think about the effort a bot account has to go to in order to even come close to gaming a system that tracks their behaviour.

Let's assume that in order to be deemed human you need at least a few characteristics like "account older than 1 day", "friends from other non-suspicious accounts", "a photo or at least comments on a friends photos", and a "certain amount of click activity".

Now let's take that criteria and make it difficult for a bot to even know if they're being flagged as a bot where the only indication is a reduced bill at the end of the day / week / month to the advertiser. So bot makers would have to pay to test and would at best receive delayed feedback on their success.

Isn't this starting to seem extremely difficult, and generally unscalable from a bot perspective? It doesn't have to be impossible, there just needs to be lower hanging fruit, of which there are plenty should FB implement a system like this.

And should this fairly simple solution still have a few holes that a very elite few bot coders are able to game, you've still got tons of data and time to use to tune your algos.

It just doesn't make sense that one of their advertiser's is being charged for 80% bot clicks, until you start to question Facebooks motivations.

I agree. This is an easy problem, considering FB's resources. Only a logged in FB user can click on ads. Not random crawlers. That alone makes this problem 100 times easier than what Google has to face with invalid clicks. No excuses.

It depends whether the bots are creating accounts or hijacking existing ones (via browser plugins or whatnot).

The bots will not be able to get accurate statistics without paying, but they might be able to get "good enough" statistics. For example, is the system moving in a direction we like.

If you do need to pay , a good bot herder probably has enough stolen credit cards to use.

Yes, there are certainly mitigation techniques. But like other spam/bot related problems it becomes an arms race.

You only need 1 good bot programmer to release his bot online for thousands of budding script kiddies to take advantage of.

Hijacking browsers? We're getting pretty sophisticated and far fetched. If you've hijacked someone's browser there are many better ways to make a profit than spiking a competitors ad buys.

So basically we've narrowed down the potentially bot makers to:

1) Has a way to hijack thousands of facebook user's browsers, and is more interested in spiking some companies ad buys than other profitable activities they could be doing.

2) Has access to stolen credit cards so they can spend the money required to debug and test their bot's humanity against the defensive algorithms. Spending money in order to indirectly make money because some company is paying them to spike a competitor's ad buys. Obviously this is going to cut into their margins, so the amount they spend should be less than they're being paid.

3) Has the patience and dedication to analyze the bills when they come in to determine how much of their activity is considered botlike and then tweak their behaviour accordingly.

4) Thousands of budding script kiddies are also being payed by companies to spike their competitor's ad buys.

5) Rides a unicorn?

Don't forget there also needs to be companies willing to pay someone to make these bots and don't mind how incredibly difficult it would be to measure it's success. In fact, if there are companies willing to pay for this, then you're going to have a whole bunch of scammers who simply claim they're able to do it that the companies will have to filter out.

We're incredibly far into the realm of unprofitability and edge-casery don't you think? An alien spaceship is going to land in my backyard before this happens.

Not really, this is just off the top of my head and I'm sufficiently unfamiliar with bot programming and FB advertising that I'm sure that plenty of better solutions must exist.

You are forgetting that there are growing numbers of programmers in developing nations for whom doing this sort of thing can be an attractive way to earn a living (or at least they are willing to try it). Look on any "hire a freelancer" type sites and you will find hundreds of postings for tasks that sound awfully like scamware development (usually at a couple of dollars an hour).

Browser hijacking is not so uncommon either, I've installed legit software in the past that has come bundled with some crapware that injects extra adverts into web-pages and other things.

The analysis would not be too difficult either, simply set a "click rate" and adjust it downwards until you are being charged > $0. Do facebook offer any method to get free trail credits? If so this could be abused, heavily.

The economics are roughly the same as with spam, if you do enough of it your "hit rate" doesn't matter so much.

Where there is an affiliate program there will be bots. Insofar as Facebook creates a way for developers to monetize their apps through a revenue share on ad revenue, developers will have an incentive to falsify app usage and ad clicks.

This is a mostly solved problem though. Or at least there are people and products out there that do this specifically for you, not with a focus on bots alone as that is a bit of a edge case but for mobile devices.

DeviceAtlas and WURFL are both products that I have used that attempt to have complete coverage of all UA strings in use in the wild even, the ones that attempt to spoof legitimate UA strings.

While their focus is on device detection and the properties of these devices for mobile content adaptation, knowing which connections are from bots is just as important if you are going to alter what content you serve up based on UA.

These product actively seek out new UAs and add them to their rule sets in a way that I can only assume will be more accurate over time than a home grown solution.

FWIW I know that FB use at least one of these products for their device detection, so if you want to match what FB thinks (knows!?) this is where you would start.

FB turning a blind eye towards shady a advertising practice because it generates revenue? See: Scamville http://techcrunch.com/2009/10/31/scamville-the-social-gaming...

It's just as easy to invest in several fake profiles that act just like real users. Pay people to work from home maintaining fake accounts for you, posting stuff that seems real, interacting with your other fake accounts and celebrity/business pages, etc. Then, run bots with these accounts, click around on a subset of good links randomly, and throw in a reasonable ratio of ad clicks per account, then move on to the next shill account.

This definitely takes non-trivial effort, but it's also definitely doable if you have a serious interest in perpetrating click fraud. While Facebook hypothetically could fight this form of click fraud, it'd likely be difficult and the method would be fragile/easily circumvented. Like most things, PPC advertising can only function in a world primarily populated with positive actors.

I am unsure if Facebook provides a service like AdSense to motivate this kind of behavior. It would be interesting if to see how they could fight this type of behavior though.

1. Make the split uneven. For example for every dollar advertiser pays, only 30 cents goes to the page owner who received the "Likes"

2. Lobby to make these kinds of activity illegal. They are after all spamming activities

3. These kinds of fake account farms will usually have very few connections to outside world. That is, accounts that do not obey the "Six Degrees of seperation" should be flagged for human review.

4. Once reviewed enlist law enforcement help to track down the owners of the all the accounts in these farms.

Painstaking work, but I guess the point is that advantage can lie with Facebook as opposed to the spammer in this game. I am not sure about the ethical implications though.

Maybe this is why Google wants to succeed badly in the social space : extra spam fighting abilities.

You're basically saying 'there is a point where the line between actual user and paid click-frauder becomes blurry'. Which is true, but that point is way beyond what is profitable, which is enough - you 'only' need to make your system hard enough to beat to make it not worth the effort.

Sure, I agree that your summary is correct. I don't really think the system I described is necessarily untenable, though. If you make 10k+/mo from ads I think it could be plenty profitable.

It's much easier to guess a weak password than create a fake profile. As for everything else, I totally agree with you.

Google seems to handle click fraud pretty well: http://www.google.com/intl/en_ALL/ads/adtrafficquality/index...

I couldn't find any resources on how Facebook handles this.

Just as a thought - facebook must be in a unique position to provide almost unbeatable captchas - just put up images of a known friend and four / forty random pictures - the one who chooses correct is likely the person logged in

ignore this is fb already does it

They do in fact do that (or something like it) when you need to recover an account/password.

They also did this the first time I logged in from a foreign country.

I suspect even that signal is not trustworthy. Given that its not 100%, what fraction of the 900+ million Facebook users are actually users? Twitter seemed to be totally plagued with that at one point anyway.

What's the grey area look like? What's the chance of falsely identifying a real user as a fake one?

Rhetorical question. Not sure FB could even answer them accurately.

> How about we just make a "blacklist" of these known bots, look up every user agent, and compare against the blacklist? So now every single request to your site has to do a substring match against every single term in this list. Depending on your site's implementation, this is probably not trivial to do without taking some sort of performance hit.

???! Excuse me? Are you programmers? Efficient substring matching is a solved problem. How many entries in that blacklist are you looking at then? Can't be more than a few thousand to catch 99.9% of the ones where a blacklist would work.

If done right it's easily faster than

> see if the client can execute Javascript.

because that requires a whole extra request roundtrip before detection. Also it's no longer true that bots don't use Javascript, the libraries are freely available.

It's also definitely faster than

> some sort of system that analyzes those clients and finds trends (for example, if they originate from a certain IP range)

> This is smarter than just matching substrings

No. It's smart to grab the 99.9% of bots with a blacklist of substrings, most importantly because it has a very very low false positive rate (unlike checking for JS support) because any human user that goes through the trouble of masking their UA as a known bot certainly knows to expect to get blocked here and there.

After that you can use more expensive checks, but at least you can bail out early with on the "properly behaving" bots with a really cheap string matching check (seriously why do you think that's an expensive operation--ever check how many MB/s GNU grep can process? just an example of how fast string matching can be, not suggesting to use grep in your project. Your blacklist is (relatively) fixed, you can use a trie or a bloomfilter it'll be faster than Apache parsing your HTTP headers)

This is entirely honor system, though. All of the user-agent checking has to be layered with the other bot-removal techniques. Malicious users will always fake user agent strings.

Of course. But you also need to make sure you don't accidentally count the non-malicious bots that have known UA strings but just happen to crawl all over the place, including ads.

I mean it's a good (and cheap) first pass to throw out quite a number of true positives.

I don't agree with this at all, user-agents...really?

Go create a really, really smart bot, you will instantly know the limitations they have. It's pretty trivial with enough data and honeypots to separate them from actual people.

Would "white listing" the dozen or so top agents that make up the vast majority of traffic work better?

Those are exactly the user agent strings the bot would spoof.

You'd want to white-list both based on user-agent string and IP / CIDR origin.

Keeping those lists maintained would be a bit of fairly constant effort, unless you could come up with a self-training mechanism, or a self-validating mechanism.

Killing Google's ability to crawl your site is pretty much as bad or worse than blocking bots. Though you should also be able to ID bad guys by noting which IPs the spoofed user-agents are coming from to, say, some honey-pot links specifically disallowed for that user-agent in your robots.txt file.

In the general case it's important to let google crawl your site, but facebook would probably be fine without it.

One quick way to detect bots is to look for a GET request of the robots.txt file

Sort of explained already, but "bad" bots would never go look for that file. And a good bot probably already identifies itself the the request, so no need to look through robots.txt.

That's it. A "bad bot" would not check robots.txt, but a legitimate user would check it. So looking for the software not checking robots.txt combined with user agent matching for good bots, you would have a good matching ratio.

Another great way of detecting bots is to redirect the visitor through an SSL connection and sniff their cipher suite. By comparing the list of ciphers the bot presents with a list of known ciphers of major browsers, you can accurately detect a majority of bots. Many bot authors use the same few libraries which never change their cipher suite.

This seems brilliant. Are there any downsides?

There is a slight performance hit, there is overhead added by creating SSL connections. However, using this method you can detect most cURL bots, practically all bots written in Ruby, and myriad other bots that are lying about their User-Agent. Combine it with other techniques for maximum effectiveness.

This sounds like a problem that should be addressed with machine learning and AI rather than the wack-a-mole style of matching strings.

Separate out and distinguish possible humans from possible bots using a set of known humans and known bots.

Assign a human likelihood probability to each user and pay out the click based on the probability.

for eg. this user is 90% likely human, therefor the cost of the 20 cent ad click is 18 cents.

This doesn't have to wait on Facebook, somebody could build this as a third-party app and then send refund requests to Facebook on clicks that are likely bots.

> How about we just make a "blacklist" of these known bots, look up every user agent, and compare against the blacklist? So now every single request to your site has to do a substring match against every single term in this list. Depending on your site's implementation, this is probably not trivial to do without taking some sort of performance hit.

Build a finite state machine which only accepts the terms in the blacklist. That should be a one-time operation.

Then feed each request into the FSM and see if you get a match. Execution time is linear in the length of the request, regardless of the number of terms in the blacklist.

A perl regular expression will test for a match in less than a microsecond. Other scripting languages have a similar speed.

There's no need to program like it's 1985 any more.

Hey, you show me a performance problem, I show you a solution. Preferably one involving finite state machines.

The big problem here isn't detecting bots. Sure, in this one company's case bots may be the problem; but in general, it's detecting behavior that you are paying for that doesn't make you any money. There may be real users who click "like" a lot; they may do it because they're paid, or they may do it because they're bored and trying to see how many "likes" they can collect. It doesn't really matter; none of them are very valuable to an advertiser.

Rather than trying to explicitly filter bot traffic, they should try to quantify the value of particular "likes". For instance, a user who demonstrably clicked through and purchased something is much more valuable than one who "likes" 10,000 companies and has no friends, whether or not they're a bot.

And heck; a bot that has thousands of followers who all buy lots of things is valuable too. Maybe someone created a particularly effective meme-generator bot, and people who follow that bot are likely to buy ironic T-shirts. Whether or not a user is a bot isn't all that relevant; it's whether that user's activity correlates with you making sales.

> So if you have any sort of business where people pay you per click

This. This is the problem -- pay-per-click is completely and utterly broken, and has been ever since people figured out how to use bots effectively. For one, PPC is simply not compatible with real-time reporting, but there are myriad other problems with it.

And honestly, I don't know if it's even really a problem worth trying to solve, since there are better models out there already. "CPA" ads [1] are by no means perfect, but they obviate almost all of the problems you mention simply by their definition: the advertiser does not pay unless whoever clicked on the ad does something "interesting". Usually, this can mean anything from signing up, to buying something, to simply generating a sales lead, but the important thing is that "interesting" is defined by the advertiser themselves. Not by, say, Facebook.

Full disclosure: I'm an engineer at a startup that does, among other things, CPA-based ad targeting.

[1] For those of us who don't work in ads, http://en.wikipedia.org/wiki/Cost_per_action has a decent explanation.

I wish their post was a bit more technical. Show us the code you used to detect javascript, how did the analytics work? Did they test their code by using their own browser to click an ad and see if the code detected javascript, or their hit in the analytic software?

More technical info and data would be nice!

While that would have been interesting to me as well, I think the post was meant for their actual users, to explain why they are moving away from FB. Technical details would have been an unnecessary distraction for their intended audience.

Maybe if they have any employees who follow HN, someone would share more light on the technicals?

Found this TechCrunch article that goes a bit more indepth. http://techcrunch.com/2012/07/30/startup-claims-80-of-its-fa...

This whole thing seems fishy to me. Facebook is one of my best sources of traffic. It's cheaper than AdWords, I can do keyword, demo, and geo targeting all built in. And if I have a hit campaign, it's pretty easily scalable. If you tie it into specific performance, like $ / email sign up, or $ / pageview, it blows away adwords. Getting in touch with someone from Facebook can be a pain, but it's drastically easier than AdWords or other ad networks.

This whole thing seems like one big linkbait/PR play. It's a really big claim from a company that doesn't even have an about us page, has only 500 Facebook likes, and basically has 0 traffic.[1]

I'd like to know if they have tried other paid traffic sources, or have concrete evidence. Inexperienced advertisers tend to spend $50-$100, expecting instant results. And when it inevitably doesn't work out, they blame anyone and everyone.

The second point seems way exaggerated. Facebook isn't "holding your name hostage" and being "scumbags". You didn't sign up with the right name from the beginning, and now you're complaining. You could have told the ad rep that you'll do $2,000 budget at some point, just change the url now. You could have also taken the 5 minutes to sign up for a Facebook Page with name@limitedrun.com and get the page you want. With only 500 facebook likes, the switching cost is low.

[1] http://siteanalytics.compete.com/limitedrun.com/ https://www.facebook.com/limitedpressing http://who.is/whois/limitedrun.com/

This whole thing seems like one big linkbait/PR play. It's a really big claim from a company that doesn't even have an about us page, has only 500 Facebook likes, and basically has 0 traffic.[1]

I don't agree with the second point. Facebook isn't "holding your name hostage" and being "scumbags". You didn't sign up with the right name from the beginning, and now you're complaining. Instead of taking 5 minutes to sign up for a Facebook Page with name@limitedrun.com and get the page you want, you decided to sling mud with no real evidence. With only 500 facebook likes, the switching cost is low.

[1] http://siteanalytics.compete.com/limitedrun.com/ https://www.facebook.com/limitedpressing http://who.is/whois/limitedrun.com/

Wow, even if this is a little true the implications are big.

Is this similar, or related to other ad space inflation? I'm reminded of Google (and others)(1) sending out free $75 adwords coupons to easily allow the costs of bids to inflate since it's not my own real money (thereby increasing the percentage of real cash being made on the ads)?

(1)Edited and clarified

How come, everytime some company does (or seems to be doing) something shady, there is always a bunch of people like this saying "not as bad as Y doing something else that might or might not be true or related". This is not middle school anymore, pointing the finger at other kids is not proper discussion ...

Raising awareness isn't necessarily bad, even though its off topic.

Because unfortunately logical reasoning and argument construction are not educational targets for most people. :(

If genuinely curious: I'm drawing a direct comparison of the potential inflation of ad costs in one playground (google) with another (facebook) through both common and unique means. Another common issue to both is click fraud and it's effects on the said cost of ads. My mistake for assuming this was obvious and known.

The relevance of such a comparison? It's not a one off, as uncommon as we think. Has nothing to do with pointing fingers.

Didn't mean to come across as off topic. To me, if someone upstream and in the past was doing this, chances are it might happen later in time downstream too, and there might be something to see in the forest instead of pointing at one tree or another.

You make a valid point.

Unless you're buying ads for popular affiliate stuff, handing new AdWord users $75 in credit doesn't have much of an effect (other than some percentage of those users going on to spend real money). I have gotten similar offers from Facebook, Twitter, Bing and a lot of othe advertising companies.

Yeah, lots send it out, but how many end up using them is relative to how big of a player in search (and search related advertising) the player is.

I've used Google coupons, none with anyone else that I can remember. I also spent ads on google way before any other system and remember the 10 cent clicks going up to $1.00 for not much more traffic.

I want to say:

>80% of $FBs revenue is made from bots! OLOL

but what I actually think is

>They're just pissed their campaign didn't work out like they'd hoped

BUT I also think that it's equally likely that $FB would, if possible, defraud the clueless. How deep are [client]'s pockets? How literate are they in working to strict ROI goals?

I think this is probably a relatively new phenomenon. Facebook went public. Sales managers need to hit targets, they're thinking quarterly and they've got stock.

People know people and the rest is history.

If this has got legs then it's the perfect storm for facebook. :)

Two other hypotheses they don't seem to have ruled out:

- Their site is so awful that 80% of people click back before the page finishes loading, thus appearing to not have javascript

- Their ad is so awful that almost no people click it and a few bots make up 80% because the total is so low.

I didn't know Facebook charges businesses for changing the name of Facebook Page. Is this a well known fact?

According to the FAQ, one cannot change the name of the page once it is set. This can be a real problem that I run into quite a few times, even with a couple of small pages, and even more as every website that has "like this page on facebook" has an automatic facebook page, where the name is determined by them, automatically, once, by the <title> element at that time. Talk about stupid.

Anyways, I don't mind since I'm a small fish, but the longer this goes on, the more it will (hopefully, i might say) blow up into their face(book).

Why not open a new page, and delete all content (but a link to the new page) off the old page?

You lose all your Likes.

Oh no!

Which is presumably why this is a policy in the first place...

Facebook allows advertisers to target friends of users who liked your page. They probably restrict name changes so users associated with a page don't get blind-sided if the company radically changes its product or service.

The only well known fact when dealing with Facebook is that if you want to manipulate their documented rules you need to either have deep pockets or a lot of influence.

We tried to change the name of our page from "Hit Tennis 2" to "Hit Tennis". (After releasing Hit Tennis 3, we realized a per version FB page was silly.) We were asked to explain the change and it was denied (as I recall they said it could lead to user confusion). I thought that was a bit picky, but I understand the position of being careful of changing their likes our from underneath users. No mention of having to pay to get our name changed.

Your name becomes locked after 200 likes: https://www.facebook.com/help/?faq=271607792873806

For the username, everyone (including users) is allowed one name change.

Presumably, Facebook can bypass these seemingly arbitrary limitations, but will essentially charge you for the right to do so (by requiring you to have purchased a certain amount of advertising with them).

This seems pretty reasonable, as the potential for abuse for arbitrary name changes is very high.

But at the same time is very normal a change of name, because companies adquires other companies, products can change the name too[1], etc... The Facebook page is supposedly owned by the creator, not Facebook and if a change of the name is needed, IMO it does not make sense to try to charge money from it.

I'm with you that F must have some restrictions on name change, but it does not seem reasonable to charge money for it.

[1] Imagine a page called "Sun Microsystems - Java" before being buyed by Oracle with a fer thousands fans. I'm sure a hight percent of the fans would not care of have a name change to "Oracle - Java" for example.

They said they would allow us to change our name. NICE! But only if we agreed to spend $2000 or more in advertising a month.

Well, apparently there is an exception from this rule. Obviously there is a huge potential for abuse when changing name of a page that has plenty of users. However, allowing only the "money making" pages to do so is a bit of an unfair move in my opinion. Reasonable but unfair.

I wonder if this situation will escalate and fb will publish some explanation. Nevertheless, it is still hard to believe for me that a company with such value would fake clicks at all or in a way that is easy to detect.

You can change your username (facebook.com/username) once, for free.

https://www.facebook.com/photo.php?fbid=10151098204101337...

After that, it's unclear.

Could it just be a badly written search spider? The ad is simply an anchor tag around an image. If you didn't filter it out specifically, you would just "click" it to get to another page.

I was thinking about it too... but then again why don't we hear the same complain from other advertisers?

When you build an advertisement network, you end up spending a bunch of time figuring out how to isolate these fake clicks and verify that they are not real. As an example, the IP addresses running the bots are going to be clicking "way too many ads to be a human".

Possibly, but aren't these ads only shown to logged in users?

If FB is not smart enough to discern curl/wget/whatever_crawler_socket_call used from genuine in browser click then the argument is already won. Hell, you could write an army of Selenium Firefox/Chrome bots to drive up costs of ad competitors, randomise the host machines, etc... but it's still hard to believe that FB anti fraud measures can't figure it out.

To me, something is off, whether with FB or these guys.

Quite interesting top comment on Reddit thread http://www.reddit.com/r/technology/comments/xel55/startup_cl...

"I can confirm this. I used Facebook to advertise my page and went from ~2000 fans to over 6000 within several months. Awesome, right? Wrong. I thought it was weird that the new fans would never, EVER interact with the pages. So I started stalking their profiles. Guess what? Bots/hijacked accounts/fake accounts. How do I know? Many of them have NO friends. Then I noticed something really scary...repeats. Actual pictures showing up more than once for new likes. Very, very few of the accounts were from the USA. I started casually messaging the accounts that looked suspicious. Not a single one with no friends responded. Out the ones that looked legitimate (although with a list of likes that is in the thousands), only a handful replied, and 100% of those people confirmed they found my page through affiliated sites, and not through Facebook itself. I ended the ad program recently and the bots stopped. tl;dr: Do not, under any circumstance, advertise with Facebook. They will feed you fake likes. If you want fake likes, use a site like fiverr to generate them, it's way cheaper and yields identical results."

Just a brief note: bot sophistication is off the hook. I worked at an ad network where one of the clients demanded we prove that the mouse had moved prior to "clicking" on one of their ads. No problem, we said, and most of the traffic passed the test. The traffic had referrers, legit user agents, javascript, and they were triggering the mousemove and click events. Out of curiosity, we sent a fraction of traffic to a test page, wherein the actual page was displayed in a full window iframe, then blocked access until the user clicked, and tracked their mouse movements. Easy enough; annoying for a user, but not a deal breaker for most.

The user movement maps we ended up with were straight lines of randomly spaced dots with sudden acute corners. A human could not replicate the pattern with a ruler and a pen pad, and we certainly didn't expect 90% of our human users going for that client's sites to be sitting around with rulers and pen pads drawing lines of randomly spaced dots. We checked the testing code and ran a dozen tests, and we couldn't figure out how a human could replicate it on any browsing device. Our conclusion is that somebody bothered to program a bot that would replicate mouse movement and we accidentally broke it by blocking programmatic access to the page, so it couldn't find a link to click and went crazy.

"We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots."

How do you tell if it's a bot? Just by user-agent string or repeated slamming by IP address, or is there some other technique?

The main technique is to notice a pattern that is only explainable by bot activity.

Generally speaking, in addition to the patterns you mentioned (invalid user-agent strings, rapid succession of same-IP access) there are also other patterns, such as failure to load referenced CSS or image files, or 100% failure on Captcha challenges.

"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files?

"100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?

"failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files?

Very much so, after all your browser does without you having to tell it to do so. It all depends on whether the hassle is worth it for the bot writer.

Of course they could -- all they need to do is parse the HTML, look for the img and link and other tags, and then request the URL. If they were trying to act more like a browser, they would. If they are just trying to click ads, they wouldn't.

> "100% failure on Captcha challenges" -- are captchas generally deployed on click-through ads?

Cloudflare has a mode (used in general for DDOS protection) which you can turn on and get stats on how many users pass a Captcha test.

> "failure to load referenced CSS or image files" -- is there a way for bots (in particular, mechanize in perl/python) to also download those files?

Use http://phantomjs.org/ and the page will loaded by webkit.

Using Captcha on an ad landing page to determine whether someone is bot seems like a bad idea. It would almost certainly increase the bounce rate on that page and you couldn't just say that people who didn't choose fill out the captcha were bots.

> is there a way for bots (in particular, mechanize in perl/python) to also download those files?

The point was mostly that if it is not downloaded, then it's a bot (or any non-browser getter, at least), not that it's not a bot if it does.

Or maybe you were asking for a way to avoid bot detection ? :)

Most likely a combination. In addition to analyzing the user's navigational path, exit points, IP ranges, etc...

They said that for 80% of their clicks , Javascript was off. Now there will be plenty of users who have disabled Javascript but nowhere near 80%.

Although they stated they built their own analytics software, so there's also a chance that it wasn't working as they thought it was.

Furthermore, facebook users in specific I would say have javascript enabled, even though its possible to control per site, I would assume most facebook users would not be aware how it works - and facebook without javascript seems rather impossible or at least, very discomforting.

How do you tell if it's a bot? Just by user-agent string or repeated slamming by IP address, or is there some other technique?

A combination of things. Off the top of my head:

* IP address ranges. Are they all from, e.g., EC2 instances?

* Are images and other content (e.g. CSS) linked from the page loaded? How long does the client take to do so? Are other pages being viewed in line with what you've seen from other users?

* What combinations of user agents are you seeing?

* What's in the headers being sent by the browser? Any foreign language support, or oddities when it comes to not accepting compressed content? Is the referer header always the same? Does it always vary?

* Do you see "normal" user behaviour that indicates activity such as clicking the back button, then clicking the ad again a few seconds later once their brain has kicked in? (nb: not sure if this is possible for Facebook with the way their site works).

* Are there any patterns in the page access times? Is there anything to indicate a cron job is kicking off once an hour at the same time?

Is there a single tool that can take access/error logs and compile these stats? This sounds like something relatively useful ...

If not, it sounds like a great place to use ML/classification algorithms.

From the article it would appear that their definition of a bot is a client with JavaScript disabled

"So we did what any good developers would do. We built our own analytic software. Here's what we found: on about 80% of the clicks Facebook was charging us for, JavaScript wasn't on."

Later:

"So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs"

I think they use a different definition of bot. Otherwise they wouldn't have to use a second logger ...

Out of curiosity, I just went back to a campaign I ran last year with Facebook to see if I was affected. I spent $2k on CPC advertisements with a filter targeting people related to phones, Cydia, Apple, etc.; the result was that I caught enough people that some smaller blogs reported on my ads.

http://www.givememind.com/saurik-advertises-cydia-tweak-on-f...

So, this campaign lasted one day, just over a year ago, on July 22nd. During this time, Facebook claims I got 5229 clicks. According to Google Analytics (which will only log users whom have JavaScript working and enabled), I got 5057 hits to that landing page.

Finally, working from logs, I have managed to find 5643 total requests for the URL. If I filter for those that explicitly sent a referer from facebook.com/ajax, I see 4644; of course, many people have browsers that do not send referers due to various reasons.

Therefore, I feel safe concluding that, for my website, a year ago, I was not having "80% of ad clicks from bots". I present this evidence in order to help people attempting to understand this issue to provide the possible clue that this is either "user-specific" or "changed in the last year".

I work in the digital advertising industry, more specifically on ad exchanges and various behavioral targeting platforms. Facebook is scrambling to monetize their traffic like never before, as a reaction to fallout from their horrifically botched IPO.

They recently announced the launch of a beta ad exchange, FBX, that for the first time will allow outside advertisers to access Facebook inventory for behavioral targeting. So if you are researching a flatscreen TV on Best Buy, you could start seeing TV ads on Facebook. This is common practice across all e-commerce websites, but until now FB has been a closed inventory platform.

This means that the clicks and actions from the Facebook audience are being scrutinized more than ever, and those have to be quality clicks (not bots). General Motors recently announced their complete withdrawal from FB because of poor advertising performance (again, bots).

Facebook is fucking up royally on the business side of things, so for the sake of sustainability I hope they are still laser-focused on product.

We dropped the Google Display Network about a year back out of very heavy suspicion of bot malfeasance... lots of clicks from frivolous, consumer-oriented, no-name sites with obviously nothing to do with our somewhat technical, LDAP-focused product.

Yeah, I'm convinced it's setup to scam users and there's a lot of click fraud happening.

Wrote a post about it here: http://peebs.org/148437517

Facebook, like Google, uses a bidding system and expects the free market to compensate for invalid clicks/impressions. This should theoretically work - advertisers should be bidding lower since the clicks/impressions are worth less. However, unnatural market forces, like companies that are (at least for a while) willing to lose money on their their advertising continue to drive unrealistic pricing for their advertising. As long as Facebook can continue to convince new advertisers to try their system, and (perhaps more importantly) keep brand advertisers measuring ROI in terms of Likes instead of cash, prices will remain unnaturally high.

>But let's move on, because who the bots belong to isn't provable.

I'd love to see someone try and find out though. Any IP doing automated requests of that nature is fair game for a bit of probing IMO.

Some background and a disclaimer. First the disclaimer, I've worked at Google (which lives of its search advertising) and now I'm with Blekko (which generates revenue from advertisements as well) so I've been on the 'inside' looking out at this problem.

The background. A lot of the web you see today is automated, which is to say that 'the web' can exert influence in the 'real' world, and influencing the web can be amplified (or leveraged) as traders would say, with automation.

Two examples, in the first example we have a thief who wants to steal money but doesn't want to go out and rob a bank. Instead he 'hires' Google (an unwilling partner here) and creates an AdSense account, makes a web site, fills it with AdSense add javascript, then writes a program that can fetch pages from his web site and click on an ad. He then feeds that to a botnet for hire (generally not an EC2 cloud) and has it go off and click on all his ads. Google congratulates him on what a great web site he launched, pays him anywhere $10,000 to $50,000 from ad revenue. And the party continues until the Advertisers complain or the traffic patterns pop out as suspicious. Obviously Google has a lot of folks who do nothing but look for this sort of abuse but when you're making numbers like they, even if you had 7 or 8 million dollars a month in click fraud it would be rounding error in their quarterly reports. The people who get pounded by this are the Advertisers which lost their $500 or $50 or whatever that month by being charged for clicks from machines that weren't ever going to buy their product.

In the second example we have an ambitious entrepreneur who wants to get the word out on their new thing. They create over time a few hundred thousand 'fake' twitter accounts, they construct a community of fake accounts following each other. Then they write a simple program to have their 'influential' tweeters push out the word and their 'followers' pick it up and retweet it. If they have let the accounts lay there for a bit they pick up their own share of robot followers (from other people doing this) and our ambitious entrepreneur creates what looks like a real groundswell of interest in what ever thing they are trying to push.

There are many rewards to automating web activities, for many different reasons. As a service provider its a full time job trying to insure I can identify the fake ones (I don't want to send ads to robots, they don't buy anything and they make me look bad to my advertising channels) That 80% of the 'clicks' this person got on Facebook fail a simple 'real user' test does not surprise me.

I expect there to be a rise of landing pages which are themselves pages requiring you to click somewhere else and then clients only paying out when a prospect clicks all the way through to some part of the web site. I know that some big names already do this where they bring up a page asking you to identify your location (which helps them bring you to the part of the site for your area) but they also don't pay for the click unless you make it through that page.

>In the second example we have an ambitious entrepreneur who wants to get the word out on their new thing. They create over time a few hundred thousand 'fake' twitter accounts, they construct a community of fake accounts following each other. Then they write a simple program to have their 'influential' tweeters push out the word and their 'followers' pick it up and retweet it. If they have let the accounts lay there for a bit they pick up their own share of robot followers (from other people doing this) and our ambitious entrepreneur creates what looks like a real groundswell of interest in what ever thing they are trying to push.

This is exactly what HBGary built. But it wasn't ads they wanted and it wasn't for money. They built sock puppet management apps for the DoD to influence public sentiment about various things the government wanted... except they got caught.

Yup, and there are several perhaps as many as a dozen 'sock puppet' systems out there today.

Generally I refer to any program which is using APIs intended for humans in order to achieve a result for the person who ran it, a 'bot'. Comes from folks who build scripted clients for computer games.

This just begs the question -- how difficult is it to do a reasonable job of detecting bots before passing through the "click"? If it is really hard to do during the request processing, surely it is not hard to do before billing?

It's an extremely hard problem whether you tackle it in realtime or post-processing. We're talking about being able to separate one group of real people at computers sometimes clicking ads, from another group of real people at computers sometimes clicking ads, where the only difference is the second group has been paid or their computers are being remotely controlled. Nothing in your access logs is going to leap out and say "hey, 123.123.123.123 is an office worker interested in this ink ad, and 123.123.123.124 is a competitor that wants to drive up the price of this ink ad".

Hey, I'm an engineering manager on the Ads team at Facebook and have been working on our Ads Marketplace Quality team for a couple of years now.

We take reporting accuracy seriously and have a variety of systems in place that may filter out some clicks and impressions before they ever appear in charges or reports. For example, we may invalidate repetitive, incomplete, or otherwise abusive clicks, and we cap the number of times any user can see or click on particular ads in a day. We also monitor traffic and click patterns across the site and try to filter clicks that appear to be generated by bots or other automated systems. We work closely with our general spam and fraud fighting engineering team called "Site Integrity", which does great work trying to detect fake users, bots, and users infected with malware.

A lot of sophisticated advertisers use Facebook, and many of them perform all kinds of validation of the effectiveness of their advertising on our site. Many advertisers also use various analytics systems to monitor the clicks they receive through our ads platform.

We are constantly working with advertisers and partners to help ensure we deliver ads that are valuable to our users and advertisers. We care about this area of work a lot.

One POSSIBLE explanation for what Facebook would use anonymous bots for: verifying the advertisement goes where the advertiser said it would go.

Facebook has problems with fraudulent or confusing advertisements. for example, iPad's being advertised for $2, but the user clicking the ad is sent to a penny auction site that bills you $100 when you sign up.... so that domain ends up getting banned, but those advertisers/spammers either create a new domain or they use a redirecting domain that redirects to a non fraudulent domain when they see an Facebook IP (in theory, the Facebook IP would mean it's an intern or Facebook bot checking the URL).

Here's an example of the penny auction crap http://www.fastcompany.com/1771099/facebook-spam-ads-damage-...

and an example of manipulating the landing page URL with a redirect http://www.larrylim.net/seo-online-marketing/facebook-ads-pr...

so maybe it's Facebook's engineers trying to verify the ads are going to the place they're intended to go, but they are obviously incorrectly counting them as billable clicks

True, but they shouldn't be charging for those verification checks.

So you think that a facebook ad checking bot is taking up 80% of the clicks?

I know facebook doesn't have the best reputation for careful engineering but that's quite a stretch.

Well why not? It is really more expensive to run such a bot at 80% rather than say 5%.

If it was coming from Facebook, I would expect not to be charged for the clicks.

Back when I was in this industry, generally speaking you would have a graylist of your own IPs that would be ignored by the ad metric system.

Ad networks tend to be very serious about anything measurable - it's easy to lie and once you've lied about one thing, no matter how small, what else have you lied about and to what magnitude?

This is a pretty interesting finding. Has your team considered acutally spinning off a consulting branch or creating a plugin that other companies can use to determin if they are paying for bot clicks? That would probably be pretty helpful for others and a great way to draw attention to the issue. I'm sure if they see a growing number of companies stop advertising on the site, they may start to pay attention.

We need some real facts here, I've done a lot of FB advertising and have not seen results like this. This smells to me like an incorrectly configured campaign(s).

Here are a few points that need addressing:

What volumes are we talking about for the clicks?

How many ads did you run per campaign and how many clicks did you receive per ad? Facebook do use bots to verify the landing page of an ad url, this will usually lead to one or two tracked conversions per ad. These will show up in the external website analytics software but won't be charged as clicks by FB.

Were you running ads that drove directly to your website, or did you direct ads to your Facebook page? - Obviously ads that land on your Facebook page won't be tracked by your analytics software.

How did you identify that links coming to your website had indeed originated from Facebook. Did you loose conversions through badly formed tracking urls and could it be these suspect page loads were actualy coming from a web crawler?

Did you run CPM or CPC campaigns? - Maybe a misunderstanding of the charges involved?

What I don't understand is why someone would target bots to only Facebook click advertisements. Facebook offer the ability to create ads that also allow users to Like a page, RSVP to an Event, or Install a Facebook app -- all of these things would require 1) Javascript to be enabled and 2) a Facebook account. If bot Facebook accounts were created they would need to be at a fairly large scale to be effective (which would be difficult given that Facebook is fairly effective at finding and disabling obvious fakes -- anyone in this space has a hard time just creating a second testing account) and would be visible to advertiser since they can run analytics reports and view individuals who have acted on this types of ads. So, I'm more inclined to believe that if anything this a competitor of the advertiser gaming the system as some others have suggested.

Still, like others I'm interested in seeing some of the data on this one.

Could FB do something like this to detect click fraud/misbehavior: FB's ad system only counts a click as valid and charges the advertiser when the browser from which the click originated has js enabled.

OP says only 1-2% of users have js disabled hence FB would only miss out on a relatively small amount of ad revenue instead of massively over-charging.

Lovely to read a post that bashes Facebook, while containing no raw data leading to the vague analysis. For all I know, they got paid by Google+ to write this. Maybe they did, maybe they did't, but my assumption about the post's credibility is as good as their assumption about the results of their analysis. What a link bait!

The data here is clear. 80% of clicks are from non JS clients. There's no getting around this. I run an ad network, a real-time analytics product and a WordPress security product that provides real-time traffic and we use JS vs non JS to distinguish between human and non-human for our customers and it is 95+% reliable.

As an ad network provider it's your responsibility to filter spam clients based on heuristics like click frequency from a geo location, JS support in clients, behavior of ISP IP blocks, etc.

It may be that this customer was specifically and exclusively targeted, but even so FB should have mechanisms in place to deal with and block this. This is not a good sign, but then as I've mentioned before I don't think ads is a viable model for FB, so perhaps their energy is being directed elsewhere.

When a Facebook user clicks on an ad, Facebook already knows the likelihood of that user being a bot, based on information like their friends and other activity in that user account. Looks like Facebook chooses to not use this information as it will reduce their revenue.

Can somebody explain to me the business model behind bot clicks? I mean I understand if Facebook is doing it to drive up income (Although I don't believe it without some kind of proof), what I don't understand is why anybody else would be doing it.

The other thing to keep in mind is if their page has any "accelerated fans" (read : purchased bot page likes), any time they promote a post, they would be promoting to the same bots that liked the page in the first place.

No accusations, but, just food for thought.

I don't really know how purchased page likes work, but why would that matter? Bots / fake accounts used to generate likes have no reason to keep running and click on ads?

Has anyone else analyzed their referrals and also seen a significant proportion of bots?

I've run a significant amount of traffic with Facebook Ads and never saw click fraud anywhere close to these kinds of numbers. By estimates it was somewhere between 3-5%, which is pretty typical across major ad networks.

It's also worth noting that Facebook has given refunds for click fraud in the past. That is if you can actually get a hold of someone in support who will talk to you, which isn't easy.

Yes but only in certain US demographics. Most of these bots are ad scrapers/landing page scrapers run by people gathering and stealing the ads/landing pages of their competition. Facebook has known about the problem for a while (years), and have gone so far as to ban some accounts that the scrapers were running out of, but lately it seems the scrapers have been given free reign.

This is not a case of Facebook using bots to deliberately drive up ad spending. It's merely scummy internet marketers at work, with Facebook failing to detect and prevent them. The phenomenon is mostly present in the US 18-30 male demographic (where all the dating ads are, and where creatives can make or break you).

A response from FB: http://www.latimes.com/business/technology/la-fi-tn-facebook...

"There must have been miscommunication with Limited Run"

"Facebook says it is investigating claims made by Musician site Limited Run early Monday regarding ad clicks by bots."

"We're currently investigating their claims," a spokeswoman for the company said in an email. "For their issue with the Page name change, there seems to be some sort of miscommunication. We do not charge Pages to have their names changed. Our team is reaching out about this now."

Pardon my naivety, but why would a third party want to set bots onto Facebook ads? I guess one could hope to exhaust a competitor's marketing budget, but surely this isn't an established tactic.

Even more Machiavellian would be Google doing it to discredit Facebook's entry into the advertising business. I doubt this because they've got too much to lose - if they were discovered, it would seriously hurt their "don't be evil" reputation, which is worth billions to them. And just imagine the anti-trust suit...

Although Facebook stands to gain the most from this, my argument about reputation damage applies there too. So who is paying for the bots?

Well, I have also tried to advertise on Facebook and saw the numbers of clicks shown in Facebook reports are much higher than the number of visits shown in the Google Analytics. It was rather strange. Besides, the support team has just ignored all my letters where I tried to define what goes wrong. As a result, I just stopped advertising campaign and wrote this article http://blog.britainloans.co.uk/2012/06/our-ambiguous-experie...

I don't understand, who are the people profiting from click fraud on Facebook? Obviously Facebook is but we can pretty safely presume they're not coding bots on their own ad network. A competitor could maybe profit from it but this seems like a rather roundabout way to damage your competitors. What other rational business case is there to build click bots on Facebook?

This is not like Google with AdSense where they pay out 3rd parties, the motivation there is clear.

What's a sure way of determining that the clicks are not coming from a proxy? I have to launch an online campaign. There are some many services that provide a high anonymous proxy and sometimes services like http://www.ip.cc/check-proxy-basic.php report a real ip as proxy. So how do I distinguish real clicks from fake users. During the signup, I can only collect Name and Email.

Seems like a simple problem for Facebook to deal with:

User clicks more than one ad per [pick some time threshold -- hour] on average ==> probably bot.

User clicks more than one ad per [pick some time threshold -- minute] ==> probably bot.

User clicked more than [pick threshold -- 20] ads in a month ==> probably bot.

FB doesn't even need to get these numbers right, it just needs to set the values so that false positives cost it an insignificant amount of money.

I guess a more general question might be: I marketing on Facebook proving to be worthwhile?

We are just finishing a new iOS app. It will be the first one in our catalog with a "Like us on Facebook" feature. I am skeptical about the value of this feature. Not sure it'll convert at all. Asked a bunch of people who use FB on a daily basis and most told me that they pay zero attention to ads or "likes". Not sure.

In January my startup ran a trial Facebook ad campaign using a free $100 advertising credit from Facebook. Extremely pleased with the results, we started paying to advertise and immediately noticed a substantial drop off in the conversion rate (down over 70%) even though we were using the same targeting, ads, and made zero changes to our website.

As a Facebook advertiser, user, and stockholder, I'm not impressed.

Relevant experiment done by the BBC: http://www.bbc.com/news/technology-18819338

(paraphrased) "Virtual Bagels": A company that does nothing gathers 4000 likes in 4 days. When set to Worldwide, it soared from hits in Egypt, Indonesia and the Philippines. When limited to the UK the needle barely moved.

I am surprised that there has been no proof released by the page owners. It feels a lot like a linkbait / FB viral campaign.

My company also experienced not getting as much traffic to my site as facebook claimed.

What I was told before I pulled my campaigns is that Facebook considers it a click when the visitor clicks your ad, but BEFORE the visitor is shown a "warning" screen that they're about to leave facebook. not sure if this has changed since. I haven't come back.

I'd like to see the actual data from this.

How much traffic are we talking about? Where were the bots coming from and how were they detected?

A related anecdote: My FB account was compromised recently (using my trivial 5 character password) and all the attackers did was occasionally sign on and like random pages. I kept getting odd company posts in my newsfeed and eventually traced the problem. These phishers/advertising bots are getting more and more creative...

Is it actually necessary to weed out bot-clicks? If all sites have equal amounts of bot-clicks, then you can just charge 80% less per click (compared to the situation where all clicks would be human). Perhaps in practice the bot percentage changes widely per-site, so it's still important to have statistics on bot-clicks.

Ran an ad a few months ago targeting twitter employee's specifically. What I found disappointed was a high rate of click throughs, (Maybe 300 Twitter employee's).

But not one of them actually used the service (Which involved typing a query into a search box).

This bot nonsense kinda makes me wonder if those 300 click throughs were really people.

Wonder how they will spin this. First they try the silence mode, see if it dies out if they give no attention. Then deny it was them and they were unaware of the problem BUT they are working on something that will totally eradicate said problem and of course more but we can't give details on us.

Just curious, could it be a competitor and Facebook doesn't block them like google does for click fraud?

Facebook couldn't be that pathetic...

I wonder if this piece of news will make it to mainstream news outlets .... If it does, I would guess it will be a week or so. In any case, it really should get out because FB is a horrible platform of any sort. I don't even use the site; Facebook, no me gusta.

Maybe it's time for the web to switch to a model where revenue is not driven by clicks, but by actual transactions. It's harder, of course, but it would sanitize things a bit. Hopefully, bots won't learn how to pay. And if they do, well, it doesn't hurt.

Why hasn't anyone asked what IP, user agent, etc. the bots were showing up as in the log?

One thing about Facebook ads, instead of buying clicks to webpages, you could buy likes to your Facebook page. This way you can at least look at the FID of the Like to get some confirmation that the Like you bought was from a real person.

I don't understand why the bots click on the ads? Of course an old school crawler would just "click" on any link on the page, but I would assume most ads are included by JavaScript and would therefore not be crawled by standard crawlers?

The comments about the company name make this look a bit more complicated. It sounds like they were increasingly unhappy about their relationship with facebook, and this new information is the final bail of hay.

Interesting. I just tried browsing facebook with Javascript off and pretty much nothing works. I can't comment, can't share, can't like. But guess what, ads show up and they can be clicked.

Isn't this why, oh, around 2005 CPA was the big thing instead of CPC? Personally, I think CPA is absolutely the right way to do internet advertising, and I wonder why FB doesn't do that.

Because they would not make a tenth of the money they do if they did that, obviously.

Most sales or acquisitions originated online are based on rational decisions, not on the appeal of annoying ads.

The day advertisers start to use a verifiable CPA system, 80% of online advertising cease to exist and 95% of today's start-ups see their cash vaporize.

Well then there's a huge market opportunity. When advertisers realize that their getting played by Facebook (and Google), paying for bad clicks, they'll look elsewhere - and CPA is pretty obvious solution (heck it's not even innovative).

Is CPAFacebook.com taken I wonder? :)

FB just gave him the best advertising ever through this publicity.

I had the same thing happen to me, but with a YouTube video instead. Facebook would report and charge me for thousands of hits, while YouTube only showed up a few hundred.

1) Buy facebook stock

2) Create massive ad clicking botfarm

3) ???

4) Profit!

More realistically, it could also be used to drain your competition's ad budget, giving you a bit of an advantage.

What benefit could a 3rd party conceivably get out of faking clicks on Facebook ads? Is there a rev share at all with publishers, like with adsense?

Driving up costs for competitors?

Tons! Charge their competitors to run out of budget and/or give up advertising on FB.

A lot of people also have bots to see who/what is advertising on FB to see what is working, what ad text people are trying, etc. Mostly aff marketers trying to stay ahead of the game. There are also SaaS companies scraping ads and landing pages and selling that data.

I'm sure there are other reasons as well...

Requiring an ad commitment before changing your name on Facebook doesn't make any sense. Anybody else hear of this kind of thing before?

The solution is obviously to make the user fill in a captcha whenever they click on an ad to confirm they are indeed human

Off topic; was any one else astounded by the misinformed and misguided comments on the FB post? Quite a contrast from HN.

The world just learned that no human being ever clicks on these ads? WOW! Amazing.

Why does this have the confident feel of a chain email, or spam? I'm suspicious.

Take this with a grain of salt. The "company" has 400 likes. Not exactly GM leaving Facebook (and then quietly returning to Facebook).

They are talking about ads, not "likes".

I think he didn't imply that. Obviously a company with 400 likes is not a big player in the Facebook ad space, it was just a measure of popularity.

Obviously the news here is not the loss of business for Facebook, but the fact that a click scam was taking place (if true).

Because they are not related. You can use them just as you would with Google Adsense. Obviously you can get a lot of free advertisement and edorsement with having people "like" you on Facebook, but that is beside of the point.

And the click scam, if true, will take away the only way for Facebook to monetize the product. Marketers are not stupid, things like this spread like wildfire.

I don't see what their number of likes has to do with bots clicking on their ads.

If true, Facebook should really have anti-fraud measures in place for this kind of thing already, and if they don't, the size of the company in question has no relevance to that.

timaelliott, I can't respond to your comment so I'm responding to its parent. Your account has been hellbanned as of 9 days ago here: http://news.ycombinator.com/item?id=4272758

The huge market represented by the "long tail" of small companies is the reason FB is valued so highly. It is (supposedly) much bigger than the GMs of the world.

Would be nice to see some actual numbers. If "80% of clicks" is only eight clicks then the OP's post doesn't carry much weight.

Make claims, get more likes.

Please explain to me why a bot would click on ads? (if anyone knows)

Is it crawlers crawling Facebook? Are they alleging that facebook is doing it to boost revenues?

Their hypothesis seems secure, given that you can't really use the facebook site at all without javascript enabled.

More alarming is Facebook holding their name for ransom for the sum of $24,000 a year!

Possibly other advertisers trying to push competitors away from facebook so BigDickPillz Inc can have it all to themselves?

Yes. This used to happen a lot on ad networks, although I personally don't know how pervasive it is now.

The game is as jiggy said. Force competing ad buyers to pay for what appears to be non-productive traffic by forcing ad clicks. Since the real views are dominated by false views, the conversion rate on whatever is being sold appears to be horrible. The competitor goes elsewhere, and the scammer maintains access to artificially cheap ad inventory, since there are fewer bidders.

How does Google prevent this? Is it simply inherently not-problematic, due to their spread of advertising?

Overall it's hard to say whether google wins or loses overall by this. In absence of a real competitor in the search ad space, people generally will swallow google's fee hikes ...

It's a cat and mouse game. Google gets good at it; scammers get more sophisticated; google catches up again; ad infinitum. Also, in addition to the scam mentioned above -- in Google's case, targeting AdWords -- they have to deal with scammers hitting their own AdSense properties.

It's a well known problem. Considering AdWords is Google's primary source of revenue, they work hard at it. For a tiny bit more directly from Google, see: http://support.google.com/adwords/bin/answer.py?hl=en&an...

As pathdependant said, this is a very common shady tactic in the ad business. You drive up your competitors costs with bots in order to drive them away from your keywords or make them think their conversions on those keywords are awful and then abandon them.

By far the most likely explanation in my opinion. Even if it's not the correct one, some of the traffic is almost certainly these kinds of bots.

The biggest conspiracy theory would say that a bot clicking ads would help earn FB money.

I don't really suspect this to be true, Facebook CTR are incredibly low so if they are faking it they are doing it badly.

I suspect the most likely reason is some sort of spam scraper thing that visits every url or something. I would have assumed Facebook would be well protected against this sort of thing in regards to charging advertisers.

Facebook does analytics...they aren't some tiny startup...surely they track the quality of their traffic...so they know that 70-80% of their ad traffic comes from bots.

So Facebook is knowingly profiting from this, and they have the resources to fix this...but they don't since this is making so much money for them.

FB does not need do that shady stuff. Here some facts.

[1] Bot registration on FB is really easy. If you try register 100 fake users in direct way you need to deal with javascript and web forms with strange ids, cookies and so on in your script. But if you enter fb.com without enabled javascript, they ask you change browser or use mobile version, which is not require* js at all.

[2] You can register number of accounts from one ip without CAPTCHA

[3] Only one "difficulty" is to find batch of emails, it is not that big issue since we have services like mailinator.com.

So 15 -25 loc script on python do all your job. It is not required use any tools like selenium or special skills , only basic understanding how http get/post works.

Why fb doing all this super easy bot registration stuff? I think we all know why.

Edit: spelling. Sorry non native English speaker here

You seem knowledgeable. What is the main purpose of bots on facebook?

The article implies they could be used by facebook or competitors to burn up ad revenue, but this approach would just kill the golden pay-per-click goose.

I think reasons is pretty strait forward:

[1] You can win a bunch of contests, competitions like those who have lot of likes can win iphone or other tech.

[2] You can sell likes.. just google and you will see, there are alot of places where you can buy it.

[3] Manipulation of public opinion.

Edit:

From FB side:

[1] they still have growth of users

[2] more bots == more clicks, more likes more FB value "on paper", more revenue

[3] and when they stabilize their market value they can easily kill those bots, just require them activation by phone.

vk.com Russian social network, had same issues, but they enabled phone activation for averybody and its not that easy to buy a lot of phone numbers in Russia, but some hackers just pay to vk.com admins...

Edit2: spelling

Thank you for that. This is the real story here, IMO.

The OP says that they are not accusing FB of using bots to drive up revenue.

They refused to overtly accuse them, but the subtext seem to argue exactly that.

Which is really taking things too far - if Facebook wanted to generate some revenue with bots then it wouldn't be very difficult for them to write ones that supported JavaScript.

That is true, but Facebook's reputation would suffer seriously if they were ever caught.

What they (Facebook) are doing now seems to be some version of DADT (Don't Ask, Don't Tell).

I don't think so, there's lots of reasons that would be a stupid idea for Facebook. But there's clearly something shady going on, and Facebook is being dicks to them in other ways, so they're out. That's the subtext.

They end the bot section with "But let's move on, because who the bots belong to isn't provable." after saying that they weren't accusing Facebook of running the bots. The subtext is screaming out.

Does me pointing out the barely shrouded subtext mean that I believe what they are saying? Of course not. It's just as likely, it seems, that their test is simply flawed (as others said, let's see the technical honey trap), that it's some other technical fault, etc. I highly doubt, just on intuition, that Facebook is running such broken bots: Come on, they'd do a better job than make something so easily detected.

It would be outrageous for FB to run the bots themselves. There's two other issues that are more realistic and unfortunately not so great for FB: they're aware of the bots and ignoring them in favor of the ad revenue or they're just becoming aware of the bots and will implement some fix which could result in a substantial decrease in ad revenue. Good to keep in mind that OP's 80% is just one data point and may not be accurate.

I actually think they were trying to say something strange is going on, which I think is pretty obvious given their stats, and that they have no idea who's doing it or why, partially because Facebook can't be bothered to look into it or at least get back to them about it.

That to me sounds like FB is aware of the issue and isn't interested in discussing it. That's the subtext imo.

My own suspicion would be because someone is gaming their ads and they haven't been able to prevent it as of yet, and not because they themselves are gaming the numbers. No doubt they believe that discussing it is tacit admission that they haven't been able to prevent it, which is bad for business.

I see what you're saying, it just isn't very hard for me to take them at their word: They're not claiming that it's Facebook, not out of some sense of propriety, but because they don't have any evidence that it is. It could be anyone.

And it doesn't matter who it is, because they're leaving for a different reason: Facebook never returns their calls unless it's with an offer of extortion.

While I doubt malicious intent on FB's part there's certainly a financial incentive to ignore this problem. How would their earnings reports change if they stopped charging at least some advertisers for 80% of clicks?

It would be a very, very short term financial incentive.

Advertisers monitor to see performance on the other end, and when the campaigns don't yield actual success they move on.

The only real exception to this are mega-brands that advertise just to build namespace. Everyone else expects direct action from some hefty percentage of their ads.

Are you saying it's short term because advertisers would notice? Sure some advertisers would notice. However I doubt the majority are capable of performing the OP's analysis (which could use some scrutiny).

Imagine a 100 person company with 5 people dedicated to advertising. Leadership says to increase on-line presence. Some recent college grad on the advertising team buys FB ads. Next month leadership is pleased that they're advertising on Facebook. Also leadership is pleased that sales increased x%. The FB ads could be completely worthless. Some non-trivial percentage of clicks could be from bots and no one would notice.

You act as if "everyone else" sees results from FB ads. They don't. This is a well known problem. They have a high rate of adoption from new sign-ons, and also a high level of departure. While the first number is greater than the second number, they are ok. Either they will manage to improve value and keep more of the advertisers, or one day the Ponzi scheme implodes.

You act as if "everyone else" sees results from FB ads. They don't.

Most ads target a specific outcome: You buy a product, create an account, joining a mailing list, etc. Even the most technically incompetent advertisers establishes these fundamental metrics such that they can say "I spent $1,000,000 on Facebook ads and only got ten signups. I spent $1,000,000 on iAds ads and got ten million signups." That sort of thing.

I think Hanlon's Razor is appropriate here: Never attribute to malice that which is adequately explained by stupidity.

I don't think so; if anything, the subtext is merely arguing that Facebook isn't doing anything to stop the problem. Whether that is just as bad as running the botnet itself, that is another thing.

Assuming the problem is as bad as many are saying, I personally think it's fair to draw the same conclusion you have -- running and letting run a botnet is one and the same. It's costing other businesses money; it's too big for Facebook to be able to play dumb on the issue.

Bots might be operated by competitors to increase the cost of advertising for Limited Pressing. This is the most likely scenario, in my opinion. However, though rather unlikely, the bots could be affiliated with Facebook itself, as all of those extra clicks translate into lots of extra Facebook revenue.

That doesn't seem like a very sustainable strategy though.

Does anyone know what kind (if any) of anti-bot measures Facebook already has in place for paid advertising?

could be ad spy tools...they crawl around clicking ads and following them to the landing pages and then follow the landing pages to the offer being promoted. Allows affiliates and other advertisers to reverse engineer the competition

I think the only fix for PPC is PPA.

Reply on news.ycombinator.com