Comment by p0pcult | Hacker News Reader

Comment by p0pcult | Hacker News Reader

Parent Full thread

<edit> I should point out that we were measuring brick & mortar sales, not clickthroughs. So no cross site tracking involved. Maybe that's the easiest way to clear things up for you? </edit>

If you look at a distribution or histogram of ROIs from our thousands of studies, you will see that it approximates a gaussian distribution...right up until the left hand side hits $1.00, where it looks like the distribution hit a brick wall.

Same thing for a distribution of incremental lifts, except that the brick wall is right above 0% lift.

It's a discontinuity that shouldn't exist, like the Polish high school exit exams. See if you can guess what the minimum score to pass* was:

https://imgur.com/4nnbyru

Except, unlike that distribution, there is virtually nothing to the left of the discontinuity at my company.

Without talking about the tech behind it, what I can say is that the fraud is not occurring on the impressions and tracking side. It's in the statistical tools that infer the incrementality of the purchasing behavior. Seriously, I urge you to listen to the Freakonomics episodes I mentioned.

*The minimum score was 30. This is pretty damning evidence that teachers were juicing the tests of kids who filed by one or two questions.

Thanks for the info. Doing brick and mortar sales attribution is a whole different ball-game.

Frankly the number of dots that have to be connected between different systems in order to make that stuff work is too difficult for me to trust the data. If I'm going to trust a black box, I'm not going to trust programmatic exchanges and attribution vendors. I'm going to trust platforms that have vertical integration at as many stages as possible.

So, Google basically.

> we were measuring brick & mortar sales, not clickthroughs

I think this is probably where you get the room to juice the numbers like this. If you were tracking online conversions, people would easily be able to compare with an incrementality study run by the provider (say Google) and see that you are lying/wrong.

Yes, and the fact that "incrementality" is inherently unobservable.

Disagree. Why do you say that?

Incrementality, as defined in my original post, is the sales that would have not occurred, had the ad not occurred.

It is impossible to actually observe what would have happened, had there been no advertisement, because there in fact was an advertisement.

Thus, the estimation of incrementality must be inferred statistically.

> It is impossible to actually observe what would have happened, had there been no advertisement, because there in fact was an advertisement.

I can see how you would have that perspective as a third-party analyst of conversion.

It's actually easy to do if you are the exchange/sell-side provider of the advertising space. In your logic, where the advertiser requesting the incrementality study would have won the auction, just randomly remove their bid from some won auctions, have the second highest bidder win, and then track that user as the "control" for conversion purposes vs. users you did display the ad to. Voilà!

This is how Google in-house incrementality studies are run, for instance.

Replying here, because I cant reply to your comment several levels down.

I can see how you would have that perspective as a third-party analyst of conversion.

It's actually easy to do if you are the exchange/sell-side provider of the advertising space. In your logic, where the advertiser running the incrementality study would have won the auction, just randomly remove their bid from some won auctions, have the second highest bidder win, and then track that user as the "control" for conversion purposes vs. users you did display the ad to. Voilà!

That's not observation. That's statistical inference. I am referring, strictly, to direct, empiricial observation. With any kind of statistical inference, you get into a pissing match about assumptions, model quality, etc... Is the "control" a good stand-in for the behavior of the "test", etc.

Incrementality is able to be estimated, but unobservable. And therein lies the potential for fraud.

Sure, if by unobservable you mean in the same way that we can't truly "observe" the population average height of humans, even if we can probably get a pretty darn good estimate.

You can observe the results of the experiment and then make statistical inference about the "true" incrementality. You don't even have to perform inference necessarily, you can just provide the direct results of your experiment to the advertiser and let them do the inference.

But I do fail to see how a third-party could possibly infer this measure, given that they have no capacity to do a true incrementality experiment on the platform.

Are you running the incrementality study, or is your provider (e.g. Google)? Because one of the points here is that providers also have an incentive to fudge the numbers - they're not in business of being accurate, they're in business of making it seem their services help you.

To riff off the average height example - you can get a pretty darn good estimate based on sampling and good statistical practice... unless the people doing the sampling have a reason to bias the measurement, or - if you're sampling on your own - people who control where you go and who you meet...

Oh that could definitely be the case, although I'll note that the larger actors have more to lose and are at more risk of being caught for engaging in fraud.

I think my usage of "you" was confusing because I work in sell-side advertising.

You nailed this. Tirole discusses this in his paper hierarchy on bueracracies. Your internal stakeholders aren't always aligned to the company, sometimes it is to a vendor.

Reply on news.ycombinator.com