Hacker News Reader

Comment by iforgotpassword | Hacker News Reader

> The worst thing for me is how the Amazon search algorithm seems to want to show you everything but the item you searched for.

Hey boss I made the site better! Through rigorous A/B testing I could figure out a way to tweak our search algorithm so people spend much more time on our site! It seems they now really enjoy browsing for products!

Ok but seriously, I have witnessed A/B testing go wrong in the past so I'm biased to blame everything on it. I wouldn't think this particular thing happened though. :)

What I could imagine is that they measure number of items bought or money spent, but even then if eg you don't also track how much of these people return stuff later you still might draw the wrong conclusions. Figuring out that a user is less likely to use your site six months down the line due to building frustration is even harder.

View on HN

Amazon definitely tracks returns in their A/B tests, along with impact on long-term projections of customer value. What they also track is ad and sponsored products revenue. The sad truth with most Internet products is that advertisers are really good customers. They will pay you a lot of money with huge margins, and it's really hard for a business to say no to that.

My best guess is the algorithm has been tweaked to return exact results maybe 1/10 or 1/20 times, like a slot machine with the psychological manipulation and “reward centre activation” that comes with it.

> rigorous A/B testing

Also known as unethical, non-consensual human experimentation for profit maximization purposes.

I hate what advertising has done to the modern web just as much as anyone, but this strikes me as hyperbole. Does making this sort of claim not make you… tired? What’s the point of arguing like this?

Nazi Germany and the Tuskegee Experiment are examples of “unethical, non-consensual human experimentation”. A/B testing features of software usually doesn’t make the same list.

That's a bit of a false dichotomy. It doesn't have to be either Nazism or totally chill.

Godwin's Law.*

If you're from a certain background it's exactly as described. In academia, frankly probably everywhere but tech, experiments as a term of art require consent when they involve humans.

* n.b. you really should have left it out, it was a good post through "hyperbole", got close-minded in the next sentence, then just sort of blew the hatch doors off. Sometimes we just don't know something someone else knows. Not understanding someone else doesn't require they have a psychological condition, much less one worth noting.

Please stop. "Godwin's Law" is irrelevant bullshit. It's not a "law" and it doesn't prove anything, or do anything except add noise to the conversation.

Right? Parent is basically saying “wow it’s so unfortunate that you have forced me to end the conversation here, I’d have really liked to continue, but it’d be against the (entirely made up, by me) law”.

Where?

N.b. "godwin's Law" is the famous joke that as an Internet discussion approaches infinity length, hitler will be brought up

Not a secret message saying discussion over lol

The *

A bank sends out two different mailers to see which gets a higher response rate. A politician tests different versions of his stump speech to see which gets more applause. A standup comedian tries different variants of a joke to see which gets more laughs. A grocery store chain tests different store layouts to see which encourages more spending on expensive high margin items. A big box store tests different doorbuster sales to see which gets more people into the store. A city government tests whether changing a traffic light pattern decreases delays at the intersection.

Unless you’re a hermit you are an unwitting participant in nonconsensual human experiments on a daily basis.

Comedians should just pick one joke and stick to it, it's unethical otherwise as some people might miss the laughs and others might laugh too much.

I said the term of art thing to try to ward off a reply like this, I didn't want you to have to put in the effort.

Using weasel words doesn’t make your claim any more correct.

It's not "weasel words" -- there's a difference between an "experiment where city government changes traffic light patterns" and "experiment as in Institutional Review Board", and I suggest relaxing in general.

No, there’s not. Any of the examples I gave could be conducted by university researchers subject to the IRB, or by corporate/government researchers not subject to an IRB and informed consent requirements. When I worked in my university’s statistical consulting center in graduate school I could have consulted on the same experiment either subject to IRB or not depending on who the client was.

Thank you for the shift in tone: I'm honestly unsure what you mean, steelmanning: you worked as a consultant at a university and not all work you did involving experiments was for IRB experiments --- I guess what I'd say is, the fact you're able to make that distinction does seem to confirm my initial observation that the grandparent of my original post in this thread was drawing on IRB-style experiments to condemn excesses of colloquial-style experiments in tech.

No, there is no distinction between “IRB-style experiments” and “colloquial-style experiments.” Exactly the same experiment could be subject to IRB or not depending on who was running it. The distinction you’re trying to make does not exist.

> there is no distinction between “IRB-style experiments” and “colloquial-style experiments

Fascinating. How were you able to draw a distinction in your previous comment, then? :)

I didn’t. I drew a distinction between experiments subject to IRB and experiments not subject to IRB, not as a function of the type of experiment, but as a function of other factors—namely who is doing the experiment. I thought this was pretty clear:

> When I worked in my university’s statistical consulting center in graduate school I could have consulted on the same experiment either subject to IRB or not depending on who the client was.

(emphasis added). Somehow you misread it.

Yeah Im sorry, I definitely don't understand the significance of the client stuff. And it's on me.

I did have a similar job for 2 years (statistics assistant farmed out to help out on different grants as needed), but clearly not long enough.

I'm a bit flummoxed, though. I had the very distinct impression an IRB imposes certain requirements.

I shouldn't even call it an impression, you're aware of it too.

Like, an experiment under the IRB has certain tasks others don't.

I don't understand what I'm missing or what's missing in our communication here.

I almost called up an old prof to ask but any question I could think of, I sound high ("does an IRB supervised experiment have different requirements from, say, a city changing traffic light timings on a street?")

Are you just trying to say in theory an IRB could always impose no requirements other than talking to the IRB, and the IRB considered human subjects and say "go ahead, ethical"?

Note that's still a distinction. FWIW that doesn't happen in tech, no IRB, no reviews of experiments. Infamously this caused some issues at Facebook

> Does making this sort of claim not make you… tired?

Nope. Being used as an unwitting guinea pig for a trillion dollar corporation sure as hell makes me tired though. It's extremely tiresome and demoralizing, knowing that just so much as browsing their website contributes to their profits.

Basically we should have to consent for them to profit off of us in any way.

You are consenting to the experimentation by accessing the software. It’s covered by the terms and conditions.

Terms and conditions? It's hard to tell when people are joking. https://www.youtube.com/watch?v=AhsyZ5V1pok

> terms and conditions

Nobody reads that stuff.

I don't expect you to back off of your take, but you should really consider how and why you came to this conclusion.

If I put two different marketing messages on two different billboards to test whether one is more effective than the other, is that unethical non-consensual experimentation? If not, how is it different from A/B testing?

In strict terms yes, if you didn't get informed consent from your test subjects that would be unethical.

Research has a lot of policies and systems set up to ensure that if your testing involves people, you must get informed consent from the persons before even trying to do the test, and it's really not hard to imagine why this is a stringent standard -- it's very easy to miss how "simple tests" can and often are adverse to those participating in the test or have unintended consequences that the researchers didn't accommodate for, regardless of the reason they did not.

Ads are often portrayed as harmless but, like, there's a reason there are restrictions on advertising for certain highly addictive products and regulations against false or misleading advertising, or certain tactics aren't allowed.

I think that's the crux of the matter here. A/B testing can be anything from which page layout leads people to complete their shopping check out process to which ad campaign has the best ad click through rate. The former is pretty inoffensive, but the latter could be bad if it involves gambling/alcoholic beverage ads to people with gambling addiction or alcoholism, for example.

> unethical

If this is based on the possibility that one or more of the ads is harmful, how is it less ethical than the time-honored alternative, which is skipping the study and just running the ads?

It's not. Ads are inherently unethical and it should be illegal to run them anywhere.

> you should really consider how and why you came to this conclusion.

Why? Is it false?

I get that reaction a lot. People have directly called me unhinged. I don't really mind.

Many of my ideas I developed by discussing this stuff with people on this site. I guess not every idea is socially acceptable. That's fine. I still want to express them.

> is that unethical non-consensual experimentation?

Yes. It's really not any different than some published psychology experiment. In fact it's much larger in scale, has much uglier interests behind it, has proprietary and unpublished results. Social sciences wish they could get away with shit like this!

Only reason it's "legitimate" is everyone depends on it to make their millions. Because money excuses everything. Just like unending amounts of first party malware corporations ship to users on a daily basis. We used to recognize that stuff as the malware it is: adware, spyware. But then corporations started doing the same thing and suddenly it's "legitimate" because they put some clause in some terms nobody reads.

You know the Nazis had pieces of flair that they made the Jews wear.

Narrowly true, but what's the difference between this and a diner trying out new blueberry pancake recipes?

It's unethical for a diner to try out new recipes. Per OpenAI policy you need to ask consent before trying out new recipes.

We do non-consensual human experimentation all the time. Whenever you try a new outfit you're doing it.

It's an extremely broad category that contains good things (installing cycle lanes to see if they encourage cycling), neutral things (making both flower mugs and wave mugs and seeing which sells) and bad things (use your imagination).

I wouldn't make as strong a claim as the parent comment myself, but someone pointed out to me recently that A/B testing is really similar to cold reading. Is it morally equivalent to suggest to someone, in bad faith, that you're able to deliver messages from their dead loved ones, and to perform an A/B test of switching around menu items or change up some language to try and get fewer people to abandon their carts?

I lean towards "no" but I have trouble either accepting or rejecting the proposition. It's hard for me to say that A/B testing is done in bad faith, but it's also hard for me to say it's entirely unmanipulative, either.

Reply on news.ycombinator.com