The control group is out of control (2014)
slatestarcodex.com
slatestarcodex.com
https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-...
Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words:
> In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.
Do you do that or
publish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ?
Incentives matter yes. Science and data should be free and transparent. But if we pay peanuts and expect the monkeys to appreciate the applause of publication, we need to change our incentive structure.
The reasons AI / ML researchers don't mind publishing the data early is because they already get 500k salaries and equity.
The way I would expect this to work is, the grad students give you credit for gathering the data, which allows you to share in the prestige associated with all of their projects.
No?
[#] I believe that last one is in actual fact the Disney legal teams' opening gambit in all copyright negotiations. Then the gloves come off.
More plausibly, if your data is that apparent,
- you publish your first paper with your data, this takes 6 months from submission to publication
- you submit a paper every three weeks (also with your data) during that 6 months. that totals 8 papers before your article and data hit the press
- after your paper is published, it is 6 weeks before anybody fully gets the data
- during that time you submit your other two papers
- the grad students you mentioned now submit 8 papers, but are forced to cite your work
You now have 10 papers submitted before the competition and have 10-20 papers being submitted that cites your data and work. Total elapsed time = 14 months.
Sounds much better than your 5 year scheme.
In those conditions, going slower than a 50-grad-student chinese lab is not slacking off.
Add on twenty more papers a week from adjacent fields where I don't even know all the techniques and the magnitude of the problem hopefully becomes clearer.
It doesn't strike me as as very practical step. Pre-registration seems much more sensible.
No, pre-registration assures that when you do a study and get a negative result, you don't simply forget to publish it - helping to reduce publication bias.
But also, it prevents you from changing what you were looking for (because that's part of the pre-registration data) and "data dredging" for something that looks like a positive result, helping to reduce p-hacking - specifically of the "Texas sharpshooter fallacy" variety.
[1] https://richardwiseman.files.wordpress.com/2020/04/expt2.pdf , found on
https://richardwiseman.wordpress.com/research/parapsychology...
I had a drug program in which we blew away the standard placebo rate … unfortunately our placebo arm did too, though not by as much. But it was just enough that we weren’t enough better than our placebo arm.
We changed modality from the current standard (we were an injection rather than a tablet and all the trial participants were super excited.
>Weirdly enough, surgery’s invasiveness may explain some of its potency. Studies have shown that invasive procedures produce a stronger placebo effect than non-invasive ones, said researcher Jonas Bloch Thorlund of the University of Southern Denmark. A pill can provoke a placebo effect, but an injection produces an even stronger one. Cutting into someone appears to be more powerful still.
https://fivethirtyeight.com/features/surgery-is-one-hell-of-...
What if you supplement with a superficial incision and stitches, or a subdermal injection that causes some localized edema (and a dressing that obscures the fact there was no incision)?
You can go further and actually do a pointless surgery.
To be fair, your attitude is quite common in cash-strapped university departments - but it leads to bad science.
The FDA wanted us to compare against the topical treatment placebo rate for various technical reasons.
Perhaps you thought that “standard” was some sort of standard for any placebo for any indication and any mode of administration? Indeed that would make no sense.
Even specifying a placebo itself is non obvious. Of course your procedure and the object itself (tablet, whatever) is ideally indistinguishable by the study participant and, ideally the clinician (hard to do, say, when it’s a surgical procedure; you can’t hide that from the clinician). But even with a pill, you can’t always just supply, say, a slug of mannitol — that might actually affect what you’re studying.
Your trial gave an appropriate placebo, regardless of the "standard" in the field, and thus you were shielded from overstating the evidence of drug effectiveness.
We've all seen that person, and almost all of us have at times _been_ that person
And unfortunately, given the replication crisis, there is no reason to assume that a published scientist is not making a fundamental error that undermines the validity of their work. It apparently happens quite frequently, and we should be quite grateful for those that question and probe at assumptions and potentially faulty analysis techniques
"standard placebo" (tablets) << "new placebo format" (injection with no active components) < "real drug" (injection).
This tells me that Schlitz actually has psychic powers and Wiseman does not. It actually makes perfect sense. People who are psychic wouldn't be skeptics, and people who are not psychic certainly would be skeptics.
"It's elementary particles, my dear Watson."
“You see,” he explained, “I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things so that he has a difficulty in laying his hands upon it. Now the skillful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.”
“But the Solar System!” I protested.
“What the deuce is it to me?” he interrupted impatiently; “you say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work.
"It's alimentary, my dear Watson!"
“It's a lemon tree, my dear Watson!”
This sounds like fertile ground for further research. What are the boundaries of these powers? In what circumstances do they work, and when do they not? What if the trial participants had all been greeted by a 3rd person, not Schiltz himself? What if he were wearing gloves, or a hat, during the trial? What if the participants were?
I'm not making fun- wouldn't such experiments go towards discovering how such observed powers work?
Or that exposure to Schlitz's and Wiseman's pheromones activate/deactivate subjects' psychic powers respectively.
> People who are psychic wouldn't be skeptics, and people who are not psychic certainly would be skeptics.
Skeptics and psychics spontaneously form self reinforcing social groups! Blah blah gender of researchers affecting mouse studies blah blah women's menstrual cycles sync up in small communities blah blah metronomes blah blah spooky action at a distance blah blah many worlds blah.
Alternatively it is possible we all have psychic powers (but not all are conscious of it) and our thoughts actually shape our reality.
This is a surprising thing I wouldn't have thought of on my own.
In this universe, we do the same with numbers generated by ASICS or GPU: it mostly gives the power of money.
Also, if every profitable phenomenon in the world is already developed, then no business should ever invest in fundamental research - i.e., the pharmaceutical companies are wasting their research dollars.
Why is it comforting to you to believe psychic powers don't exist?
Just because something isn't particularly useful doesn't mean it cannot exist.
A little over 53% of days are up (green) days and yet you can’t use that 3x edge over a 51% predictor to make all the money.
> A little over 53% of days are up (green) days and yet you can’t use that 3x edge over a 51% predictor to make all the money.
You can make money, just not ALL the money. And that's because other people know that too.
Assuming these information sources are independent - even if the trends predicted the market in 99% and your prediction was only 51% - you can still combine them to get a better estimate than the generally known one.
IIRC, doing better than chance by 2^-32 is enough for the randomization system to be thrown out, and marching the output 51% is way better than that.
But yes, I’d agree with a steelmanned version of your point with a much smaller improvement over chance. But then, you would have to do a realllllly long test and it wouldn’t actually look that impressive.
So red queen effect [1]. Maybe telepathy evolves in tandem with resistance to telepathy.
So until the 1880s, there was no commercial exploitation of these compounds, yet they nevertheless existed either in actuality or potential. Once there was an understanding of the principles and the development of some related chemical and biochemical processing, this exploded and had a dramatic impact on human life.
If someone postulated in, say, 1785, the idea that perhaps there were specific compounds that could be used to cure infection, and these could be produced en masse, would it not be analogous to respond that since nobody is commercially exploiting (at the time), they clearly do not exist?
None of the ideas included in the xkcd comic are particularly new or obscure. Despite being pigeonholed as a certain kind of bullshit, they are still ideas that a lot of people have heard about and been hearing about for a very long time—more than enough time for someone to get around to commercially exploiting if that were in fact practical. Ghostbusters came out 37 years ago, and the stuff it made fun of was familiar enough to its audience.
However, in 1785 the principles of chemical engineering were largely unknown, precluding the development of a modern-style pharmaceutical industry. It did not, however, prevent widespread use of antibiotic treatments that we now understand the mechanism for and can refine or synthesize into more effective forms.
(2) It is intriguing to consider commercial music as an industrial utilization of lyric poetry, but the point is that the individual activity of lyric poetry is not amenable to industrialization IN ITSELF, but it still exists. But perhaps this example doesn't work -- the really important argument is above, namely, that the cultivation of these practices -- astrology, dousing, etc, is individuated, and oriented around virtues directly opposed to their mass utilization.
(3) As other commentators have noticed, we also don't know that these sciences aren't used on a mass technological/industrial fashion --- it seems likely, for instance, that there are corporate entities that use i ching divination, and likely, too, that there are financial firms that utilize astrological methods -- but wouldn't reveal this because of the likely opprobrium from a still dominant (if clearly declining) orthodox materialism...
But you could do that, or just make enough money gambling in poker rooms and racetracks.
Hell, if you could see a few days into the future, it might be smartest to just break that out when the lottery gets big enough one time.
It is known that many people who consider that they might have psychic powers are on the highly sensitive side.
Knowing people who are highly sensitive, I can easily imagine that the enormous stress, pressure and greed in a hedge fund environment would make hedge funds a horrible work environment for such people.
Maybe Simons is psychic and all the quant stuff is just a coverup.
Note, “just hold index funds” as a strategy should be dominated by “borrow a lot of money and put it in index funds” by the same arguments. Don’t see robos offering that though.
What about his track record cannot be adequately explained by mundane insider trading?
Edit: Puzzled by the downvotes. I think it's entirely fair to ask whether Cohen is enough of an outlier that it requires a psychic explanation, even after accounting for the degree of success that can be reasonably attributed to luck and non-psychic skill (effects which other hedge fund managers are also subject to) plus the insider trading he's been involved in (which other hedge fund managers are not necessarily subject to).
In a world with a finite number of hedge fund managers, there is always going to be somebody who is the most successful out of the bunch, and that somebody is almost certainly going to appear to be an outlier even without a supernatural influence on their success. So before attributing anything to psychic powers, we first have to establish whether he's too successful to be accounted for by the non-psychic explanations. I genuinely don't know enough about Steve Cohen's track record to know whether he's that much of an outlier, and I'd appreciate some real information about the degree of his success rather than just downvoting.
You can't prove they aren't using paranormal powers.
Ergo the case isn't closed yet.
But that's still a LOT less profit exploitation than e.g. GPS which is ubiquitous. Curses haven't replaced drones in the military. Crystals aren't in billions of electronic devices wait
A flipside example: Financial world could instead be used by individuals on casinos, betting, stock markets, etc. and they'd have incentive to keep it secret so they keep winning.
The more enormous the secret and the larger the group of people who supposedly are in on it, the more the probability of the secret not being a secret very long approaches one.
I find it very unlikely that a shortcut to making a lot of money would be secret for very long.
multiple and costly military invasions were to offer democracy as goodwill and for world safety.
a certain vaccine I can't dare to name is a vaccine as it does provide immunity.
the institutions of most democracies surely can't be so corrupted.
worldwide cocoa isn't mostly harvested by enslaved workers.
HIV is so far more prevalent in Africa because they can't afford treatment and understand prevention measures there.
JFK.
I could go on and on but not sure how many examples you would need to accept that the chances for enormous secrets known by even a significantly large group to not take a freaking long time before blowing up is rather close to zero. thus reconsidering your opinion on the existence of conspiracies.
and, about shortcuts to making a lot of money being kept rather well secrets: dark budgets, secret and hidden inflation, supply fudging
But I am not sure if we both mean the same things when we use the word "conspiracy". It's not a secret, for example. that cocolate has a slave labor problem. Planned obsolescence is not a secret (but it's more complex than just someone trying to screw you over). I don't think I understand the conspiracy aspect of the rest of your examples, save for the obvious JFK conspiracy theories.
> about shortcuts to making a lot of money being kept rather well secrets: dark budgets, secret and hidden inflation, supply fudging
I would put to you: If the chances of this blowing up is rather close to zero, how do you know about it ?
what is more straight to see is that you've just made a false attribution by distorting what I wrote about close to zero chances of a blow up. I don't question your intention and honesty, but read my comment more carefully, or quote verbatim without distortion, and resist the use of fallacies in general if you would like to engage in honest and fair argumentative discution.
planned obsolescence was a well kept from public industrial scheme, decided by a small group, screwing billions of consumers. it isn't a secret anymore, it blew up, it took several decades to blow up, kept being refuted by some who haven't caught up with the evidences.
and if you accept the JFK case then I suppose you are admitting to beleive in some conspiracies, while at the same time refuting the concept while it's enduring its long process of battle against deniers, during the inevitable period of evidences remaining arguably to few or too weak, simplifications of reality. consider at least accepting there is some contradiction this logic.
edit: had pressed send before finishing the comment.
It seems to help people relax going into surgery and improves outcomes.
Thus strengthening your argument that if it works, capitalism will exploit it.
A while back I looked at the CIA "Star Gate" files[1], which, are a collection of documents released under FOIA detailing the CIA's investigation into psychic powers and whether or not those powers serve an intelligence gathering purpose. I just opened up random files and skimmed them - I wish I had taken notes when doing so, but my impression was that the CIA was actually producing some better than chance results with their experiments.
I don't really know what to make of parapsychology or psychic powers in general. Sometimes I'll idly wonder if I should dedicate time to trying to uncover my psychic powers.
Did his studies get debunked or he is still able to prove things? And the other guy runs same studies and gets different result ? WhT ?
The Control Group Is Out of Control - https://news.ycombinator.com/item?id=7666575 - April 2014 (74 comments)
Isn't this an "isolated demand for rigor"? [1]
Doesn't this prove that science as practiced is not just the empirical process and some raw axioms that guide it (which these "psychic research" both follows), which scientists pretend to ascribe to, but also part blind faith/prejudice?
I'm not saying physic phenomena are real. I'm saying if tests are conducted with the same principles as those about physical phenomena, and results are equally carefully collected, and they have the same condidence, science in principle shouldn't treat them any differently than results about "regular" subjects.
[1] https://slatestarcodex.com/2014/08/14/beware-isolated-demand...
That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed.
Personally, I'm leaning heavily toward the second option. The methods in ESP are identical to methods in psychology, social sciences, etc, but not physics or chemistry. I'm just wondering where to draw the line. Maybe the whole method of statistical inference needs to be flushed? Maybe it cannot be saved by preregistration, publishing all the data, transparency and other bells and whistles.
That's my point them. "If we are going to treat ESP research the same as research in other fields", according to the principles science pays lip service to this shouldn't be a question to begin with, because science shouldn't have a priori ideas about what can be treated "the same as other fields" and what shouldn't.
Research (results) should determine that (which means: if there are results in its favor, and we can't see a problem with the research that other - non ESP - accepted research doesn't have, we should accept it, as much as we accept in other such - non ESP - results).
That's an a priori assumption though. Who arrived at the conclusion it was "extraordinary" and with what scientific methods? Not based on research, for one, because that's what we're evaluating here: research results positive to that claim. We could say: "We have a bias against it being true", but the "extraordinary claim" part is BS within the context of the scientific method.
As for applying Bayes, the "prior" (in fields where it's not mathematically determined, like e.g. judging whether a coin is fair), is just another name for "at will bias".
https://www.youtube.com/watch?v=hic18Xyk9is
and then later he gave a talk called "The Science Delusion" which got banned from TED:
BTW, YouTube title says “TED TALK”; note that it’s TEDx.
Therefore psychic phenomena exist? I don’t see the problem.
Maybe I am pedantic, but it should be "which most reasonable people don't believe exist"
You are allowed to expect that psychic phenomena being real would lead to the existence of clear evidence that psychic phenomena are real, and to conclude from the lack of such evidence where it ought to be found that it is less likely that psychic phenomena are real.
> Suppose you have a target that is half red and half blue; you are aiming for red. You would have to be very very confident in your dart skills to say there is only a one in a million chance you will miss it. But if there is a target that is 999,999 millionths red, and 1 millionth blue, then you do not have to be at all good at darts to say confidently that there is only a one in a million chance you will miss the red area.
> Suppose a Christian says “Jesus might be God. And he might not be God. 50-50 chance. So you would have to be incredibly overconfident to say you’re sure he isn’t.” The atheist might respond “The target is full of all of these zillions of hypotheses – Jesus is God, Allah is God, Ahura Mazda is God, Vishnu is God, a random guy we’ve never heard of is God. You are taking a tiny tiny submillimeter-sized fraction of a huge blue target, painting it red, and saying that because there are two regions of the target, a blue region and a red region, you have equal chance of hitting either.” Eliezer Yudkowsky calls this “privileging the hypothesis”. [0]
> There’s a tougher case. Suppose the Christian says “Okay, I’m not sure about Jesus. But either there is a Hell, or there isn’t. Fifty fifty. Right?”
> I think the argument against this is that there are way more ways for there not to be Hell than there are for there to be Hell. If you take a bunch of atoms and shake them up, they usually end up as not-Hell, in much the same way as the creationists’ fabled tornado-going-through-a-junkyard usually ends up as not-a-Boeing-747. For there to be Hell you have to have some kind of mechanism for judging good vs. evil – which is a small part of the space of all mechanisms, let alone the space of all things – some mechanism for diverting the souls of the evil to a specific place, which same, some mechanism for punishing them – again same – et cetera. Most universes won’t have Hell unless you go through a lot of work to put one there. Therefore, Hell existing is only a very tiny part of the target. Making this argument correctly would require an in-depth explanation of formalizations of Occam’s Razor, which is outside the scope of this essay but which you can find on the LW Sequences.
[0] https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privilegin...
subject: most reasonable people
object: psi phenomena
verbs: to believe, to exist
The full statement can be stated in at least two ways:
"[ these people ] don't believe that [ these phenomena ] exist."
"[ these people ] believe that [ these phenomena ] don't exist."
These are closely related things, but not identical, and I imagine the author likely chose carefully.
Yes, this is the answer to this entire article, and his attempt to dismiss this for some reason is strange to me.
I often wondered why this is, because he seems like a very smart guy with interesting ideas sometimes, so it seemed kind of weird and conflicting.
Then I learned he's trying to hide his beliefs about scientific racism and political views (1) while often writing in the tone of "pained centrist just wanting to understand things scientifically" and it made much more sense as (in the worst case) an old school propaganda technique of FUD in science and rigorous science to make the HBD go down better and (more charitably) as Mr. Siskind just being another bright-but-human person who when writing on things he has actual expertise on can be very well worth reading, but simply shouldn't be trusted on anything aside from this. Rationalwiki has a decent writeup on him (2)
1: https://web.archive.org/web/20210217195335/https://twitter.c...
I thought it was a fascinating write up about how if you assume ESP doesn't exist then parapsychology research has something important to say about the replication crisis in social sciences.
Going into how double blind solves all this means skipping the interesting bit: that parapsychology can be viewed as a control group for the scientific method.
It was the Sandifer article "The Beigeness" that cleared it up for me: https://www.eruditorumpress.com/blog/the-beigeness-or-how-to...
But what first got my attention (well, second, since the first thing was that Sandifer's article is as long and rambling as Alexander's is--both of them could have made what valid points they had with far fewer words) was this statement, a little earlier, about Scott Aaronson (referring to the comment in the blog post of Aaronson's that Alexander's article links to):
> Aaronson was declaring that, as a shy and nerdy man, he was a member of "one of society's least privileged classes" in response to someone saying they were a rape victim.
The reason this got my attention was that I had previously read the Aaronson post in question, and the comments, and I didn't remember any such thing. So I read through the post and comments again. The commenter that Aaronson was responding to when he talked about his personal history (the comment of his that Alexander quoted) did say she was a rape victim in that comment thread--but she said it after the quoted comment from Aaronson. Aaronson's quoted comment was in response to an earlier comment of hers, which did not say anything about her having been raped.
So Sandifer lied in her article about a fact that was key to her argument. I understand this is just one example, but there isn't room here for a multi-thousand-word critique and I don't have time to write one anyway. But the example I have given is not the only questionable thing I saw.
Amy did imply she was a victim of sexual assault previously, just not rape. So at the very least it was an error of degree. And regardless, Aaronson, had no reason to think he wasn't talking to a rape victim. The odds were something he surely knew. So I think Sandifer's correct that "this was neither the time nor the place to express them". And that's coming from a "shy and nerdy" guy.
Given that you're very up in arms about "questionable rhetorical tricks and shoddy reasoning", you'll see why I would expect you to do better than turning a reasonable and non-fatal error into a supposed lie about a key fact. But it is about what I'd expect from somebody working hard to defend a cryptofascist.
Obviously we are not going to find any common ground here if that's your opinion of Scott Alexander.
But if you can't talk to me or take responsibility for your own words because I have a negative opinion of one of your faves, that's a choice you're making.
https://en.wikipedia.org/wiki/Human_genetic_variation
but more this variation
https://en.wikipedia.org/wiki/Scientific_racism.
Even if it was correct, I think if there are no nefarious consequences being pushed for, I think people who want to argue HBD should rise to the challenge of explaining in plain english what they mean and what consequences they would draw from their assumption being correct, instead of hiding behind obfuscation. While I am not necessarily 100% aligned with groups like AntiFa chapters, BLM etc., at least they are honest about what they think and want to achieve.
For hundreds of years people have energetically misused science to justify both their racist feelings and systemic racism. And other people have showed up to disingenuously defend that in many ways, including your evasive, just-asking-questions routine.
There is no reason to think that activity has stopped. And indeed, the shift in the last 50 years to make overt racism moderately less socially acceptable suggests that we might see an increase in the amount of covert racism. So I feel no obligation to take the latest iteration any more seriously than I would the 1850s version. It's the same deal with perpetual motion machines or multi-level marketing schemes or religious door-knockers: the next one to come along could finally be the one that isn't horseshit. But that's not the way to bet.
If you'd like to make the argument that your personal favorite scientific racism is better science than the scientific racism favored by other racists, feel free to take a swing at it. But just looking over your comment history, I'm not going to give it a lot of weight.
Is it possible that one day somebody will come along with a real perpetual motion machine and I'll miss out for a while? Sure. But there are other people who enjoy evaluating that stuff enough that I'll eventually hear about it. And it's the same deal here. I don't get my theology from evangelical Christians, and I don't get my bioanthropology from evangelical racists.
The results? Schlitz’s trials found strong evidence of psychic powers, Wiseman’s trials found no evidence whatsoever.
Take a second to reflect on how this makes no sense. Two experimenters in the same laboratory, using the same apparatus, having no contact with the subjects except to introduce themselves and flip a few switches – and whether one or the other was there that day completely altered the result. For a good time, watch the gymnastics they have to do to in the paper to make this sound sufficiently sensical to even get published. This is the only journal article I’ve ever read where, in the part of the Discussion section where you’re supposed to propose possible reasons for your findings, both authors suggest maybe their co-author hacked into the computer and altered the results.
Well, if you would take the "psi can never exist" blinders off, Mr. Codex, then this seems to be a pointer towards "this works better if you believe it will work", and it makes perfect sense if you accept this.
Schlitz's belief that it would work resulted in it working. Wiseman's belief that it wouldn't resulted in it not working.
The whole idea behind psi is that the human mind can sense and/or effect the world in currently-inexplicable ways, and here it is doing just that. With at least ten other experiments on this idea getting similar results. But parapsychology is completely and utterly a scam in your eyes so that is not a result you will consider.