Dogs are sensitive to small variations of the Earth’s magnetic field (2013)
frontiersinzoology.biomedcentral.com
frontiersinzoology.biomedcentral.com
and 10 months ago (44 comments) https://news.ycombinator.com/item?id=32736851
A good blog post on the seeming ridiculousness of the claim: https://www.shapeoperator.com/2020/02/15/how-to-lie-with-uni...
The authors are pretty clear that they didn’t find what they were looking for and so they just tried whatever they could with the remaining data and landed on dog pooping orientation.
When I put it like that, it evokes more pity than outrage. Imagine what they had to go through to collect enough data.
Last time I looked into it, I couldn't understand why it was seen as a problem.
The problem is if what you actually did wasn't one experiment, but 1000 simultaneous experiments. That happens quickly if you start looking for correlations in your data. The issue is that if you then use the same 1% threshold for significance, on average 10 experiments will clear that threshold by pure chance. Your statistical test has become worthless. That's fine if you account for it, but if you don't and just publish those 10 results that's called p-hacking. People hate it because it makes purely random results look as if they were significant.
You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring.
Well duh. It would be weird if you didn’t find such an outcome. This particular study openly acknowledged doing this, and also did some creative binning of their numbers to help nudge the stats.
You will always be able to find something. P hacking doesn’t mean the thing you found is false, but it is at best a sloppy estimate of something to explore further.
The more egregious examples of p hacking is adjusting outliers and filtering out specific datapoints until you can make the data claim whatever you want.
(I have this in my head when p-hacking comes up)
The real problem is when you set out to test 20 things, and they don’t work, so you test 20 more things; and then much worse when you only tell us about the analysis that lead to the 1/40 that mattered.
The thing about p hacking specifically is changing your analysis until it finds something that looks good. A lot of people don’t understand math and will do this and not understand that they’re gaming it. A lot of people will understand this and do it anyway because academia is soul crushing and they need to publish.
> You do your experiment. You don’t find any evidence you were correct. So you test 20 other ideas. Lo and behold, 1 out of the 20 new unregistered ideas passes the test of seeming to occur with less than a 1 in 20 probability of occurring.
I think this ^^^ is the part that's tripping me up: the notion of "registered ideas".
Suppose we have 21 researchers, each testing one of those 21 ideas. They pool their funds and hire someone to generate that one data set.
20 of those researchers would find their ideas unsupported by the data. But one researcher would find their idea was supported. And that's the only study that gets published in Nature.
I don't hear about this ^^^^ issue being criticized the same way P-hacking is. But AFAICT it's a similar kind of statistical factor.
I'd think that (the extent to which data support a particular model) is unrelated to (subjective details such as what hypothesis the researchers intended to test).
IIUC, the criticism of P-hacking hinges on such subjective details.
If the main goal is to find useful models, why would we care about the researchers' intent when originally designing the experiment?
Further a lot of science finds things pass the statistical test but are still not useful, which is a different kind of problem.
If you do an experiment and see something interesting you should probably follow up on it. But it’s the difference between saying “there’s a 5% chance that this is just sample bias” using basic methods and “there’s a 90% chance this is sample bias using methods that account for how many things we tried”. Who knows? Maybe your finding is novel, important, statistically strong, and useful, without you having planned to explore it. Maybe you accidentally found penicillin.
The two big problems is that people don’t know how to do these calcs (a lot of which don’t really make any sense because they try to assert independence into things where independence is impossible) and they don’t mention the things they tried which didn’t work which makes evaluating their claims impossible.
So is the fundamental issue that an experiment is probably designed to avoid bias for the data related to the hypothesis being tested, but probably isn't designed to avoid bias for other hypotheses you might generate once the results are in?
Think of sample bias as a dice roll. Every time you check a new hypothesis, you roll the die. If it rolls a 1, you get a false positive.
Pre registering a hundred different things you want to test doesn’t make that better, but it does create confidence that you didn’t continually test new ideas one at a time until number 100 was something that looked good.
Sure.
When your experiment has a P value of 0.05, what that means is that 5% of the time you expect completely incorrect results purely due to random chance.
So you test $THEORY, and expect that 1 out of every 20 times the results are going to be wrong purely due to random chance.
P-hacking is when you repeat the test 20 times[1], and publish that 1 random result while throwing away the other 19 results.
[1] Or however many times you need to
He works for the state department of land management or somesuch. Apparently they all dowse there. They even have a preferred "dowsing rod". A retractable thing like a radio antenna attached to a swivelling handle. (According to the dowser rather overpriced)
Compared to what?
I can almost guarantee that they aren't getting results any better than random chance.
I mean, really, if dowsing is every proven to work under rigorous scientific conditions, there'd be a Nobel prize in it for someone, because we'd have a whole new field of scientific study that has never before in the history of mankind produced any evidence of it's existence.
I mean you're naysaying the testimony of hundreds of field researchers on the weight of, let's face it, mere convention.
And these aren't necessarily dumb people. They value time and money just like we do
It does depend on the sensitivity of the dowser after all. That sounds squishy, like psychology. Certainly less obvious than weights and measures.
Who knows, maybe it takes a certain kind of person to do it.
Many hundreds of the most successful dowsers (i.e. the most sensitive) could not beat random chance.
Until they can, I'll remain skeptical.
Broad area mag surveys are flown in grids 80m above the ground with nine nano-tesla senistive magnetometers (wing tips and tail boom each have three on cartesian axis).
To produce a level map they also run local base stations to record the daily diurnal mag fluctuations to subtract and normalise.
That gets feed into the IGRF (International Geomagnetic Reference Field) and|or WMM (World Magnetic Model) to create parameters for five year epoh predictions.
On a smaller scale many archeology groups and or soil engineers run smaller tighter grids dragging skids of walking arrays across an acre or more to create sub ground imaging.
https://www.ncei.noaa.gov/products/international-geomagnetic...
https://geomag.bgs.ac.uk/research/modelling/IGRF.html
https://www.ncei.noaa.gov/products/world-magnetic-model
Here's some broad area airborne magnetic maps:
https://www.energymining.sa.gov.au/industry/geological-surve...
Perhaps another could be a trusted, HIPAA-compliant third-party log service for raw data that's immutable with a blockchain-like hash and timestamp.