Doubts over a ‘possible sign of life’ on Venus
sciencenews.org
sciencenews.org
Numerous comments about the promoter's non-scientific background, the sensationalism of the original announcement and skepticism over the findings. Please note that the scientists themselves weren't sensationalizing their findings, but others were.
As this article points out: it was a "big, if true" story that had numerous red flags from the start.
It also means just about no one read the real paper...a twelfth-order polynomial would certainly be a red flag for me.
Interesting moral here for interpreting science: surprising results don't usually have interesting explanations. It's usually a bug.
I made a comment in that thread, but I didn't notice the 12 degree polynomial.
Perhaps I'm too optimistic, but I expect someone to read the paper and make a comment about that in HN.
Making any claims based on that kind of curve fitting is a huge red flag, especially if you don't discuss that.
One of the reasons we would prefer lower-order fits to higher-order fits is that there is a very real risk of overfitting in the interior of a data set but then providing completely inaccurate results anywhere but at the data points. Seeing that kind of overfitting in a scientific paper without any justification suggests that the author of the paper is making statements without any basis in science.
If you want to make it simple, think of using a 12th order curve as being similar to picking whatever 12 points you want to match and then drawing lines from point to point.
We later discovered that the tame functions that usually work well for extrapolating from a sample are things called splines.
It like saying that a rule with as many special cases as data points is not actually a general rule. It is just a collection of special cases that explains nothing.
More precisely, if you give a model enough parameters, you can always have it describe your data well. The question is whether it is likely to predict future data. And the answer is no.
See the last example here: https://xkcd.com/2048/
Highest I know is Lighthill's (aptly called) eighth power law which says that the sound power created by a turbulent flow scales with the eight power of the characteristic turbulent velocity: https://en.wikipedia.org/wiki/Lighthill%27s_eighth_power_law, anyone knows something higher?
The difference is that in my example and in your example there are only a few coefficients to tweak to fit the data. So the shape of the curve if fixed and it is very difficult to overfit the data.
In the paper they used a full polynomial of degree 12, that has 13 coefficients to tweak and it is very easy to get weird shapes.
> they use a 3rd degree poly to remove the noise it seems... not to fit very well
They are not trying to fit the noise, they are trying to fit the hidden smooth signal that if hidden by the noise. In some cases it is difficult to make a formula for the real signal, so you can approximate it locally with a polynomial.
The idea is that after you subtract the smooth part, you get only the noise. So you can calculate the expected noise level.
And if the "noise" has a big peak, you can guess there is something strange, like a big absorption line.
See also my other comment: https://news.ycombinator.com/item?id=24985680
EDIT: Also see the last panel of this XKCD: https://xkcd.com/2048/
(a) a smooth function that is not interesting
(b) noise
(c) a big unexpected peak in an interesting part
There is a nice preprint posted a few days ago: https://arxiv.org/abs/2010.09761 https://arxiv.org/abs/2010.09761.pdf
They reanalyze the data than in the phosphine paper.
---
If you look at figure 3, they have the data that is the skyscraper-like line, and they use a polynomial of degree 3 to approximate the signal, that is that curved unhappy smooth line.
The smooth line is (a), when you subtract this smooth function, you get the other part of figure 3, that is the noise (b). There are some high and low parts, but nothing too high or low that look special. So their conclusion is that there is no interesting part (c).
---
If you look at figure 2, top left, they use a slightly smaller interval, but now they use a polynomial of degree 12 instead of a polynomial of degree 3. This is the smooth function (a).
This is a reconstruction of the process in the original paper. They fit the polynomial using the data, but excluding the central part.
The problem with the polynomial of degree 12 is that is has too much freedom, so it fits the actual smooth curve, but it also fit the noise.
The polynomial of degree 3 has to go somewhat in the middle of the data, because it can't go up and down too many times. The polynomial of degree 12 can follow the local bumps and fit the noise.
When you subtract the polynomial of degree 12 you get the the graph in the third row, with the noise that is (b). It is copied in Figure 2, and it is very similar to the graph in the original paper.
Since the polynomial of degree 12 fit the noise, the noise is too small, so you underestimate the noise level.
And since the central part was skip in the fit, in some case you get a big bump like here. It is bigger than the apparent level of noise so it looks like an unexpected peak (c).
But here the problem is that you are comparing the peak with the surrounding noise level, but the noise level is underestimated because the polynomial of degree 12 overfit.
---
They repeat the same kind of analysis in other regions, and they get a few additional fake peaks. This are the other 5 graph it the top of Figure 3.
Often it's better to use a spline fit or other interpolation technique, once you find yourself needing to go beyond five- or six-degree polynomial fits.
Yeah right.
They were saying things like "please prove us wrong" while doing interview rounds, and (I'm sure) celebrating, loving and looking forward to more and more limelight, so strictly speaking they were not "sensationalizing their findings".
In a related story, OJ and Casey Anthony did not kill anyone.
Hah. Wish that had been more emphasized in the original popular reports.
https://fermatslibrary.com/s/drawing-an-elephant-with-four-c...
It's perhaps less clever or cool than the hack above (by virtue of us being used to Turing machines), but serves as a friendly reminder that you can encode a lot of information in a long enough number.
Consider the two sequences:
1 2 3 4 5 6
And
6 4 6 3 8 2
Do these sequences really have equivalent entropy? Welcome to algorithmic information theory :)
Yes, algorithmic information theory is a thing. But neither it nor your example refute what I said (because what I said is in fact true).
Sinclair: https://en.wikipedia.org/wiki/Sinclair_Coefficients
Wilks: https://en.wikipedia.org/wiki/Wilks_Coefficient
discontinuities for men in Wolfram Alpha https://www.wolframalpha.com/input/?i=discontinuities+in+y%3...
discontinuities for women https://www.wolframalpha.com/input/?i=discontinuities+in+y%3...
A simple calculation of number of parameters in the human brain divided by seconds of lifetime yields: 10^14 synapses / (72.6 * 365 * 24 * 3600 seconds) =~ 43000 synapses/second, not all of them related to language and reasoning.
So we use tens of thousands of synapses for each second of our life, which is quite a contrast to the 100 bits of information per second which is estimated for conscious processing.
One reason people stall investigations is they know the press will forget after a few months.
I googled for some evidence:
Our climate data was wrong, which shows how science works: https://scripps.ucsd.edu/news/research-highlight-pointing-ou...
Re the replication crisis: "Negative opinions about psychology as a science already exist. We have the opportunity to show how science works by directly investigating and addressing the problems." https://www.psychologicalscience.org/observer/champions-of-p...
I didn't find any others, but if you doubt my thesis, that's just the magic of science at work!
You say this as if it's a small feat. The willingness to rollback a claim you made, ambitious or mundane, is surprisingly difficult for people to do. Most of science is about making claims based on the best evidence that you have with some level of confidence.
Where we run into problems, just look in the news, is when claims become invariant, regardless of new data, methods, theories.
And it's hard to argue that the social structure of contemporary scientific discovery doesn't incentives this sort of sloppiness.
There is a difference, but since both kinds of errors are widespread among humans, and science addresses both problems with a common solution, and nonscientific methods of generating systems of belief about the material world are prone to error due to their failures to correct both problems, I don't think there's really that material of a difference unless you mean “you” in the narrow individual sense of the specific actor making the initial error, in which case—and only in which case—is the difference at all significant.
This is such a non statement. Climate always changes. The correct statement is "the earth is definitely warming".
The term "climate change" almost always refers to the current rapid changes over a century as opposed to natural rates of change; if someone misunderstands that I suspect that it is because they have decided to misunderstand it.
Don't telescopes have some lens coatings that could dampen certain wavelengths? Or some form of giant sun glasses?
I don't understand how you would approach removing noise basically. A 12th order polynomical can fit many, many curves with really good detail already. But what signal were they trying to interpolate?
which turned out to be so completely wrong it's ridiculous:
https://www.nature.com/news/arsenic-life-bacterium-prefers-p...
Coupled with NASA's constant touting of crappy life science aboard ISS, I've more or less just tuned them out (with a prior probability of 0.01).
What happened though was that they went public early with too little information, because they were worried about being scooped in the media. When they got asked questions that they didn’t have answers to, the result started to look like it was on shaky grounds. The opinion was formed that it was a bad result. After that it was pretty hard to get anyone to pay attention after that. Follow-on results pretty much confirmed the original result though.
(Astrobiology was my field ~10 years ago.)
https://spaceflightnow.com/news/n1001/09marslife/
I think there's a more recent book that came out a few years ago, but a quick google search isn't finding the name, sorry.
For Wikipedia which relies on citations, it's going to reflect the majority opinion which is "ALH84001 is insufficient evidence for life." However the reasoning for saying that morphological features are insufficient signs of life (in contrast with biologists studying Earth life who routinely rely on such evidence) is actually a direct response to ALH84001. Like the Pluto-Is-A-Planet? debacle, the generally accepted requirements and definitions were drawn up after the fact specifically to exclude ALH84001, in order to end an embarrassing controversy and provide funding for more missions (why go to Mars when you can just pick up rocks in Antarctica instead?) rather than answer a scientific question.
There just isn't enough information here to conclude anything so it's easier to reject the hypothesis (I have to admit, it took me a while to accept the idea that there is that much mass exchange between planets in the first place!)
Still, in the years that followed further analysis was done by the original authors and their teams. Isotope ratios and captured atmosphere within and around the tracks confirmed that the fossil features were ancient and of Martian origin. Comparative characterization of terrestrial rock-worm fossils was done and it was noted that like the Earth fossils--and unlike abiotic processes--the ALH84001 features were of uniform size and were anisotropic on the sub-microscopic scale, consistent with fossilization of biological structures rather than geologic crystallization processes. Finally, testing showed that the inside of the ALH84001 meteorite itself never exceeded 40 degrees when it was blasted from the Martian surface, or when it entered Earth's atmosphere, so any biological life deep within the rocks could have survived (no one thought the fossils were live, but by implication Mars could have, and almost certainly was seeded by Earth, making it not surprising that we would see a the telltale signs of Earth microbes in a rock from Mars.
TL;DR: At the time of the press conference and the weeks after, a bunch of reasonable questions were raised about the ALH84001 work. In the years that followed a few of the scientists involved followed up with further tests which confirmed or at least did not disprove their martian-life interpretation of the fossils. But the rest of the world moved on.
However, touting very surprising, very unsure results using the megaphone of media, is not science. It is clickbait. It is BS. And it is intensely damaging to science.
What happens is there is a selection effect. The most surprising findings are the ones that are most likely to be touted by the media. However, the most surprising results are the most likely to be wrong. So over time, you end up with the result that much of the "science" touted by the mainstream media ends up being wrong.
And we cannot just blame this phenomenon on the media. Many times it is the institutions that are supporting the research that are contacting the media and putting out press releases and getting on twitter with very clickbaity summaries of the research.
It is great for the scientist. It makes them famous for a day. And as with most stories, the original story gets a lot more play than the correction.
However, it is very damaging to our society. The regular person sees science as a stream of exciting results that are later proven wrong. All the while people are shouting, "This is fine, because this is how science works." And it damages trust.
Imagine if you went to your doctor for a routine blood test and the doctor came back and said, "You have an elevated white blood cell count. This could mean you have cancer." Then a couple of days later, you got called back in to be told, "Well actually, the count was increased because you likely had a cold. But this is how medicine works. One test can suggest something, but different tests can give a different explanation."
If this scenario happened on a regular basis, your confidence in medicine would really be messed up.
If we present science as this ever changing, always being proven wrong entity, then how can we ask people to depend on "science" and make potentially very expensive and inconvenient adjustments to their lives such as with COVID or climate change?
People often ask what HN was like in the good old days. I don't remember drivel like this in the first 5 years.
I think the underlying reason for these types of threads is that HNers want to virtue signal that they're interested in science, but too lazy to do the actual study and work to understand and contribute. Pro tip: just follow the Twitter NASA channel. You'll get pretty pictures, with no real effort.
An analogy is when the Linux kernel team started a kernel newbie project, and got nothing but white space patches. So the project had to be abandoned because of low quality and the burden of wasting time for, you know, actual kernel contributors.
Science is science when an author can be removed from a study and the experiment can be repeated. A 12 year old or science hater should be able to reproduce the results if it's science.
Things that aren't science-
>Authority- A PhD/medical doctor/a scientist, these are human opinions, not science. They can cite studies to be scientific.
>Art
>Tradition
There's value in non scientific disciplines, but they have their own burdens to be aware of.
As a side note, I'd push back against the idea that replicability is the key ingredient of science: it's neither necessary nor sufficient to discover scientifically "true" knowledge, though it's definitely a nice to have. Plenty of important measurements have been made confirming theories we see as true today (or true enough) that turn out to have been irreproducible, plagued by experimenter bias, or even outright fraudulent. But it's a particular research programme, embedded in a social context, that generates knowledge, not a single reproducible experiment.
Can you give an example of something that isn't reproducible but was still true?
Of course there is an epistemological distinction here between a thing that cannot be reproduced by its nature vs. one that cannot be reproduced by societal norms.
He excluded values to reduce his error (seems like), but that doesn't make the experiment non-reproducible. You can do the same experiment today and get an approximate value for e.
Will you get the exact same results as Millikan? Of course not. But that's not what reproducible means.
Reproducible means that if you follow the method outlined by Millikan, you should get an approximate value for e. This is indeed the case, so the experiment is reproducible.
A non-reproducible experiment would be one where you follow Millikan's method but get something that is nowhere close to being a value for e.
ETA: I encourage the interested to read https://plato.stanford.edu/entries/scientific-reproducibilit... which delves into these semantic issues.
Not every experiment needs to be reproducible in exactly the same way, just like not every study needs to be double-blind. You need to interrogate the reason the experiment exists in the first place.
I guess another way of saying it is this: even though Millikan's actual final result is not reproducible because he fudged his numbers, the scientific method employed in the experiment is valid, which is really all you need in the case of that experiment.
The problem with medicine / psychology / other less "hard" research is that in many instances it doesn't matter if the method is reproducible if the results are not. If the goal is to show that eating pancakes cause depression the result is actually all that matters.
Thanks for the link, I will give it a read.
This link clearly talks about reproducible results, not reproducible data.
Wherever there is some degree of randomness, we would not expect to get the same data, but we would expect to draw the same conclusion.
See also: the Mendelian paradox - it is sometimes argued that Gregor Mendel's results were too perfect, and yet he is honoured as the father of genetics.
To me it seems you're conflating reproducibility with insight. Of course insight is also key, and it may be useful even if it's wrong, as long as it causes others to recursively have new insights and create hypotheses and do experiments that eventually turn out to be true.
- we want to be able to understand and predict things
- we can come up with various explanations that might be able to do this
- if our explanation fails to predict an observation, it is wrong and we need to update or replace it
If you're doing that, you're doing science.
I would say some of the ancients did science too, to the best of their knowledge.
This passage reminds me of how neural nets learn and often fail to generalise.
" Guess -> Compute Consequences -> Compare to Nature / Experiment / Experience / Observation
If it disagrees with experiment, it's wrong. In that simple statement is the key to science."
science has told us so far that masks might help at the margins, but hasn’t proven it, which indicates it probably shouldn’t be counted on as a primary mitigative measure, but the public signaling value is just too irresistible for the believers (transmission principally happens in private where social norms oppose mitigative measures).
in other words, expertise is narrow, and treatment experience isn't research.
Somewhat disagree. That particular research programme must still have reproducible experiment as a unit. A million irreproducible experiments will generate exactly zero knowledge.
And if you knew what needs to be put in, you would not need the simulation in the first place.
Note that a simulation is not the same as a model. A model is tested and refined based on real world data, until the model returns the same results.
Well, that's going to be pretty limiting. A 12yo isn't going to have the budget to build their own LHC -- reproducibility is certainly an important goal, but "a 12yo or science hater" is an unreasonable place to set the bar.
But we can educate others that we are the sign of life and we are the life. Take a breath, and cherish that.
At one point, it wouldn't matter if there is life outside our galaxy or life outside our solar system or outside our planet or outside us.
Only thing that will matter is there is life within us and our focus will shift there.
I guarantee you that scientists are as smart and capable as the best of the non-scientist HN population, and were the story roles reversed, that scientists on the whole would be more humble about the limits of their knowledge about programming practice and design, etc. They would not revel in the "idiocy" of your bugs, security lapses, or petty spy-based mobile apps. Well maybe the last.
Have some humility. Stop building up fragile egos by tearing others down, pretending that you wouldn't make similar "mistakes" or produce designs with similar weaknesses or limits. Edit 1:Yes I'll defend my tribe. Each down-vote is a badge.
I believe you are wrong in saying that scientists aren't idiots: they make colossal errors, obvious ones, all the time! Frankly, I think most successful (in terms of adulation and $$$) scientists are ones who learned how to play the games scientists play, and publish known incorrect work, or unknowingly publish overly strong estimates of their confidence, because they've mastered the incentive system, not because they're not idiots.
I see very little true humility amongst scientists- nearly every successful one I've met was aggresively egotistical.
That said, after another 10-15 years of being a SWE, I've come to the conclusion most SWEs are idiots, too- too focused on clever algorithms and microoptimizations.
As an extreme, let us suppose that we know that there are 6 sources of noise spread out over the x-axis, and that the power of the noise will decrease x units away from each source. In that case, you might suspect that noise function needs a 12th order polynomial to fit the undulation pattern, just to get the necessary number of humps.
The general population are idiots. The amount of scientists studying is huge compared to 40 years ago. It has to be full of idiots.
We are not talking mistakes. We are talking fundamental structural flaws. The fact people in the scientific community defend themselves with rhetoric show they are idiots.
If scientists weren't such idiots they would attack the tech community and how useless it is. But it's scientists who support the AI lies for instance. They are a big part of the problem in tech.