Hacker News Reader

Comment by AstralStorm | Hacker News Reader

Whoever choose letter p (suggesting probability) over s (significance) or e1 (type 1 error value) did a major disservice to the world of science.

P value is a measure of error (And not a probability but maybe one kind of likelihood), not evidence, how likely it is we are seeing the result of a chance. (How the chance is defined is important and hidden in the method used to compute the p value.) And any hard cutoff is really missing the point, there is a reason this is a number not a binary value.

Effect size estimate or risk ratio are much more useful numbers anyway.

View on HN

>"P value is a measure of error (And not a probability but maybe one kind of likelihood), not evidence, how likely it is we are seeing the result of a chance"

Nope, please stop trying to "help" people understand p-values without understanding them yourself. Reminds me of this just a few days ago: https://news.ycombinator.com/item?id=14646844

Honestly, most of the problem is people who don't know what they are talking about teaching statistics/science to each other... They have created an insane vortex of misinformation and ignorance that keeps sucking in new areas of research.

PS: If you or someone you know is doing research and don't recognize almost unbelievably huge problems with the way things are done, you are most likely part of the problem.

P is a kind of likelihood with very strong assumption. Likelihood of type 1 error (accidentally getting a result) if error distribution matches student t squared - if you are using the Fischer test to compute it. (ANOVA that is.) And if the hypotheses are independent which is supposed to be the case for null.

Specifically a logarithm of a likelihood ratio of two hypotheses given the assumption.

Note specific three ifs. Then read what the difference between likelihood and probability is before correcting people. Or what a difference is between likelihood and likelihood ratio is.

(Not much in case of null hypothesis testing, unless you didn't check the null to be true in untreated population. Or how true it is, as in didn't get likelihood for placebo. You get the actual likelihood approximation from Wilks' theorem after relating F distribution to chi squared.)

Now this fails if:

The actual distribution of the observations is not close to T or normal. (There are some more technical reasons, this is a shorthand.) For example multimodal or highly skewed. This is typically glossed over. (Will usually cause a false positive.) Has to be checked for null too.

The hypotheses are related. (Say multiple observations over different dosages.) There are different tests that should be used instead.

A few more technical reasons.

This is wrong in multiple ways. The p-value is not an error rate even if the null model is correct, "alpha" (the cutoff, usually 0.05) is the error rate.

The p-value also has nothing to do with "how likely it is we are seeing the result of a chance." As you note regarding the t-distribution, the p-value calculation assumes the null model is true (which not necessarily, but usually amounts to "chance did it").

How can a procedure both at once assume something is true and give you the probability it is true? You are also using the term "likelihood" in a strange way (it has a specific meaning when it comes to statistics). Anyway, your comments seem to indicate you hold two standard misinterpretations of p-values:

1 If P = .05, the null hypothesis has only a 5% chance of being true.

2 A nonsignificant difference (eg, P >= .05) means there is no difference between groups.

3 A statistically significant finding is clinically important.

4 Studies with P values on opposite sides of .05 are conflicting.

5 Studies with the same P value provide the same evidence against the null hypothesis.

6 P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis.

7 P = .05 and P <=.05 mean the same thing.

8 P values are properly written as inequalities (eg, “P <=.02” when P =.015)

9 P = .05 means that if you reject the null hypothesis, the probability of a type I error is only 5%.

10 With a P = .05 threshold for significance, the chance of a type I error will be 5%.

11 You should use a one-sided P value when you don’t care about a result in one direction, or a difference in

that direction is impossible.

12 A scientific conclusion or treatment policy should be based on whether or not the P value is significant.

https://www.ncbi.nlm.nih.gov/pubmed/18582619

What's wrong about #6? Is it just blurring the difference between "less than .05" and "only .05"?

#7 isn't clear to me either, though I'm guessing it's a pithy way of saying that two results that are both "significant" can easily have different evidential value.

There was a problem with pasting the symbols (= vs < vs <=, etc), I think I have it fixed now. Please check again.

Here are the reasons given by Goodman (2008):

>"Misconception #6: P = .05 means that we have observed data that would occur only 5% of the time under the null hypothesis. That this is not the case is seen immediately from the P value’s definition, the probability of the observed data, plus more extreme data, under the null hypothesis. The result with the P value of exactly .05 (or any other value) is the most probable of all the other possible results included in the “tail area” that defines the P value. The probability of any individual result is actually quite small, and Fisher said he threw in the rest of the tail area “as an approximation.” As we will see later in this chapter, the inclusion of these rarer outcomes poses serious logical and quantitative problems for the P value, and using comparative rather than single probabilities to measure evidence eliminates the need to include outcomes other than what was observed.

[...]

Misconception #7: P = .05 and P <= .05 mean the same thing. This misconception shows how diabolically difficult it is to either explain or understand P values. There is a big difference between these results in terms of weight of evidence, but because the same number (5%) is associated with each, that difference is literally impossible to communicate. It can be calculated and seen clearly only using a Bayesian evidence metric.16"

Thanks for liberating those quotes from the shackles of Elsevier (and fixing the parent).

About the letter 'p': I'd ban the word 'significant' instead if we're going to go there. You could say something like 'null-improbable' if you insisted on using NHST -- such a term would make it harder to forget that it's a fiddly technical concept.

Alpha is the error rate of the binary significance test if at all. Not related to any of the observations.

However, since p is a loglikelihood ratio, it is related to the observation and null themselves. To get actual likelihood you need to exponentiate, know the degrees of freedom and know one of the actual likelihoods. (Typically null is easier to come by as the other side is assumed to be null+treatment.) This exact likelihood is of course tiny. Which is why inequality is supposed to be used.

The process is not valid if any of the test's assumptions is violated in a big way.

Other than this, all the points are true.

How is the p-value a log-likelihood ratio?

A mathematical sleigh of hand depending on big N, chi-squared statistics and Wilks' theorem. This is obviously inaccurate for small N.

Otherwise p depends on the exact test employed. (Specifically the distributions employed. For Fischer's exact, hypergeometric, notoriously hard to calculate, but if you can, you can recover likelihoods too.)

A p-value can be calculated for any statistic. The p-value is often calculated using the sample average. I wouldn't say that it is an average, though.

The standard meaning of likelihood is P(data|Hypothesis)[1] where, for the purposes here, hypothesis will refer to "chance/accident generated the data". Do you agree with this? (I understand you say "kind of likelihood" because we are dealing with the inequality)

If so, can you clarify what you mean by "P is ... Likelihood of type 1 error (accidentally getting a result)..." ?

[1] https://en.wikipedia.org/wiki/Likelihood_function

The standard meaning of likelihood is P(parameters|data).

Source?

Actually I should have written L(parameters|data), I'm sorry for the confusion. Which is equal, for a given value of data and parameters, to P(data|parameters). But the likelihood function is not a function of the data (with the parameters fixed), it is a function of the parameters (with the data fixed). Its "meaning" is not a probability distribution of different outcomes given the hypothesis. But I maybe misinterpreted your comment.

It seems to me that model selection is usually the problem when good evidence in the data is discarded.

Most researchers are taught statistics using linear models and linear correlation. They learn how to use them well and they look statistical significance using linear models. Linear models match only linear patterns. Everyone knows about Anscombe's quartet, but it seems that there is very strong tendency to overdo linear models.

You sometimes see scatter plot that shows clear nonlinear interesting pattern that differs significantly from the null hypothesis, but authors use linear model to get P-values.

The letter p was chosen to suggest probability because the p value is a probability.

Reply on news.ycombinator.com