r/statistics • u/Sunlightn1ng • 17d ago
Question [Q] Why is the p-value measured against 0.05?
This may have been asked before, but all I can find is explaining what it means. I want to know why we've chosen 0.05 and not 0.04 or 0.06 etc for a lot of disciplines.
Edit: Thank you!!! I was hoping for a more exciting answer than "bc this one guy a while back thought we should" but that's interesting too!!
132
u/Biofelip 17d ago edited 17d ago
Ok no one has mentioned this so far but it's actually the result of a copyright dispute between Fisher and Pearson. At the time, the only way to know the density of a function at specific quantiles was to look at a table. Pearson owned the rights to the most used statistical tables and apparently refused the proposal of Fisher of including some of them in his SMRW book. Consequently, Fisher could only include a reduced version of the tables with values he considered important and focus the discussion in these values. In reality Fisher's approach to p values was always more on the side of a continuous measure of evidence against hypothesis rather than strict thresholds, but without the tables it was impossible to suggest a workflow like that om his book so he went with the now commonly known thresholds.
This article about it very cool
Scientific Apophenia in Strategic Management Research
17
u/eager_wayfarer 16d ago
Rights for statistical tables is just diabolical lmao
21
u/2spam2care2 16d ago
i mean, back when “computer” was a profession instead of a machine, those numbers represented a substantial investment in time and money. not saying it’s good, just an understandable consequence of capitalism.
4
6
100
u/Khornatejester 17d ago
It’s just convenient.
"The value for which P=0.05, or 1 in 20, is 1.96 or nearly 2; it is convenient to take this point as a limit in judging whether a deviation ought to be considered significant or not.”
Statistical Methods for Research Workers (1925)
82
40
u/LiesDamnLiesAndStatz 17d ago
Not only is it arbitrary, it isn’t always reasonable. What you really need to think about is the cost of a type I error and the cost of a type II error. If you lower type I you tend to increase type II, and vice versa. If the cost of type I is great, you want a lower alpha, but if cost of type II is greater you might want a larger alpha. In any case, the issue is avoided if you just specify p values and let the reader do their own interpretation.
8
u/Calm_Interaction_268 17d ago
I love your comment. I have not seen many people write on the connection between type I/II error.
Separately, the real issue is making actionable decisions off of arbitrary set p value thresholds.
3
25
u/IaNterlI 17d ago
It's totally arbitrary and I think it came from Fisher himself as a reasonably low probability.
2
u/Unbearablefrequent 15d ago
Incorrect. Using 0.05 predates Fisher.....
1
u/IaNterlI 15d ago
Oh I didn't know! I thought he had written it somewhere, either a paper or a letter. Curious, where was being used before his time?
3
u/Unbearablefrequent 15d ago
By Francis Edgeworth. Even when Fisher was around, his adoption was from Gosset.
11
8
u/glavglavglav 17d ago
It actually deepnds on the field. In social sciences it makes sense to have it 0.05 (2 sigma), while in high energy physics it would be like 0.0000633 (4 sigma).
3
u/2spam2care2 16d ago
and often when doing child development research they use a threshold of 0.1, because children are just so inherently chaotic that it’s almost impossible to get a .05
1
u/glavglavglav 16d ago
Yep. When it comes to analysing specific results, I would not even be operating with the notion of a threshold. One has a dataset, calculates the p-value, reports it, and that's it. It depends on the context whether it is reasonable to include that dataset into further analysis. For example, if most of your data come with the p in the range of 0.01-0.03, then you probably won't trust the one with p=0.1. While if most of your data have p between 0.09-0.15, then p=0.1 is good enough.
4
u/Calm_Interaction_268 17d ago
I don’t think this is true of “it makes sense”, theoretically/universally speaking.
0
u/glavglavglav 17d ago
the notion of "makes sense" is the opposite of "theoretically/universally".
2
u/Calm_Interaction_268 17d ago
How? You may be right but to me I thought makes sense means like theoretically/universally?
1
u/glavglavglav 17d ago
Well, usually if something is "theoretically/universally/formally/exactly" correct, one uses these words explicitly. Because they would indicate that some sort of a mathematically rigorous procedure was applied. In contrast, "makes sense" would be in the same row with "intuition", "rule of thumb", "practical". This does not mean it is arbitrary or not right, quite the contrary. It just does not imply mathematical rigor.
6
u/Calm_Interaction_268 17d ago
Ya but how does a threshold p value of 0.05 make sense? Isn’t it just an arbitrary threshold value?
2
u/icantfindadangsn 17d ago
Lol I love how they were here for the pedantic points but as soon as you bring the actual question back to the front, c ya.
I'm with you. There's so "sense" in 0.05 in any field. It's arbitrary. And even if we were talking standard deviations, 0.05 isn't even 2 sigma. Maybe 0.05 is easier to cross in some fields that are e.g., prone to measurement error.
0
u/glavglavglav 16d ago
To understand "how" it makes sense, you first need to understand what the purpose of any threshold is. Then you should do a fair bit of working with practical data to understand what kind of patterns you may observe. Then you can try to see which values of this threshold serve best to your initial purpose. And then you will see "how" this all makes sense.
1
u/Calm_Interaction_268 16d ago
Very interesting!
Do you have a way to determine which initial purpose should be used/is better than another? So for example, should one prefer type I error reduction or type II error reduction? If the answer is it depends, can you give me some examples of both then?
Or is your answer essentially that you first pick any purpose, and then pick the threshold that best serves that purpose? If yes, are you able to pick any purpose and then tell me how a specific threshold value, whether 0.05 or something else, best serves that purpose?
1
u/glavglavglav 16d ago
1
u/Calm_Interaction_268 16d ago edited 16d ago
This makes my point beautifully.
Why set it as low as 0.0001? Why not 0.00001 or 0.000001? Why not make it 0? This example answers that question by saying that decreasing α increases β. However, now you are creating a loss function of how much should we accept the decrease of α and the increase of β.
And then your example makes my entire point. It says “perhaps 0.1”. It does not provide a reason that “makes sense” of why to set it at that threshold over another. Why 0.1? Why not 0.2? Why not 0.3?
Furthermore, even at the end, it says that the threshold should take into account the “relative costs of each type of error,” and that if the cost of a Type I error is high relative to a Type II error, α should be “relatively low,” and vice versa. But that is exactly my question. How low? How high? What determines the relative cost of a Type I versus a Type II error, and how do those relative costs give you a specific threshold?
If the only purpose in this example is minimizing Type I error, then the threshold that best serves that purpose is 0. If your only purpose is minimizing Type II error, then it is 1. The moment you choose something between 0 and 1 because you care about both, you have introduced a tradeoff between the two. So what determines that tradeoff and therefore the specific threshold?
→ More replies (0)
5
4
u/efrique 17d ago edited 16d ago
First thing to note is not every area that uses statistics is stuck in 0.05-land. Look at physics, for example with its "5σ" standard.
But many areas are, so here goes:
There's a decent paper on it here that someone posted to a previous thread on /r/askstatistics:
I have seen some other historical accounts with details not there but it's got a decent coverage; I recommend reading it.
While the origins of the use of 5% and 1% go back at least to Fisher and arguably predate him, once you have people making tables for distributions that have shape parameters (such as chi-squared with its degrees of freedom parameter, and the related t- and F-distributions) you will have those people seeking a small number of significance levels to make available (unless they are publishing a book devoted to tables), so if they see mention of 5% and 1% by some influential person(s), that is where they will start. In turn, people using said tables will then choose to use significance levels that are in the tables. And then those will get taught, since that's what's in the available tables, and you're firmly in the grip of technological inertia. There's been liittle need to rely on tables in real applications since before I took my first stats class in the early 80s, and so nothing to stop people making better choices for about the last 5 decades. However, many people don't want to invest effort in doing so when their intended readership already expects to see 5%; that's the nature of technological inertia.
5
4
6
u/naturalis99 17d ago
It's an accepted risk for "standard" simplistic modeling to have a p value be below 0.05 when it shouldn't be for 1 in 20 times from classical statistic times that just stuck. The idea is that every study should be repeated. But modernization sort of neglects that by having a publication bias towards original research. Which is sort of solved by Meta studies.
More advanced researchers will lower the alpha value based on the number of tests in their study.
Ever since I went from student to professional it has always bothered me how many of my peers in social studies would neglect this tough. They put a lot more emphasis on data-collection-bias and limitations. I think it simply because they fundamentally enjoy contemplating those issues more than mathematical issues. Until they run into p values between 0.01 and 0.05, then they always find me grining maliciously in the corner of their eyes :3
The alpha value and mathematical issues with modelling are taken a lot more serious in biostatistics because they are drowning in parameters. More so when they run into high dimensional data where there are more parameters than subjects.
8
1
u/naturalis99 17d ago
In short: setting the alpha to 0.05 is just a default. You can actually do with it whatever you want. Although you should probably include an argument for you deviation in the essay/report/paper/code-comment ;)
1
1
u/PadisarahTerminal 17d ago
I'd bet that it's more like most research in biology don't bother having a strong statistical study design and just do whatever. And that understanding statistics is hard.
7
u/CatOfGrey 17d ago
You'll see a lot of people answering different versions of "Arbitrary" or similar answers.
I don't know the citation, but I'm a litigation analyst - I do math for attorneys and legal cases. And there are various court cases where the p = 0.05 value is upheld as a reasonable standard of practice. In my particular area of work, estimates from certain types of samples require a '95% confidence band', which is often approximated by a '2-standard deviation range' or similar measure.
6
u/JarryBohnson 17d ago
I’m curious about this job, can you give me an example of when people debating the 0.05 threshold would come up in court?
3
u/CatOfGrey 17d ago
I've never had a debate in court (or in my usual practice, in litigation, with an opposing expert) about that particular topic.
However, some of my cases use data from questionnaires or 'surveys'. And I often worry that our response rate is too small, meaning that margins of error are too large. Thankfully, I have never had work discarded on that basis. But part of that success is because of a trade-off - a large margin of error means a smaller 'lower bound' or larger 'upper bound' on monetary damages, so it sometimes impacts the 'final dollar amount' of a case!
6
u/naturalis99 17d ago
Oh yeah, this an interesting one. Statistical analysis becomes a thing in convicting someone of crimes that are easily hidden or by definition out of sight. The best example I know is cases where nurses (or caretakers/docters/etc) are accused of murdering patients. Do they murder patients or are they just unlucky with the amount of patients that die on their watch? Statistics!
4
u/PortiaLynnTurlet 17d ago
I'm curious - how do you account for potential, unobserved causal factors in this kind of situation? Are there baselines for supply chain contamination, computer errors, etc. (for example)?
1
u/komodorian 17d ago
This feels like you need to answer whether the nurse is a murderer or it was straight up bad luck of being around when it happened.
(Your comment game me flashbacks of stats classes in uni)
4
u/EmsMTN 17d ago
That is alarming depending on the context.
2
u/CatOfGrey 17d ago
That's well said, and surprisingly precise.
My own catchphrase is something like "Although law intends to reflect common sense, it's rarely wise to assume that the law should actually reflect common sense."
Usually, it all works out pretty well. Expert witness are pretty good at explaining stuff to juries.
On the other hand, Supreme Court Justices can be shockingly ignorant, literally rejecting statistical evidence when relevant - the recent Voting Rights/Gerrymandering case comes to mind...
1
u/EmsMTN 17d ago
How often is a Bayesian called as an expert witness?
2
u/CatOfGrey 17d ago
How often is a Bayesian called as an expert witness?
So many angles, so little time....
I'm actually more actuarial that statistician, so the actuary joke "How often did that happen last year?" answers the question moderately well.
The other one in my office is "Statistics means never having to say "I'm certain"...."
11
u/SalvatoreEggplant 17d ago
5% probability of a type 1 error. Seems reasonable.
22
4
u/ohcsrcgipkbcryrscvib 17d ago
If the null is true
10
u/SalvatoreEggplant 17d ago
Isn't that "type 1 error" means ?
5
u/ohcsrcgipkbcryrscvib 17d ago
If the null is false, there's a zero percent chance of Type I error. In frequentist statistics the hypotheses are not random.
5
u/cmdrtestpilot 17d ago
I like that after not adding anything substantive, you doubled down and made a new comment which also failed to contribute in any way. Gutsy discourse.
8
u/ohcsrcgipkbcryrscvib 17d ago
I'm simply pointing out that the size of a test, here 5%, is not the probability of a Type I error in general.
1
2
u/valegrete 17d ago
Because Fisher felt that was the threshold at which researchers in most disciplines could be morally certain that they had a meaningful result.
2
2
1
u/Icy_Kaleidoscope_546 17d ago
Its a low probability but not too low.
1
u/Calm_Interaction_268 16d ago
No. The actual answer is we don’t know if it’s not too low or low enough.
1
u/ANewPope23 17d ago
Arbitrary. It is lower in genomics research and higher in business studies.
1
u/Significant-Boot-260 17d ago
Yeah, it's pretty much just an assertion of how comfortable we are with being wrong lol.
2
u/Calm_Interaction_268 17d ago
I don’t think this statement is correct?
1
u/Significant-Boot-260 16d ago
Type 1 error rate alpha refers to the probability which we reject the null hypothesis when it is true. Generally, we choose 0.05 because we think a 5% chance of falsely rejecting is arbitrarily rare enough.
2
u/Calm_Interaction_268 16d ago
After you say “when it is true”, I could be wrong but I think there is an important statement after it. But now I’m nitpicking.
1
u/Significant-Boot-260 16d ago
I put my statement though ai after posting it just to make sure it was correct. I may be missing a comma though, I mean to express when the null hypothesis it true.
2
u/Calm_Interaction_268 16d ago
Ask ai “what would Sander Greenland say goes after that statement” and let me know what it says.
1
u/Significant-Boot-260 16d ago
AI suggested this: However, the choice of (alpha) is arbitrary, and a 5% Type I error rate is not necessarily acceptable in every situation. Rather than relying on a rigid threshold, researchers should consider the strength of the evidence, the consequences of incorrect conclusions, and the broader context of the study.
1
u/Calm_Interaction_268 16d ago
Now I’m really nitpicking. But I was specifically saying to ask ai what would Sander Greenland add to make your sentence more complete: “Type 1 error rate alpha refers to the probability which we reject the null hypothesis when it is true.” There should be a comma after the “when it is true”, not a period. Or something to that effect. Because it’s not actually “when it is true”.
1
u/Significant-Boot-260 16d ago
I'm interested to see what you're referring too since I'm not sure what it is.
→ More replies (0)
1
u/GreatBigBagOfNope 17d ago
Because Fisher pulled it out of his ass in 1925.
I wish I was making it up. Fisher basically gave the academic equivalent of a shrug and an "I dunno, that sounds about right", and it went on to set the standard in fields from law to demography and economics and biology and psychology.
1
1
u/thefringthing 17d ago
Relatedly, there is a tradition among Bayesians of citing 89% credible intervals instead of 90% or 95% as a way of reminding the reader that the width of the interval is arbitrary.
1
1
1
u/big_data_mike 17d ago
In the oil industry they actually use a higher p value as a threshold because one well with oil can pay for many other wells without oil.
1
1
u/waterless2 16d ago
I'd argue that the *primary* point is that you choose something a priori and then stick to it, whether it's .05 or .1 or whatever. As in, following an objective(*) procedure of agreeing on what will count as evidence under transparency is the main thing. The specific details can evolve within a field.
(*) Yeah, it's a contentious term, but in the sense of - you're not allowed to *really believe* your p-value from above to below the threshold, no matter how much you want to or how great the story would be if you could change your mind.
1
u/Mother___Night 16d ago
Because sometimes the mere existence of a standard is what’s important, regardless of why it was chosen
1
1
u/Professional_Toe346 16d ago
It is a tolerance, essentially. You’re saying “for such and such a procedure, I’m tolerant of a 5 per cent chance that I falsely reject the null hypothesis”. Ronald Fisher chose 5% because he found a 1 in 20 chance acceptable.
1
1
u/Specialist-Hyena-946 16d ago
I think there are actually two parts to this question:
- Should there be a threshold?
- If so, why should the threshold be 0.05?
Most answers here focus on the second question, to which the answer is that it is arbitrary.
But I think the first question is far more interesting. There, my answer would be that the guarantee offered by the p-value P(p <= a) <= a only works for values of ‘a’ that were prespecified. To avoid accusations of post-hoc specification, it is therefore useful that there is one agreed upon standard.
This recent paper in Statistical Science discusses exactly this problem, and what can go wrong if there is not one agreed upon standard in a field: https://projecteuclid.org/journals/statistical-science/volume-41/issue-1/Choosing-Alpha-Post-Hoc--The-Danger-of-Multiple-Standard/10.1214/25-STS983.full
This related paper offers an interesting solution: https://arxiv.org/abs/2312.08040.
A bit more on why we must compare to a threshold: P-values are quite problematic when used as a continuous measure of evidence: a p-value of say 0.02 is far more likely under the null than under the alternative if the alternative is very absurd. The problem is that the p-value is not properly scaled…
1
u/grandzooby 16d ago
I suspect there are two related reasons. First, people tend to go with what they've done in the past, particularly what they were taught. Second, before computers could easily give you values from distributions at any point, they used to be only easily available at the end of statistics books, only at values computed at specific points like 0.025, 0.05, etc. There likely wasn't much value in setting a threshold of 0.03 when your look-up tables didn't provide those.
1
u/haditwithyoupeople 16d ago
It's convenient. It's about 2 SDs (1.96) away from the mean. 0.01 is also used. Sometimes 0.1 is used. You can use any number you like. As others have mentioned, in the real world (vs. school) you need to consider Type 1 (reject a true null) and Type 2 errors (fail to reject a false null).
1
1
1
u/Admirable_Pie_6609 15d ago
It’s arbitrary and really depends on the application whether .05 is reasonable. It’s your tolerance for a false acceptance. How often are you okay assuming there is a signal when there really isn’t? What’s more costly: a false acceptance, or failing to detect a real signal?
All of these questions should impact where you set alpha
1
u/Unbearablefrequent 15d ago edited 15d ago
Anyone saying it's arbitrary is just repeating ignorance. Especially at this point.
I can go all out on the history of 0.05 (btw, its predates Fisher! You're all wrong!), but today its used as convention. In fact, if everyone uses 0.05, then we have a very good idea of typical error rates.
But just so you know, you can and should use your own alpha level. You should be able to argue why you're using it. Anyone remotely familiar with the theory will know this.
1
u/doodoomrpoopyman 15d ago
Its essentially arbitrary, but its a good enough estimate to say you’re quite confident.
1
1
u/DubGrips 14d ago
In my field (tech) even getting non technical stakeholders to understand the basics is such a battle that when I set other values they got extremely confused. I deal with enough BS at work to where I don't want to constantly be playing stats prof so I relented to an arbitrary standard
1
1
u/Mission-Direction-23 14d ago
It isn’t always. In some fields where the false discovery rate is known to be higher, they require more extreme p values as a matter of course to counteract the problem.
1
u/SchoolPuzzleheaded87 14d ago
Your edit is wrong way to think about it.
What is a p value? Its the area under a probability distribution.
Lets give a tangible and easy exempel. We are testing to see if the mean value of taking a medicine is equal to the mean value of a placebo.
I can then arbitarily write that i choose a significance level of 0,2. Then if p < 0,2 i can say that the results are significant. Because thats how i chose to define it. Will you accept this?
No, of course not. This is way to high value and could definitely be random. I can say its 0,05 and then that will be the threshold for when I choose to call it as significant. This is standard for most statistic test but if you were to try a drug and tell call the change significant just because the p value was 0.04, im not gonna be convinced. Why? Because risk / reward reasons.
My point is that its arbitrary, and that 0,05 is what has become common. 1/20 that we observe this result given that the bull is true. But it is still a value that come from a probability distribution and should therefore be interpreted as a continuous measure of evidence against the null hypothesis, rather than an absolute, magical line between "truth" and "falsehood."
1
1
u/MCHDeAnza 12d ago
It’s conventional/traditional in psychology, with the expectation that effect sizes are reported along with it. Then the onus is on the consumer of the research to determine how meaningful the difference (if comparing two samples) really is in that specific context.
The issue is with really large data sets, which Fischer didn’t live long enough to experience. If you analyze, for example, 5,000 lines of data (like from the a large scale organizational survey) you can have a laugh about all the significant results on very small mean differences or .10 correlations.
1
u/Aggravating-Dance707 6d ago
It’s saying that you’d see results as extreme as the results you have 5% of the time. Which is basically a guess of saying it’s rare to produce these results therefore you should consider this extremity to not be pure chance.
5% of the time is significant because it holds no matter the context. But if you’re working in a field where 5% of the time is actually a really big problem lowering it to 1% of the time is common
1
u/ParticularCorrect436 1d ago
the same way H0 is a convenient way to judge the direction your data is going.
1
u/soynumerito 10h ago
Late to this and people already explained the fisher part, but this is how I think about it in my day to day, I work running A/B tests.
0.05 basically means that if my change does nothing, around 1 in 20 tests will still come out "significant". you can actually see it with A/A tests (same thing in both groups), about 5% of them give p < 0.05 just by chance.
So for me the question is not really why 0.05, is more how many false positives can I live with. And that depends on what you are testing. For example, if it's a small change that's easy to revert (a copy change, a button), 0.05 or even 0.1 is fine, being too strict just make you lose real wins. If it's something like pricing, I want to be more sure. And if you look at 20 metrics at the same time you should expect one of them to be "significant" by pure luck lol
So yeah 0.04 or 0.06 would be equally ok, 0.05 is just the default that stuck. The important thing is to decide it before looking at the results.
1
u/Current-Ad1688 17d ago
The answer to one of the most foundational questions in modern science is "🤷"
-1
u/DigThatData 17d ago
because a 5% threshold is a rough shorthand for "the variation is smaller than is causally perceptible".
1
u/Calm_Interaction_268 16d ago
No, lol.
1
u/DigThatData 16d ago
literally yes.
the alternative to picking a threshold out of the air based on a heuristic like "people generally just use 5% anyway so yolo" is to do a power analysis. when you perform a power analysis, you are calibrating what resolution of causal perceptibility is meaningful or interesting to you, i.e. beneath which that if there is some effect, it's so small that you literally don't care.
so when you adopt a threshold like this without doing the power analysis first, you're implicitly adopting an "effect is small enough that I don't care" threshold based on a heuristic intead of simulation. I.e. it's a shorthand for doing the simulation.
literally yes.
2
u/Calm_Interaction_268 16d ago
No lol. “Too small for my study to reliably detect” and “too small for me to care about” are not the same thing.
I can use alpha .05 in a huge study and get statistical significance for an effect I don’t care about. I can use the exact same alpha in a small study and miss an effect I care about a lot. Same alpha. It told you nothing about what I consider meaningful.
Alpha is about controlling the probability of rejecting the null when it is true. It is not an effect-size threshold.
And bringing in a power analysis does not change that. You can choose an effect you care about and work out how large your study needs to be to detect it with a specified power. But then you supplied the “I care about this” part. The calculation didn’t establish that for you. You didn’t eliminate the choice of a threshold. You just moved the choice to the effect size.
So no… choosing .05 without doing a power analysis does not mean someone implicitly decided smaller effects don’t matter. Having low power to detect something does not mean someone doesn’t care about it. You’re treating those as interchangeable. They are not.
2
1
-8
u/Corruptionss 17d ago
The world did a study once and we changed all p-value rules from 0.05 to 0.06. That's what created the CrowdStrike global outage happened and almost all online went down
321
u/JarryBohnson 17d ago
It’s essentially arbitrary - I can’t remember exactly which paper it is but the person who first proposed them wrote in the paper something along the lines of “you should set your own reasonable threshold depending on your data” and nobody did.