What do textbooks do wrong

Textbook Mistakes: Statistical Significance Incorrectly Explained

But getting it right is also really complicated. A statistical finding is generally considered to be “significant” or “significant” if the associated p-value is less than 0.05. And that means: "Assuming that the null hypothesis is correct and the study is repeated an infinite number of times by drawing a random sample from the same population, then less than five percent of the results will be more extreme than the present result."

Certainly a number of textbooks have already explained this in simpler terms. But many of them make common mistakes. At least this is the conclusion reached by psychologists in a small field study based on the definition cited at the beginning.

David Stanley's team from the University of Guelph near Toronto used the example of 30 psychology textbooks to check whether first-year students were correctly conveyed the term "statistical significance". In 28 of the 30 books - according to the authors, the best-selling in the US and Canada in 2017 and 2018 - the term was defined or explained at least once. And of these, 25 (around 89 percent) contained at least one error.

By far the most common was: Statistically significant means that the probability that a result will come about by chance is less than five percent. The best way to see why this is a mistake is to anticipate the condition on which the calculation is based: "if the null hypothesis is true". The null hypothesis is the assumption that there is no effect, i.e. that, for example, two characteristics Not related. Under this condition, however, it is by definition not possible that the null hypothesis is incorrect. Rather, the p-value indicates, as a measure of significance, how often an effect that is at least as large as the present one occurs due to an infinite number of samples - although there is none.

This also reveals another fallacy: "A statistically significant result confirms the research hypothesis," for example the assumption that two features are related. Nor can p<  0,05="" ableiten,="" dass="" es="" mit="" 95-prozentiger="" wahrscheinlichkeit="" einen="" effekt="" gibt="" oder="" dass="" die="" wahrscheinlichkeit,="" dass="" es="" keinen="" gibt,="" weniger="" als="" 5 prozent="">

The significance test is the most common statistical criterion for research findings in psychology. In view of their reading, Stanley and his colleagues attribute the fact that it is often misinterpreted by students and researchers alike, at least partly to errors in the textbooks. If you are still hoping for a simple definition, you can stick to the British statistician Ronald Fisher: According to him, greatly simplified, the p-value indicates how well a given result can be reconciled with the null hypothesis. However, he does not reveal anything about the truth or the quality of the study.