Evidence-Based Approaches to Public Health: Biostatistics – Inferential Statistics: p-Values and Statistical Significance

In this tutorial, we will explore the concepts of p-values and statistical significance, which are fundamental to interpreting results in inferential statistics. p-values help researchers determine whether the results of a study are likely due to chance or represent a true effect in the population. Understanding p-values and statistical significance is essential for public health professionals and a key topic on the Certified in Public Health (CPH) exam.

By the end of this tutorial, you will understand what a p-value is, how to interpret it, and how it relates to statistical significance. Practice questions are included to help reinforce your knowledge.

Table of Contents:

1. Introduction to p-Values and Statistical Significance
2. What is a p-Value?
   • Definition of a p-Value
   • p-Value Threshold (e.g., 0.05)
   • Interpreting p-Values
3. Statistical Significance
   • Definition of Statistical Significance
   • Type I and Type II Errors
   • Limitations of p-Values
4. Practice Questions
5. Conclusion

1. Introduction to p-Values and Statistical Significance

In public health research, researchers often test hypotheses to determine whether an observed effect or association is real or if it could have occurred by chance. The p-value is a statistical measure that helps assess this probability. It is used in hypothesis testing to determine whether the results of a study are statistically significant—that is, unlikely to have occurred by chance alone.

A result is considered statistically significant if the p-value is below a predetermined threshold (usually 0.05), indicating that there is strong evidence against the null hypothesis (the assumption that there is no effect or association).

2. What is a p-Value?

2.1 Definition of a p-Value

A p-value is the probability of observing the data, or something more extreme, assuming that the null hypothesis is true. The null hypothesis typically states that there is no effect or no association between the variables being studied.

The p-value helps us determine whether the observed results could have occurred by chance. A smaller p-value indicates stronger evidence against the null hypothesis, while a larger p-value suggests that the results are more likely due to random variation.

• Example: In a clinical trial testing the effectiveness of a new drug, the null hypothesis would be that the drug has no effect. A p-value of 0.03 would indicate a 3% probability that the observed difference in outcomes occurred by chance if the drug had no effect.

2.2 p-Value Threshold (e.g., 0.05)

The most commonly used threshold for statistical significance is 0.05. This means that if the p-value is less than 0.05, the result is considered statistically significant, and the null hypothesis is rejected.

• p-value < 0.05: Statistically significant. There is less than a 5% probability that the observed effect occurred by chance.
• p-value ≥ 0.05: Not statistically significant. There is a higher probability that the observed effect occurred by chance.

2.3 Interpreting p-Values

It is important to interpret p-values in the context of the study:

• p < 0.01: Strong evidence against the null hypothesis. The results are highly statistically significant.
• p < 0.05: Moderate evidence against the null hypothesis. The results are statistically significant.
• p > 0.05: Weak or no evidence against the null hypothesis. The results are not statistically significant, meaning the effect could be due to chance.

However, a statistically significant result (p < 0.05) does not prove that the effect is large or important; it simply means that the effect is unlikely to be due to chance. Conversely, a non-significant result (p ≥ 0.05) does not necessarily mean that there is no effect; it could indicate that the study lacked sufficient power to detect a difference.

3. Statistical Significance

3.1 Definition of Statistical Significance

Statistical significance refers to the likelihood that an observed effect or relationship in a study is not due to random chance. If a result is statistically significant, it means that the evidence is strong enough to reject the null hypothesis, suggesting that the effect is real. The threshold for statistical significance is typically set at p < 0.05, but this threshold can vary depending on the context of the study.

It is important to note that statistical significance does not necessarily imply clinical or practical significance. A small p-value may indicate a statistically significant result, but the effect size might still be too small to have meaningful public health implications.

3.2 Type I and Type II Errors

In hypothesis testing, there are two types of errors that researchers need to be aware of:

• Type I Error (False Positive): This occurs when the null hypothesis is incorrectly rejected. In other words, the study finds a statistically significant result when there is actually no true effect. The probability of a Type I error is denoted by α (the significance level, often set at 0.05).
• Type II Error (False Negative): This occurs when the null hypothesis is incorrectly accepted. In this case, the study fails to find a statistically significant result even though a true effect exists. The probability of a Type II error is denoted by β, and the power of the study (1 – β) represents the ability to detect a true effect.

Researchers aim to minimize both types of errors by choosing an appropriate significance level and ensuring that the study has sufficient power (large enough sample size) to detect meaningful effects.

3.3 Limitations of p-Values

While p-values are a useful tool for hypothesis testing, they have limitations:

• p-values do not measure the size or importance of an effect: A small p-value only indicates that the effect is unlikely to be due to chance, not that the effect is large or important.
• p-values can be influenced by sample size: With a very large sample, even small, unimportant effects can become statistically significant. Conversely, with a small sample, large effects may not reach statistical significance.
• p-values do not prove causation: Statistical significance does not imply a cause-and-effect relationship between variables.

4. Practice Questions

Test your understanding of p-values and statistical significance with these practice questions. Try answering them before checking the solutions.

Question 1:

A study finds that a new treatment reduces blood pressure, and the p-value for this result is 0.02. Is this result statistically significant?

Answer 1:

Question 2:

A clinical trial testing a new vaccine reports a p-value of 0.08. Assuming a p-value cut-off of 0.05,  What does this mean in terms of statistical significance?

Answer 2:

Question 3:

When might a researcher have a p-value cut-off 0.01 instead of 0.05?

Answer 3:

5. Conclusion

p-values and statistical significance are key concepts in inferential statistics that help researchers determine whether an observed effect is likely due to chance. A statistically significant result provides evidence against the null hypothesis, but it does not measure the size or importance of the effect.

Humanities Moment

The Featured Image for this article is Klänge Pl.19 (1913) by Wassily Kandinsky (Russian, 1866 – 1944). Kandinsky was a Russian painter and art theorist recognized as a pioneering figure in abstract art. Initially trained in law and economics, he turned to art at age 30, studying in Munich and later teaching at Germany’s Bauhaus school until its closure by the Nazis. He spent his final years in France, where he created many significant works and became a French citizen before his death in 1944.