Skip to content

Search Broadly Epi

Parametric and nonparametric tests

In this article, we will explore the differences between parametric and non-parametric tests, two types of statistical tests commonly used in hypothesis testing. Both types of tests have their own assumptions and applications, and public health professionals must understand when to use each type to analyze data correctly. Mastering parametric and non-parametric tests is essential for the Certified in Public Health (CPH) exam and for conducting research in public health settings.

By the end of this article, you will understand the key differences between parametric and non-parametric tests, when to use each, and how they apply to public health research. Practice questions are included to help reinforce your understanding.

1. Introduction to Hypothesis Testing
2. What Are Parametric Tests?
3. What Are Non-parametric Tests?
4. Differences Between Parametric and Non-parametric Tests
5. When to Use Parametric vs. Non-parametric Tests
6. Common Parametric Tests
7. Common Non-parametric Tests
8. Practice Questions
9. Conclusion

1. Introduction to Hypothesis Testing

Hypothesis testing is a statistical method used to determine whether there is enough evidence in a sample of data to infer that a certain condition holds true for the entire population. In public health research, hypothesis testing is widely used to compare groups, assess associations, and evaluate the effectiveness of interventions. Depending on the nature of the data and the assumptions that can be made about the population, researchers may use either parametric or non-parametric tests to analyze the data.

2. What Are Parametric Tests?

Parametric tests are statistical tests that rely on assumptions about the underlying population distribution, typically that the data follow a normal distribution. These tests use parameters such as the mean and standard deviation to make inferences about the population. Because parametric tests make more assumptions about the data, they tend to be more powerful (i.e., they have a higher likelihood of detecting a true effect) when the assumptions are met.

However, parametric tests are not suitable for all types of data, especially when the data do not meet the assumptions of normality or when the sample size is small.

Key Features of Parametric Tests:

– Assume the data follow a specific distribution, usually a normal distribution.
– Use population parameters (mean, standard deviation) to make inferences.
– Tend to be more powerful when assumptions are met.
– Require interval or ratio-level data.

3. What Are Non-parametric Tests?

Non-parametric tests, also known as distribution-free tests, do not assume that the data follow a specific distribution. These tests are often used when the sample size is small, when the data are not normally distributed, or when the data are ordinal or nominal. Non-parametric tests are more flexible than parametric tests because they make fewer assumptions about the data, but they are generally less powerful (i.e., they may require a larger sample size to detect a true effect).

Non-parametric tests are particularly useful in public health research when dealing with skewed data, small sample sizes, or data that do not meet the assumptions required for parametric tests.

Key Features of Non-parametric Tests:

– Do not assume a specific population distribution.
– Use ranks or medians instead of means.
– Suitable for ordinal, nominal, or non-normally distributed data.
– Less powerful than parametric tests but more robust when assumptions are violated.

4. Differences Between Parametric and Non-parametric Tests

Aspect Parametric Tests Non-parametric Tests
Assumptions Require assumptions about the population distribution (e.g., normality). Do not require assumptions about the population distribution.
Data Type Require interval or ratio-level data. Can be used with ordinal, nominal, or non-normally distributed data.
Use of Parameters Use population parameters like mean and standard deviation. Use medians or ranks rather than means.
Power More powerful if assumptions are met. Less powerful but more flexible when assumptions are not met.
When to Use When data are normally distributed or the sample size is large. When data are not normally distributed, the sample size is small, or data are ordinal/nominal.

5. When to Use Parametric vs. Non-parametric Tests

The decision to use a parametric or non-parametric test depends on several factors, including the distribution of the data, the level of measurement, and the sample size. Here are general guidelines for when to use each type of test:

6. Common Parametric Tests

Here are some common parametric tests used in public health research:

7. Common Non-parametric Tests

Here are some common non-parametric tests used in public health research:

8. Practice Questions

Test your understanding of parametric and non-parametric tests with these practice questions. Try answering them before checking the solutions.

Question 1:

A researcher wants to compare the mean cholesterol levels between two groups of patients, but the data are not normally distributed. Which type of test should the researcher use?

Answer 1:

Answer: Click to reveal

The researcher should use a non-parametric test, such as the Mann-Whitney U Test, because the data are not normally distributed.

Question 2:

A study compares the blood pressure of patients before and after they undergo a new treatment. The data are normally distributed. What test should be used?

Answer 2:

Answer: Click to reveal

The appropriate test is a paired t-test because the data are normally distributed, and the same patients are being measured at two different times.

Question 3:

A researcher is analyzing the effectiveness of three different diets on weight loss. The data are not normally distributed. What test should the researcher use?

Answer 3:

Answer: Click to reveal

The researcher should use the Kruskal-Wallis Test, a non-parametric test, to compare the three groups since the data are not normally distributed and are in 3 independent groups.

9. Conclusion

Both parametric and non-parametric tests are essential tools in public health research. Parametric tests are more powerful but require certain assumptions about the data, while non-parametric tests are more flexible and can be used when those assumptions are not met. Understanding when to use each type of test and how they differ is crucial for accurately analyzing data and making valid conclusions in public health research. Make sure you understand the assumptions behind parametric and non-parametric tests, and practice identifying which test is appropriate for different types of data and research scenarios. This knowledge will be essential not only for the Certified in Public Health (CPH) exam but also for conducting sound public health research.

Humanities Moment

The featured image for this CPH Focus is Olive Trees (1889) by Vincent van Gogh (Dutch, 1853–1890), a post-impressionist painter whose emotionally charged use of color and expressive brushwork helped lay the groundwork for modern art. Despite creating over 2,000 artworks in just a decade, van Gogh struggled with mental health and poverty throughout his life and achieved lasting fame only after his death. Today, his deeply personal paintings are celebrated worldwide and his legacy is preserved at the Van Gogh Museum in Amsterdam.

Type I and Type II errors

In this tutorial, we explore the fundamental concepts of Type I and Type II errors in hypothesis testing. These error types are critical to understanding the risk of false positives and false negatives within a research or applied epidemiology context. Recognizing these errors and their implications is vital for the Certified in Public Health (CPH) exam and for interpreting research findings accurately.

By the end of this tutorial, you will understand what Type I and Type II errors are, how they impact research, and how to minimize them. Practice questions with answers are included to help reinforce your understanding.

1. Introduction to Hypothesis Testing

Hypothesis testing is a statistical method used to make decisions about a population based on sample data. The process involves formulating two hypotheses:

The goal of hypothesis testing is to determine whether there is enough evidence in the sample data to reject the null hypothesis in favor of the alternative hypothesis. However, this process is subject to two types of errors: Type I error and Type II error.

2. Type I Error (False Positive)

A Type I error, also known as a false positive, occurs when the null hypothesis is incorrectly rejected when it is actually true. In other words, a Type I error happens when we conclude that there is an effect or difference when, in reality, there is none.

2.1 Definition of Type I Error

A Type I error occurs when the data lead us to reject the null hypothesis (H0) even though it is true. This means that the conclusion that an effect exists is incorrect.

2.2 Significance Level (α)

The probability of making a Type I error is denoted by α, which is known as the significance level. Common significance levels are 0.05 (5%) or 0.01 (1%), meaning there is a 5% or 1% chance of incorrectly rejecting the null hypothesis.

2.3 Consequences of Type I Error

In public health research, a Type I error can have serious consequences. For example, concluding that a new treatment is effective when it is not could lead to widespread use of an ineffective or harmful intervention. Therefore, controlling the significance level is crucial to minimize the risk of a Type I error.

3. Type II Error (False Negative)

A Type II error, also known as a false negative, occurs when the null hypothesis is not rejected when it is actually false. This means that the test fails to detect an effect or difference that truly exists.

3.1 Definition of Type II Error

A Type II error occurs when we fail to reject the null hypothesis (H0) even though the alternative hypothesis (HA) is true. This means that the conclusion that there is no effect is incorrect because an effect does exist.

3.2 Power of a Test

The probability of making a Type II error is denoted by β. The power of a statistical test is the probability of correctly rejecting the null hypothesis when the alternative hypothesis is true, and it is given by 1 – β.

Increasing the sample size or using more precise measurements can increase the power of a test, thereby reducing the likelihood of a Type II error.

3.3 Consequences of Type II Error

In public health research, a Type II error can be problematic because it means that a true effect goes undetected. For example, failing to identify an effective treatment could result in missed opportunities for improving health outcomes. Therefore, minimizing the probability of a Type II error is important to avoid overlooking meaningful findings.

4. Balancing Type I and Type II Errors

There is a trade-off between Type I and Type II errors. Reducing the risk of a Type I error (by lowering α) increases the risk of a Type II error, and vice versa. This balance must be carefully managed based on the context of the research.

In public health research, the cost of making one type of error may be higher than the other, depending on the situation. For example, in the case of a new drug, it may be more important to avoid a Type I error (false positive) to prevent the approval of an ineffective or harmful drug.

5. Practice Questions

Test your understanding of Type I and Type II errors with these practice questions. Try answering them before checking the solutions.

Question 1:

A researcher concludes that a new public health intervention is effective when it actually is not. What type of error is this?

Answer 1:

Answer: Click to reveal

This is a Type I error because the researcher incorrectly rejected the null hypothesis (concluding the intervention works when it does not).

Question 2:

A study fails to detect a significant effect of a new treatment, even though the treatment is actually effective. What type of error is this?

Answer 2:

Answer: Click to reveal

This is a Type II error because the study failed to reject the null hypothesis even though the alternative hypothesis is true.

Question 3:

If the significance level (α) is set to 0.01 instead of 0.05, what happens to the risk of Type I and Type II errors?

Answer 3:

Answer: Click to reveal

Lowering the significance level to 0.01 reduces the risk of a Type I error (false positive), but it increases the risk of a Type II error (false negative) because the criteria for rejecting the null hypothesis become more stringent.

6. Conclusion

Understanding Type I and Type II errors is essential for making accurate decisions in public health research. Researchers must carefully balance the risk of false positives and false negatives by setting appropriate significance levels and ensuring studies have sufficient power. This balance helps ensure that research findings are both reliable and actionable.

Key Takeaways:

Always be aware of the consequences of both types of errors, and ensure that your study design reduces their impact as much as possible to avoid the pitfalls in evidence-based public health scenarios.

Humanities Moment

The featured image for this CPH Focus is Interior with Two Figures, Open Window (1922) by Henri Matisse (French, 1869 – 1954). Mattise was a visionary artist whose innovative use of color and expressive line influenced the course of modern art in the twentieth century. Renowned for his painting, drawing, sculpture, and later his striking cut-paper collages, Matisse helped define movements such as Fauvism and continued evolving his style throughout his life. His ability to balance rigorous structure with decorative beauty secured his legacy as one of the most important artists of his era.

Null and alternative hypotheses

Understanding the concepts of null and alternative hypotheses is fundamental to the Certified in Public Health (CPH) exam. Hypothesis testing is a crucial statistical tool used in public health to make decisions based on sample data. In this tutorial, we will explore what null and alternative hypotheses are, how they are used in research, and their importance in hypothesis testing. We will also include practice questions with answers to reinforce your knowledge.

By the end of this tutorial, you will have a solid understanding of null and alternative hypotheses and how they apply in public health research. This knowledge is essential for public health professionals and for tackling questions related to hypothesis testing on the CPH exam.

1. Introduction to Hypothesis Testing

Hypothesis testing is a statistical method used to decide whether there is enough evidence in a sample of data to infer that a certain condition is true for the entire population. Researchers start with two competing hypotheses: the null hypothesis and the alternative hypothesis. The goal of hypothesis testing is to determine whether the sample data provide enough evidence to reject the null hypothesis in favor of the alternative hypothesis.

2. What Is a Null Hypothesis?

The null hypothesis (denoted as H₀) is a statement that there is no effect, no difference, or no relationship between variables. It assumes that any observed outcome is due to random chance or sampling error. In hypothesis testing, the null hypothesis represents the default or status quo assumption.

Key Points about the Null Hypothesis:

3. What Is an Alternative Hypothesis?

The alternative hypothesis (denoted as H₁ or Ha) is a statement that contradicts the null hypothesis. It suggests that there is an effect, a difference, or a relationship between variables. The alternative hypothesis is what the researcher aims to provide evidence for through hypothesis testing.

Key Points about the Alternative Hypothesis:

4. Types of Null and Alternative Hypotheses

Hypotheses can be classified into two main types based on the direction of the effect being tested:

5. Steps in Hypothesis Testing

The process of hypothesis testing involves several key steps:

  1. State the Hypotheses: Formulate the null and alternative hypotheses based on the research question.
  2. Select the Significance Level (α): The significance level is the threshold for deciding when to reject the null hypothesis. Commonly used values are 0.05 or 0.01.
  3. Collect and Analyze Data: Gather sample data and calculate the appropriate test statistic (e.g., t-test, chi-square test).
  4. Make a Decision: Compare the test statistic to a critical value or use a p-value to decide whether to reject or fail to reject the null hypothesis.
  5. Draw a Conclusion: Based on the decision, draw conclusions about the population. If the null hypothesis is rejected, it suggests that there is evidence in favor of the alternative hypothesis.

6. Importance of Hypothesis Testing in Public Health

Hypothesis testing is critical in public health research because it allows researchers to make informed decisions about the effectiveness of interventions, the relationship between risk factors and health outcomes, and the impact of public health policies. It helps determine whether observed differences or associations are statistically significant, providing evidence to guide public health practice.

7. Common Pitfalls in Hypothesis Testing

While hypothesis testing is a powerful tool, there are common pitfalls that researchers should be aware of:

8. Practice Questions

Test your understanding of null and alternative hypotheses with these practice questions. Try answering them before checking the solutions.

Question 1:

A researcher wants to test whether a new drug lowers blood pressure. What is the null hypothesis and the alternative hypothesis?

Answer 1:

Answer: Click to reveal

Null Hypothesis (H₀): The new drug does not lower blood pressure (the mean difference in blood pressure is zero).

Alternative Hypothesis (H₁): The new drug lowers blood pressure (the mean difference in blood pressure is less than zero).

Question 2:

In a study comparing two diets, the researchers want to test whether there is any difference in weight loss between the diets. What are the null and alternative hypotheses?

Answer 2:

Answer: Click to reveal

Null Hypothesis (H₀): There is no difference in weight loss between the two diets (mean weight loss for diet A equals mean weight loss for diet B).

Alternative Hypothesis (H₁): There is a difference in weight loss between the two diets (mean weight loss for diet A does not equal mean weight loss for diet B).

Question 3:

A study is conducted to determine whether a new public health campaign increases the rate of vaccination. What is the null hypothesis, and what is the alternative hypothesis?

Answer 3:

Answer: Click to reveal

Null Hypothesis (H₀): The public health campaign does not increase the vaccination rate (vaccination rate is less than or equal to the previous rate).

Alternative Hypothesis (H₁): The public health campaign increases the vaccination rate (vaccination rate is greater than the previous rate).

9. Conclusion

Hypothesis testing, with its focus on null and alternative hypotheses, is a foundational concept in public health research. It allows researchers to make evidence-based decisions about the effectiveness of interventions, relationships between variables, and public health outcomes. Here is a recap of the key points from this tutorial:

Humanities Moment

The featured work for this CPH Focus is Jaroslava Mucha by Alphonse Mucha (Czech, 1860-1939), a celebrated painter and graphic artist best known for helping define the Art Nouveau movement through his elegant posters and illustrations, especially those of Sarah Bernhardt. After achieving fame in Paris, Mucha returned to his homeland, where he devoted himself to The Slav Epic, a monumental series of canvases depicting Slavic history, which he regarded as his greatest legacy and gifted to the Czech nation.
By practicing the formulation of null and alternative hypotheses and understanding the importance of hypothesis testing in public health, you will be well-prepared for questions on the Certified in Public Health (CPH) exam and for applying these concepts in real-world public health research.

Probability distributions

In this tutorial, we will explore three key probability distributions that are commonly used in public health research: the normal distribution, the binomial distribution, and the Poisson distribution. Each distribution describes the behavior of data in different situations, and understanding them is essential for making accurate statistical inferences. This knowledge is critical for public health professionals and a key part of the Certified in Public Health (CPH) exam.

By the end of this tutorial, you will understand what each probability distribution represents, how to calculate probabilities using these distributions, and when to use each distribution. Practice questions are provided to reinforce your understanding.

1. Introduction to Probability Distributions

A probability distribution describes how the values of a random variable are distributed. It shows the likelihood of different outcomes or events occurring. In public health research, different types of probability distributions are used to model different types of data, such as the number of new disease cases in a given time period or the distribution of health measures like blood pressure.

The three most commonly used distributions in public health are the normal, binomial, and Poisson distributions.

2. The Normal Distribution

The normal distribution (also called the Gaussian distribution) is the most widely used probability distribution in statistics. It describes data that are symmetrically distributed around the mean, with most values clustering around the center and fewer values occurring as you move away from the mean. The normal distribution is often used to model continuous data such as height, weight, or cholesterol levels.

2.1 Definition of the Normal Distribution

A normal distribution is a continuous probability distribution that is symmetric around its mean. The probability density function (PDF) of a normal distribution is bell-shaped and defined by two parameters: the mean (μ) and the standard deviation (σ).

The formula for the normal distribution is:

f(x)=1σ2πe−(x−μ)22σ2 f(x) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{(x – \mu)^2}{2\sigma^2}}

Where:

2.2 Properties of the Normal Distribution

2.3 Using Z-Scores with the Normal Distribution

Z-scores are used to standardize values in a normal distribution, allowing comparisons between different datasets or variables. A Z-score represents the number of standard deviations a value is from the mean.

The formula for calculating a Z-score is:

Z=x−μσ Z = \frac{x – \mu}{\sigma}

Where x is the value of interest, μ is the mean, and σ is the standard deviation.

3. The Binomial Distribution

The binomial distribution is used to model the number of successes in a fixed number of trials, where each trial has only two possible outcomes (e.g., success or failure). It is often used in public health research to model outcomes such as the number of individuals who develop a disease after exposure or the number of individuals who test positive in a diagnostic test.

3.1 Definition of the Binomial Distribution

A binomial distribution describes the probability of obtaining a fixed number of successes in a fixed number of independent trials, where each trial has the same probability of success (p).

The formula for binomial probability is:

P(X=k)=(nk)pk(1−p)n−k P(X = k) = \binom{n}{k} p^k (1 – p)^{n-k}

Where:

3.2 Properties of the Binomial Distribution

4. The Poisson Distribution

The Poisson distribution is used to model the number of events that occur in a fixed interval of time or space, where the events occur randomly and independently. It is commonly used in public health to model the number of disease cases or the number of emergency room visits within a given time period.

4.1 Definition of the Poisson Distribution

A Poisson distribution describes the probability of a given number of events occurring in a fixed interval, given the average number of events (λ) expected in that interval.

The formula for Poisson probability is:

P(X=k)=λke−λk! P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}

Where:

4.2 Properties of the Poisson Distribution

5. Practice Questions

Test your understanding of probability distributions with these practice questions. Try answering them before checking the solutions.

Question 1:

In a population, the average number of asthma-related emergency room visits per day is 3. What is the probability of exactly 5 visits occurring on a given day? Assume a Poisson distribution.

Answer 1:

Answer, click to reveal

Here, λ = 3, and we want to find the probability of k = 5 visits. Using the Poisson formula:

P(X=5)=35e−35!=243×0.0498120=0.1008 P(X = 5) = \frac{3^5 e^{-3}}{5!} = \frac{243 \times 0.0498}{120} = 0.1008

Question 2:

A clinical trial involves 10 patients, and the probability that a patient will respond to the treatment is 0.7. What is the probability that exactly 7 patients will respond? Assume a binomial distribution.

Answer 2:

Answer, click to reveal

Using the binomial formula:

P(X=7)=(107)(0.7)7(0.3)3=10!7!(10−7)!×(0.7)7×(0.3)3 P(X = 7) = \binom{10}{7} (0.7)^7 (0.3)^3 = \frac{10!}{7!(10 – 7)!} \times (0.7)^7 \times (0.3)^3

Calculating this gives a probability of approximately 0.2668.

Question 3:

What proportion of data lies between one standard deviation above and below the mean in a normal distribution?

Answer, click to reveal

Answer 3:

Approximately 68% of the data in a normal distribution lie within one standard deviation of the mean.

6. Conclusion

Probability distributions are essential tools in public health research, helping us model different types of data and make inferences about populations. The normal, binomial, and Poisson distributions are widely used to represent different data types and scenarios.

Remember:

Final Tip for the CPH Exam:

Make sure you understand the key properties of each probability distribution and how to apply them to real-world public health scenarios. In particular, make sure to remember the key differences between normal, binomial, and Poisson distributions and when they are used. Practice using the formulas to calculate probabilities, as this knowledge will be crucial for answering questions related to probability distributions on the Certified in Public Health (CPH) exam.

Humanities Moment

The featured image for this CPH Focus is The Wave (1917) by Christopher R. W. Nevinson (English, 1889 – 1946). Nevinson was a British painter and printmaker renowned for his powerful depictions of World War I, blending Futurism and Cubism to portray the mechanized brutality of modern warfare. He also occasionally used these styles in pieces showing natural beauty, as above. Initially a radical modernist, he later shifted to realism as the horrors of war outpaced abstraction. Though once hailed as one of Britain’s most important war artists, his postwar career was marred by personal conflict, exaggeration, and declining influence.

Basic probability concepts

In this tutorial, we will explore basic probability concepts, which are foundational to understanding statistical methods in public health research. Probability helps quantify the likelihood of an event occurring and is critical in making predictions based on data. Mastering these concepts is essential for public health professionals and will be covered on the Certified in Public Health (CPH) exam.

By the end of this tutorial, you will understand the basic rules of probability, key terms, and how to calculate probabilities in different scenarios. Practice questions are included to reinforce your understanding.

1. Introduction to Probability

Probability is the measure of how likely an event is to occur. It ranges from 0 to 1, where 0 means the event is impossible and 1 means the event is certain to happen. In public health research, probability is used to assess risks, predict outcomes, and analyze the likelihood of health events based on data.

The formula for calculating probability is:

P(A)=Number of favorable outcomesTotal number of possible outcomes P(A) = \frac{\text{Number of favorable outcomes}}{\text{Total number of possible outcomes}}

Where P(A) is the probability of event A occurring.

2. Basic Probability Terms

Before diving into the rules of probability, it’s important to understand some key terms:

2.1 Experiment, Sample Space, Event

2.2 Mutually Exclusive Events

Mutually exclusive events are events that cannot happen at the same time. If one event occurs, the other cannot.

2.3 Independent Events

Independent events are events where the outcome of one event does not affect the outcome of the other.

3. Rules of Probability

There are several key rules of probability that help us calculate the likelihood of events occurring, whether they are mutually exclusive, independent, or otherwise related.

3.1 Addition Rule for Mutually Exclusive Events

If two events are mutually exclusive (i.e., they cannot happen at the same time), the probability that either event occurs is the sum of their individual probabilities.

The formula is:

P(A or B)=P(A)+P(B) P(A \text{ or } B) = P(A) + P(B)

Example: If the probability of rolling a 3 on a die is 1/6, and the probability of rolling a 5 is also 1/6, then the probability of rolling either a 3 or a 5 is:

P(3 or 5)=16+16=26=13 P(3 \text{ or } 5) = \frac{1}{6} + \frac{1}{6} = \frac{2}{6} = \frac{1}{3}

3.2 Multiplication Rule for Independent Events

If two events are independent, the probability that both events occur is the product of their individual probabilities.

The formula is:

P(A and B)=P(A)×P(B) P(A \text{ and } B) = P(A) \times P(B)

Example: If the probability of flipping heads on a coin is 1/2 and the probability of rolling a 6 on a die is 1/6, the probability of flipping heads and rolling a 6 is:

P(Heads and 6)=12×16=112 P(\text{Heads and 6}) = \frac{1}{2} \times \frac{1}{6} = \frac{1}{12}

3.3 Complement Rule

The complement rule is used to find the probability that an event does not occur. The complement of event A is denoted by A’, and the sum of the probabilities of A and A’ is always 1.

The formula is:

P(A′)=1−P(A) P(A’) = 1 – P(A)

Example: If the probability of it raining tomorrow is 0.30, the probability that it will not rain is:

P(No Rain)=1−0.30=0.70 P(\text{No Rain}) = 1 – 0.30 = 0.70

4. Practice Questions

Test your understanding of basic probability concepts with these practice questions. Try answering them before checking the solutions.

Question 1:

A die is rolled. What is the probability of rolling a 4 or a 6?

Answer 1:

Answer, click to reveal

These are mutually exclusive events, so we use the addition rule:

P(4 or 6)=P(4)+P(6)=16+16=26=13 P(4 \text{ or } 6) = P(4) + P(6) = \frac{1}{6} + \frac{1}{6} = \frac{2}{6} = \frac{1}{3}

Question 2:

A coin is flipped and a die is rolled. What is the probability of flipping heads and rolling an odd number?

Answer 2:

Answer, click to reveal

These are independent events, so we use the multiplication rule:

P(Heads and Odd)=P(Heads)×P(Odd)=12×36=12×12=14 P(\text{Heads and Odd}) = P(\text{Heads}) \times P(\text{Odd}) = \frac{1}{2} \times \frac{3}{6} = \frac{1}{2} \times \frac{1}{2} = \frac{1}{4}

Question 3:

The probability that a person selected at random has a certain disease is 0.02. What is the probability that a randomly selected person does not have the disease?

Answer 3:

Answer, click to reveal

We use the complement rule:

P(No Disease)=1−P(Disease)=1−0.02=0.98 P(\text{No Disease}) = 1 – P(\text{Disease}) = 1 – 0.02 = 0.98

5. Conclusion

Basic probability concepts are essential for understanding the likelihood of events occurring in public health research. These concepts help researchers make predictions, assess risks, and interpret data in a meaningful way.

Always remember:

Final Tip for the CPH Exam:

Make sure you understand how to apply basic probability rules and calculate probabilities for different types of events. Practice solving problems involving mutually exclusive events, independent events, and complements, as these concepts are likely to be tested on the Certified in Public Health (CPH) exam. Additionally, and perhaps more importantly, learn to determine what category a set of events might belong to so as to apply the right rule at the right time.

Humanities Moment

For this CPH Focus, the featured image is Czardas dancers by Ernst Ludwig Kirchner (German, 1880-1938). Kirchner was a pioneering German expressionist and founding member of Die Brücke, a movement that broke from academic tradition to embrace raw emotion, bold color, and modern forms. After early success, World War I and mental health struggles shaped both his life and art, leading to periods of isolation and prolific output in Davos, Switzerland. Condemned as “degenerate” by the Nazis, Kirchner’s later years were marked by political disillusionment and deteriorating health, ending with his death in 1938. He left behind a body of work that helped define early 20th-century modernism.

P-values and statistical significance

In this tutorial, we will explore the concepts of p-values and statistical significance, which are fundamental to interpreting results in inferential statistics. p-values help researchers determine whether the results of a study are likely due to chance or represent a true effect in the population. Understanding p-values and statistical significance is essential for public health professionals and a key topic on the Certified in Public Health (CPH) exam.

By the end of this tutorial, you will understand what a p-value is, how to interpret it, and how it relates to statistical significance. Practice questions are included to help reinforce your knowledge.

1. Introduction to p-Values and Statistical Significance

In public health research, researchers often test hypotheses to determine whether an observed effect or association is real or if it could have occurred by chance. The p-value is a statistical measure that helps assess this probability. It is used in hypothesis testing to determine whether the results of a study are statistically significant—that is, unlikely to have occurred by chance alone.

A result is considered statistically significant if the p-value is below a predetermined threshold (usually 0.05), indicating that there is strong evidence against the null hypothesis (the assumption that there is no effect or association).

2. What is a p-Value?

2.1 Definition of a p-Value

A p-value is the probability of observing the data, or something more extreme, assuming that the null hypothesis is true. The null hypothesis typically states that there is no effect or no association between the variables being studied.

The p-value helps us determine whether the observed results could have occurred by chance. A smaller p-value indicates stronger evidence against the null hypothesis, while a larger p-value suggests that the results are more likely due to random variation.

2.2 p-Value Threshold (e.g., 0.05)

The most commonly used threshold for statistical significance is 0.05. This means that if the p-value is less than 0.05, the result is considered statistically significant, and the null hypothesis is rejected.

2.3 Interpreting p-Values

It is important to interpret p-values in the context of the study:

However, a statistically significant result (p < 0.05) does not prove that the effect is large or important; it simply means that the effect is unlikely to be due to chance. Conversely, a non-significant result (p ≥ 0.05) does not necessarily mean that there is no effect; it could indicate that the study lacked sufficient power to detect a difference.

3. Statistical Significance

3.1 Definition of Statistical Significance

Statistical significance refers to the likelihood that an observed effect or relationship in a study is not due to random chance. If a result is statistically significant, it means that the evidence is strong enough to reject the null hypothesis, suggesting that the effect is real. The threshold for statistical significance is typically set at p < 0.05, but this threshold can vary depending on the context of the study.

It is important to note that statistical significance does not necessarily imply clinical or practical significance. A small p-value may indicate a statistically significant result, but the effect size might still be too small to have meaningful public health implications.

3.2 Type I and Type II Errors

In hypothesis testing, there are two types of errors that researchers need to be aware of:

Researchers aim to minimize both types of errors by choosing an appropriate significance level and ensuring that the study has sufficient power (large enough sample size) to detect meaningful effects.

3.3 Limitations of p-Values

While p-values are a useful tool for hypothesis testing, they have limitations:

4. Practice Questions

Test your understanding of p-values and statistical significance with these practice questions. Try answering them before checking the solutions.

Question 1:

A study finds that a new treatment reduces blood pressure, and the p-value for this result is 0.02. Is this result statistically significant?

Answer 1:

Answer, click to reveal

It depends! If the cut-off is 0.05 then yes, the result is statistically significant because the p-value (0.02) is less than 0.05. This suggests that the effect is unlikely to be due to chance. Keep in mind that other cut-offs may exist (0.01 is another common cut-off).

Question 2:

A clinical trial testing a new vaccine reports a p-value of 0.08. Assuming a p-value cut-off of 0.05,  What does this mean in terms of statistical significance?

Answer 2:

Answer, click to reveal

The p-value of 0.08 is greater than 0.05, meaning the result is not statistically significant at the 0.05 level. The study does not provide strong enough evidence to reject the null hypothesis, so the effect could be due to chance.

Question 3:

When might a researcher have a p-value cut-off 0.01 instead of 0.05?

Answer 3:

Answer, click to reveal

Typically, a higher cut-off p-value is used when incorrect conclusions run the risk of having more severe consequences, especially in the case of conclusions drawn from Type I errors.

5. Conclusion

p-values and statistical significance are key concepts in inferential statistics that help researchers determine whether an observed effect is likely due to chance. A statistically significant result provides evidence against the null hypothesis, but it does not measure the size or importance of the effect.

Humanities Moment

The Featured Image for this article is Klänge Pl.19 (1913) by Wassily Kandinsky (Russian, 1866 – 1944). Kandinsky was a Russian painter and art theorist recognized as a pioneering figure in abstract art. Initially trained in law and economics, he turned to art at age 30, studying in Munich and later teaching at Germany’s Bauhaus school until its closure by the Nazis. He spent his final years in France, where he created many significant works and became a French citizen before his death in 1944.

Confidence intervals

In this tutorial, we will explore the concept of confidence intervals, a fundamental tool in inferential statistics. Confidence intervals are used to estimate the range within which a population parameter (e.g., a mean or proportion) is likely to lie, based on sample data. Understanding confidence intervals is essential for making informed decisions in public health research and is also a key topic for the Certified in Public Health (CPH) exam.

By the end of this tutorial, you will understand what a confidence interval is, how to calculate it, and how to interpret it. Practice questions are provided to help reinforce your understanding.

1. Introduction to Confidence Intervals

In inferential statistics, researchers often use data from a sample to make inferences about a population. A confidence interval (CI) is a range of values, derived from sample data, that is likely to contain the true population parameter (such as a mean or proportion). Unlike a single point estimate (e.g., a sample mean), a confidence interval provides a range of plausible values for the population parameter, along with an associated level of confidence.

For example, instead of estimating that the average blood pressure in a population is exactly 120 mmHg, a confidence interval might state that the true average is between 118 and 122 mmHg, with 95% confidence.

2. What is a Confidence Interval?

2.1 Definition of Confidence Intervals

A confidence interval is a range of values around a sample estimate (such as a sample mean) that is likely to contain the true population parameter. It is typically expressed with a confidence level, such as 95%, which indicates how confident we are that the interval contains the true parameter. A 95% confidence interval means that if we were to take 100 different samples and calculate a confidence interval for each, we would expect about 95 of them to contain the true population parameter.

2.2 Confidence Level (e.g., 95%, 99%)

The confidence level reflects the degree of certainty that the confidence interval contains the true population parameter. Common confidence levels are 90%, 95%, and 99%, with 95% being the most frequently used in public health research. A higher confidence level means a wider confidence interval, providing greater assurance that the interval contains the true value, but at the cost of precision.

2.3 Interpretation of Confidence Intervals

Interpreting a confidence interval involves understanding what it tells you about the population parameter:

3. How to Calculate Confidence Intervals

The calculation of a confidence interval depends on the type of data and whether the population standard deviation is known or not. In most public health applications, we calculate confidence intervals for the population mean or proportion based on sample data.

3.1 Confidence Interval for a Population Mean

The formula for calculating a confidence interval for a population mean when the sample standard deviation (s) is known or estimated is:

CI=x‾±Z(sn) \text{CI} = \overline{x} \pm Z \left( \frac{s}{\sqrt{n}} \right)

Where:

Steps to Calculate the Confidence Interval for a Mean:

3.2 Confidence Interval for a Population Proportion

The formula for calculating a confidence interval for a population proportion is:

CI=p±Z(p(1−p)n) \text{CI} = p \pm Z \left( \sqrt{\frac{p(1 – p)}{n}} \right)

Where:

Steps to Calculate the Confidence Interval for a Proportion:

4. Practice Questions

Test your understanding of confidence intervals with these practice questions. Try answering them before checking the solutions.

Question 1:

A study reports that the mean cholesterol level in a sample of 100 adults is 200 mg/dL, with a standard deviation of 15 mg/dL. Calculate the 95% confidence interval for the mean cholesterol level in the population.

Answer 1:

Answer, click to reveal

Step 1: Sample mean (x̄) = 200 mg/dL

Step 2: Standard error (SE) = s / √n = 15 / √100 = 1.5

Step 3: Z-score for 95% confidence = 1.96

Step 4: Margin of error = 1.96 * 1.5 = 2.94

Step 5: Confidence interval = 200 ± 2.94 = [197.06, 202.94]

Question 2:

A survey finds that 40% of respondents support a new public health policy, with a sample size of 500. Calculate the 95% confidence interval for the population proportion.

Answer 2:

Answer, click to reveal

Step 1: Sample proportion (p) = 0.40

Step 2: Standard error (SE) = √[p(1 – p) / n] = √[0.40(0.60) / 500] = 0.0219

Step 3: Z-score for 95% confidence = 1.96

Step 4: Margin of error = 1.96 * 0.0219 = 0.0429

Step 5: Confidence interval = 0.40 ± 0.0429 = [0.3571, 0.4429]

5. Conclusion

Confidence intervals are an essential tool in inferential statistics that provide a range of plausible values for a population parameter, based on sample data. They offer more information than a single point estimate and allow researchers to make informed conclusions about the population.

Remember:

Final Tip for the CPH Exam:

Ensure you understand how to calculate and interpret confidence intervals for both means and proportions, and keep in mind that the standard error and margin of error are unique values, as there might be a question or two on that point as well.

Humanities Moment

The featured image of this article is Great Falls of Snake River, Idaho Territory (1876) by Thomas Moran (American, 1837-1926). Moran was a prominent American landscape painter of the Hudson River School and the Rocky Mountain School, renowned for his vivid depictions of the American West, notably Yellowstone National Park, which his artwork helped establish. Influenced significantly by British artist J.M.W. Turner, Moran excelled in multiple mediums including oils, watercolors, and chromolithography. His celebrated works such as The Grand Canyon of the Yellowstone (1872) solidified his reputation, earning him national acclaim and securing his legacy as one of America’s premier landscape artists.

Sampling methods and sample size

In this tutorial, we will explore two key concepts in inferential statistics: sampling methods and sample size calculation. Sampling refers to the process of selecting a subset of individuals from a population to represent the whole population, while sample size calculation ensures that the sample is large enough to draw valid and reliable conclusions.

By the end of this tutorial, you will understand different types of sampling methods, how to calculate the appropriate sample size, and why these steps are crucial in research design. Practice questions are included to help reinforce your knowledge.

1. Introduction to Sampling and Sample Size

Sampling is the process of selecting a subset (or sample) of individuals from a larger population to estimate population parameters and make inferences. Since it is often impractical to study an entire population, sampling allows researchers to make generalizations about a population based on a smaller, manageable group.

Sample size calculation determines the number of individuals needed in a sample to ensure that the results are statistically valid and reliable. If the sample size is too small, the study may lack the power to detect significant differences. If the sample is too large, resources may be wasted.

2. Types of Sampling Methods

There are several methods of sampling, each with its strengths and weaknesses. The choice of sampling method depends on the research design, the population, and the goals of the study.

2.1 Random Sampling

Random sampling (also called simple random sampling) is a method where each individual in the population has an equal chance of being selected for the sample. Random sampling helps minimize bias and ensures that the sample is representative of the population.

2.2 Systematic Sampling

Systematic sampling involves selecting individuals from a population at regular intervals (e.g., every 10th person on a list). While this method is easier to implement than random sampling, it can introduce bias if there is a pattern in the population list.

2.3 Stratified Sampling

Stratified sampling involves dividing the population into subgroups (or strata) based on characteristics such as age, gender, or income, and then selecting a random sample from each subgroup. This method ensures that all subgroups are represented in the sample.

2.4 Cluster Sampling

Cluster sampling involves dividing the population into clusters (e.g., geographic areas or schools) and then randomly selecting entire clusters to include in the study. This method is often used in large, geographically dispersed populations, but it can introduce bias if clusters are not representative.

2.5 Convenience Sampling

Convenience sampling involves selecting individuals who are readily available or easy to reach. While this method is inexpensive and easy to implement, it is prone to bias and may not be representative of the population.

3. Sample Size Calculation

Sample size calculation is an important step in research design because it ensures that the sample is large enough to detect meaningful differences or associations. The calculation is based on several factors, including the desired level of precision, confidence level, and expected effect size.

3.1 Importance of Sample Size

Choosing the correct sample size is crucial for the validity of a study. If the sample size is too small, the study may not have enough power to detect significant differences or associations, leading to false-negative results. On the other hand, if the sample size is too large, the study may be unnecessarily expensive or time-consuming.

3.2 Factors Influencing Sample Size

Several factors influence the calculation of sample size:

3.3 Sample Size Formula

The formula for calculating the sample size depends on whether the study is estimating a proportion or a mean. A basic formula for estimating sample size when proportions are involved is:

n=Z2p(1−p)E2 n = \frac{Z^2 p(1-p)}{E^2}

Where:

For estimating the sample size for means, the formula is:

n=Z2σ2E2 n = \frac{Z^2 \sigma^2}{E^2}

Where:

4. Practice Questions

Test your understanding of sampling methods and sample size calculation with these practice questions. Try answering them before checking the solutions.

Question 1:

A researcher wants to study the prevalence of diabetes in different income groups. What type of sampling method should the researcher use?

Answer 1:

Answer, click to reveal

The researcher should use stratified sampling to ensure that all income groups are represented in the sample.

Question 2:

A study is conducted to evaluate the effectiveness of a new vaccine, and participants are selected by choosing every 10th person from a list of patients. What type of sampling method is this?

Answer 2:

Answer, click to reveal

This is an example of systematic sampling, where individuals are selected at regular intervals.

Question 3:

A public health study requires a 95% confidence level. The population has an estimated variance of 25, and the desired margin of error is 5. What is the minimum sample size required for the study?

Answer 3:

Answer, click to reveal

Using the formula for sample size when estimating the mean:

n=Z2σ2E2 n = \frac{Z^2 \sigma^2}{E^2}

For a 95% confidence level, Z = 1.96, σ² = 25, and E = 5:

n=(1.96)2(25)52=3.8416×2525=3.8416×1=3.8416≈4 n = \frac{(1.96)^2 (25)}{5^2} = \frac{3.8416 \times 25}{25} = 3.8416 \times 1 = 3.8416 \approx 4

Thus, a minimum sample size of 4 participants is required.

5. Conclusion

Sampling methods and sample size calculation are fundamental aspects of research design in public health. The choice of sampling method ensures that the sample is representative of the population, while sample size calculation ensures that the study is adequately powered to detect meaningful differences or associations.

Remember:

Final Tip for the CPH Exam:

Make sure you understand how to choose the appropriate sampling method and how to calculate the required sample size for different types of studies. Practice identifying different sampling techniques mentioned in various studies, and practice calculating sample sizes.

Humanities Moment

The featured image for this CPH Focus is La Cène (1898) by Alexandre Falguière (French, 1831 – 1900). Falguière  was a renowned French sculptor and painter, celebrated for works like Victor of the Cockfight, Joan of Arc, and The Dancer. A Prix de Rome winner and Officer of the Legion of Honor, he was also a prominent teacher and member of the Académie des Beaux-Arts, leaving behind a legacy of iconic sculptures and paintings now housed in major museums like the Musée d’Orsay.

Data visualization

In this tutorial, we will explore the use of data visualization techniques, including charts, graphs, and histograms, to present and interpret data effectively. Visualizing data allows public health professionals to understand patterns, trends, and distributions within datasets and to communicate findings clearly. Understanding these methods is key for the Certified in Public Health (CPH) exam and for presenting public health data.

By the end of this tutorial, you will understand different types of data visualizations, how to create and interpret them, and when to use each type. Practice questions are included to help reinforce your knowledge.

1. Introduction to Data Visualization

Data visualization is the process of creating graphical representations of data to make complex data easier to understand. Charts, graphs, and histograms are widely used in public health research to display descriptive statistics and to identify trends, distributions, and relationships within datasets.

Effective data visualization helps communicate data insights to diverse audiences, from policymakers to the public. The choice of visualization depends on the type of data and the information you want to convey.

2. Bar Charts

Bar charts are used to display categorical data by representing each category as a bar, with the length of the bar corresponding to the frequency or value of that category. Bar charts are useful for comparing the sizes of different groups or categories.

2.1 Definition of Bar Charts

A bar chart consists of rectangular bars, where the length of each bar represents the magnitude or frequency of the corresponding category. Bars can be arranged vertically or horizontally, and they are typically separated by gaps to emphasize that the data are categorical rather than continuous.

2.2 When to Use Bar Charts

Bar charts are best used when you are comparing discrete categories or groups, such as age groups, gender, or types of diseases. They are especially useful for illustrating the frequency of occurrences or for comparing quantities across different categories.

2.3 How to Create and Interpret Bar Charts

To create a bar chart:

When interpreting a bar chart, look at the lengths of the bars to compare the values or frequencies across categories. For example, a bar chart might show the number of people vaccinated in different age groups, with the longest bar representing the age group with the highest vaccination rate.

3. Line Graphs

Line graphs are used to display trends in data over time or continuous data. A line graph connects individual data points with a line, allowing viewers to see how a variable changes over time or across a continuous range.

3.1 Definition of Line Graphs

A line graph uses points connected by a line to represent data points in a sequence. It is most commonly used to track changes over time, but it can also be used to display continuous data. Line graphs are particularly effective for showing trends, patterns, and relationships over time.

3.2 When to Use Line Graphs

Line graphs are best used when you want to visualize changes over time or trends in continuous data. They are ideal for displaying data such as disease incidence rates over time, temperature changes, or population growth trends.

3.3 How to Create and Interpret Line Graphs

To create a line graph:

When interpreting a line graph, look for trends over time or patterns in the data. For example, a line graph might show the rise and fall of flu cases throughout a year, with peaks during the winter months.

4. Histograms

Histograms are used to display the distribution of continuous data. They group data into intervals (bins) and display the frequency of data points that fall within each interval. Histograms provide a visual representation of the shape of the data distribution, such as whether the data are skewed or normally distributed.

4.1 Definition of Histograms

A histogram is similar to a bar chart, but it is used to display continuous data rather than categorical data. The x-axis represents the intervals (bins) of the continuous data, and the y-axis represents the frequency of data points within each bin. Unlike bar charts, the bars in a histogram touch each other to indicate that the data are continuous.

4.2 When to Use Histograms

Histograms are best used when you want to visualize the distribution of continuous data. They are particularly useful for showing the frequency of data points within specified ranges and for identifying patterns such as skewness, modality, or the presence of outliers.

4.3 How to Create and Interpret Histograms

To create a histogram:

When interpreting a histogram, look for the shape of the distribution. For example, a histogram might show that most data points are clustered around a central value, indicating a normal distribution, or it might reveal a skewed distribution with more data points on one side.

5. Practice Questions

Test your understanding of data visualization with these practice questions. Try answering them before checking the solutions.

Question 1:

A public health researcher wants to compare the number of flu cases in different cities. What type of graph should the researcher use?

Answer 1:

Answer, click to reveal

The researcher should use a bar chart to compare the number of flu cases across different cities, as bar charts are best for comparing categorical data.

Question 2:

A study tracks the daily number of new COVID-19 cases over the course of a year. What type of graph is most appropriate to display this data?

Answer 2:

Answer, click to reveal

A line graph is most appropriate, as it can display trends in the number of cases over time.

Question 3:

A public health researcher is analyzing the distribution of ages in a population. The ages range from 18 to 80. What type of visualization should the researcher use to show the distribution of ages?

Answer 3:

Answer, click to reveal

A histogram should be used, as it can show the frequency of ages within specified intervals, making it easy to see the overall distribution of the data.

6. Conclusion

Data visualization is a critical tool for understanding and communicating public health data. Charts, graphs, and histograms help present complex data in a clear and interpretable way, allowing public health professionals to identify trends, distributions, and relationships in their data. Each type of visualization has its strengths, and choosing the right one depends on the type of data and the message you want to convey.

Remember:

Final Tip for the CPH Exam:

Ensure you understand when each type of visualization is best suited for what sort of underlying data. It’s encouraged you look at several studies, examine what data types they are using in their study, and how that data is then represented. Ask yourself if it was appropriate or if a better choice was available.

Humanities Moment

The featured image for this CPH Focus is Fruit Still Life With Two Parrots (1623) by Balthasar van der Ast (Dutch, 1593-1657). Balthasar van der Ast, a Dutch Golden Age painter born in Middelburg around 1593-1594, was a pioneer of shell still lifes and known for his intricate still life paintings of flowers, fruit, insects, and lizards. Trained by his brother-in-law Ambrosius Bosschaert the Elder, van der Ast influenced the “Bosschaert dynasty” and other prominent painters, eventually settling in Delft, where he joined the Guild of St. Luke and lived until his death in 1657.

Text data types for NLP in Python

Welcome to PyFriday! If this is your first time exploring Natural Language Processing (NLP), you’re in the right place. NLP is all about teaching computers to understand, interpret, and interact with human language. To do that effectively, we use various text data types to represent and process text. Before we dig into the big NLP topics that will go from data cleaning to building out your own Machine Learning models, we’ll need to take some time to learn about these text data types.

Please keep in mind, while we aim to keep this tutorial very beginner friendly, there is some assumptions of experience in Python. If you're completely new to Python, we recommend checking out our earlier PyFriday entries!   Normally, we host all code presented on Cody’s Github. This time however, since it's a bunch of smaller examples, we'll keep the examples in the article.

What is Text Data in NLP?

In NLP, text data refers to the words, sentences, or paragraphs we process. Computers can’t understand text like humans do—they need structured representations of text to work with it. These representations are the text data types we’ll discuss. Imagine you want a computer to analyze this sentence:
“Welcome to PyFriday!”
The computer must:
  1. Store the text.
  2. Break it into parts (e.g., words or letters).
  3. Analyze it (e.g., find the most important words).
  4. Apply meaning (e.g., recognize “PyFriday” as a name).
We use different data processing for each step, which often includes breaking up the data into several types.

Strings: Definition, Use Cases, and Examples

What Are Strings?

A string is a basic data type that represents text. You can think of it as a single piece of text, like a word, sentence, or paragraph. Computers use strings to store and manipulate text, and can be as short as one letter, to entire books.

Why Use Strings in NLP?

Strings are fundamental for:
  • Storing raw text data, like a tweet or a paragraph. It’s basically as fundamental a data type as integers or boolean values.
  • Doing simple text processing, such as converting to lowercase or removing punctuation.

Example

Here, we assign a sentence or string as text, and print it. Next, we turn all letters lowercase and print again. Lastly, we split the string into a list of tokens (both terms will be discussed shortly), and print that. These are quick examples of how string data is read in, and can be cleaned.

Lists of Strings: Definition, Use Cases, and Examples

What Are Lists of Strings?

A list is like a container that can hold multiple pieces of text (strings). When you split a sentence into words, you usually end up with a list of strings.

Why Use Lists of Strings in NLP?

  • Storing tokenized text (text broken into smaller pieces like words).
  • Processing multiple pieces of text in order (e.g., sentences in a paragraph).
  • Can better organize/categorize data (e.g. a list of strings can give you an organized way to look at chief complaints by visit).
    • These can be further abstracted into lists of lists of strings, like if you want to see chief complaint by patient over time, by hospital.

Examples

Here, we take our original sentence, split it by word to form a list, and then rejoin the words from a list back into a string, with a space between each list entry to make things intelligible.

Dictionaries: Definition, Use Cases, and Examples

What Are Dictionaries?

A dictionary is a data type that pairs values with keys. Think of it like a mini-database where each key has a value. In NLP, dictionaries are often used to store information about words, like their frequency in a document.

Why Use Dictionaries in NLP?

  • To count words (important for understanding which words are common or rare).
  • To map words to other data, like their synonyms or embeddings (discussed later).

Examples

For this example, we’re taking a sentence and putting it into a variable. Then, we set up an empty dictionary called word_counts. From there, we use a for loop to iterate over every word, and each time a word shows up, we add to its count value in the dictionary.

DataFrames: Definition, Use Cases, and Example

What Are DataFrames?

A DataFrame is a table-like data structure (think of an Excel sheet). It’s great for handling text data with additional information, such as labels or metadata.

Why Use DataFrames in NLP?

  • To organize large datasets, like medical records with lots of subfields.
  • To store text and related attributes for machine learning, analysis, etc.

Example

Token Objects:  Tokens for NLP, Use Cases, and Example

A token object is a specialized representation of a word or a segment of text used by NLP libraries such as spaCy or NLTK. Tokens go beyond just being individual words—they include additional information that helps computers understand the structure and meaning of the text. For instance, a token might know whether it’s a noun, verb, or adjective, its base form (lemma), or even its position in a sentence. When raw text is processed by an NLP library, the library “tokenizes” the text, breaking it into smaller parts (tokens). Each token becomes a rich object containing metadata about that piece of text.

Key Features of Token Objects

  1. Text Content: The actual word or segment of text represented by the token.
  2. Lemma: The base form of the word (e.g., “running” becomes “run”).
  3. Part of Speech (POS): The grammatical category of the word (e.g., noun, verb, adjective).
  4. Dependencies: Relationships between words in the sentence (e.g., subject, object).
  5. Offsets: Start and end positions of the token in the original text.
  6. Attributes: Whether the token is a punctuation mark, stop word (common but unimportant word like “the” or “and”), or other special text.

Why Use Token Objects in NLP?

Token objects are invaluable because they allow detailed linguistic analysis without having to write custom code for parsing and annotating text. These objects make it easy to:
  • Analyze grammatical structure: Understand how words function in a sentence.
  • Extract metadata: Retrieve information like lemmas and part-of-speech tags for downstream tasks.
  • Annotate text efficiently: Use built-in attributes to quickly identify features like stop words or punctuation. Some functions within NLTK and SpaCy actually do the part of speech annotation for you in some circumstances, making some tasks much less troublesome.

Examples

For this example, we’re loading in Spacy and from it, loading in a small English pipeline that’s very feature rich. We then use this pipeline to take core data from our sentence using the nlp method. Then for each word, we print out the text, the lemma, and the part of speech the word or symbol belongs to.

Sparse Matrices: Efficient Data Structures, Use Cases, and Examples

What Are Sparse Matrices?

A sparse matrix is a compact way to store data with lots of zeroes. In NLP, it’s used to represent text as numbers, like in the bag-of-words model, where each word’s presence is encoded as 1 (present) or 0 (absent). It should be noted that “compact” does not mean small; While the storage requirements are usually much smaller than some other matrix based interaction tables, for large corpi you can quickly hit a matrix with billions of cells.

Why Use Sparse Matrices in NLP?

  • To efficiently store large datasets.
  • To prepare text for simpler machine learning models. Dense Matrices, which are much less memory efficient are often used for more complex tasks.

Examples

This one is a bit trickier without some underlying knowledge, but let’s dig into it. First, we’ll import SciKitLearn, a machine learning package, and assign the tool for converting text into a numerical matrix (CountVectorizer()) to an easier to use “vectorizer”. Then, we have our documents that we want to fit into a matrix. When we say fit, we mean it looks at the word choices for each document and identifies each unique word, and converts this information into a matrix showing if any of the unique words actually show up in the document. In this case, the words “fun”, “nlp”, “is”, “with” and “pyfriday” are all unique terms. Normally this function will also arrange words in alphabetical order, so we need to keep that in mind when examining whether a word is or is not present in the subsequent matrix.

Vectors and Embeddings: Dense Vector Representations, Use Cases, and Examples

What Are Vectors and Embeddings?

In NLP, vectors are numerical representations of text. They are a way to convert words, phrases, or even entire documents into a format that machines can process: arrays of numbers. Each dimension in a vector represents some aspect of the text, such as its occurrence in a corpus, its relationship to other words, or its position in a semantic space. Embeddings are a specific type of vector representation that go beyond simple numerical representations like one-hot encoding or term frequency. Embeddings are dense vectors learned from data, and they capture the semantic meaning of words by placing similar words closer together in the vector space.

Why Use Vectors and Embeddings in NLP?

  • To represent text numerically for machine learning.
  • To capture relationships between words (e.g., “king” and “queen” are related).

Examples

here, we have a slightly different bit of code than we did with the sparse matrix. Namely, we’re using a vectorizer that takes into account relative importance as well as position, so we don’t just have 1’s and 0’s, but rather the relationship dynamic as well that ranges from 0-1 showing how critical a term is to the text.

Sequences: Ordered Collections for Modeling, and Examples

What Are Sequences?

A sequence in NLP refers to an ordered list of elements that can represent words, characters, or tokens. Sequences preserve the order in which these elements appear in the text, which is crucial for understanding the context and meaning of the text. Unlike individual words or isolated tokens, sequences allow models to capture the relationships between elements over time or across positions. Sequences are used in models designed to process and understand text step-by-step, such as Recurrent Neural Networks (RNNs), Long Short-Term Memory networks (LSTMs), and modern architectures like Transformers.

Why Use Sequences in NLP?

  • Language inherently depends on order. For example:
    • The sentence “I ate the cake” is vastly different from “The cake ate I.”
    • Words gain meaning from their position and their relationship with surrounding words.Language inherently depends on order. For example:
  • Sequences help capture:
    • Context: Understanding how earlier elements in the sequence influence later ones (e.g., “not happy” conveys a different meaning than “happy”).
    • Order: The specific arrangement of elements matters (e.g., the difference between “He eats quickly” and “Quickly, he eats”).
    • Dependency: Words or tokens often depend on one another over long distances in a text.

Examples

This itself is an incredibly simple example, mostly as we’ll cover sequences much later and it’ll require much more depth. For now though, the most basic of basics will do.

Conclusion

In this tutorial, we explored the foundational text data types used in natural language processing: strings for raw text; lists of strings for tokenized words or sentences; dictionaries for mapping words to values; and DataFrames for organizing datasets with metadata. We also introduced token objects, sparse matrices, vectors and embeddings, and sequences. Together, these data types provide a foundation for processing, analyzing, and modeling text.

Happy learning, and see you at the next PyFriday!

Humanities Moment

The featured image for this PyFriday is Manuscript Book mural in Evolution of the Book series (1896) by John White Alexander (American, 1856-1915). John White Alexander was an American painter and illustrator known for his portraits, decorative works, and murals. Orphaned as a child, he was raised by his grandparents and later mentored by Edward J. Allen, who nurtured his artistic talent. Starting as an illustrator for Harper’s Weekly, he pursued formal training in Europe, gaining influence from artists like Whistler and Duveneck. Returning to New York, he achieved acclaim for his portraits of prominent figures and murals like "Apotheosis of Pittsburgh," with his works displayed in major museums across the United States and Europe.