What a 95% Vaccine Efficacy Rate Actually Means

After over a year filled with huge amounts of negativity in the news, it has been refreshing and exciting to hear the numbers of COVID-19 vaccines being administered. (Could 'life as we knew it' be just around the corner?)

But along with the hope and good cheer surrounding the vaccinations, also comes the concern over just how effective they really are. Headlines have told us that the trials showed the vaccines to be "XX% effective", but often that number isn't actually explained, or is explained very briefly.

In this post, I'll take the time to explain what it does mean, and what it doesn't. I'll then provide some ideas for how to approach this topic in a hands on way in a math classroom, to get your students engaged and involved.

What Vaccine Efficacy Isn't

First of all, 95% vaccine efficacy rate does NOT mean that:

• 5% of vaccine recipients got sick with COVID-19. This would be wild, and rather terrifying. Even in many countries that have been hit hardest by the pandemic, the total percentage of the population to have a confirmed case of COVID-19 is approximately 3-5% (The United States is at approximately 3.7%). If the people getting vaccinated were getting sick at a higher rate than the general population, the vaccine would actually be detrimental. Yikes! Thankfully, this is not the case.
• 95% of the coronavirus is blocked, so you won't get as sick when you do come down with the virus. While some of the studies have shown a reduced severity of COVID-19 in those who were vaccinated, when they still got sick with the disease, there is nothing to quantitatively suggest that they only got 5% as sick as those who were unvaccinated.
• the vaccine will work on 95% of the population, but has no impact on the remaining 5%. Contrary to this, early returns on the vaccines have shown that there is a similar level of effectiveness, regardless of population group. Further testing, however, will have to be done to confirm that the vaccine will work equally well for all target groups.

That gives you an idea of what the 95% doesn't mean, so let's look at what it does mean. In particular, we'll investigate how the effectiveness number is calculated

Calculating the vaccine efficacy

Step 1: Divide your study into two groups and administer the vaccine

Split your participants into two randomized groups of equal size. The importance of equal size will become apparent later. One group will receive the vaccine being tested, and the other group will receive a placebo. In Moderna's study, they had a sample size of 30,000 individuals, which were split into two groups of 15,000.

Step 2: Track participants for incidence of COVID-19 after injection

To assess the effectiveness of a vaccine, ideally those who received the vaccine should get sick at a much lower rate than those who received the placebo. Since we will be comparing the number of positive cases in each group, it is important that they are the same size. The fact that there are more cases of COVID-19 in the group that received the placebo doesn't mean much if there were also far more people who were given the placebo. In Pfizer's trial, there were 170 total cases of COVID-19 in their studied population, including 162 in the placebo group, and 8 in the vaccine group.

Step 3: Calculate the vaccine efficacy

Edit: Initially, I had an incorrect calculation here. Thanks to my colleague Nick Nesbitt for pointing me in the right direction. Updated calculation is below.

To calculate the effectiveness as a percentage, first subtract the number of COVID-19 cases in the vaccine population from the number in the placebo group, and then divide by the number of COVID-19 cases in the placebo group.

For Pfizer's trial, the efficacy was (162-8)/162 = 95.1%. For Moderna, there were 95 total cases of COVID, of which 90 were in the placebo group, so the effectiveness was (90-5)/90 = 94.4%

So what does this actually mean?
We've discussed what vaccine efficacy isn't, and shown how to calculate it, but how can we define it simply. Essentially, a vaccine with efficacy of X% will prevent X% of occurrences of the virus that would have otherwise occurred. In Moderna's case, there were 90 cases of COVID in the control group. Since there were only 5 cases in the test group, 85 out of 90 cases that would have occurred did not, presumably because of the vaccine (assuming the same size group, properly created sample groups and consistent reporting).So even vaccines with a comparatively lower efficacy rate are still preventing a substantial proportion of infections!

Questions for Further Discussion

1. If the two control groups are different sizes, how could this impact the calculation of the efficacy? How might an unscrupulous study use this to make their vaccine seem more effective? If the groups are different sizes, design a calculation that would take this into account when calculating the efficacy.
2. If a vaccine is 95% effective, how many times less likely are you to get sick if you get the vaccine compared to if you don't? What if the vaccine is 90% effective? 50% effective?
3. Other than the effectiveness of the vaccine, what other factors should be considered when deciding which vaccine or vaccines should be mass produced?

Design your own experiment

Here is an activity that students can do to simulate the design of a study and the calculation of an efficiency.

1. Have your students develop some sort of simple, but not too simple, pass/fail test, like a spelling test of made-up words, or a quiz about obscure animal facts.
2. Divide the class into two groups of equal size.
3. Provide one of the groups with some sort of assistance for the test, like a summary sheet, or access to the questions beforehand. The other group, your control group, will receive no advance help or support
4. Conduct the test and determine the results. Feel free to make the results anonymous, as it isn't important at all which specific students pass the test, and which ones do not. The only thing that matters is which group the students are in.
5. Calculate the efficacy of your support technique.

A version of this article first appeared on LinkedIn