Category Archives: confidence interval

Covid 19 Vaccine: Statistics, Safety, Efficacy, What is a Confidence Interval?

This week began the saga of vaccine implementation for the mRNA vaccine developed by the Pfizer-BioNtech partnership. The New England Journal of Medicine published a peer-reviewed article reporting the data from the RCT (Randomized Controlled Trial) and an Editorial appeared on the same day in the NEJM. You can view both here:

http://Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine

http://SARS-COV-2 Vaccination — An Ounce (Actually, Much Less) of Prevention

If you go to the first link you can click on a link to a 3 minute video that summarizes the results. The video is informative and succinct.

Here is the most important table in the peer-reviewed article:

Age > 55 includes >=65 and >=75, Age >= 65 includes age >=75. If age>= 75 were not listed separately, all age groups would have statistical significance. This is why overall number of infections in the vaccine group is 8 but the total of the four age groups is 9. One of the 3 infections in age >55 was 65 or older. Got it?

Look at the third column labeled “Surveillance Time”. The first number in this column is described under the caption as “total time in 1000 person-years for the given endpoint..” So multiply by 1000 and you get a number that represents “person years” of observation. This is an odd number but a reasonable way to present information. The trial data covers a period of 6 months starting from when the first patient was vaccinated. The median duration of observation was 2 months after vaccination with the second dose. (50% of patients had been vaccinated for more than 2 months and 50% of patients less than 2 months)

The number in parentheses gives the number of people in each category of age, sex, race/ethnic group, country.

Notice that as we descend through the age groups from youngest to oldest, the number of participants decreases significantly in >= 65 and >=75 years old.

Now go to the last column labeled “Vaccine Efficacy, %”. The first number is the overall efficacy for that age group. But how confident are we about that efficacy % for each age group? The numbers in the parentheses represents a range or “confidence interval” which is determined statistically. This range is determined by the number of people in each (age) group and by the difference between the placebo and drug treatment outcome in that age group.

Note that the confidence interval widens as you go down from younger to older age groups. That is primarily because the number of people represented by the groups age >=65 and >= 75 are much smaller than the first two age groups.

For the age group > 65, the efficacy is reported at 94.7%. But that is an average number that shows that compared to the placebo group, the symptomatic infection rate in the vaccinated group was 94.7% lower. For the age group between >=65, there were 19 cases of symptomatic infection in the placebo group and only 1 in the vaccine group. 1/19 equals 0.0526. 1 minus 0.0526 equals 0.947 or 94.7% (rounding).

What is a 95% confidence interval?

The 95% confidence interval is the the range over which we are 95% confident the “true” value falls within. In other words, for age >=65 there is a 95% probability that the true vaccine efficacy falls somewhere between 66.7% and 99.9%. As more people age >=65 are entered into the trial, the confidence interval will get narrower for that age group provided the difference between the placebo and vaccine group remains the same. If the difference between placebo and vaccine groups increases AND the number of people in this age group increases, the confidence interval will shrink further.

This is a fantastic result!

But now look at the confidence interval for age >=75. It is -13.1-100. A minus number is in the range. That means we do not have enough people in the study age 75 or older to reach a “statistically significant” conclusion. There were only 774 folks who received the vaccine and 785 people who received the placebo in this age group. 5 in the placebo group got sick, 1 in the vaccine group got sick. This difference shows a “trend” but does not reach “statistical significance”. We need more data for this group.

But again, do not lose heart. Based on this data, which includes efficacy and safety overall, I recommend that my 92 year old mother-in-law receive the vaccine as soon as it is available.

What is “statistical significance”?

Simply stated, a result is “statistically significant” in a drug trial if the probability that the difference between the placebo group and the drug group occurred by accident (or by chance) is less than or equal to 5%. Or to put it another way, the probability that the difference between placebo and drug did not occur by chance but rather represents a true difference in outcome is 95% or greater. 5% is the standard cutoff point in medical trials. This is usually reported as a “p value”. If a p value is less than or equal to 0.05 the difference is considered statistically significant.

“Clinical significance” is another issue. For example, a drug might only decrease relative risk of an event by 5%, but the result could be “statistically significant”.

In the case of this vaccine, the clinical significance is outstanding.

Here is another important chart.

Protection begins at day 14 after first dose

This chart demonstrates that 14 days after the first dose of vaccine protection from symptomatic infection begins. The blue line shows infections in the placebo group, the red line is the vaccine group. They diverge at day 14. Thereafter the divergence increases. This makes sense. It is consistent with our understanding of how the immune system works. This time course is very reassuring.

Now let us look at side effects.

These are the usual side effects expected with any vaccine. Symptoms such as fever, malaise, local reaction at the injection site, muscle aches are all to be expected. This profile is a little worse than the flu vaccine, but less compared to the shingles vaccine.

So far 2 episodes of anaphylaxis have occurred in the US and in UK with the vaccine rollout. (millions of doses already given) This can happen with any vaccine, that is why 15 minutes of observation following vaccination is part of the protocol and the vaccine should only be administered in a location that has epinephrine available to treat a severe allergic reaction.

This vaccine time sequence is historically unprecedented. This was made possible by a combination of advances in science and incredible action on the part of the public and private sectors in response to a global crisis. Prior to this, the shortest time to develop an effective vaccine was 4.5 years (measles mumps, and that was done by Maurice Hilleman with the help of a few lab assistants, more about Maurice in my next post).

I received my first dose of the vaccine today (as a member of hospital medical staff). I was directed to a CDC website to register as a vaccine recipient. The CDC will send me periodic questions on side effects which I will answer. If everyone participates in this program more data will become available on safety and efficacy. So please participate when you get the vaccine (vsafe.cdc.gov).

Many questions remain. Here is a quote from the NEJM editorial:

  1. Will unexpected safety issues arise when the number grows to millions and possibly billions of people?
  2. Will side effects emerge with longer follow-up?
  3. Implementing a vaccine that requires two doses is challenging. What happens to the inevitable large number of recipients who miss their second dose?
  4. How long will the vaccine remain effective?
  5. Does the vaccine prevent asymptomatic disease and limit transmission?
  6. And what about the groups of people who were not represented in this trial, such as children, pregnant women, and immunocompromised patients of various sorts?

The logistic challenges of manufacturing and delivering a vaccine remain daunting. This vaccine, in particular, requires storage at −70°C, a factor that may limit its deployment in some areas. Nevertheless, the remarkable level of safety and efficacy the vaccine has demonstrated thus far make this a problem that we should welcome solving. What appears to be a dramatic success for vaccination holds the promise of saving uncounted lives and giving us a pathway out of what has been a global disaster.

In the context of the COVID 19 pandemic I will close with the usual summary.

  1. Avoid alcohol consumption (alcohol wreaks havoc with your immunity)
  2. Get plenty of sleep (without adequate sleep your immune system does not work well )
  3. Follow good sleep habits
  4. Exercise, especially out of doors in a green space, supports the immune system
  5. Get some sunshine and make sure you have adequate Vitamin D levels.
  6. Eat an anti-inflammatory diet rich in micronutrients.
  7. Practice stress reduction like meditation and yoga which improves the immune system
  8. Eliminate sugar-added foods and beverages from your diet. These increase inflammation, cause metabolic dysfunction, and suppress immunity.
  9. Eliminate refined-inflammatory “vegetable oils” from your diet, instead eat healthy fat.
  10. Clean up your home environment and minimize your family’s exposure to environmental toxins by following recommendations at EWG.org with regards to household products, personal care products, and organic foods. (https://www.ewg.org/)

THIS WEBSITE PROVIDES INFORMATION FOR EDUCATIONAL PURPOSES ONLY. CONSULT YOUR HEALTH CARE PROVIDER FOR MEDICAL ADVICE.

Eat clean, drink filtered water, love, laugh, exercise outdoors in a greenspace, get some morning sunlight, block the blue light before bed, engage in meaningful work, find a sense of purpose, spend time with those you love, AND sleep well tonight.

Doctor Bob