There are 13 video-based lessons in this course. Work through each lesson at your own pace. Discussion board
Post questions to the discussion board. I will post answers using text or custom-made videos. Public zoom meetings
We will hold group zoom meetings several times a week. Private zoom meetings
I will also host private zoom meetings by appointment.
At the conclusion of this course, you will understand:
Basic issues in meta-analysis
How to choose a statistical model
The meaning of the summary effect size
How we can (and cannot) generalize from the meta-analysis to other populations
How to choose an effect size index
How to choose between narrow vs. broad inclusion criteria for selecting studies
How to interpret each of the statistics associated with heterogeneity
What I-squared tells us (and does not tell us)
How to compute a prediction interval
How to plot the distribution of true effects
How to understand the difference between a confidence interval and a prediction interval
How to write a report that explains the results of the analysis
How to avoid common mistakes
Avoid mistakes in choosing a statistical model
Understand limitations of the fixed-effect analysis
Understand limitations of the random-effects analysis
Understand the common misinterpretation of the I-squared statistic
Is this course too basic for me?
Probably not. While the course is geared to researchers, statisticians who attend our workshops invariably report that they found the course well worth their time. The overwhelming majority of meta-analyses contain some serious mistakes. You will learn what these mistakes are and how to avoid them in your own work.
Is the course too advanced for me?
Probably not. The course assumes only basic knowledge of research techniques and no knowledge of meta-analysis. The course is intended for researchers rather than statisticians. I focus on a conceptual approach to each issue and use practical examples from published analyses. Statistical formulas are typically addressed separately and explained in a way that focuses on the logic rather than the math.
Can I see a sample video?
To get a sense of the course, watch this video on heterogeneity. This should give you a sense of how the course works.
Do I need to be online at any specific time?
No. The videos are available to watch at your leisure.
The zoom sessions will be scheduled at various times of the day/night so you can attend any that work for you.
You can post questions at your convenience. We will generally post a response within 1 business day. In some cases this will be written, and in some cases I will create a video in response. If the issue calls for an extended discussion, I will set up a private zoom based on your time zone.
How much time will I spend on this course?
To work through the videos, and carry out all the analyses demonstrated in the videos, will take roughly 15 hours. To get the most out of this course I recommend that you also participate in some of the zoom meetings, which will take as much time as you’d like. You may also want to use your own data sets to practice the concepts explained in the videos. If you have questions, we can address these in written correspondence or via zoom.
The subscription runs for the number of months you select. This allows you to complete the course and apply it to your own work. Then, you can choose to re-watch the videos and ask additional questions.
May I ask questions about my own data?
I assume that some participants will want to apply the concepts to their own data and will have questions when they do so. I encourage that and will be happy to address these questions. I try to address general questions in the group chats and questions of a more specific nature in private chats.
I am happy to address questions about your analysis (for example “What does it mean that I-squared is 80% despite the fact that the effects all seem to be consistent” or “The journal said that we should apply this statistical model – how should we respond?”). However, I am not set up to review entire papers (for example, “Could you critique this paper?”). To review entire papers properly requires an amount of time that would not be possible.
What This Course Does Not Cover
This course covers basic meta-analysis. The specific topics are detailed in the table of contents. Over the next few months we will be adding intermediate and advanced courses in meta-analysis. Persons who attend the basic course will be offered a discounted rate for those courses.
The course does not cover network meta-analysis, meta-analysis of diagnostic tests, or Bayesian methods.
The course is taught by Michael Borenstein. Dr. Borenstein is the co-author (with Larry Hedges, Julian Higgins, and Hannah Rothstein) of the text Introduction to Meta-Analysis. He is also the author of the text Common Mistakes in Meta-Analysis and How to Avoid Them. Dr. Borenstein has been teaching workshops on meta-analysis for fifteen years. These include invited workshops at the FDA, CDC, NIH, and various pharmaceutical companies and universities. It also includes workshops open to the public in the United States, the UK, Australia, Singapore, Israel, and Switzerland. These workshops have been attended by some two thousand people. He regularly reviews papers for journals in the fields of medicine and social science.
The goal of this course is to teach meta-analysis, rather than primarily teaching how to use any specific program. However, I will be using the software Comprehensive Meta-Analysis (CMA) in course examples. Course registration includes free access to the program for one month.
There is no required text. However, the course is based on the following texts, which can be ordered on Amazon:
Modules marked with an asterisk (*) should not be skipped. These contain information that is important for the modules that follow.
Introduction 60 Minutes
[ More Information ]We start with a simple meta-analysis to compare the relative utility of two treatments for preventing cardiovascular events. I use this to outline the elements of a meta-analysis that we will be exploring in later modules. I also use this to show how we can perform a simple analysis from start to finish, including generating a high-resolution plot and writing a report.
How a Meta-Analysis Works 21 Minutes
[ More Information ]In this module I use a series of fictional studies to show what happens as we add studies to a meta-analysis. When the effect size is consistent across studies, we focus on the common effect size. When the effect size varies across studies we estimate the mean effect size, but we also need to estimate the dispersion in effects and consider the implications of this dispersion for the utility of the intervention.
Fixed Effect vs. Random Effects * 74 Minutes
[ More Information ]Every meta-analysis must be based on a statistical model. The model tells us how the studies were sampled and how we can generalize from them to other studies or populations. There is a widespread belief that the fixed-effect model applies when the true effect size is the same in all populations, and the random-effects model applies otherwise. The reality is more complicated. The fixed-effect model applies when our goal is to estimate the common effect (or the mean effect) for the studies actually included in the analysis. The random-effects model applies when our goal is to generalize from these studies to a wider universe of comparable studies or populations. We discuss how to select a model, and also how to avoid common mistakes related to this issue.
Effect Sizes 58 Minutes
[ More Information ]The effect size is the unit of currency in a meta-analysis. We compute an effect size for each study. Then, we pool these values to estimate the common (or mean) effect size, we estimate the dispersion in effects, and we sometimes try to explain the variation in effects. In this module, I discuss some of the common effect-size indices. We discuss how to choose an effect size index, and how to understand and explain the meaning of the effect size.
Tamiflu Symptom Relief Time TBD
[ More Information ]Tamiflu is a drug that is widely prescribed for treatment of flu symptoms. This is an analysis of randomized controlled trials (RCTs) that compared the duration of flu symptoms in patients treated with Tamiflu to those treated with a placebo. This is a relatively simple example. It provides experience working with a difference in means where the effect size is consistent across studies. I show how to perform the analysis and report the results.
Effect Size vs. P-value 27 Minutes
[ More Information ]In primary studies that compare outcomes in two groups, there are two general approaches that researchers apply. One is to test the null hypothesis of no effect and report a p-value. The other is to estimate the effect size, and report that effect size along with a confidence interval. In this module, I show how the two approaches are related to each other and why we almost always want to focus on the effect size approach. Then I extend this to meta-analysis, where it is imperative that we work with the effect size from each study rather than the p-value.
What Studies to Include 27 Minutes
[ More Information ]In any meta-analysis we can choose to work with a narrowly defined population and a specific variant of the intervention. In this case, we assume that the effect size will be reasonably consistent across studies, and our goal will be to estimate this effect size. Alternatively, we can choose to include an array of populations and/or variants of the intervention. In this case, our goal will be to assess the dispersion in effects and possibly to see what moderators are associated with this dispersion. We explain how to make these decisions and how to map them to the inclusion/exclusion criteria.
Heterogeneity * 88 Minutes
[ More Information ]In most meta-analyses, the effect size varies from study to study. We would all agree that it’s important to understand how much the effect size varies, and to consider the clinical or substantive implications of this variation. In practice, however, this is rarely done. The vast majority of meta-analyses focus on the mean effect size, while little attention is paid to the dispersion in effects. This is because researchers don’t understand how much the effect size varies. In fact, the statistics typically reported for heterogeneity don’t actually tell us how much the effect size varies. For example, the most common index for reporting heterogeneity is the I-squared index, with I-squared values of 25%, 50%, and 75% often assumed to reflect low, moderate, and high levels of heterogeneity. While this use of I-squared is widespread, it is nevertheless incorrect. A meta-analysis where I-squared is 25% could have substantial variation in effects, while a meta-analysis where I-squared is 75% could have only trivial variation in effects. In fact, I present examples where this is true.
In this module, I start by reviewing how we think about heterogeneity in a primary study. Then, I show that the same ideas apply in a meta-analysis. In a section called “Forget what you know,” I show that most of what researchers “know” about heterogeneity is wrong. Statistics such as the Q-value, the p-value, I-squared and Tau-squared, do not tell us how much the effect size varies. Then, I discuss the statistics that do actually tell us how much the effect size varies – these include Tau (in some cases) and the prediction interval. I show how to compute and report these values. I then discuss how to use the heterogeneity, in conjunction with the mean effect size, to consider the clinical utility of the treatment or (more generally) the substantive implications of the findings. I also discuss what the other statistics do tell us. The module ends with an appendix that shows how the various statistics are related to each other, using clear and intelligent graphics.
ADHD 46 Minutes
[ More Information ]In the module on heterogeneity, I refer to a meta-analysis that I call ADHD. This is an analysis of seventeen randomized controlled trials (RCTs) that assessed the impact of methylphenidate on adults with ADHD (attention deficit hyperactivity disorder). Now, I show how to run this analysis from start to finish, quantify the variation in effects, report it, and consider the clinical implications. This is an opportunity to develop a feel (in practice) for the concepts we earlier discussed in the abstract. This is also an opportunity to become familiar with the software Comprehensive Meta-Analysis. This example uses the standardized difference in means as the effect-size index.
I-squared 64 Minutes
[ More Information ]In the module on heterogeneity, I explain that I-squared does not tell us how much the effect size varies. When I teach this course in person, researchers who are new to meta-analysis accept this fact, but those who have been performing meta-analyses (especially in medicine) often find this idea hard to accept.
If you’ve never heard of I-squared before this course, you may want to skip this module. But if you have been working with I-squared, you are likely to find this module enlightening. A few years ago I was teaching this workshop in London, and Julian Higgins (who created the I-squared statistic with Simon Thompson) was kind enough to drop in at the workshop and explain to the participants that what I said about I-squared is correct. This module includes a video clip of Julian’s remarks to the group.
Prediction Intervals 33 Minutes
[ More Information ]In prior modules I explained why it’s important to understand how much the effect size varies across studies. I also explained that the statistics typically reported for heterogeneity (such as I-squared) don’t actually tell us how much the effect size varies. The one statistic that does provide this information is the prediction interval. It tells us how much the effect size varies. Additionally, it combines this with information about the mean. As such, it allows us to distinguish between the case where the effects vary from trivial to large (on the one hand) versus harmful to helpful (on the other). In this module I show how to compute and report the prediction interval.
Viagra 52 Minutes
[ More Information ]This is an analysis of randomized controlled trials (RCTs) that assessed the impact of Viagra for patients with erectile dysfunction. I show how to run this analysis from start to finish, quantify the variation in effects, report it, and consider the clinical implications. This example uses the risk ratio as the effect-size index.
PTSD 37 Minutes
[ More Information ]This is an analysis of studies that assessed the prevalence of PTSD (post-traumatic stress disorder) in mothers whose children were being treated for serious chronic illness. I show how to run this analysis from start to finish, quantify the variation in effects, report it, and consider the clinical implications. This example uses prevalence as the effect-size index.