
Day1
1. What is a metaanalysis?
[ Expand ]
Metaanalysis refers to the statistical procedures for synthesizing data from a set of studies. We start with the summary effect (such as the mean difference, risk ratio, or correlation) from each study, and the metaanalysis enables us to view these effects in context. If the effect size is consistent, we can report that the treatment effect is robust, and can also estimate the effect size with better precision than we could with a single study. If the effect size varies, we can describe the variation and may also be able to explain it.
2. The shift from narrative reviews to systematic reviews
[ Expand ]
Prior to the 1990s, the primary mechanism for synthesizing data from multiple studies was the narrative review, where an expert would study the literature, critique and summarize the results, and come to a conclusion. Beginning in the early 1990s metaanalysis began to replace the narrative review, and it now dominates the field of research synthesis in such fields in medicine, education, ecology, psychology, business, criminal justice, and others. We will discuss the key differences between metaanalysis and narrative reviews, how researchers' thinking about these differences has evolved, and why metaanalysis is now generally embraced as the gold standard.
3. Fixedeffect and randomeffects models
[ Expand ]
In a metaanalysis we want to assign a weight to each study, with more precise studies getting more weight and less precise studies getting less weight. It turns out that the definition of "precise" depends on our understanding of the sampling frame. The fixedeffect model is appropriate in some cases, while the randomeffects model is appropriate in others. We explain the conceptual and practical differences between the models. We also discuss common (and serious) mistakes in selecting a model, and how to avoid these mistakes.
4. Effect sizes for means, binary, and correlational data
[ Expand ]
The effect size (or treatment effect) is the unit of currency in a metaanalysis. This could be a mean difference, a risk ratio, a correlation, or another index, but the basic idea is the always the same. We compute an effect size for each study, and then use the metaanalysis to synthesize these values. We also need to compute the variance of each effect size, since this affects the weight that is allocated to each study in the synthesis. We will discuss several effect sizes, and show how to compute these using Excel™ and CMA™.
5. Performing a metaanalysis using CMA
[ Expand ]
We will show how to use CMA to perform a simple metaanalysis. Steps include entering the data, running the analysis, studying the results, creating a forest plot, and exporting the plot to PowerPoint© and to Word©. We will also use the educational features of CMA to illustrate conceptual issues in metaanalysis. For example, CMA is able to display the study weights under the fixedeffect and randomeffects models, and how these weights affect the results.
6. Avoiding common mistakes, Part 1
[ Expand ]
Over a period of years we have reviewed hundreds of metaanalyses and have compiled a list of mistakes that people make on a fairly regular basis. At the conclusion of every session we'll discuss common mistakes related to that session. This will help you to avoid these mistakes, and also to defend your analysis if the person reviewing your paper should raise questions about the procedures. Some of the mistakes we'll discuss on Day1 include mistakes in the selection of a statistical model, and mistakes in interpreting the mean effect and its confidence interval.
Day2
7. Heterogeneity in treatment effects
[ Expand ]
We will explain how to quantify and to understand heterogeneity in effect sizes. We explain this conceptually, and then show how the concept leads to a series of distinct indices, each with a different meaning and purpose.
8. Comparing the effect size in different groups of studies
[ Expand ]
Suppose we are working with 20 studies that tested the impact of a drug. Ten of the studies employed a standard dose of the drug while the others employed a high dose. Was the impact greater in one group then other? Or, suppose we are working with 20 studies that tested the impact of an intervention for students. Some of the studies ran the intervention for eight weeks while the others ran it for sixteen weeks. Was the longer intervention more effective? In a primary study we would address these kinds of questions by using a ttest or analysis of variance, and we can apply similar procedures in metaanalysis. We'll show how to perform these kinds of analyses using CMA.
9. Using regression to assess the impact of continuous moderators on effect size
[ Expand ]
In a primary study we use multipleregression to assess the relationship between covariates and outcome. This technique is simple enough to be applied with a single predictor, but also allows us to work with sets of predictors, and to assess the impact of one set with another set held constant. We will show how the same techniques can be employed with metaanalysis, and how to use CMA for this purpose.
10. Avoiding common mistakes, Part 2
[ Expand ]
Some of the mistakes we'll discuss on Day1 include mistakes in reporting heterogeneity. For example, the "common" wisdom is that the Isquared index reflects the amount of dispersion in treatment effects, with certain values (typically 25%, 50%, 80%) taken to mean small, medium, and large dispersion. In fact, Isquared is NOT a measure of heterogeneity. We will discuss what Isquared does measure. We will also discuss indices that do reflect the dispersion in treatment effects, and explain how to use these.
Day3
11. Working with studies that report effects for two or more independent subgroups
[ Expand ]
Sometimes studies report the impact of a treatment separately for two (or more) independent samples – for example, the impact of an intervention for males and also for females. We explain the options for working with this kind of data, and how to implement these options in CMA.
12. Working with studies that report effects for two or more outcomes
[ Expand ]
Suppose studies report the impact of an intervention on both math and reading scores. In the analysis we may want to assess the impact for each outcome separately, or we may want to assess the impact on a composite score called "Academic achievement". We explain the options for working with this kind of data, and how to implement these options in CMA.
13. Working with studies that compare several treatment groups to a common control group
[ Expand ]
Suppose studies report the impact of an intervention on both math and reading scores. In the analysis we may want to assess the impact for each outcome separately, or we may want to assess the impact on a composite score called "Academic achievement". We explain the options for working with this kind of data, and how to implement these options in CMA.
14. Important conceptual issues in metaanalysis
[ Expand ]
When does it make sense to perform a metaanalysis? How many studies do we need? How similar do the studies need to be? What is the role of a cumulative metaanalysis? What if the metaanalysis appears to conflict with a largescale trial? We discuss these and other issues that arise frequently and are not wellunderstood.
15. Criticisms of metaanalysis
[ Expand ]
In 1990 Charles Mann published a paper about the future of metaanalysis. While many researchers were convinced that metaanalysis would eventually serve as the basis for evidencebased practice, others were less enthusiastic. One person compared metaanalysis to alchemy and another compared it to terrorism. In 1993, an editorial in the New England Journal of Medicine suggested that metaanalyses are so likely to be flawed that it would be preferable to stick with narrative reviews. It's important to understand the criticisms outlined in these papers (and others), both to rebut the objections that are not valid and also to learn from the criticisms that are valid.
16. Resources for metaanalysis
[ Expand ]
We provide a synopsis of various texts. Some of these show how to work with metaanalysis in a specific substantive area such as medicine, social science, or ecology. Others are intended as general texts or as handbooks. We also discuss a number of web sites and professional groups that serve as resources for systematic reviews and metaanalysis.
17. Avoiding common mistakes, Part 3
[ Expand ]
The objective of this session is similar to the corresponding sessions on Days 1 and 2. We will discuss mistakes to avoid when working with complex data sets. We will discuss some of the key mistakes people make when discussing publication bias. We will discuss the argument that a metaanalysis should not be performed when the effect size varies across studies.
Expand / Hide All « Previous Back to Top Next » 

Testimonials
The CMA workshop presented by Dr. Michael Borenstein is one of the best we had. Dr. Borenstein delivered the workshop on site to our study team of eight members in Rockville, Maryland. Since most of the participants did not have previous experience with metaanalysis, the oneday workshop covered both conceptual issues related to metaanalysis as well as handson demonstration of the CMA software. It was extremely thorough and informative. We were most appreciative of the followups he provided. Dr. Borenstein was always very accessible and helpful whenever we had questions about CMA in the subsequent months as we coded the studies and analyzed the data.
Xiaodong Zhang, Senior Study Director, Westat, Rockville. MD
As an NIH funded metaanalytic researcher, I can honestly say that Comprehensive MetaAnalysis is currently the most comprehensive standalone software package for conducting highquality metaanalyses. I especially like its ease of use, including, but not limited to, the ability to calculate effect sizes from a large array of reported statistics as well as the ability to easily combine results from multiple groups in the same study into one common effect size. In my opinion, this is a "must" purchase for any serious metaanalyst.
George A. Kelley, DA, FACSM, Professor & Director, MetaAnalytic Research Group, School of Medicine. Department of Community Medicine, West Virginia University, Morgantown, WV
We perform a variety of metaanalyses for academic, regulatory, and international clients. Each presents a different set of challenges regarding study design and outcome measurement. We have found CMA to be invaluable in this work. The ability of the software to capture a variety of data elements (study design, multiple outcomes, covariates/ confounders) and present details of computations is important in the credibility of our work. The ease of use and ability to produce graphics in a variety of formats aids in preparation of the report. In many instances, we are required to replicate the results of CMA in another package (e.g., SAS). We have always found the support staff at CMA very helpful in these replications and the results of CMA have been replicated in every instance. CMA is a great tool in the scientific credibility of our metaanalytic studies.
Donna F. Stroup, PhD, MSc, Data for Solutions, Inc.

