meta analysis quantitative methods for research synthesis

• Methodology
• Open access
• Published: 24 April 2023

Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences

• Shinichi Nakagawa   ORCID: orcid.org/0000-0002-7765-5182 1 , 2 ,
• Yefeng Yang   ORCID: orcid.org/0000-0002-8610-4016 1 ,
• Erin L. Macartney   ORCID: orcid.org/0000-0003-3866-143X 1 ,
• Rebecca Spake   ORCID: orcid.org/0000-0003-4671-2225 3 &
• Malgorzata Lagisz   ORCID: orcid.org/0000-0002-3993-6127 1  

Environmental Evidence volume  12 , Article number:  8 ( 2023 ) Cite this article

7972 Accesses

19 Citations

29 Altmetric

Metrics details

Meta-analysis is a quantitative way of synthesizing results from multiple studies to obtain reliable evidence of an intervention or phenomenon. Indeed, an increasing number of meta-analyses are conducted in environmental sciences, and resulting meta-analytic evidence is often used in environmental policies and decision-making. We conducted a survey of recent meta-analyses in environmental sciences and found poor standards of current meta-analytic practice and reporting. For example, only ~ 40% of the 73 reviewed meta-analyses reported heterogeneity (variation among effect sizes beyond sampling error), and publication bias was assessed in fewer than half. Furthermore, although almost all the meta-analyses had multiple effect sizes originating from the same studies, non-independence among effect sizes was considered in only half of the meta-analyses. To improve the implementation of meta-analysis in environmental sciences, we here outline practical guidance for conducting a meta-analysis in environmental sciences. We describe the key concepts of effect size and meta-analysis and detail procedures for fitting multilevel meta-analysis and meta-regression models and performing associated publication bias tests. We demonstrate a clear need for environmental scientists to embrace multilevel meta-analytic models, which explicitly model dependence among effect sizes, rather than the commonly used random-effects models. Further, we discuss how reporting and visual presentations of meta-analytic results can be much improved by following reporting guidelines such as PRISMA-EcoEvo (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Ecology and Evolutionary Biology). This paper, along with the accompanying online tutorial, serves as a practical guide on conducting a complete set of meta-analytic procedures (i.e., meta-analysis, heterogeneity quantification, meta-regression, publication bias tests and sensitivity analysis) and also as a gateway to more advanced, yet appropriate, methods.

Evidence synthesis is an essential part of science. The method of systematic review provides the most trusted and unbiased way to achieve the synthesis of evidence [ 1 , 2 , 3 ]. Systematic reviews often include a quantitative summary of studies on the topic of interest, referred to as a meta-analysis (for discussion on the definitions of ‘meta-analysis’, see [ 4 ]). The term meta-analysis can also mean a set of statistical techniques for quantitative data synthesis. The methodologies of the meta-analysis were initially developed and applied in medical and social sciences. However, meta-analytic methods are now used in many other fields, including environmental sciences [ 5 , 6 , 7 ]. In environmental sciences, the outcomes of meta-analyses (within systematic reviews) have been used to inform environmental and related policies (see [ 8 ]). Therefore, the reliability of meta-analytic results in environmental sciences is important beyond mere academic interests; indeed, incorrect results could lead to ineffective or sometimes harmful environmental policies [ 8 ].

As in medical and social sciences, environmental scientists frequently use traditional meta-analytic models, namely fixed-effect and random-effects models [ 9 , 10 ]. However, we contend that such models in their original formulation are no longer useful and are often incorrectly used, leading to unreliable estimates and errors. This is mainly because the traditional models assume independence among effect sizes, but almost all primary research papers include more than one effect size, and this non-independence is often not considered (e.g., [ 11 , 12 , 13 ]). Furthermore, previous reviews of published meta-analyses in environmental sciences (hereafter, ‘environmental meta-analyses’) have demonstrated that less than half report or investigate heterogeneity (inconsistency) among effect sizes [ 14 , 15 , 16 ]. Many environmental meta-analyses also do not present any sensitivity analysis, for example, for publication bias (i.e., statistically significant effects being more likely to be published, making collated data unreliable; [ 17 , 18 ]). These issues might have arisen for several reasons, for example, because of no clear conduct guideline for the statistical part of meta-analyses in environmental sciences and rapid developments in meta-analytic methods. Taken together, the field urgently requires a practical guide to implement correct meta-analyses and associated procedures (e.g., heterogeneity analysis, meta-regression, and publication bias tests; cf. [ 19 ]).

To assist environmental scientists in conducting meta-analyses, the aims of this paper are five-fold. First, we provide an overview of the processes involved in a meta-analysis while introducing some key concepts. Second, after introducing the main types of effect size measures, we mathematically describe the two commonly used traditional meta-analytic models, demonstrate their utility, and introduce a practical, multilevel meta-analytic model for environmental sciences that appropriately handles non-independence among effect sizes. Third, we show how to quantify heterogeneity (i.e., consistencies among effect sizes and/or studies) using this model, and then explain such heterogeneity using meta-regression. Fourth, we show how to test for publication bias in a meta-analysis and describe other common types of sensitivity analysis. Fifth, we cover other technical issues relevant to environmental sciences (e.g., scale and phylogenetic dependence) as well as some advanced meta-analytic techniques. In addition, these five aims (sections) are interspersed with two more sections, named ‘Notes’ on: (1) visualisation and interpretation; and (2) reporting and archiving. Some of these sections are accompanied by results from a survey of 73 environmental meta-analyses published between 2019 and 2021; survey results depict current practices and highlight associated problems (for the method of the survey, see Additional file 1 ). Importantly, we provide easy-to-follow implementations of much of what is described below, using the R package, metafor [ 20 ] and other R packages at the webpage ( https://itchyshin.github.io/Meta-analysis_tutorial/ ), which also connects the reader to the wealth of online information on meta-analysis (note that we also provide this tutorial as Additional file 2 ; see also [ 21 ]).

Overview with key concepts

Statistically speaking, we have three general objectives when conducting a meta-analysis [ 12 ]: (1) estimating an overall mean , (2) quantifying consistency ( heterogeneity ) between studies, and (3) explaining the heterogeneity (see Table 1 for the definitions of the terms in italic ). A notable feature of a meta-analysis is that an overall mean is estimated by taking the sampling variance of each effect size into account: a study (effect size) with a low sampling variance (usually based on a larger sample size) is assigned more weight in estimating an overall mean than one with a high sampling variance (usually based on a smaller sample size). However, an overall mean estimate itself is often not informative because one can get the same overall mean estimates in different ways. For example, we may get an overall estimate of zero if all studies have zero effects with no heterogeneity. In contrast, we might also obtain a zero mean across studies that have highly variable effects (e.g., ranging from strongly positive to strongly negative), signifying high heterogeneity. Therefore, quantifying indicators of heterogeneity is an essential part of a meta-analysis, necessary for interpreting the overall mean appropriately. Once we observe non-zero heterogeneity among effect sizes, then, our job is to explain this variation by running meta-regression models, and, at the same time, quantify how much variation is accounted for (often quantified as R 2 ). In addition, it is important to conduct an extra set of analyses, often referred to as publication bias tests , which are a type of sensitivity analysis [ 11 ], to check the robustness of meta-analytic results.

Choosing an effect size measure

In this section, we introduce different kinds of ‘effect size measures’ or ‘effect measures’. In the literature, the term ‘effect size’ is typically used to refer to the magnitude or strength of an effect of interest or its biological interpretation (e.g., environmental significance). Effect sizes can be quantified using a range of measures (for details, see [ 22 ]). In our survey of environmental meta-analyses (Additional file 1 ), the two most commonly used effect size measures are: the logarithm of response ratio, lnRR ([ 23 ]; also known as the ratio of means; [ 24 ]) and standardized mean difference, SMD (often referred to as Hedges’ g or Cohen’s d [ 25 , 26 ]). These are followed by proportion (%) and Fisher’s z -transformation of correlation, or Zr . These four effect measures nearly fit into the three categories, which are named: (1) single-group measures (a statistical summary from one group; e.g., proportion), (2) comparative measures (comparing between two groups e.g., SMD and lnRR), and (3) association measures (relationships between two variables; e.g., Zr ). Table 2 summarizes effect measures that are common or potentially useful for environmental scientists. It is important to note that any measures with sampling variance can become an ‘effect size’. The main reason why SMD, lnRR, Zr, or proportion are popular effect measures is that they are unitless, while a meta-analysis of mean, or mean difference, can only be conducted when all effect sizes have the same unit (e.g., cm, kg).

Table 2 also includes effect measures that are likely to be unfamiliar to environmental scientists; these are effect sizes that characterise differences in the observed variability between samples, (i.e., lnSD, lnCV, lnVR and lnCVR; [ 27 , 28 ]) rather than central tendencies (averages). These dispersion-based effect measures can provide us with extra insights along with average-based effect measures. Although the literature survey showed none of these were used in our sample, these effect sizes have been used in many fields, including agriculture (e.g., [ 29 ]), ecology (e.g., [ 30 ]), evolutionary biology (e.g., [ 31 ]), psychology (e.g., [ 32 ]), education (e.g., [ 33 ]), psychiatry (e.g., [ 34 ]), and neurosciences (e.g. [ 35 ],),. Perhaps, it is not difficult to think of an environmental intervention that can affect not only the mean but also the variance of measurements taken on a group of individuals or a set of plots. For example, environmental stressors such as pesticides and eutrophication are likely to increase variability in biological systems because stress accentuates individual differences in environmental responses (e.g. [ 36 , 37 ],). Such ideas are yet to be tested meta-analytically (cf. [ 38 , 39 ]).

Choosing a meta-analytic model

Fixed-effect and random-effects models.

Two traditional meta-analytic models are called the ‘fixed-effect’ model and the ‘random-effects’ model. The former assumes that all effect sizes (from different studies) come from one population (i.e., they have one true overall mean), while the latter does not have such an assumption (i.e., each study has different overall means or heterogeneity exists among studies; see below for more). The fixed-effect model, which should probably be more correctly referred to as the ‘common-effect’ model, can be written as [ 9 , 10 , 40 ]:

where the intercept, \({\beta }_{0}\) is the overall mean, z j (the response/dependent variable) is the effect size from the j th study ( j  = 1, 2,…, N study ; in this model, N study  = the number of studies = the number of effect sizes), m j is the sampling error, related to the j th sampling variance ( v j ), which is normally distributed with the mean of 0 and the ‘study-specific’ sampling variance, v j (see also Fig.  1 A).

Visualisation of the three statistical models of meta-analysis: A a fixed-effect model (1-level), B a random-effects model (2-level), and C a multilevel model (3-level; see the text for what symbols mean)

The overall mean needs to be estimated and often done so as the weighted average with the weights, \({w}_{j}=1/{v}_{j}\) (i.e., the inverse-variance approach). An important, but sometimes untenable, assumption of meta-analysis is that sampling variance is known. Indeed, we estimate sampling variance, using formulas, as in Table 2 , meaning that vj is submitted by sampling variance estimates (see also section ‘ Scale dependence ’). Of relevance, the use of the inverse-variance approach has been recently criticized, especially for SMD and lnRR [ 41 , 42 ] and we note that the inverse-variance approach using the formulas in Table 2 is one of several different weighting approaches used in meta-analysis (e.g., for adjusted sampling-variance weighing, see [ 43 , 44 ]; for sample-size-based weighting, see [ 41 , 42 , 45 , 46 ]). Importantly, the fixed-effect model assumes that the only source of variation in effect sizes ( z j ) is the effect due to sampling variance (which is inversely proportional to the sample size, n ; Table 2 ).

Similarly, the random-effects model can be expressed as:

where u j is the j th study effect, which is normally distributed with the mean of 0 and the between-study variance, \({\tau }^{2}\) (for different estimation methods, see [ 47 , 48 , 49 , 50 ]), and other notations are the same as in Eq.  1 (Fig.  1 B). Here, the overall mean can be estimated as the weighted average with weights \({w}_{j}=1/\left({\tau }^{2}+{v}_{j}^{2}\right)\) (note that different weighting approaches, mentioned above, are applicable to the random-effects model and some of them are to the multilevel model, introduced below). The model assumes each study has its specific mean, \({b}_{0}+{u}_{j}\) , and (in)consistencies among studies (effect sizes) are indicated by \({\tau }^{2}\) . When \({\tau }^{2}\) is 0 (or not statistically different from 0), the random-effects model simplifies to the fixed-effect model (cf. Equations  1 and 2 ). Given no studies in environmental sciences are conducted in the same manner or even at exactly the same place and time, we should expect different studies to have different means. Therefore, in almost all cases in the environmental sciences, the random-effects model is a more ‘realistic’ model [ 9 , 10 , 40 ]. Accordingly, most environmental meta-analyses (68.5%; 50 out of 73 studies) in our survey used the random-effects model, while only 2.7% (2 of 73 studies) used the fixed-effect model (Additional file 1 ).

Multilevel meta-analytic models

Although we have introduced the random-effects model as being more realistic than the fixed-effect model (Eq.  2 ), we argue that the random-effects model is rather limited and impractical for the environmental sciences. This is because random-effects models, like fixed-effect models, assume all effect sizes ( z j ) to be independent. However, when multiple effect sizes are obtained from a study, these effect sizes are dependent (for more details, see the next section on non-independence). Indeed, our survey showed that in almost all datasets used in environmental meta-analyses, this type of non-independence among effect sizes occurred (97.3%; 71 out of 73 studies, with two studies being unclear, so effectively 100%; Additional file 1 ). Therefore, we propose the simplest and most practical meta-analytic model for environmental sciences as [ 13 , 40 ] (see also [ 51 , 52 ]):

where we explicitly recognize that N effect ( i  = 1, 2,…, N effect ) >  N study ( j  = 1, 2,…, N study ) and, therefore, we now have the study effect (between-study effect), u j[i] (for the j th study and i th effect size) and effect-size level (within-study) effect, e i (for the i th effect size), with the between-study variance, \({\tau }^{2}\) , and with-study variance, \({\sigma }^{2}\) , respectively, and other notations are the same as above. We note that this model (Eq.  3 ) is an extension of the random-effects model (Eq.  2 ), and we refer to it as the multilevel/hierarchical model (used in 7 out of 73 studies: 9.6% [Additional file 1 ]; note that Eq.  3 is also known as a three-level meta-analytic model; Fig.  1 C). Also, environmental scientists who are familiar with (generalised) linear mixed-models may recognize u j (the study effect) as the effect of a random factor which is associated with a variance component, i.e., \({\tau }^{2}\) [ 53 ]; also, e i and m i can be seen as parts of random factors, associated with \({\sigma }^{2}\) and v i (the former is comparable to the residuals, while the latter is sampling variance, specific to a given effect size).

It seems that many researchers are aware of the issue of non-independence so that they often use average effect sizes per study or choose one effect size (at least 28.8%, 21 out of 73 environmental meta-analyses; Additional file 1 ). However, as we discussed elsewhere [ 13 , 40 ], such averaging or selection of one effect size per study dramatically reduces our ability to investigate environmental drivers of variation among effect sizes [ 13 ]. Therefore, we strongly support the use of the multilevel model. Nevertheless, this proposed multilevel model, formulated as Eq.  3 does not usually deal with the issue of non-independence completely, which we elaborate on in the next section.

Non-independence among effect sizes and among sampling errors

When you have multiple effect sizes from a study, there are two broad types and three cases of non-independence (cf. [ 11 , 12 ]): (1) effect sizes are calculated from different cohorts of individuals (or groups of plots) within a study (Fig.  2 A, referred to as ‘shared study identity’), and (2) effects sizes are calculated from the same cohort of individuals (or group of plots; Fig.  2 B, referred to as ‘shared measurements’) or partially from the same individuals and plots, more concretely, sharing individuals and plots from the control group (Fig.  2 C, referred to as ‘shared control group’). The first type of non-independence induces dependence among effect sizes, but not among sampling variances, and the second type leads to non-independence among sampling variances. Many datasets, if not almost all, will have a combination of these three cases (or even are more complex, see the section " Complex non-independence "). Failing to deal with these non-independences will inflate Type 1 error (note that the overall estimate, b 0 is unlikely to be biased, but standard error of b 0 , se( b 0 ), will be underestimated; note that this is also true for all other regression coefficients, e.g., b 1 ; see Table 1 ). The multilevel model (as in Eq.  3 ) only takes care of cases of non-independence that are due to the shared study identity but neither shared measurements nor shared control group.

Visualisation of the three types of non-independence among effect sizes: A due to shared study identities (effect sizes from the same study), B due to shared measurements (effect sizes come from the same group of individuals/plots but are based on different types of measurements), and C due to shared control (effect sizes are calculated using the same control group and multiple treatment groups; see the text for more details)

There are two practical ways to deal with non-independence among sampling variances. The first method is that we explicitly model such dependence using a variance–covariance (VCV) matrix (used in 6 out of 73 studies: 8.2%; Additional file 1 ). Imagine a simple scenario with a dataset of three effect sizes from two studies where two effects sizes from the first study are calculated (partially) using the same cohort of individuals (Fig.  2 B); in such a case, the sampling variance effect, \({m}_{i}\) , as in Eq.  3 , should be written as:

where M is the VCV matrix showing the sampling variances, \({v}_{1\left[1\right]}\) (study 1 and effect size 1), \({v}_{1\left[2\right]}\) (study 1 and effect size 2), and \({v}_{2\left[3\right]}\) (study 2 and effect size 3) in its diagonal, and sampling covariance, \(\rho \sqrt{{v}_{1\left[1\right]}{v}_{1\left[2\right]}}= \rho \sqrt{{v}_{1\left[2\right]}{v}_{1\left[1\right]}}\) in its off-diagonal elements, where \(\rho \) is a correlation between two sampling variances due to shared samples (individuals/plots). Once this VCV matrix is incorporated into the multilevel model (Eq.  3 ), all the types of non-independence, as in Fig.  2 , are taken care of. Table 3 shows formulas for the sampling variance and covariance of the four common effect sizes (SDM, lnRR, proportion and Zr ). For comparative effect measures (Table 2 ), exact covariances can be calculated under the case of ‘shared control group’ (see [ 54 , 55 ]). But this is not feasible for most circumstances because we usually do not know what \(\rho \) should be. Some have suggested fixing this value at 0.5 (e.g., [ 11 ]) or 0.8 (e.g., [ 56 ]); the latter is a more conservative assumption. Or one can run both and use one for the main analysis and the other for sensitivity analysis (for more, see the ‘ Conducting sensitivity analysis and critical appraisal " section).

The second method overcomes this very issue of unknown \(\rho \) by approximating average dependence among sampling variance (and effect sizes) from the data and incorporating such dependence to estimate standard errors (only used in 1 out of 73 studies; Additional file 1 ). This method is known as ‘robust variance estimation’, RVE, and the original estimator was proposed by Hedges and colleagues in 2010 [ 57 ]. Meta-analysis using RVE is relatively new, and this method has been applied to multilevel meta-analytic models only recently [ 58 ]. Note that the random-effects model (Eq.  2 ) and RVE could correctly model both types of non-independence. However, we do not recommend the use of RVE with Eq.  2 because, as we will later show, estimating \({\sigma }^{2}\) as well as \({\tau }^{2}\) will constitute an important part of understanding and gaining more insights from one’s data. We do not yet have a definite recommendation on which method to use to account for non-independence among sampling errors (using the VCV matrix or RVE). This is because no simulation work in the context of multilevel meta-analysis has been done so far, using multilevel meta-analyses [ 13 , 58 ]. For now, one could use both VCV matrices and RVE in the same model [ 58 ] (see also [ 21 ]).

Quantifying and explaining heterogeneity

Measuring consistencies with heterogeneity.

As mentioned earlier, quantifying heterogeneity among effect sizes is an essential component of any meta-analysis. Yet, our survey showed only 28 out of 73 environmental meta-analyses (38.4%; Additional file 1 ) report at least one index of heterogeneity (e.g., \({\tau }^{2}\) , Q , and I 2 ). Conventionally, the presence of heterogeneity is tested by Cochrane’s Q test. However, Q (often noted as Q T or Q total ), and its associated p value, are not particularly informative: the test does not tell us about the extent of heterogeneity (e.g. [ 10 ],), only whether heterogeneity is zero or not (when p  < 0.05). Therefore, for environmental scientists, we recommend two common ways of quantifying heterogeneity from a meta-analytic model: absolute heterogeneity measure (i.e., variance components, \({\tau }^{2}\) and \({\sigma }^{2}\) ) and relative heterogeneity measure (i.e., I 2 ; see also the " Notes on visualisation and interpretation " section for another way of quantifying and visualising heterogeneity at the same time, using prediction intervals; see also [ 59 ]). We have already covered the absolute measure (Eqs.  2 & 3 ), so here we explain I 2 , which ranges from 0 to 1 (for some caveats for I 2 , see [ 60 , 61 ]). The heterogeneity measure, I 2 , for the random-effect model (Eq.  2 ) can be written as:

Where \(\overline{v}\) is referred to as the typical sampling variance (originally this is called ‘within-study’ variance, as in Eq.  2 , and note that in this formulation, within-study effect and the effect of sampling error is confounded; see [ 62 , 63 ]; see also [ 64 ]) and the other notations are as above. As you can see from Eq.  5 , we can interpret I 2 as relative variation due to differences between studies (between-study variance) or relative variation not due to sampling variance.

By seeing I 2 as a type of interclass correlation (also known as repeatability [ 65 ],), we can generalize I 2 to multilevel models. In the case of Eq.  3 ([ 40 , 66 ]; see also [ 52 ]), we have:

Because we can have two more I 2 , Eq.  7 is written as \({I}_{total}^{2}\) ; these other two are \({I}_{study}^{2}\) and \({I}_{effect}^{2}\) , respectively:

\({I}_{total}^{2}\) represents relative variance due to differences both between and within studies (between- and within-study variance) or relative variation not due to sampling variance, while \({I}_{study}^{2}\) is relative variation due to differences between studies, and \({I}_{effect}^{2}\) is relative variation due to differences within studies (Fig.  3 A). Once heterogeneity is quantified (note almost all data will have non-zero heterogeneity and an earlier meta-meta-analysis suggests in ecology, we have on average, I 2 close to 90% [ 66 ]), it is time to fit a meta-regression model to explain the heterogeneity. Notably, the magnitude of \({I}_{study}^{2}\) (and \({\tau }^{2}\) ) and \({I}_{effect}^{2}\) (and \({\sigma }^{2}\) ) can already inform you which predictor variable (usually referred to as ‘moderator’) is likely to be important, which we explain in the next section.

Visualisation of variation (heterogeneity) partitioned into different variance components: A quantifying different types of I 2 from a multilevel model (3-level; see Fig.  1 C) and B variance explained, R 2 , by moderators. Note that different levels of variances would be explained, depending on which level a moderator belongs to (study level and effect-size level)

Explaining variance with meta-regression

We can extend the multilevel model (Eq.  3 ) to a meta-regression model with one moderator (also known as predictor, independent, explanatory variable, or fixed factor), as below:

where \({\beta }_{1}\) is a slope of the moderator ( x 1 ), \({x}_{1j\left[i\right]}\) denotes the value of x 1 , corresponding to the j th study (and the i th effect sizes). Equation ( 10 ) (meta-regression) is comparable to the simplest regression with the intercept ( \({\beta }_{0}\) ) and slope ( \({\beta }_{1}\) ). Notably, \({x}_{1j\left[i\right]}\) differs between studies and, therefore, it will mainly explain the variance component, \({\tau }^{2}\) (which relates to \({I}_{study}^{2}\) ). On the other hand, if noted like \({x}_{1i}\) , this moderator would vary within studies or at the level of effect sizes, therefore, explaining \({\sigma }^{2}\) (relating to \({I}_{effect}^{2}\) ). Therefore, when \({\tau }^{2}\) ( \({I}_{study}^{2}\) ), or \({\sigma }^{2}\) ( \({I}_{effect}^{2}\) ), is close to zero, there will be little point fitting a moderator(s) at the level of studies, or effect sizes, respectively.

As in multiple regression, we can have multiple (multi-moderator) meta-regression, which can be written as:

where \(\sum_{h=1}^{q}{\beta }_{h}{x}_{h\left[i\right]}\) denotes the sum of all the moderator effects, with q being the number of slopes (staring with h  = 1). We note that q is not necessarily the number of moderators. This is because when we have a categorical moderator, which is common, with more than two levels (e.g., method A, B & C), the fixed effect part of the formula is \({\beta }_{0}+{\beta }_{1}{x}_{1}+{\beta }_{2}{x}_{2}\) , where x 1 and x 2 are ‘dummy’ variables, which code whether the i th effect size belongs to, for example, method B or C, with \({\beta }_{1}\) and \({\beta }_{2}\) being contrasts between A and B and between A and C, respectively (for more explanations of dummy variables, see our tutorial page [ https://itchyshin.github.io/Meta-analysis_tutorial/ ]; also see [ 67 , 68 ]). Traditionally, researchers conduct separate meta-analyses per different groups (known as ‘sub-group analysis’), but we prefer a meta-regression approach with a categorical variable, which is statistically more powerful [ 40 ]. Also, importantly, what can be used as a moderator(s) is very flexible, including, for example, individual/plot characteristics (e.g., age, location), environmental factors (e.g., temperature), methodological differences between studies (e.g., randomization), and bibliometric information (e.g., publication year; see more in the section ‘Checking for publication bias and robustness’). Note that moderators should be decided and listed a priori in the meta-analysis plan (i.e., a review protocol or pre-registration).

As with meta-analysis, the Q -test ( Q m or Q moderator ) is often used to test the significance of the moderator(s). To complement this test, we can also quantify variance explained by the moderator(s) using R 2 . We can define R 2 using Eq. ( 11 ) as:

where R 2 is known as marginal R 2 (sensu [ 69 , 70 ]; cf. [ 71 ]), \({f}^{2}\) is the variance due to the moderator(s), and \({(f}^{2}+{\tau }^{2}+{\sigma }^{2})\) here equals to \(({\tau }^{2}+{\sigma }^{2})\) in Eq.  7 , as \({f}^{2}\) ‘absorbs’ variance from \({\tau }^{2}\) and/or \({\sigma }^{2}\) . We can compare the similarities and differences in Fig.  3 B where we denote a part of \({f}^{2}\) originating from \({\tau }^{2}\) as \({f}_{study}^{2}\) while \({\sigma }^{2}\) as \({f}_{effect}^{2}\) . In a multiple meta-regression model, we often want to find a model with the ‘best’ or an adequate set of predictors (i.e., moderators). R 2 can potentially help such a model selection process. Yet, methods based on information criteria (such as Akaike information criterion, AIC) may be preferable. Although model selection based on the information criteria is beyond the scope of the paper, we refer the reader to relevant articles (e.g., [ 72 , 73 ]), and we show an example of this procedure in our online tutorial ( https://itchyshin.github.io/Meta-analysis_tutorial/ ).

Notes on visualisation and interpretation

Visualization and interpretation of results is an essential part of a meta-analysis [ 74 , 75 ]. Traditionally, a forest plot is used to display the values and 95% of confidence intervals (CIs) for each effect size and the overall effect and its 95% CI (the diamond symbol is often used, as shown in Fig.  4 A). More recently, adding a 95% prediction interval (PI) to the overall estimate has been strongly recommended because 95% PIs show a predicted range of values in which an effect size from a new study would fall, assuming there is no sampling error [ 76 ]. Here, we think that examining the formulas for 95% CIs and PIs for the overall mean (from Eq.  3 ) is illuminating:

where \({t}_{df\left[\alpha =0.05\right]}\) denotes the t value with the degree of freedom, df , at 97.5 percentile (or \(\alpha =0.05\) ) and other notations are as above. In a meta-analysis, it has been conventional to use z value 1.96 instead of \({t}_{df\left[\alpha =0.05\right]}\) , but simulation studies have shown the use of t value over z value reduces Type 1 errors under many scenarios and, therefore, is recommended (e.g., [ 13 , 77 ]). Also, it is interesting to note that by plotting 95% PIs, we can visualize heterogeneity as Eq.  15 includes \({\tau }^{2}\) and \({\sigma }^{2}\) .

Different types of plots useful for a meta-analysis using data from Midolo et al. [ 133 ]: A a typical forest plot with the overall mean shown as a diamond at the bottom (20 effect sizes from 20 studies are used), B a caterpillar plot (100 effect sizes from 24 studies are used), C an orchard plot of categorical moderator with seven levels (all effect sizes are used), and D a bubble plot of a continuous moderator. Note that the first two only show confidence intervals, while the latter two also show prediction intervals (see the text for more details)

A ‘forest’ plot can become quickly illegible as the number of studies (effect sizes) becomes large, so other methods of visualizing the distribution of effect sizes have been suggested. Some suggested to present a ‘caterpillar’ plot, which is a version of the forest plot, instead (Fig.  4 B; e.g., [ 78 ]). We here recommend an ‘orchard’ plot, as it can present results across different groups (or a result of meta-regression with a categorical variable), as shown in Fig.  4 C [ 78 ]. For visualization of a continuous variable, we suggest what is called a ‘bubble’ plot, shown in Fig.  4 D. Visualization not only helps us interpret meta-analytic results, but can also help to identify something we may not see from statistical results, such as influential data points and outliers that could threaten the robustness of our results.

Checking for publication bias and robustness

Detecting and correcting for publication bias.

Checking for and adjusting for any publication bias is necessary to ensure the validity of meta-analytic inferences [ 79 ]. However, our survey showed almost half of the environmental meta-analyses (46.6%; 34 out of 73 studies; Additional file 1 ) neither tested for nor corrected for publication bias (cf. [ 14 , 15 , 16 ]). The most popular methods used were: (1) graphical tests using funnel plots (26 studies; 35.6%), (2) regression-based tests such as Egger regression (18 studies; 24.7%), (3) Fail-safe number tests (12 studies; 16.4%), and (4) trim-and-fill tests (10 studies; 13.7%). We recently showed that these methods are unsuitable for datasets with non-independent effect sizes, with the exception of funnel plots [ 80 ] (for an example of funnel plots, see Fig.  5 A). This is because these methods cannot deal with non-independence in the same way as the fixed-effect and random-effects models. Here, we only introduce a two-step method for multilevel models that can both detect and correct for publication bias [ 80 ] (originally proposed by [ 81 , 82 ]), more specifically, the “small study effect” where an effect size value from a small-sample-sized study can be much larger in magnitude than a ‘true’ effect [ 83 , 84 ]. This method is a simple extension of Egger’s regression [ 85 ], which can be easily implemented by using Eq.  10 :

where \({\widetilde{n}}_{i}\) is known as effective sample size; for Zr and proportion it is just n i , and for SMD and lnRR, it is \({n}_{iC}{n}_{iT}/\left({n}_{iC}+{n}_{iT}\right)\) , as in Table 2 . When \({\beta }_{1}\) is significant, we conclude there exists a small-study effect (in terms of a funnel plot, this is equivalent to significant funnel asymmetry). Then, we fit Eq.  17 and we look at the intercept \({\beta }_{0}\) , which will be a bias-corrected overall estimate [note that \({\beta }_{0}\) in Eq. ( 16 ) provides less accurate estimates when non-zero overall effects exist [ 81 , 82 ]; Fig.  5 B]. An intuitive explanation of why \({\beta }_{0}\) (Eq.  17 ) is the ‘bias-corrected’ estimate is that the intercept represents \(1/\widetilde{{n}_{i}}=0\) (or \(\widetilde{{n}_{i}}=\infty \) ); in other words, \({\beta }_{0}\) is the estimate of the overall effect when we have a very large (infinite) sample size. Of note, appropriate bias correction requires a selection-mode-based approach although such an approach is yet to be available for multilevel meta-analytic models [ 80 ].

Different types of plots for publication bias tests: A a funnel plot using model residuals, showing a funnel (white) that shows the region of statistical non-significance (30 effect sizes from 30 studies are used; note that we used the inverse of standard errors for the y -axis, but for some effect sizes, sample size or ‘effective’ sample size may be more appropriate), B a bubble plot visualising a multilevel meta-regression that tests for the small study effect (note that the slope was non-significant: b  = 0.120, 95% CI = [− 0.095, 0.334]; all effect sizes are used), and C a bubble plot visualising a multilevel meta-regression that tests for the decline effect (the slope was non-significant: b  = 0.003, 95%CI = [− 0.002, 0.008])

Conveniently, this proposed framework can be extended to test for another type of publication bias, known as time-lag bias, or the decline effect, where effect sizes tend to get closer to zero over time, as larger or statistically significant effects are published more quickly than smaller or non-statistically significant effects [ 86 , 87 ]. Again, a decline effect can be statistically tested by adding year to Eq. ( 3 ):

where \(c\left(yea{r}_{j\left[i\right]}\right)\) is the mean-centred publication year of a particular study (study j and effect size i ); this centring makes the intercept \({\beta }_{0}\) meaningful, representing the overall effect estimate at the mean value of publication years (see [ 68 ]). When the slope is significantly different from 0, we deem that we have a decline effect (or time-lag bias; Fig.  5 C).

However, there may be some confounding moderators, which need to be modelled together. Indeed, Egger’s regression (Eqs.  16 and 17 ) is known to detect the funnel asymmetry when there is little heterogeneity; this means that we need to model \(\sqrt{1/{\widetilde{n}}_{i}}\) with other moderators that account for heterogeneity. Given this, we probably should use a multiple meta-regression model, as below:

where \(\sum_{h=3}^{q}{\beta }_{h}{x}_{h\left[i\right]}\) is the sum of the other moderator effects apart from the small-study effect and decline effect, and other notations are as above (for more details see [ 80 ]). We need to carefully consider which moderators should go into Eq.  19 (e.g., fitting all moderators or using an AIC-based model selection method; see [ 72 , 73 ]). Of relevance, when running complex models, some model parameters cannot be estimated well, or they are not ‘identifiable’ [ 88 ]. This is especially so for variance components (random-effect part) rather than regression coeffects (fixed-effect part). Therefore, it is advisable to check whether model parameters are all identifiable, which can be checked using the profile function in metafor (for an example, see our tutorial webpage [ https://itchyshin.github.io/Meta-analysis_tutorial/ ]).

Conducting sensitivity analysis and critical appraisal

Sensitivity analysis explores the robustness of meta-analytic results by running a different set of analyses from the original analysis, and comparing the results (note that some consider publication bias tests a part of sensitivity analysis; [ 11 ]). For example, we might be interested in assessing how robust results are to the presence of influential studies, to the choice of method for addressing non-independence, or weighting effect sizes. Unfortunately, in our survey, only 37% of environmental meta-analyses (27 out of 73) conducted sensitivity analysis (Additional file 1 ). There are two general and interrelated ways to conduct sensitivity analyses [ 73 , 89 , 90 ]. The first one is to take out influential studies (e.g., outliers) and re-run meta-analytic and meta-regression models. We can also systematically take each effect size out and run a series of meta-analytic models to see whether any resulting overall effect estimates are different from others; this method is known as ‘leave-one-out’, which is considered less subjective and thus recommended.

The second way of approaching sensitivity analysis is known as subset analysis, where a certain group of effect sizes (studies) will be excluded to re-run the models without this group of effect sizes. For example, one may want to run an analysis without studies that did not randomize samples. Yet, as mentioned earlier, we recommend using meta-regression (Eq.  13 ) with a categorical variable of randomization status (‘randomized’ or ‘not randomized’), to statistically test for an influence of moderators. It is important to note that such tests for risk of bias (or study quality) can be considered as a way of quantitatively evaluating the importance of study features that were noted at the stage of critical appraisal, which is an essential part of any systematic review (see [ 11 , 91 ]). In other words, we can use meta-regression or subset analysis to quantitatively conduct critical appraisal using (study-level) moderators that code, for example, blinding, randomization, and selective reporting. Despite the importance of critical appraisal ([ 91 ]), only 4 of 73 environmental meta-analyses (5.6%) in our survey assessed the risk of bias in each study included in a meta-analysis (i.e., evaluating a primary study in terms of the internal validity of study design and reporting; Additional file 1 ). We emphasize that critically appraising each paper or checking them for risk of bias is an extremely important topic. Also, critical appraisal is not restricted to quantitative synthesis. Therefore, we do not cover any further in this paper for more, see [ 92 , 93 ]).

Notes on transparent reporting and open archiving

For environmental systematic reviews and maps, there are reporting guidelines called RepOrting standards for Systematic Evidence Syntheses in environmental research, ROSES [ 94 ] and synthesis assessment checklist, the Collaboration for Environmental Evidence Synthesis Appraisal Tool (CEESAT; [ 95 ]). However, these guidelines are somewhat limited in terms of reporting quantitative synthesis because they cover only a few core items. These two guidelines are complemented by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Ecology and Evolutionary Biology (PRISMA-EcoEvo; [ 96 ]; cf. [ 97 , 98 ]), which provides an extended set of reporting items covering what we have described above. Items 20–24 from PRISMA-EcoEvo are most relevant: these items outline what should be reported in the Methods section: (i) sample sizes and study characteristics, (ii) meta-analysis, (iii) heterogeneity, (iv) meta-regression and (v) outcomes of publication bias and sensitivity analysis (see Table 4 ). Our survey, as well as earlier surveys, suggest there is a large room for improvement in the current practice ([ 14 , 15 , 16 ]). Incidentally, the orchard plot is well aligned with Item 20, as this plot type shows both the number of effect sizes and studies for different groups (Fig.  4 C). Further, our survey of environmental meta-analyses highlighted the poor standards of data openness (with 24 studies sharing data: 32.9%) and code sharing (7 studies: 29.2%; Additional file 1 ). Environmental scientists must archive their data as well as their analysis code in accordance with the FAIR principles (Findable, Accessible, Interoperable, and Reusable [ 99 ]) using dedicated depositories such as Dryad, FigShare, Open Science Framework (OSF), Zenodo or others (cf. [ 100 , 101 ]), preferably not on publisher’s webpages (as paywall may block access). However, archiving itself is not enough; data requires metadata (detailed descriptions) and the code needs to also be FAIR [ 102 , 103 ].

Other relevant and advanced issues

Scale dependence.

The issue of scale dependence is a unique yet widespread problem in environmental sciences (see [ 7 , 104 ]); our literature survey indicated three quarters of the environmental meta-analyses (56 out of 73 studies) have inferences that are potentially vulnerable to scale-dependence [ 105 ]. For example, studies that set out to compare group means in biodiversity measures, such as species richness, can vary as a function of the scale (size) of the sampling unit. When the unit of replication is a plot (not an individual animal or plant), the aerial size of a plot (e.g., 100 cm 2 or 1 km 2 ) will affect both the precision and accuracy of effect size estimates (e.g., lnRR and SMD). In general, a study with larger plots might have more accurately estimated species richness differences, but less precisely than a study with smaller plots and greater replication. Lower replication means that our sampling variance estimates are likely to be misestimated, and the study with larger plots will generally have less weight than the study with smaller plots, due to higher sampling variance. Inaccurate variance estimates in little-replicated ecological studies are known to cause an accumulating bias in precision-weighted meta-analysis, requiring correction [ 43 ]. To assess the potential for scale-dependence, it is recommended that analysts test for possible covariation among plot size, replication, variances, and effect sizes [ 104 ]. If detected, analysts should use an effect size measure that is less sensitive to scale dependence (lnRR), and could use the size of a plot as a moderator in meta-regression, or alternatively, they consider running an unweighted model ([ 7 ]; note that only 12%, 9 out of 73 studies, accounted for sampling area in some way; Additional file 1 ).

• Missing data

In many fields, meta-analytic data almost always encompass missing values see [ 106 , 107 , 108 ]. Broadly, we have two types of missing data in meta-analyses [ 109 , 110 ]: (1) missing data in standard deviations or sample sizes, associated with means, preventing effect size calculations (Table 2 ), and (2) missing data in moderators. There are several solutions for both types. The best, and first to try, should be contacting the authors. If this fails, we can potentially ‘impute’ missing data. Single imputation methods using the strong correlation between standard deviation and mean values (known as mean–variance relationship) are available, although single imputation can lead to Type I error [ 106 , 107 ] (see also [ 43 ]) because we do not model the uncertainty of imputation itself. Contrastingly, multiple imputation, which creates multiple versions of imputed datasets, incorporates such uncertainty. Indeed, multiple imputation is a preferred and proven solution for missing data in effect sizes and moderators [ 109 , 110 ]. Yet, correct implementation can be challenging (see [ 110 ]). What we require now is an automated pipeline of merging meta-analysis and multiple imputation, which accounts for imputation uncertainty, although it may be challenging for complex meta-analytic models. Fortunately, however, for lnRR, there is a series of new methods that can perform better than the conventional method and which can deal with missing SDs [ 44 ]; note that these methods do not deal with missing moderators. Therefore, where applicable, we recommend these new methods, until an easy-to-implement multiple imputation workflow arrives.

Complex non-independence

Above, we have only dealt with the model that includes study identities as a clustering/grouping (random) factor. However, many datasets are more complex, with potentially more clustering variables in addition to the study identity. It is certainly possible that an environmental meta-analysis contains data from multiple species. Such a situation creates an interesting dependence among effect sizes from different species, known as phylogenetic relatedness, where closely related species are more likely to be similar in effect sizes compared to distantly related ones (e.g., mice vs. rats and mice vs. sparrows). Our multilevel model framework is flexible and can accommodate phylogenetic relatedness. A phylogenetic multilevel meta-analytic model can be written as [ 40 , 111 , 112 ]:

where \({a}_{k\left[i\right]}\) is the phylogenetic (species) effect for the k th species (effect size i ; N effect ( i  = 1, 2,…, N effect ) >  N study ( j  = 1, 2,…, N study ) >  N species ( k  = 1, 2,…, N species )), normally distributed with \({\omega }^{2}{\text{A}}\) where is the phylogenetic variance and A is a correlation matrix coding how close each species are to each other and \({\omega }^{2}\) is the phylogenetic variance, \({s}_{k\left[i\right]}\) is the non-phylogenetic (species) effect for the k th species (effect size i ), normally distributed with the variance of \({\gamma }^{2}\) (the non-phylogenetic variance), and other notations are as above. It is important to realize that A explicitly models relatedness among species, and we do need to provide this correlation matrix, using a distance relationship usually derived from a molecular-based phylogenetic tree (for more details, see [ 40 , 111 , 112 ]). Some may think that the non-phylogenetic term ( \({s}_{k\left[i\right]}\) ) is unnecessary or redundant because \({s}_{k\left[i\right]}\) and the phylogenetic term ( \({a}_{k\left[i\right]}\) ) are both modelling variance at the species level. However, a simulation recently demonstrated that failing to have the non-phylogenetic term ( \({s}_{k\left[i\right]}\) ) will often inflate the phylogenetic variance \({\omega }^{2}\) , leading to an incorrect conclusion that there is a strong phylogenetic signal (as shown in [ 112 ]). The non-phylogenetic variance ( \({\gamma }^{2}\) ) arises from, for example, ecological similarities among species (herbivores vs. carnivores or arboreal vs. ground-living) not phylogeny [ 40 ].

Like phylogenetic relatedness, effect sizes arising from closer geographical locations are likely to be more correlated [ 113 ]. Statistically, spatial correlation can be also modelled in a manner analogous to phylogenetic relatedness (i.e., rather than a phylogenetic correlation matrix, A , we fit a spatial correlation matrix). For example, Maire and colleagues [ 114 ] used a meta-analytic model with spatial autocorrelation to investigate the temporal trends of fish communities in the network of rivers in France. We note that a similar argument can be made for temporal correlation, but in many cases, temporal correlations could be dealt with, albeit less accurately, as a special case of ‘shared measurements’, as in Fig.  2 . An important idea to take away is that one can model different, if not all, types of non-independence as the random factor(s) in a multilevel model.

Advanced techniques

Here we touch upon five advanced meta-analytic techniques with potential utility for environmental sciences, providing relevant references so that interested readers can obtain more information on these advanced topics. The first one is the meta-analysis of magnitudes, or absolute values (effect sizes), where researchers may be interested in deviations from 0, rather than the directionality of the effect [ 115 ]. For example, Cohen and colleagues [ 116 ] investigated absolute values of phenological responses, as they were concerned with the magnitudes of changes in phenology rather than directionality.

The second method is the meta-analysis of interaction where our focus is on synthesizing the interaction effect of, usually, 2 × 2 factorial design (e.g., the effect of two simultaneous environmental stressors [ 54 , 117 , 118 ]; see also [ 119 ]). Recently, Siviter and colleagues [ 120 ] showed that agrochemicals interact synergistically (i.e., non-additively) to increase the mortality of bees; that is, two agrochemicals together caused more mortality than the sum of mortalities of each chemical.

Third, network meta-analysis has been heavily used in medical sciences; network meta-analysis usually compares different treatments in relation to placebo and ranks these treatments in terms of effectiveness [ 121 ]. The very first ‘environmental’ network meta-analysis, as far as we know, investigated the effectives of ecosystem services among different land types [ 122 ].

Fourth, a multivariate meta-analysis is where one can model two or more different types of effect sizes with the estimation of pair-wise correlations between different effect sizes. The benefit of such an approach is known as the ‘borrowing of strength’, where the error of fixed effects (moderators; e.g., b 0 and b 1 ) can be reduced when different types of effect sizes are correlated (i.e., se ( b 0 ) and se ( b 1 ) can be smaller [ 123 ]) For example, it is possible for lnRR (differences in mean) and lnVR (differences in SDs) to be modelled together (cf. [ 124 ]).

Fifth, as with network meta-analysis, there has been a surge in the use of ‘individual participants data’, called ‘IPD meta-analysis’, in medical sciences [ 125 , 126 ]. The idea of IPD meta-analysis is simple—rather than using summary statistics reported in papers (sample means and variances), we directly use raw data from all studies. We can either model raw data using one complex multilevel (hierarchical) model (one-step method) or calculate statistics for each study and use a meta-analysis (two-step method; note that both methods will usually give the same results). Study-level random effects can be incorporated to allow the response variable of interest to vary among studies, and overall effects correspond to fixed, population-level estimates. The use of IPD or ‘full-data analyses’ has also surged in ecology, aided by open-science policies that encourage the archival of raw data alongside articles, and initiatives that synthesise raw data (e.g., PREDICTS [ 127 ], BioTime [ 128 ]). In health disciplines, such meta-analyses are considered the ‘gold standard’ [ 129 ], owing to their potential for resolving issues regarding study-specific designs and confounding variation, and it is unclear whether and how they might resolve issues such as scale dependence in environmental meta-analyses [ 104 , 130 ].

Conclusions

In this article, we have attempted to describe the most practical ways to conduct quantitative synthesis, including meta-analysis, meta-regression, and publication bias tests. In addition, we have shown that there is much to be improved in terms of meta-analytic practice and reporting via a survey of 73 recent environmental meta-analyses. Such improvements are urgently required, especially given the potential influence that environmental meta-analyses can have on policies and decision-making [ 8 ]. So often, meta-analysts have called for better reporting of primary research (e.g. [ 131 , 132 ]), and now this is the time to raise the standards of reporting in meta-analyses. We hope our contribution will help to catalyse a turning point for better practice in quantitative synthesis in environmental sciences. We remind the reader most of what is described is implemented in the R environment on our tutorial webpage and researchers can readily use the proposed models and techniques ( https://itchyshin.github.io/Meta-analysis_tutorial/ ). Finally, meta-analytic techniques are always developing and improving. It is certainly possible that in the future, our proposed models and related methods will become dated, just as the traditional fixed-effect and random-effects models already are. Therefore, we must endeavour to be open-minded to new ways of doing quantitative research synthesis in environmental sciences.

Availability of data and materials

All data and material are provided as additional files.

Higgins JP, Thomas JE, Chandler JE, Cumpston ME, Li TE, Page MJ, Welch VA. Cochrane handbook for systematic reviews of interventions. 2nd ed. Chichester: Wikey; 2019.

Book   Google Scholar  

Cooper HM, Hedges LV, Valentine JC. The handbook of research synthesis and meta-analysis . 3rd ed. New York: Russell Sage Foundation; 2019.

Google Scholar  

Schmid CH, Stijnen TE, White IE. Handbook of meta-analysis. 1st ed. Boca Ranton: CRC; 2021.

Vetter D, Rucker G, Storch I. Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere. 2013;4(6):1.

Article   Google Scholar  

Koricheva J, Gurevitch J, Mengersen K, editors. Handbook of meta-analysis in ecology and evolution. Princeton: Princeton Univesity Press; 2017.

Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555(7695):175–82.

Article   CAS   Google Scholar  

Spake R, Doncaster CP. Use of meta-analysis in forest biodiversity research: key challenges and considerations. Forest Ecol Manag. 2017;400:429–37.

Bilotta GS, Milner AM, Boyd I. On the use of systematic reviews to inform environmental policies. Environ Sci Policy. 2014;42:67–77.

Hedges LV, Vevea JL. Fixed- and random-effects models in meta-analysis. Psychol Methods. 1998;3(4):486–504.

Borenstein M, Hedges LV, Higgins JPT, Rothstein H. Introduction to meta-analysis. 2nd ed. Chichester: Wiley; 2021.

Noble DWA, Lagisz M, Odea RE, Nakagawa S. Nonindependence and sensitivity analyses in ecological and evolutionary meta-analyses. Mol Ecol. 2017;26(9):2410–25.

Nakagawa S, Noble DWA, Senior AM, Lagisz M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. Bmc Biol. 2017;15:1.

Nakagawa S, Senior AM, Viechtbauer W, Noble DWA. An assessment of statistical methods for nonindependent data in ecological meta-analyses: comment. Ecology. 2022;103(1): e03490.

Romanelli JP, Meli P, Naves RP, Alves MC, Rodrigues RR. Reliability of evidence-review methods in restoration ecology. Conserv Biol. 2021;35(1):142–54.

Koricheva J, Gurevitch J. Uses and misuses of meta-analysis in plant ecology. J Ecol. 2014;102(4):828–44.

O’Leary BC, Kvist K, Bayliss HR, Derroire G, Healey JR, Hughes K, Kleinschroth F, Sciberras M, Woodcock P, Pullin AS. The reliability of evidence review methodology in environmental science and conservation. Environ Sci Policy. 2016;64:75–82.

Rosenthal R. The “file drawer problem” and tolerance for null results. Psychol Bull. 1979;86(3):638–41.

Nakagawa S, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, Sánchez-Tójar A, Yang Y, O’Dea RE. Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods Ecol Evol. 2022;13(1):4–21.

Cheung MWL. A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychol Rev. 2019;29(4):387–96.

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1–48.

Yang Y, Macleod M, Pan J, Lagisz M, Nakagawa S. Advanced methods and implementations for the meta-analyses of animal models: current practices and future recommendations. Neurosci Biobehav Rev. 2022. https://doi.org/10.1016/j.neubiorev.2022.105016:105016 .

Nakagawa S, Cuthill IC. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev. 2007;82(4):591–605.

Hedges LV, Gurevitch J, Curtis PS. The meta-analysis of response ratios in experimental ecology. Ecology. 1999;80(4):1150–6.

Friedrich JO, Adhikari NKJ, Beyene J. The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: A simulation study. BMC Med Res Methodol. 2008;8:5.

Hedges L, Olkin I. Statistical methods for meta-analysis. New York: Academic Press; 1985.

Cohen J. Statistical power analysis for the beahvioral sciences. 2nd ed. Hillsdale: Lawrence Erlbaum; 1988.

Senior AM, Viechtbauer W, Nakagawa S. Revisiting and expanding the meta-analysis of variation: the log coefficient of variation ratio. Res Synth Methods. 2020;11(4):553–67.

Nakagawa S, Poulin R, Mengersen K, Reinhold K, Engqvist L, Lagisz M, Senior AM. Meta-analysis of variation: ecological and evolutionary applications and beyond. Methods Ecol Evol. 2015;6(2):143–52.

Knapp S, van der Heijden MGA. A global meta-analysis of yield stability in organic and conservation agriculture. Nat Commun. 2018;9:3632.

Porturas LD, Anneberg TJ, Cure AE, Wang SP, Althoff DM, Segraves KA. A meta-analysis of whole genome duplication and theeffects on flowering traits in plants. Am J Bot. 2019;106(3):469–76.

Janicke T, Morrow EH. Operational sex ratio predicts the opportunity and direction of sexual selection across animals. Ecol Lett. 2018;21(3):384–91.

Chamberlain R, Brunswick N, Siev J, McManus IC. Meta-analytic findings reveal lower means but higher variances in visuospatial ability in dyslexia. Brit J Psychol. 2018;109(4):897–916.

O’Dea RE, Lagisz M, Jennions MD, Nakagawa S. Gender differences in individual variation in academic grades fail to fit expected patterns for STEM. Nat Commun. 2018;9:3777.

Brugger SP, Angelescu I, Abi-Dargham A, Mizrahi R, Shahrezaei V, Howes OD. Heterogeneity of striatal dopamine function in schizophrenia: meta-analysis of variance. Biol Psychiat. 2020;87(3):215–24.

Usui T, Macleod MR, McCann SK, Senior AM, Nakagawa S. Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. Plos Biol. 2021;19(5): e3001009.

Hoffmann AA, Merila J. Heritable variation and evolution under favourable and unfavourable conditions. Trends Ecol Evol. 1999;14(3):96–101.

Wood CW, Brodie ED 3rd. Environmental effects on the structure of the G-matrix. Evolution. 2015;69(11):2927–40.

Hillebrand H, Donohue I, Harpole WS, Hodapp D, Kucera M, Lewandowska AM, Merder J, Montoya JM, Freund JA. Thresholds for ecological responses to global change do not emerge from empirical data. Nat Ecol Evol. 2020;4(11):1502.

Yang YF, Hillebrand H, Lagisz M, Cleasby I, Nakagawa S. Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology. Global Change Biol. 2022;28(3):969–89.

Nakagawa S, Santos ESA. Methodological issues and advances in biological meta-analysis. Evol Ecol. 2012;26(5):1253–74.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of response ratios. BMC Med Res Methodol. 2020;20(1):1.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of mean difference and standardized mean difference. Stat Med. 2020;39(2):171–91.

Doncaster CP, Spake R. Correction for bias in meta-analysis of little-replicated studies. Methods Ecol Evol. 2018;9(3):634–44.

Nakagawa S, Noble DW, Lagisz M, Spake R, Viechtbauer W, Senior AM. A robust and readily implementable method for the meta-analysis of response ratios with and without missing standard deviations. Ecol Lett. 2023;26(2):232–44

Hamman EA, Pappalardo P, Bence JR, Peacor SD, Osenberg CW. Bias in meta-analyses using Hedges’ d. Ecosphere. 2018;9(9): e02419.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. On the Q statistic with constant weights for standardized mean difference. Brit J Math Stat Psy. 2022;75(3):444–65.

DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: an update. Contemp Clin Trials. 2007;28(2):105–14.

Veroniki AA, Jackson D, Viechtbauer W, Bender R, Bowden J, Knapp G, Kuss O, Higgins JPT, Langan D, Salanti G. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7(1):55–79.

Langan D, Higgins JPT, Simmonds M. Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Synth Methods. 2017;8(2):181–98.

Panityakul T, Bumrungsup C, Knapp G. On estimating residual heterogeneity in random-effects meta-regression: a comparative study. J Stat Theory Appl. 2013;12(3):253–65.

Bishop J, Nakagawa S. Quantifying crop pollinator dependence and its heterogeneity using multi-level meta-analysis. J Appl Ecol. 2021;58(5):1030–42.

Cheung MWL. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol Methods. 2014;19(2):211–29.

Bolker BM, Brooks ME, Clark CJ, Geange SW, Poulsen JR, Stevens MHH, White JSS. Generalized linear mixed models: a practical guide for ecology and evolution. Trends Ecol Evol. 2009;24(3):127–35.

Lajeunesse MJ. On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology. 2011;92(11):2049–55.

Gleser LJ, Olkin I. Stochastically dependent effect sizes. In: Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and meta-analysis. New York: Russell Sage Foundation; 2009.

Tipton E, Pustejovsky JE. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J Educ Behav Stat. 2015;40(6):604–34.

Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates (vol 1, pg 39, 2010). Res Synth Methods. 2010;1(2):164–5.

Pustejovsky JE, Tipton E. Meta-analysis with robust variance estimation: expanding the range of working models. Prev Sci. 2021. https://doi.org/10.1007/s11121-021-01246-3 .

Cairns M, Prendergast LA. On ratio measures of heterogeneity for meta-analyses. Res Synth Methods. 2022;13(1):28–47.

Borenstein M, Higgins JPT, Hedges LV, Rothstein HR. Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Res Synth Methods. 2017;8(1):5–18.

Hoaglin DC. Practical challenges of I-2 as a measure of heterogeneity. Res Synth Methods. 2017;8(3):254–254.

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. Brit Med J. 2003;327(7414):557–60.

Xiong CJ, Miller JP, Morris JC. Measuring study-specific heterogeneity in meta-analysis: application to an antecedent biomarker study of Alzheimer’s disease. Stat Biopharm Res. 2010;2(3):300–9.

Nakagawa S, Schielzeth H. Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists. Biol Rev. 2010;85(4):935–56.

Senior AM, Grueber CE, Kamiya T, Lagisz M, O’Dwyer K, Santos ESA, Nakagawa S. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology. 2016;97(12):3293–9.

Gelman A, Hill J. Data analysis using regression and multilevel/hierarchical models. Cambridge: Cambridge University Press; 2007.

Schielzeth H. Simple means to improve the interpretability of regression coefficients. Methods Ecol Evol. 2010;1(2):103–13.

Nakagawa S, Schielzeth H. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods Ecol Evol. 2013;4(2):133–42.

Nakagawa S, Johnson PCD, Schielzeth H. The coefficient of determination R-2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. J R Soc Interface. 2017;14(134):20170213.

Aloe AM, Becker BJ, Pigott TD. An alternative to R-2 for assessing linear models of effect size. Res Synth Methods. 2010;1(3–4):272–83.

Cinar O, Umbanhowar J, Hoeksema JD, Viechtbauer W. Using information-theoretic approaches for model selection in meta-analysis. Res Synth Methods. 2021. https://doi.org/10.1002/jrsm.1489 .

Viechtbauer W. Model checking in meta-analysis. In: Schmid CH, Stijnen T, White IR, editors. Handbook of meta-analysis. Boca Raton: CRC; 2021.

Anzures-Cabrera J, Higgins JPT. Graphical displays for meta-analysis: An overview with suggestions for practice. Res Synth Methods. 2010;1(1):66–80.

Kossmeier M, Tran US, Voracek M. Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis. Bmc Med Res Methodol. 2020;20(1):1.

Intout J, Ioannidis JPA, Rovers MM, Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open. 2016;6(7): e010247.

Moeyaert M, Ugille M, Beretvas SN, Ferron J, Bunuan R, Van den Noortgate W. Methods for dealing with multiple outcomes in meta-analysis a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. Int J Soc Res Methodol. 2017;20:559.

Nakagawa S, Lagisz M, O’Dea RE, Rutkowska J, Yang YF, Noble DWA, Senior AM. The orchard plot: cultivating a forest plot for use in ecology, evolution, and beyond. Res Synth Methods. 2021;12(1):4–12.

Rothstein H, Sutton AJ, Borenstein M. Publication bias in meta-analysis : prevention, assessment and adjustments. Hoboken: Wiley; 2005.

Nakagawa S, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, Sanchez-Tojar A, Yang YF, O’Dea RE. Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods Ecol Evol. 2022;13(1):4–21.

Stanley TD, Doucouliagos H. Meta-regression analysis in economics and business. New York: Routledge; 2012.

Stanley TD, Doucouliagos H. Meta-regression approximations to reduce publication selection bias. Res Synth Methods. 2014;5(1):60–78.

Sterne JAC, Becker BJ, Egger M. The funnel plot. In: Rothstein H, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester: Wiley; 2005. p. 75–98.

Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rucker G, Harbord RM, Schmid CH, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. Br Med J. 2011;343:4002.

Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Brit Med J. 1997;315(7109):629–34.

Jennions MD, Moller AP. Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. P Roy Soc B-Biol Sci. 2002;269(1486):43–8.

Koricheva J, Kulinskaya E. Temporal instability of evidence base: a threat to policy making? Trends Ecol Evol. 2019;34(10):895–902.

Raue A, Kreutz C, Maiwald T, Bachmann J, Schilling M, Klingmuller U, Timmer J. Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics. 2009;25(15):1923–9.

Matsushima Y, Noma H, Yamada T, Furukawa TA. Influence diagnostics and outlier detection for meta-analysis of diagnostic test accuracy. Res Synth Methods. 2020;11(2):237–47.

Viechtbauer W, Cheung MWL. Outlier and influence diagnostics for meta-analysis. Res Synth Methods. 2010;1(2):112–25.

Haddaway NR, Macura B. The role of reporting standards in producing robust literature reviews comment. Nat Clim Change. 2018;8(6):444–7.

Frampton G, Whaley P, Bennett M, Bilotta G, Dorne JLCM, Eales J, James K, Kohl C, Land M, Livoreil B, et al. Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews. Environ Evid. 2022;11(1):12.

Stanhope J, Weinstein P. Critical appraisal in ecology: what tools are available, and what is being used in systematic reviews? Res Synth Methods. 2022. https://doi.org/10.1002/jrsm.1609 .

Haddaway NR, Macura B, Whaley P, Pullin AS. ROSES RepOrting standards for systematic evidence syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ Evid. 2018;7(1):1.

Woodcock P, Pullin AS, Kaiser MJ. Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: a methodology. Biol Conserv. 2014;176:54–62.

O’Dea RE, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, Gurevitch J, Page MJ, Stewart G, Moher D, et al. Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension. Biol Rev. 2021;96(5):1695–722.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Plos Med. 2009;6(7):e1000097.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Plos Med. 2021;18(3): e1003583.

Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, Blomberg N, Boiten JW, Santos LBD, Bourne PE, et al. Comment: the FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3: 160018.

Culina A, Baglioni M, Crowther TW, Visser ME, Woutersen-Windhouwer S, Manghi P. Navigating the unfolding open data landscape in ecology and evolution. Nat Ecol Evol. 2018;2(3):420–6.

Roche DG, Lanfear R, Binning SA, Haff TM, Schwanz LE, Cain KE, Kokko H, Jennions MD, Kruuk LE. Troubleshooting public data archiving: suggestions to increase participation. Plos Biol. 2014;12(1): e1001779.

Roche DG, Kruuk LEB, Lanfear R, Binning SA. Public data archiving in ecology and evolution: how well are we doing? Plos Biol. 2015;13(11): e1002295.

Culina A, van den Berg I, Evans S, Sanchez-Tojar A. Low availability of code in ecology: a call for urgent action. Plos Biol. 2020;18(7): e3000763.

Spake R, Mori AS, Beckmann M, Martin PA, Christie AP, Duguid MC, Doncaster CP. Implications of scale dependence for cross-study syntheses of biodiversity differences. Ecol Lett. 2021;24(2):374–90.

Osenberg CW, Sarnelle O, Cooper SD. Effect size in ecological experiments: the application of biological models in meta-analysis. Am Nat. 1997;150(6):798–812.

Noble DWA, Nakagawa S. Planned missing data designs and methods: options for strengthening inference, increasing research efficiency and improving animal welfare in ecological and evolutionary research. Evol Appl. 2021;14(8):1958–68.

Nakagawa S, Freckleton RP. Missing inaction: the dangers of ignoring missing data. Trends Ecol Evol. 2008;23(11):592–6.

Mavridis D, Chaimani A, Efthimiou O, Leucht S, Salanti G. Addressing missing outcome data in meta-analysis. Evid-Based Ment Health. 2014;17(3):85.

Ellington EH, Bastille-Rousseau G, Austin C, Landolt KN, Pond BA, Rees EE, Robar N, Murray DL. Using multiple imputation to estimate missing data in meta-regression. Methods Ecol Evol. 2015;6(2):153–63.

Kambach S, Bruelheide H, Gerstner K, Gurevitch J, Beckmann M, Seppelt R. Consequences of multiple imputation of missing standard deviations and sample sizes in meta-analysis. Ecol Evol. 2020;10(20):11699–712.

Hadfield JD, Nakagawa S. General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J Evol Biol. 2010;23(3):494–508.

Cinar O, Nakagawa S, Viechtbauer W. Phylogenetic multilevel meta-analysis: a simulation study on the importance of modelling the phylogeny. Methods Ecol Evol. 2021. https://doi.org/10.1111/2041-210X.13760 .

Ives AR, Zhu J. Statistics for correlated data: phylogenies, space, and time. Ecol Appl. 2006;16(1):20–32.

Maire A, Thierry E, Viechtbauer W, Daufresne M. Poleward shift in large-river fish communities detected with a novel meta-analysis framework. Freshwater Biol. 2019;64(6):1143–56.

Morrissey MB. Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J Evol Biol. 2016;29(10):1882–904.

Cohen JM, Lajeunesse MJ, Rohr JR. A global synthesis of animal phenological responses to climate change. Nat Clim Change. 2018;8(3):224.

Gurevitch J, Morrison JA, Hedges LV. The interaction between competition and predation: a meta-analysis of field experiments. Am Nat. 2000;155(4):435–53.

Macartney EL, Lagisz M, Nakagawa S. The relative benefits of environmental enrichment on learning and memory are greater when stressed: a meta-analysis of interactions in rodents. Neurosci Biobehav R. 2022. https://doi.org/10.1016/j.neubiorev.2022.104554 .

Spake R, Bowler DE, Callaghan CT, Blowes SA, Doncaster CP, Antão LH, Nakagawa S, McElreath R, Chase JM. Understanding ‘it depends’ in ecology: a guide to hypothesising, visualising and interpreting statistical interactions. Biol Rev. 2023. https://doi.org/10.1111/brv.12939 .

Siviter H, Bailes EJ, Martin CD, Oliver TR, Koricheva J, Leadbeater E, Brown MJF. Agrochemicals interact synergistically to increase bee mortality. Nature. 2021;596(7872):389.

Salanti G, Schmid CH. Research synthesis methods special issue on network meta-analysis: introduction from the editors. Res Synth Methods. 2012;3(2):69–70.

Gomez-Creutzberg C, Lagisz M, Nakagawa S, Brockerhoff EG, Tylianakis JM. Consistent trade-offs in ecosystem services between land covers with different production intensities. Biol Rev. 2021;96(5):1989–2008.

Jackson D, White IR, Price M, Copas J, Riley RD. Borrowing of strength and study weights in multivariate and network meta-analysis. Stat Methods Med Res. 2017;26(6):2853–68.

Sanchez-Tojar A, Moran NP, O’Dea RE, Reinhold K, Nakagawa S. Illustrating the importance of meta-analysing variances alongside means in ecology and evolution. J Evol Biol. 2020;33(9):1216–23.

Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ. 2010;340:c221.

Riley RD, Tierney JF, Stewart LA. Individual participant data meta-analysis : a handbook for healthcare research. 1st ed. Hoboken: Wiley; 2021.

Hudson LN, Newbold T, Contu S, Hill SLL, Lysenko I, De Palma A, Phillips HRP, Alhusseini TI, Bedford FE, Bennett DJ, et al. The database of the PREDICTS (projecting responses of ecological diversity in changing terrestrial systems) project. Ecol Evol. 2017;7(1):145–88.

Dornelas M, Antao LH, Moyes F, Bates AE, Magurran AE, Adam D, Akhmetzhanova AA, Appeltans W, Arcos JM, Arnold H, et al. BioTIME: a database of biodiversity time series for the anthropocene. Glob Ecol Biogeogr. 2018;27(7):760–86.

Mengersen K, Gurevitch J, Schmid CH. Meta-analysis of primary data. In: Koricheva J, Gurevitch J, Mengersen K, editors. Handbook of meta-analysis in ecology and evolution. Priceton: Princeton university; 2013. p. 300–12.

Spake R, O’Dea RE, Nakagawa S, Doncaster CP, Ryo M, Callaghan CT, Bullock JM. Improving quantitative synthesis to achieve generality in ecology. Nat Ecol Evol. 2022;6(12):1818–28.

Gerstner K, Moreno-Mateos D, Gurevitch J, Beckmann M, Kambach S, Jones HP, Seppelt R. Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting. Methods Ecol Evol. 2017;8(6):777–84.

Haddaway NR. A call for better reporting of conservation research data for use in meta-analyses. Conserv Biol. 2015;29(4):1242–5.

Midolo G, De Frenne P, Holzel N, Wellstein C. Global patterns of intraspecific leaf trait responses to elevation. Global Change Biol. 2019;25(7):2485–98.

White IR, Schmid CH, Stijnen T. Choice of effect measure and issues in extracting outcome data. In: Schmid CH, Stijnen T, White IR, editors. Handbook of meta-analysis. Boca Raton: CRC; 2021.

Lajeunesse MJ. Bias and correction for the log response ratio in ecological meta-analysis. Ecology. 2015;96(8):2056–63.

Download references

Acknowledgements

SN, ELM, and ML were supported by the ARC (Australian Research Council) Discovery grant (DP200100367), and SN, YY, and ML by the ARC Discovery grant (DP210100812). YY was also supported by the National Natural Science Foundation of China (32102597). A part of this research was conducted while visiting the Okinawa Institute of Science and Technology (OIST) through the Theoretical Sciences Visiting Program (TSVP) to SN.

Australian Research Council Discovery grant (DP200100367); Australian Research Council Discovery grant (DP210100812); The National Natural Science Foundation of China (32102597).

Author information

Authors and affiliations.

Evolution & Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, NSW, 2052, Australia

Shinichi Nakagawa, Yefeng Yang, Erin L. Macartney & Malgorzata Lagisz

Theoretical Sciences Visiting Program, Okinawa Institute of Science and Technology Graduate University, Onna, 904-0495, Japan

Shinichi Nakagawa

School of Biological Sciences, Whiteknights Campus, University of Reading, Reading, RG6 6AS, UK

Rebecca Spake

You can also search for this author in PubMed   Google Scholar

Contributions

SN was commissioned to write this article so he assembled a team of co-authors. SN discussed the idea with YY, ELM, RS and ML, and all of them contributed to the design of this review. ML led the survey working with YY and ELM, while YY led the creation of the accompanying webpage working with RS. SN supervised all aspects of this work and wrote the first draft, which was commented on, edited, and therefore, significantly improved by the other co-authors. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Shinichi Nakagawa or Yefeng Yang .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Contest for publication

The authors provide consent for publication.

Competing interests

The authors report no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:.

The survey of meta-analyses in environmnetal sciences.

Additional file 2:

The hands-on R tutorial.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Nakagawa, S., Yang, Y., Macartney, E.L. et al. Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences. Environ Evid 12 , 8 (2023). https://doi.org/10.1186/s13750-023-00301-6

Download citation

Received : 13 January 2023

Accepted : 23 March 2023

Published : 24 April 2023

DOI : https://doi.org/10.1186/s13750-023-00301-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Hierarchical models
• Robust variance estimation
• Spatial dependency
• Variance–covariance matrix
• Meta-analysis of variance
• Network meta-analysis
• Multivariate meta-analysis

Environmental Evidence

ISSN: 2047-2382

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical Literature
• Classical Reception
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Language Acquisition
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Culture
• Music and Religion
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Science
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Society
• Law and Politics
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Games
• Computer Security
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business History
• Business Strategy
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Methodology
• Economic Systems
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Politics and Law
• Politics of Development
• Public Policy
• Public Administration
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

30 Meta-Analysis and Quantitative Research Synthesis

Noel A. Card, Family Studies and Human Development, University of Arizona, Tucson, AZ

Deborah M. Casper, Family Studies and Human Development, University of Arizona, Tucson, AZ

• Published: 01 October 2013
• Cite Icon Cite
• Permissions Icon Permissions

Meta-analysis is an increasingly common method of quantitatively synthesizing research results, with substantial advantages over traditional (i.e., qualitative or narrative) methods of literature review. This chapter is an overview of meta-analysis that provides the foundational knowledge necessary to understand the goals of meta-analysis and the process of conducting a meta-analysis, from the initial formulation of research questions through the interpretation of results. The chapter provides insights into the types of research questions that can and cannot be answered through meta-analysis as well as more practical information on the practices of meta-analysis. Finally, the chapter concludes with some advanced topics intended to alert readers to further possibilities available through meta-analysis.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• DOI: 10.2307/3151396
• Corpus ID: 144610380

Meta-Analysis: Quantitative Methods for Research Synthesis

• Published 1 November 1987

1,635 Citations

Meta-analytic procedures for estimation of effect sizes in experiments using complex analysis of variance, meta-analysis as a tool for understanding existing research literature., meta-analysis: a primer, meta-analysis of studies on cpr: a better route to new practice guidelines, understanding the role and methods of meta-analysis in is research.

• Highly Influenced

The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory.

Applications of meta-analysis for marketing and public policy: a review, meta-analysis: methods of accumulating results across research domains, meta-analysis in medical research review articles:, the reliability of meta-analytic review, related papers.

Showing 1 through 3 of 0 Related Papers

How to conduct a meta-analysis in eight steps: a practical guide

• Open access
• Published: 30 November 2021
• Volume 72 , pages 1–19, ( 2022 )

Cite this article

You have full access to this open access article

• Christopher Hansen 1 ,
• Holger Steinmetz 2 &
• Jörn Block 3 , 4 , 5  

158k Accesses

52 Citations

157 Altmetric

Explore all metrics

Avoid common mistakes on your manuscript.

1 Introduction

“Scientists have known for centuries that a single study will not resolve a major issue. Indeed, a small sample study will not even resolve a minor issue. Thus, the foundation of science is the cumulation of knowledge from the results of many studies.” (Hunter et al. 1982 , p. 10)

Meta-analysis is a central method for knowledge accumulation in many scientific fields (Aguinis et al. 2011c ; Kepes et al. 2013 ). Similar to a narrative review, it serves as a synopsis of a research question or field. However, going beyond a narrative summary of key findings, a meta-analysis adds value in providing a quantitative assessment of the relationship between two target variables or the effectiveness of an intervention (Gurevitch et al. 2018 ). Also, it can be used to test competing theoretical assumptions against each other or to identify important moderators where the results of different primary studies differ from each other (Aguinis et al. 2011b ; Bergh et al. 2016 ). Rooted in the synthesis of the effectiveness of medical and psychological interventions in the 1970s (Glass 2015 ; Gurevitch et al. 2018 ), meta-analysis is nowadays also an established method in management research and related fields.

The increasing importance of meta-analysis in management research has resulted in the publication of guidelines in recent years that discuss the merits and best practices in various fields, such as general management (Bergh et al. 2016 ; Combs et al. 2019 ; Gonzalez-Mulé and Aguinis 2018 ), international business (Steel et al. 2021 ), economics and finance (Geyer-Klingeberg et al. 2020 ; Havranek et al. 2020 ), marketing (Eisend 2017 ; Grewal et al. 2018 ), and organizational studies (DeSimone et al. 2020 ; Rudolph et al. 2020 ). These articles discuss existing and trending methods and propose solutions for often experienced problems. This editorial briefly summarizes the insights of these papers; provides a workflow of the essential steps in conducting a meta-analysis; suggests state-of-the art methodological procedures; and points to other articles for in-depth investigation. Thus, this article has two goals: (1) based on the findings of previous editorials and methodological articles, it defines methodological recommendations for meta-analyses submitted to Management Review Quarterly (MRQ); and (2) it serves as a practical guide for researchers who have little experience with meta-analysis as a method but plan to conduct one in the future.

2 Eight steps in conducting a meta-analysis

2.1 step 1: defining the research question.

The first step in conducting a meta-analysis, as with any other empirical study, is the definition of the research question. Most importantly, the research question determines the realm of constructs to be considered or the type of interventions whose effects shall be analyzed. When defining the research question, two hurdles might develop. First, when defining an adequate study scope, researchers must consider that the number of publications has grown exponentially in many fields of research in recent decades (Fortunato et al. 2018 ). On the one hand, a larger number of studies increases the potentially relevant literature basis and enables researchers to conduct meta-analyses. Conversely, scanning a large amount of studies that could be potentially relevant for the meta-analysis results in a perhaps unmanageable workload. Thus, Steel et al. ( 2021 ) highlight the importance of balancing manageability and relevance when defining the research question. Second, similar to the number of primary studies also the number of meta-analyses in management research has grown strongly in recent years (Geyer-Klingeberg et al. 2020 ; Rauch 2020 ; Schwab 2015 ). Therefore, it is likely that one or several meta-analyses for many topics of high scholarly interest already exist. However, this should not deter researchers from investigating their research questions. One possibility is to consider moderators or mediators of a relationship that have previously been ignored. For example, a meta-analysis about startup performance could investigate the impact of different ways to measure the performance construct (e.g., growth vs. profitability vs. survival time) or certain characteristics of the founders as moderators. Another possibility is to replicate previous meta-analyses and test whether their findings can be confirmed with an updated sample of primary studies or newly developed methods. Frequent replications and updates of meta-analyses are important contributions to cumulative science and are increasingly called for by the research community (Anderson & Kichkha 2017 ; Steel et al. 2021 ). Consistent with its focus on replication studies (Block and Kuckertz 2018 ), MRQ therefore also invites authors to submit replication meta-analyses.

2.2 Step 2: literature search

2.2.1 search strategies.

Similar to conducting a literature review, the search process of a meta-analysis should be systematic, reproducible, and transparent, resulting in a sample that includes all relevant studies (Fisch and Block 2018 ; Gusenbauer and Haddaway 2020 ). There are several identification strategies for relevant primary studies when compiling meta-analytical datasets (Harari et al. 2020 ). First, previous meta-analyses on the same or a related topic may provide lists of included studies that offer a good starting point to identify and become familiar with the relevant literature. This practice is also applicable to topic-related literature reviews, which often summarize the central findings of the reviewed articles in systematic tables. Both article types likely include the most prominent studies of a research field. The most common and important search strategy, however, is a keyword search in electronic databases (Harari et al. 2020 ). This strategy will probably yield the largest number of relevant studies, particularly so-called ‘grey literature’, which may not be considered by literature reviews. Gusenbauer and Haddaway ( 2020 ) provide a detailed overview of 34 scientific databases, of which 18 are multidisciplinary or have a focus on management sciences, along with their suitability for literature synthesis. To prevent biased results due to the scope or journal coverage of one database, researchers should use at least two different databases (DeSimone et al. 2020 ; Martín-Martín et al. 2021 ; Mongeon & Paul-Hus 2016 ). However, a database search can easily lead to an overload of potentially relevant studies. For example, key term searches in Google Scholar for “entrepreneurial intention” and “firm diversification” resulted in more than 660,000 and 810,000 hits, respectively. Footnote 1 Therefore, a precise research question and precise search terms using Boolean operators are advisable (Gusenbauer and Haddaway 2020 ). Addressing the challenge of identifying relevant articles in the growing number of database publications, (semi)automated approaches using text mining and machine learning (Bosco et al. 2017 ; O’Mara-Eves et al. 2015 ; Ouzzani et al. 2016 ; Thomas et al. 2017 ) can also be promising and time-saving search tools in the future. Also, some electronic databases offer the possibility to track forward citations of influential studies and thereby identify further relevant articles. Finally, collecting unpublished or undetected studies through conferences, personal contact with (leading) scholars, or listservs can be strategies to increase the study sample size (Grewal et al. 2018 ; Harari et al. 2020 ; Pigott and Polanin 2020 ).

2.2.2 Study inclusion criteria and sample composition

Next, researchers must decide which studies to include in the meta-analysis. Some guidelines for literature reviews recommend limiting the sample to studies published in renowned academic journals to ensure the quality of findings (e.g., Kraus et al. 2020 ). For meta-analysis, however, Steel et al. ( 2021 ) advocate for the inclusion of all available studies, including grey literature, to prevent selection biases based on availability, cost, familiarity, and language (Rothstein et al. 2005 ), or the “Matthew effect”, which denotes the phenomenon that highly cited articles are found faster than less cited articles (Merton 1968 ). Harrison et al. ( 2017 ) find that the effects of published studies in management are inflated on average by 30% compared to unpublished studies. This so-called publication bias or “file drawer problem” (Rosenthal 1979 ) results from the preference of academia to publish more statistically significant and less statistically insignificant study results. Owen and Li ( 2020 ) showed that publication bias is particularly severe when variables of interest are used as key variables rather than control variables. To consider the true effect size of a target variable or relationship, the inclusion of all types of research outputs is therefore recommended (Polanin et al. 2016 ). Different test procedures to identify publication bias are discussed subsequently in Step 7.

In addition to the decision of whether to include certain study types (i.e., published vs. unpublished studies), there can be other reasons to exclude studies that are identified in the search process. These reasons can be manifold and are primarily related to the specific research question and methodological peculiarities. For example, studies identified by keyword search might not qualify thematically after all, may use unsuitable variable measurements, or may not report usable effect sizes. Furthermore, there might be multiple studies by the same authors using similar datasets. If they do not differ sufficiently in terms of their sample characteristics or variables used, only one of these studies should be included to prevent bias from duplicates (Wood 2008 ; see this article for a detection heuristic).

In general, the screening process should be conducted stepwise, beginning with a removal of duplicate citations from different databases, followed by abstract screening to exclude clearly unsuitable studies and a final full-text screening of the remaining articles (Pigott and Polanin 2020 ). A graphical tool to systematically document the sample selection process is the PRISMA flow diagram (Moher et al. 2009 ). Page et al. ( 2021 ) recently presented an updated version of the PRISMA statement, including an extended item checklist and flow diagram to report the study process and findings.

2.3 Step 3: choice of the effect size measure

2.3.1 types of effect sizes.

The two most common meta-analytical effect size measures in management studies are (z-transformed) correlation coefficients and standardized mean differences (Aguinis et al. 2011a ; Geyskens et al. 2009 ). However, meta-analyses in management science and related fields may not be limited to those two effect size measures but rather depend on the subfield of investigation (Borenstein 2009 ; Stanley and Doucouliagos 2012 ). In economics and finance, researchers are more interested in the examination of elasticities and marginal effects extracted from regression models than in pure bivariate correlations (Stanley and Doucouliagos 2012 ). Regression coefficients can also be converted to partial correlation coefficients based on their t-statistics to make regression results comparable across studies (Stanley and Doucouliagos 2012 ). Although some meta-analyses in management research have combined bivariate and partial correlations in their study samples, Aloe ( 2015 ) and Combs et al. ( 2019 ) advise researchers not to use this practice. Most importantly, they argue that the effect size strength of partial correlations depends on the other variables included in the regression model and is therefore incomparable to bivariate correlations (Schmidt and Hunter 2015 ), resulting in a possible bias of the meta-analytic results (Roth et al. 2018 ). We endorse this opinion. If at all, we recommend separate analyses for each measure. In addition to these measures, survival rates, risk ratios or odds ratios, which are common measures in medical research (Borenstein 2009 ), can be suitable effect sizes for specific management research questions, such as understanding the determinants of the survival of startup companies. To summarize, the choice of a suitable effect size is often taken away from the researcher because it is typically dependent on the investigated research question as well as the conventions of the specific research field (Cheung and Vijayakumar 2016 ).

2.3.2 Conversion of effect sizes to a common measure

After having defined the primary effect size measure for the meta-analysis, it might become necessary in the later coding process to convert study findings that are reported in effect sizes that are different from the chosen primary effect size. For example, a study might report only descriptive statistics for two study groups but no correlation coefficient, which is used as the primary effect size measure in the meta-analysis. Different effect size measures can be harmonized using conversion formulae, which are provided by standard method books such as Borenstein et al. ( 2009 ) or Lipsey and Wilson ( 2001 ). There also exist online effect size calculators for meta-analysis. Footnote 2

2.4 Step 4: choice of the analytical method used

Choosing which meta-analytical method to use is directly connected to the research question of the meta-analysis. Research questions in meta-analyses can address a relationship between constructs or an effect of an intervention in a general manner, or they can focus on moderating or mediating effects. There are four meta-analytical methods that are primarily used in contemporary management research (Combs et al. 2019 ; Geyer-Klingeberg et al. 2020 ), which allow the investigation of these different types of research questions: traditional univariate meta-analysis, meta-regression, meta-analytic structural equation modeling, and qualitative meta-analysis (Hoon 2013 ). While the first three are quantitative, the latter summarizes qualitative findings. Table 1 summarizes the key characteristics of the three quantitative methods.

2.4.1 Univariate meta-analysis

In its traditional form, a meta-analysis reports a weighted mean effect size for the relationship or intervention of investigation and provides information on the magnitude of variance among primary studies (Aguinis et al. 2011c ; Borenstein et al. 2009 ). Accordingly, it serves as a quantitative synthesis of a research field (Borenstein et al. 2009 ; Geyskens et al. 2009 ). Prominent traditional approaches have been developed, for example, by Hedges and Olkin ( 1985 ) or Hunter and Schmidt ( 1990 , 2004 ). However, going beyond its simple summary function, the traditional approach has limitations in explaining the observed variance among findings (Gonzalez-Mulé and Aguinis 2018 ). To identify moderators (or boundary conditions) of the relationship of interest, meta-analysts can create subgroups and investigate differences between those groups (Borenstein and Higgins 2013 ; Hunter and Schmidt 2004 ). Potential moderators can be study characteristics (e.g., whether a study is published vs. unpublished), sample characteristics (e.g., study country, industry focus, or type of survey/experiment participants), or measurement artifacts (e.g., different types of variable measurements). The univariate approach is thus suitable to identify the overall direction of a relationship and can serve as a good starting point for additional analyses. However, due to its limitations in examining boundary conditions and developing theory, the univariate approach on its own is currently oftentimes viewed as not sufficient (Rauch 2020 ; Shaw and Ertug 2017 ).

2.4.2 Meta-regression analysis

Meta-regression analysis (Hedges and Olkin 1985 ; Lipsey and Wilson 2001 ; Stanley and Jarrell 1989 ) aims to investigate the heterogeneity among observed effect sizes by testing multiple potential moderators simultaneously. In meta-regression, the coded effect size is used as the dependent variable and is regressed on a list of moderator variables. These moderator variables can be categorical variables as described previously in the traditional univariate approach or (semi)continuous variables such as country scores that are merged with the meta-analytical data. Thus, meta-regression analysis overcomes the disadvantages of the traditional approach, which only allows us to investigate moderators singularly using dichotomized subgroups (Combs et al. 2019 ; Gonzalez-Mulé and Aguinis 2018 ). These possibilities allow a more fine-grained analysis of research questions that are related to moderating effects. However, Schmidt ( 2017 ) critically notes that the number of effect sizes in the meta-analytical sample must be sufficiently large to produce reliable results when investigating multiple moderators simultaneously in a meta-regression. For further reading, Tipton et al. ( 2019 ) outline the technical, conceptual, and practical developments of meta-regression over the last decades. Gonzalez-Mulé and Aguinis ( 2018 ) provide an overview of methodological choices and develop evidence-based best practices for future meta-analyses in management using meta-regression.

2.4.3 Meta-analytic structural equation modeling (MASEM)

MASEM is a combination of meta-analysis and structural equation modeling and allows to simultaneously investigate the relationships among several constructs in a path model. Researchers can use MASEM to test several competing theoretical models against each other or to identify mediation mechanisms in a chain of relationships (Bergh et al. 2016 ). This method is typically performed in two steps (Cheung and Chan 2005 ): In Step 1, a pooled correlation matrix is derived, which includes the meta-analytical mean effect sizes for all variable combinations; Step 2 then uses this matrix to fit the path model. While MASEM was based primarily on traditional univariate meta-analysis to derive the pooled correlation matrix in its early years (Viswesvaran and Ones 1995 ), more advanced methods, such as the GLS approach (Becker 1992 , 1995 ) or the TSSEM approach (Cheung and Chan 2005 ), have been subsequently developed. Cheung ( 2015a ) and Jak ( 2015 ) provide an overview of these approaches in their books with exemplary code. For datasets with more complex data structures, Wilson et al. ( 2016 ) also developed a multilevel approach that is related to the TSSEM approach in the second step. Bergh et al. ( 2016 ) discuss nine decision points and develop best practices for MASEM studies.

2.4.4 Qualitative meta-analysis

While the approaches explained above focus on quantitative outcomes of empirical studies, qualitative meta-analysis aims to synthesize qualitative findings from case studies (Hoon 2013 ; Rauch et al. 2014 ). The distinctive feature of qualitative case studies is their potential to provide in-depth information about specific contextual factors or to shed light on reasons for certain phenomena that cannot usually be investigated by quantitative studies (Rauch 2020 ; Rauch et al. 2014 ). In a qualitative meta-analysis, the identified case studies are systematically coded in a meta-synthesis protocol, which is then used to identify influential variables or patterns and to derive a meta-causal network (Hoon 2013 ). Thus, the insights of contextualized and typically nongeneralizable single studies are aggregated to a larger, more generalizable picture (Habersang et al. 2019 ). Although still the exception, this method can thus provide important contributions for academics in terms of theory development (Combs et al., 2019 ; Hoon 2013 ) and for practitioners in terms of evidence-based management or entrepreneurship (Rauch et al. 2014 ). Levitt ( 2018 ) provides a guide and discusses conceptual issues for conducting qualitative meta-analysis in psychology, which is also useful for management researchers.

2.5 Step 5: choice of software

Software solutions to perform meta-analyses range from built-in functions or additional packages of statistical software to software purely focused on meta-analyses and from commercial to open-source solutions. However, in addition to personal preferences, the choice of the most suitable software depends on the complexity of the methods used and the dataset itself (Cheung and Vijayakumar 2016 ). Meta-analysts therefore must carefully check if their preferred software is capable of performing the intended analysis.

Among commercial software providers, Stata (from version 16 on) offers built-in functions to perform various meta-analytical analyses or to produce various plots (Palmer and Sterne 2016 ). For SPSS and SAS, there exist several macros for meta-analyses provided by scholars, such as David B. Wilson or Andy P. Field and Raphael Gillet (Field and Gillett 2010 ). Footnote 3 Footnote 4 For researchers using the open-source software R (R Core Team 2021 ), Polanin et al. ( 2017 ) provide an overview of 63 meta-analysis packages and their functionalities. For new users, they recommend the package metafor (Viechtbauer 2010 ), which includes most necessary functions and for which the author Wolfgang Viechtbauer provides tutorials on his project website. Footnote 5 Footnote 6 In addition to packages and macros for statistical software, templates for Microsoft Excel have also been developed to conduct simple meta-analyses, such as Meta-Essentials by Suurmond et al. ( 2017 ). Footnote 7 Finally, programs purely dedicated to meta-analysis also exist, such as Comprehensive Meta-Analysis (Borenstein et al. 2013 ) or RevMan by The Cochrane Collaboration ( 2020 ).

2.6 Step 6: coding of effect sizes

2.6.1 coding sheet.

The first step in the coding process is the design of the coding sheet. A universal template does not exist because the design of the coding sheet depends on the methods used, the respective software, and the complexity of the research design. For univariate meta-analysis or meta-regression, data are typically coded in wide format. In its simplest form, when investigating a correlational relationship between two variables using the univariate approach, the coding sheet would contain a column for the study name or identifier, the effect size coded from the primary study, and the study sample size. However, such simple relationships are unlikely in management research because the included studies are typically not identical but differ in several respects. With more complex data structures or moderator variables being investigated, additional columns are added to the coding sheet to reflect the data characteristics. These variables can be coded as dummy, factor, or (semi)continuous variables and later used to perform a subgroup analysis or meta regression. For MASEM, the required data input format can deviate depending on the method used (e.g., TSSEM requires a list of correlation matrices as data input). For qualitative meta-analysis, the coding scheme typically summarizes the key qualitative findings and important contextual and conceptual information (see Hoon ( 2013 ) for a coding scheme for qualitative meta-analysis). Figure  1 shows an exemplary coding scheme for a quantitative meta-analysis on the correlational relationship between top-management team diversity and profitability. In addition to effect and sample sizes, information about the study country, firm type, and variable operationalizations are coded. The list could be extended by further study and sample characteristics.

Exemplary coding sheet for a meta-analysis on the relationship (correlation) between top-management team diversity and profitability

2.6.2 Inclusion of moderator or control variables

It is generally important to consider the intended research model and relevant nontarget variables before coding a meta-analytic dataset. For example, study characteristics can be important moderators or function as control variables in a meta-regression model. Similarly, control variables may be relevant in a MASEM approach to reduce confounding bias. Coding additional variables or constructs subsequently can be arduous if the sample of primary studies is large. However, the decision to include respective moderator or control variables, as in any empirical analysis, should always be based on strong (theoretical) rationales about how these variables can impact the investigated effect (Bernerth and Aguinis 2016 ; Bernerth et al. 2018 ; Thompson and Higgins 2002 ). While substantive moderators refer to theoretical constructs that act as buffers or enhancers of a supposed causal process, methodological moderators are features of the respective research designs that denote the methodological context of the observations and are important to control for systematic statistical particularities (Rudolph et al. 2020 ). Havranek et al. ( 2020 ) provide a list of recommended variables to code as potential moderators. While researchers may have clear expectations about the effects for some of these moderators, the concerns for other moderators may be tentative, and moderator analysis may be approached in a rather exploratory fashion. Thus, we argue that researchers should make full use of the meta-analytical design to obtain insights about potential context dependence that a primary study cannot achieve.

2.6.3 Treatment of multiple effect sizes in a study

A long-debated issue in conducting meta-analyses is whether to use only one or all available effect sizes for the same construct within a single primary study. For meta-analyses in management research, this question is fundamental because many empirical studies, particularly those relying on company databases, use multiple variables for the same construct to perform sensitivity analyses, resulting in multiple relevant effect sizes. In this case, researchers can either (randomly) select a single value, calculate a study average, or use the complete set of effect sizes (Bijmolt and Pieters 2001 ; López-López et al. 2018 ). Multiple effect sizes from the same study enrich the meta-analytic dataset and allow us to investigate the heterogeneity of the relationship of interest, such as different variable operationalizations (López-López et al. 2018 ; Moeyaert et al. 2017 ). However, including more than one effect size from the same study violates the independency assumption of observations (Cheung 2019 ; López-López et al. 2018 ), which can lead to biased results and erroneous conclusions (Gooty et al. 2021 ). We follow the recommendation of current best practice guides to take advantage of using all available effect size observations but to carefully consider interdependencies using appropriate methods such as multilevel models, panel regression models, or robust variance estimation (Cheung 2019 ; Geyer-Klingeberg et al. 2020 ; Gooty et al. 2021 ; López-López et al. 2018 ; Moeyaert et al. 2017 ).

2.7 Step 7: analysis

2.7.1 outlier analysis and tests for publication bias.

Before conducting the primary analysis, some preliminary sensitivity analyses might be necessary, which should ensure the robustness of the meta-analytical findings (Rudolph et al. 2020 ). First, influential outlier observations could potentially bias the observed results, particularly if the number of total effect sizes is small. Several statistical methods can be used to identify outliers in meta-analytical datasets (Aguinis et al. 2013 ; Viechtbauer and Cheung 2010 ). However, there is a debate about whether to keep or omit these observations. Anyhow, relevant studies should be closely inspected to infer an explanation about their deviating results. As in any other primary study, outliers can be a valid representation, albeit representing a different population, measure, construct, design or procedure. Thus, inferences about outliers can provide the basis to infer potential moderators (Aguinis et al. 2013 ; Steel et al. 2021 ). On the other hand, outliers can indicate invalid research, for instance, when unrealistically strong correlations are due to construct overlap (i.e., lack of a clear demarcation between independent and dependent variables), invalid measures, or simply typing errors when coding effect sizes. An advisable step is therefore to compare the results both with and without outliers and base the decision on whether to exclude outlier observations with careful consideration (Geyskens et al. 2009 ; Grewal et al. 2018 ; Kepes et al. 2013 ). However, instead of simply focusing on the size of the outlier, its leverage should be considered. Thus, Viechtbauer and Cheung ( 2010 ) propose considering a combination of standardized deviation and a study’s leverage.

Second, as mentioned in the context of a literature search, potential publication bias may be an issue. Publication bias can be examined in multiple ways (Rothstein et al. 2005 ). First, the funnel plot is a simple graphical tool that can provide an overview of the effect size distribution and help to detect publication bias (Stanley and Doucouliagos 2010 ). A funnel plot can also support in identifying potential outliers. As mentioned above, a graphical display of deviation (e.g., studentized residuals) and leverage (Cook’s distance) can help detect the presence of outliers and evaluate their influence (Viechtbauer and Cheung 2010 ). Moreover, several statistical procedures can be used to test for publication bias (Harrison et al. 2017 ; Kepes et al. 2012 ), including subgroup comparisons between published and unpublished studies, Begg and Mazumdar’s ( 1994 ) rank correlation test, cumulative meta-analysis (Borenstein et al. 2009 ), the trim and fill method (Duval and Tweedie 2000a , b ), Egger et al.’s ( 1997 ) regression test, failsafe N (Rosenthal 1979 ), or selection models (Hedges and Vevea 2005 ; Vevea and Woods 2005 ). In examining potential publication bias, Kepes et al. ( 2012 ) and Harrison et al. ( 2017 ) both recommend not relying only on a single test but rather using multiple conceptionally different test procedures (i.e., the so-called “triangulation approach”).

2.7.2 Model choice

After controlling and correcting for the potential presence of impactful outliers or publication bias, the next step in meta-analysis is the primary analysis, where meta-analysts must decide between two different types of models that are based on different assumptions: fixed-effects and random-effects (Borenstein et al. 2010 ). Fixed-effects models assume that all observations share a common mean effect size, which means that differences are only due to sampling error, while random-effects models assume heterogeneity and allow for a variation of the true effect sizes across studies (Borenstein et al. 2010 ; Cheung and Vijayakumar 2016 ; Hunter and Schmidt 2004 ). Both models are explained in detail in standard textbooks (e.g., Borenstein et al. 2009 ; Hunter and Schmidt 2004 ; Lipsey and Wilson 2001 ).

In general, the presence of heterogeneity is likely in management meta-analyses because most studies do not have identical empirical settings, which can yield different effect size strengths or directions for the same investigated phenomenon. For example, the identified studies have been conducted in different countries with different institutional settings, or the type of study participants varies (e.g., students vs. employees, blue-collar vs. white-collar workers, or manufacturing vs. service firms). Thus, the vast majority of meta-analyses in management research and related fields use random-effects models (Aguinis et al. 2011a ). In a meta-regression, the random-effects model turns into a so-called mixed-effects model because moderator variables are added as fixed effects to explain the impact of observed study characteristics on effect size variations (Raudenbush 2009 ).

2.8 Step 8: reporting results

2.8.1 reporting in the article.

The final step in performing a meta-analysis is reporting its results. Most importantly, all steps and methodological decisions should be comprehensible to the reader. DeSimone et al. ( 2020 ) provide an extensive checklist for journal reviewers of meta-analytical studies. This checklist can also be used by authors when performing their analyses and reporting their results to ensure that all important aspects have been addressed. Alternative checklists are provided, for example, by Appelbaum et al. ( 2018 ) or Page et al. ( 2021 ). Similarly, Levitt et al. ( 2018 ) provide a detailed guide for qualitative meta-analysis reporting standards.

For quantitative meta-analyses, tables reporting results should include all important information and test statistics, including mean effect sizes; standard errors and confidence intervals; the number of observations and study samples included; and heterogeneity measures. If the meta-analytic sample is rather small, a forest plot provides a good overview of the different findings and their accuracy. However, this figure will be less feasible for meta-analyses with several hundred effect sizes included. Also, results displayed in the tables and figures must be explained verbally in the results and discussion sections. Most importantly, authors must answer the primary research question, i.e., whether there is a positive, negative, or no relationship between the variables of interest, or whether the examined intervention has a certain effect. These results should be interpreted with regard to their magnitude (or significance), both economically and statistically. However, when discussing meta-analytical results, authors must describe the complexity of the results, including the identified heterogeneity and important moderators, future research directions, and theoretical relevance (DeSimone et al. 2019 ). In particular, the discussion of identified heterogeneity and underlying moderator effects is critical; not including this information can lead to false conclusions among readers, who interpret the reported mean effect size as universal for all included primary studies and ignore the variability of findings when citing the meta-analytic results in their research (Aytug et al. 2012 ; DeSimone et al. 2019 ).

2.8.2 Open-science practices

Another increasingly important topic is the public provision of meta-analytical datasets and statistical codes via open-source repositories. Open-science practices allow for results validation and for the use of coded data in subsequent meta-analyses ( Polanin et al. 2020 ), contributing to the development of cumulative science. Steel et al. ( 2021 ) refer to open science meta-analyses as a step towards “living systematic reviews” (Elliott et al. 2017 ) with continuous updates in real time. MRQ supports this development and encourages authors to make their datasets publicly available. Moreau and Gamble ( 2020 ), for example, provide various templates and video tutorials to conduct open science meta-analyses. There exist several open science repositories, such as the Open Science Foundation (OSF; for a tutorial, see Soderberg 2018 ), to preregister and make documents publicly available. Furthermore, several initiatives in the social sciences have been established to develop dynamic meta-analyses, such as metaBUS (Bosco et al. 2015 , 2017 ), MetaLab (Bergmann et al. 2018 ), or PsychOpen CAMA (Burgard et al. 2021 ).

3 Conclusion

This editorial provides a comprehensive overview of the essential steps in conducting and reporting a meta-analysis with references to more in-depth methodological articles. It also serves as a guide for meta-analyses submitted to MRQ and other management journals. MRQ welcomes all types of meta-analyses from all subfields and disciplines of management research.

Gusenbauer and Haddaway ( 2020 ), however, point out that Google Scholar is not appropriate as a primary search engine due to a lack of reproducibility of search results.

One effect size calculator by David B. Wilson is accessible via: https://www.campbellcollaboration.org/escalc/html/EffectSizeCalculator-Home.php .

The macros of David B. Wilson can be downloaded from: http://mason.gmu.edu/~dwilsonb/ .

The macros of Field and Gillet ( 2010 ) can be downloaded from: https://www.discoveringstatistics.com/repository/fieldgillett/how_to_do_a_meta_analysis.html .

The tutorials can be found via: https://www.metafor-project.org/doku.php .

Metafor does currently not provide functions to conduct MASEM. For MASEM, users can, for instance, use the package metaSEM (Cheung 2015b ).

The workbooks can be downloaded from: https://www.erim.eur.nl/research-support/meta-essentials/ .

Aguinis H, Dalton DR, Bosco FA, Pierce CA, Dalton CM (2011a) Meta-analytic choices and judgment calls: Implications for theory building and testing, obtained effect sizes, and scholarly impact. J Manag 37(1):5–38

Google Scholar  

Aguinis H, Gottfredson RK, Joo H (2013) Best-practice recommendations for defining, identifying, and handling outliers. Organ Res Methods 16(2):270–301

Article   Google Scholar  

Aguinis H, Gottfredson RK, Wright TA (2011b) Best-practice recommendations for estimating interaction effects using meta-analysis. J Organ Behav 32(8):1033–1043

Aguinis H, Pierce CA, Bosco FA, Dalton DR, Dalton CM (2011c) Debunking myths and urban legends about meta-analysis. Organ Res Methods 14(2):306–331

Aloe AM (2015) Inaccuracy of regression results in replacing bivariate correlations. Res Synth Methods 6(1):21–27

Anderson RG, Kichkha A (2017) Replication, meta-analysis, and research synthesis in economics. Am Econ Rev 107(5):56–59

Appelbaum M, Cooper H, Kline RB, Mayo-Wilson E, Nezu AM, Rao SM (2018) Journal article reporting standards for quantitative research in psychology: the APA publications and communications BOARD task force report. Am Psychol 73(1):3–25

Aytug ZG, Rothstein HR, Zhou W, Kern MC (2012) Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analyses. Organ Res Methods 15(1):103–133

Begg CB, Mazumdar M (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics 50(4):1088–1101. https://doi.org/10.2307/2533446

Bergh DD, Aguinis H, Heavey C, Ketchen DJ, Boyd BK, Su P, Lau CLL, Joo H (2016) Using meta-analytic structural equation modeling to advance strategic management research: Guidelines and an empirical illustration via the strategic leadership-performance relationship. Strateg Manag J 37(3):477–497

Becker BJ (1992) Using results from replicated studies to estimate linear models. J Educ Stat 17(4):341–362

Becker BJ (1995) Corrections to “Using results from replicated studies to estimate linear models.” J Edu Behav Stat 20(1):100–102

Bergmann C, Tsuji S, Piccinini PE, Lewis ML, Braginsky M, Frank MC, Cristia A (2018) Promoting replicability in developmental research through meta-analyses: Insights from language acquisition research. Child Dev 89(6):1996–2009

Bernerth JB, Aguinis H (2016) A critical review and best-practice recommendations for control variable usage. Pers Psychol 69(1):229–283

Bernerth JB, Cole MS, Taylor EC, Walker HJ (2018) Control variables in leadership research: A qualitative and quantitative review. J Manag 44(1):131–160

Bijmolt TH, Pieters RG (2001) Meta-analysis in marketing when studies contain multiple measurements. Mark Lett 12(2):157–169

Block J, Kuckertz A (2018) Seven principles of effective replication studies: Strengthening the evidence base of management research. Manag Rev Quart 68:355–359

Borenstein M (2009) Effect sizes for continuous data. In: Cooper H, Hedges LV, Valentine JC (eds) The handbook of research synthesis and meta-analysis. Russell Sage Foundation, pp 221–235

Borenstein M, Hedges LV, Higgins JPT, Rothstein HR (2009) Introduction to meta-analysis. John Wiley, Chichester

Book   Google Scholar  

Borenstein M, Hedges LV, Higgins JPT, Rothstein HR (2010) A basic introduction to fixed-effect and random-effects models for meta-analysis. Res Synth Methods 1(2):97–111

Borenstein M, Hedges L, Higgins J, Rothstein H (2013) Comprehensive meta-analysis (version 3). Biostat, Englewood, NJ

Borenstein M, Higgins JP (2013) Meta-analysis and subgroups. Prev Sci 14(2):134–143

Bosco FA, Steel P, Oswald FL, Uggerslev K, Field JG (2015) Cloud-based meta-analysis to bridge science and practice: Welcome to metaBUS. Person Assess Decis 1(1):3–17

Bosco FA, Uggerslev KL, Steel P (2017) MetaBUS as a vehicle for facilitating meta-analysis. Hum Resour Manag Rev 27(1):237–254

Burgard T, Bošnjak M, Studtrucker R (2021) Community-augmented meta-analyses (CAMAs) in psychology: potentials and current systems. Zeitschrift Für Psychologie 229(1):15–23

Cheung MWL (2015a) Meta-analysis: A structural equation modeling approach. John Wiley & Sons, Chichester

Cheung MWL (2015b) metaSEM: An R package for meta-analysis using structural equation modeling. Front Psychol 5:1521

Cheung MWL (2019) A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychol Rev 29(4):387–396

Cheung MWL, Chan W (2005) Meta-analytic structural equation modeling: a two-stage approach. Psychol Methods 10(1):40–64

Cheung MWL, Vijayakumar R (2016) A guide to conducting a meta-analysis. Neuropsychol Rev 26(2):121–128

Combs JG, Crook TR, Rauch A (2019) Meta-analytic research in management: contemporary approaches unresolved controversies and rising standards. J Manag Stud 56(1):1–18. https://doi.org/10.1111/joms.12427

DeSimone JA, Köhler T, Schoen JL (2019) If it were only that easy: the use of meta-analytic research by organizational scholars. Organ Res Methods 22(4):867–891. https://doi.org/10.1177/1094428118756743

DeSimone JA, Brannick MT, O’Boyle EH, Ryu JW (2020) Recommendations for reviewing meta-analyses in organizational research. Organ Res Methods 56:455–463

Duval S, Tweedie R (2000a) Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56(2):455–463

Duval S, Tweedie R (2000b) A nonparametric “trim and fill” method of accounting for publication bias in meta-analysis. J Am Stat Assoc 95(449):89–98

Egger M, Smith GD, Schneider M, Minder C (1997) Bias in meta-analysis detected by a simple, graphical test. BMJ 315(7109):629–634

Eisend M (2017) Meta-Analysis in advertising research. J Advert 46(1):21–35

Elliott JH, Synnot A, Turner T, Simmons M, Akl EA, McDonald S, Salanti G, Meerpohl J, MacLehose H, Hilton J, Tovey D, Shemilt I, Thomas J (2017) Living systematic review: 1. Introduction—the why, what, when, and how. J Clin Epidemiol 91:2330. https://doi.org/10.1016/j.jclinepi.2017.08.010

Field AP, Gillett R (2010) How to do a meta-analysis. Br J Math Stat Psychol 63(3):665–694

Fisch C, Block J (2018) Six tips for your (systematic) literature review in business and management research. Manag Rev Quart 68:103–106

Fortunato S, Bergstrom CT, Börner K, Evans JA, Helbing D, Milojević S, Petersen AM, Radicchi F, Sinatra R, Uzzi B, Vespignani A (2018) Science of science. Science 359(6379). https://doi.org/10.1126/science.aao0185

Geyer-Klingeberg J, Hang M, Rathgeber A (2020) Meta-analysis in finance research: Opportunities, challenges, and contemporary applications. Int Rev Finan Anal 71:101524

Geyskens I, Krishnan R, Steenkamp JBE, Cunha PV (2009) A review and evaluation of meta-analysis practices in management research. J Manag 35(2):393–419

Glass GV (2015) Meta-analysis at middle age: a personal history. Res Synth Methods 6(3):221–231

Gonzalez-Mulé E, Aguinis H (2018) Advancing theory by assessing boundary conditions with metaregression: a critical review and best-practice recommendations. J Manag 44(6):2246–2273

Gooty J, Banks GC, Loignon AC, Tonidandel S, Williams CE (2021) Meta-analyses as a multi-level model. Organ Res Methods 24(2):389–411. https://doi.org/10.1177/1094428119857471

Grewal D, Puccinelli N, Monroe KB (2018) Meta-analysis: integrating accumulated knowledge. J Acad Mark Sci 46(1):9–30

Gurevitch J, Koricheva J, Nakagawa S, Stewart G (2018) Meta-analysis and the science of research synthesis. Nature 555(7695):175–182

Gusenbauer M, Haddaway NR (2020) Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synth Methods 11(2):181–217

Habersang S, Küberling-Jost J, Reihlen M, Seckler C (2019) A process perspective on organizational failure: a qualitative meta-analysis. J Manage Stud 56(1):19–56

Harari MB, Parola HR, Hartwell CJ, Riegelman A (2020) Literature searches in systematic reviews and meta-analyses: A review, evaluation, and recommendations. J Vocat Behav 118:103377

Harrison JS, Banks GC, Pollack JM, O’Boyle EH, Short J (2017) Publication bias in strategic management research. J Manag 43(2):400–425

Havránek T, Stanley TD, Doucouliagos H, Bom P, Geyer-Klingeberg J, Iwasaki I, Reed WR, Rost K, Van Aert RCM (2020) Reporting guidelines for meta-analysis in economics. J Econ Surveys 34(3):469–475

Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, Orlando

Hedges LV, Vevea JL (2005) Selection methods approaches. In: Rothstein HR, Sutton A, Borenstein M (eds) Publication bias in meta-analysis: prevention, assessment, and adjustments. Wiley, Chichester, pp 145–174

Hoon C (2013) Meta-synthesis of qualitative case studies: an approach to theory building. Organ Res Methods 16(4):522–556

Hunter JE, Schmidt FL (1990) Methods of meta-analysis: correcting error and bias in research findings. Sage, Newbury Park

Hunter JE, Schmidt FL (2004) Methods of meta-analysis: correcting error and bias in research findings, 2nd edn. Sage, Thousand Oaks

Hunter JE, Schmidt FL, Jackson GB (1982) Meta-analysis: cumulating research findings across studies. Sage Publications, Beverly Hills

Jak S (2015) Meta-analytic structural equation modelling. Springer, New York, NY

Kepes S, Banks GC, McDaniel M, Whetzel DL (2012) Publication bias in the organizational sciences. Organ Res Methods 15(4):624–662

Kepes S, McDaniel MA, Brannick MT, Banks GC (2013) Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-Analytic Reporting Standards). J Bus Psychol 28(2):123–143

Kraus S, Breier M, Dasí-Rodríguez S (2020) The art of crafting a systematic literature review in entrepreneurship research. Int Entrepreneur Manag J 16(3):1023–1042

Levitt HM (2018) How to conduct a qualitative meta-analysis: tailoring methods to enhance methodological integrity. Psychother Res 28(3):367–378

Levitt HM, Bamberg M, Creswell JW, Frost DM, Josselson R, Suárez-Orozco C (2018) Journal article reporting standards for qualitative primary, qualitative meta-analytic, and mixed methods research in psychology: the APA publications and communications board task force report. Am Psychol 73(1):26

Lipsey MW, Wilson DB (2001) Practical meta-analysis. Sage Publications, Inc.

López-López JA, Page MJ, Lipsey MW, Higgins JP (2018) Dealing with effect size multiplicity in systematic reviews and meta-analyses. Res Synth Methods 9(3):336–351

Martín-Martín A, Thelwall M, Orduna-Malea E, López-Cózar ED (2021) Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations. Scientometrics 126(1):871–906

Merton RK (1968) The Matthew effect in science: the reward and communication systems of science are considered. Science 159(3810):56–63

Moeyaert M, Ugille M, Natasha Beretvas S, Ferron J, Bunuan R, Van den Noortgate W (2017) Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. Int J Soc Res Methodol 20(6):559–572

Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 6(7):e1000097

Mongeon P, Paul-Hus A (2016) The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics 106(1):213–228

Moreau D, Gamble B (2020) Conducting a meta-analysis in the age of open science: Tools, tips, and practical recommendations. Psychol Methods. https://doi.org/10.1037/met0000351

O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4(1):1–22

Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A (2016) Rayyan—a web and mobile app for systematic reviews. Syst Rev 5(1):1–10

Owen E, Li Q (2021) The conditional nature of publication bias: a meta-regression analysis. Polit Sci Res Methods 9(4):867–877

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E,McDonald S,McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372. https://doi.org/10.1136/bmj.n71

Palmer TM, Sterne JAC (eds) (2016) Meta-analysis in stata: an updated collection from the stata journal, 2nd edn. Stata Press, College Station, TX

Pigott TD, Polanin JR (2020) Methodological guidance paper: High-quality meta-analysis in a systematic review. Rev Educ Res 90(1):24–46

Polanin JR, Tanner-Smith EE, Hennessy EA (2016) Estimating the difference between published and unpublished effect sizes: a meta-review. Rev Educ Res 86(1):207–236

Polanin JR, Hennessy EA, Tanner-Smith EE (2017) A review of meta-analysis packages in R. J Edu Behav Stat 42(2):206–242

Polanin JR, Hennessy EA, Tsuji S (2020) Transparency and reproducibility of meta-analyses in psychology: a meta-review. Perspect Psychol Sci 15(4):1026–1041. https://doi.org/10.1177/17456916209064

R Core Team (2021). R: A language and environment for statistical computing . R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/ .

Rauch A (2020) Opportunities and threats in reviewing entrepreneurship theory and practice. Entrep Theory Pract 44(5):847–860

Rauch A, van Doorn R, Hulsink W (2014) A qualitative approach to evidence–based entrepreneurship: theoretical considerations and an example involving business clusters. Entrep Theory Pract 38(2):333–368

Raudenbush SW (2009) Analyzing effect sizes: Random-effects models. In: Cooper H, Hedges LV, Valentine JC (eds) The handbook of research synthesis and meta-analysis, 2nd edn. Russell Sage Foundation, New York, NY, pp 295–315

Rosenthal R (1979) The file drawer problem and tolerance for null results. Psychol Bull 86(3):638

Rothstein HR, Sutton AJ, Borenstein M (2005) Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Chichester

Roth PL, Le H, Oh I-S, Van Iddekinge CH, Bobko P (2018) Using beta coefficients to impute missing correlations in meta-analysis research: Reasons for caution. J Appl Psychol 103(6):644–658. https://doi.org/10.1037/apl0000293

Rudolph CW, Chang CK, Rauvola RS, Zacher H (2020) Meta-analysis in vocational behavior: a systematic review and recommendations for best practices. J Vocat Behav 118:103397

Schmidt FL (2017) Statistical and measurement pitfalls in the use of meta-regression in meta-analysis. Career Dev Int 22(5):469–476

Schmidt FL, Hunter JE (2015) Methods of meta-analysis: correcting error and bias in research findings. Sage, Thousand Oaks

Schwab A (2015) Why all researchers should report effect sizes and their confidence intervals: Paving the way for meta–analysis and evidence–based management practices. Entrepreneurship Theory Pract 39(4):719–725. https://doi.org/10.1111/etap.12158

Shaw JD, Ertug G (2017) The suitability of simulations and meta-analyses for submissions to Academy of Management Journal. Acad Manag J 60(6):2045–2049

Soderberg CK (2018) Using OSF to share data: A step-by-step guide. Adv Methods Pract Psychol Sci 1(1):115–120

Stanley TD, Doucouliagos H (2010) Picture this: a simple graph that reveals much ado about research. J Econ Surveys 24(1):170–191

Stanley TD, Doucouliagos H (2012) Meta-regression analysis in economics and business. Routledge, London

Stanley TD, Jarrell SB (1989) Meta-regression analysis: a quantitative method of literature surveys. J Econ Surveys 3:54–67

Steel P, Beugelsdijk S, Aguinis H (2021) The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews. J Int Bus Stud 52(1):23–44

Suurmond R, van Rhee H, Hak T (2017) Introduction, comparison, and validation of Meta-Essentials: a free and simple tool for meta-analysis. Res Synth Methods 8(4):537–553

The Cochrane Collaboration (2020). Review Manager (RevMan) [Computer program] (Version 5.4).

Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, Glasziou P, Shemilt I, Synnot A, Turner T, Elliot J (2017) Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol 91:31–37

Thompson SG, Higgins JP (2002) How should meta-regression analyses be undertaken and interpreted? Stat Med 21(11):1559–1573

Tipton E, Pustejovsky JE, Ahmadi H (2019) A history of meta-regression: technical, conceptual, and practical developments between 1974 and 2018. Res Synth Methods 10(2):161–179

Vevea JL, Woods CM (2005) Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychol Methods 10(4):428–443

Viechtbauer W (2010) Conducting meta-analyses in R with the metafor package. J Stat Softw 36(3):1–48

Viechtbauer W, Cheung MWL (2010) Outlier and influence diagnostics for meta-analysis. Res Synth Methods 1(2):112–125

Viswesvaran C, Ones DS (1995) Theory testing: combining psychometric meta-analysis and structural equations modeling. Pers Psychol 48(4):865–885

Wilson SJ, Polanin JR, Lipsey MW (2016) Fitting meta-analytic structural equation models with complex datasets. Res Synth Methods 7(2):121–139. https://doi.org/10.1002/jrsm.1199

Wood JA (2008) Methodology for dealing with duplicate study effects in a meta-analysis. Organ Res Methods 11(1):79–95

Download references

Open Access funding enabled and organized by Projekt DEAL. No funding was received to assist with the preparation of this manuscript.

Author information

Authors and affiliations.

University of Luxembourg, Luxembourg, Luxembourg

Christopher Hansen

Leibniz Institute for Psychology (ZPID), Trier, Germany

Holger Steinmetz

Trier University, Trier, Germany

Erasmus University Rotterdam, Rotterdam, The Netherlands

Wittener Institut Für Familienunternehmen, Universität Witten/Herdecke, Witten, Germany

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jörn Block .

Ethics declarations

Conflict of interest.

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table 1 .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Hansen, C., Steinmetz, H. & Block, J. How to conduct a meta-analysis in eight steps: a practical guide. Manag Rev Q 72 , 1–19 (2022). https://doi.org/10.1007/s11301-021-00247-4

Download citation

Published : 30 November 2021

Issue Date : February 2022

DOI : https://doi.org/10.1007/s11301-021-00247-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• AIMS Public Health
• v.3(1); 2016

What Synthesis Methodology Should I Use? A Review and Analysis of Approaches to Research Synthesis

Kara schick-makaroff.

1 Faculty of Nursing, University of Alberta, Edmonton, AB, Canada

Marjorie MacDonald

2 School of Nursing, University of Victoria, Victoria, BC, Canada

Marilyn Plummer

3 College of Nursing, Camosun College, Victoria, BC, Canada

Judy Burgess

4 Student Services, University Health Services, Victoria, BC, Canada

Wendy Neander

Associated data, additional file 1.

When we began this process, we were doctoral students and a faculty member in a research methods course. As students, we were facing a review of the literature for our dissertations. We encountered several different ways of conducting a review but were unable to locate any resources that synthesized all of the various synthesis methodologies. Our purpose is to present a comprehensive overview and assessment of the main approaches to research synthesis. We use ‘research synthesis’ as a broad overarching term to describe various approaches to combining, integrating, and synthesizing research findings.

We conducted an integrative review of the literature to explore the historical, contextual, and evolving nature of research synthesis. We searched five databases, reviewed websites of key organizations, hand-searched several journals, and examined relevant texts from the reference lists of the documents we had already obtained.

We identified four broad categories of research synthesis methodology including conventional, quantitative, qualitative, and emerging syntheses. Each of the broad categories was compared to the others on the following: key characteristics, purpose, method, product, context, underlying assumptions, unit of analysis, strengths and limitations, and when to use each approach.

Conclusions

The current state of research synthesis reflects significant advancements in emerging synthesis studies that integrate diverse data types and sources. New approaches to research synthesis provide a much broader range of review alternatives available to health and social science students and researchers.

1. Introduction

Since the turn of the century, public health emergencies have been identified worldwide, particularly related to infectious diseases. For example, the Severe Acute Respiratory Syndrome (SARS) epidemic in Canada in 2002-2003, the recent Ebola epidemic in Africa, and the ongoing HIV/AIDs pandemic are global health concerns. There have also been dramatic increases in the prevalence of chronic diseases around the world [1] – [3] . These epidemiological challenges have raised concerns about the ability of health systems worldwide to address these crises. As a result, public health systems reform has been initiated in a number of countries. In Canada, as in other countries, the role of evidence to support public health reform and improve population health has been given high priority. Yet, there continues to be a significant gap between the production of evidence through research and its application in practice [4] – [5] . One strategy to address this gap has been the development of new research synthesis methodologies to deal with the time-sensitive and wide ranging evidence needs of policy makers and practitioners in all areas of health care, including public health.

As doctoral nursing students facing a review of the literature for our dissertations, and as a faculty member teaching a research methods course, we encountered several ways of conducting a research synthesis but found no comprehensive resources that discussed, compared, and contrasted various synthesis methodologies on their purposes, processes, strengths and limitations. To complicate matters, writers use terms interchangeably or use different terms to mean the same thing, and the literature is often contradictory about various approaches. Some texts [6] , [7] – [9] did provide a preliminary understanding about how research synthesis had been taken up in nursing, but these did not meet our requirements. Thus, in this article we address the need for a comprehensive overview of research synthesis methodologies to guide public health, health care, and social science researchers and practitioners.

Research synthesis is relatively new in public health but has a long history in other fields dating back to the late 1800s. Research synthesis, a research process in its own right [10] , has become more prominent in the wake of the evidence-based movement of the 1990s. Research syntheses have found their advocates and detractors in all disciplines, with challenges to the processes of systematic review and meta-analysis, in particular, being raised by critics of evidence-based healthcare [11] – [13] .

Our purpose was to conduct an integrative review of the literature to explore the historical, contextual, and evolving nature of research synthesis [14] – [15] . We synthesize and critique the main approaches to research synthesis that are relevant for public health, health care, and social scientists. Research synthesis is the overarching term we use to describe approaches to combining, aggregating, integrating, and synthesizing primary research findings. Each synthesis methodology draws on different types of findings depending on the purpose and product of the chosen synthesis (see Additional File 1 ).

3. Method of Review

Based on our current knowledge of the literature, we identified these approaches to include in our review: systematic review, meta-analysis, qualitative meta-synthesis, meta-narrative synthesis, scoping review, rapid review, realist synthesis, concept analysis, literature review, and integrative review. Our first step was to divide the synthesis types among the research team. Each member did a preliminary search to identify key texts. The team then met to develop search terms and a framework to guide the review.

Over the period of 2008 to 2012 we extensively searched the literature, updating our search at several time points, not restricting our search by date. The dates of texts reviewed range from 1967 to 2015. We used the terms above combined with the term “method* (e.g., “realist synthesis” and “method*) in the database Health Source: Academic Edition (includes Medline and CINAHL). This search yielded very few texts on some methodologies and many on others. We realized that many documents on research synthesis had not been picked up in the search. Therefore, we also searched Google Scholar, PubMed, ERIC, and Social Science Index, as well as the websites of key organizations such as the Joanna Briggs Institute, the University of York Centre for Evidence-Based Nursing, and the Cochrane Collaboration database. We hand searched several nursing, social science, public health and health policy journals. Finally, we traced relevant documents from the references in obtained texts.

We included works that met the following inclusion criteria: (1) published in English; (2) discussed the history of research synthesis; (3) explicitly described the approach and specific methods; or (4) identified issues, challenges, strengths and limitations of the particular methodology. We excluded research reports that resulted from the use of particular synthesis methodologies unless they also included criteria 2, 3, or 4 above.

Based on our search, we identified additional types of research synthesis (e.g., meta-interpretation, best evidence synthesis, critical interpretive synthesis, meta-summary, grounded formal theory). Still, we missed some important developments in meta-analysis, for example, identified by the journal's reviewers that have now been discussed briefly in the paper. The final set of 197 texts included in our review comprised theoretical, empirical, and conceptual papers, books, editorials and commentaries, and policy documents.

In our preliminary review of key texts, the team inductively developed a framework of the important elements of each method for comparison. In the next phase, each text was read carefully, and data for these elements were extracted into a table for comparison on the points of: key characteristics, purpose, methods, and product; see Additional File 1 ). Once the data were grouped and extracted, we synthesized across categories based on the following additional points of comparison: complexity of the process, degree of systematization, consideration of context, underlying assumptions, unit of analysis, and when to use each approach. In our results, we discuss our comparison of the various synthesis approaches on the elements above. Drawing only on documents for the review, ethics approval was not required.

We identified four broad categories of research synthesis methodology: Conventional, quantitative, qualitative, and emerging syntheses. From our dataset of 197 texts, we had 14 texts on conventional synthesis, 64 on quantitative synthesis, 78 on qualitative synthesis, and 41 on emerging syntheses. Table 1 provides an overview of the four types of research synthesis, definitions, types of data used, products, and examples of the methodology.

Although we group these types of synthesis into four broad categories on the basis of similarities, each type within a category has unique characteristics, which may differ from the overall group similarities. Each could be explored in greater depth to tease out their unique characteristics, but detailed comparison is beyond the scope of this article.

Additional File 1 presents one or more selected types of synthesis that represent the broad category but is not an exhaustive presentation of all types within each category. It provides more depth for specific examples from each category of synthesis on the characteristics, purpose, methods, and products than is found in Table 1 .

4.1. Key Characteristics

4.1.1. what is it.

Here we draw on two types of categorization. First, we utilize Dixon Woods et al.'s [49] classification of research syntheses as being either integrative or interpretive . (Please note that integrative syntheses are not the same as an integrative review as defined in Additional File 1 .) Second, we use Popay's [80] enhancement and epistemological models .

The defining characteristics of integrative syntheses are that they involve summarizing the data achieved by pooling data [49] . Integrative syntheses include systematic reviews, meta-analyses, as well as scoping and rapid reviews because each of these focus on summarizing data. They also define concepts from the outset (although this may not always be true in scoping or rapid reviews) and deal with a well-specified phenomenon of interest.

Interpretive syntheses are primarily concerned with the development of concepts and theories that integrate concepts [49] . The analysis in interpretive synthesis is conceptual both in process and outcome, and “the product is not aggregations of data, but theory” [49] , [p.12]. Interpretive syntheses involve induction and interpretation, and are primarily conceptual in process and outcome. Examples include integrative reviews, some systematic reviews, all of the qualitative syntheses, meta-narrative, realist and critical interpretive syntheses. Of note, both quantitative and qualitative studies can be either integrative or interpretive

The second categorization, enhancement versus epistemological , applies to those approaches that use multiple data types and sources [80] . Popay's [80] classification reflects the ways that qualitative data are valued in relation to quantitative data.

In the enhancement model , qualitative data adds something to quantitative analysis. The enhancement model is reflected in systematic reviews and meta-analyses that use some qualitative data to enhance interpretation and explanation. It may also be reflected in some rapid reviews that draw on quantitative data but use some qualitative data.

The epistemological model assumes that quantitative and qualitative data are equal and each has something unique to contribute. All of the other review approaches, except pure quantitative or qualitative syntheses, reflect the epistemological model because they value all data types equally but see them as contributing different understandings.

4.1.2. Data type

By and large, the quantitative approaches (quantitative systematic review and meta-analysis) have typically used purely quantitative data (i.e., expressed in numeric form). More recently, both Cochrane [81] and Campbell [82] collaborations are grappling with the need to, and the process of, integrating qualitative research into a systematic review. The qualitative approaches use qualitative data (i.e., expressed in words). All of the emerging synthesis types, as well as the conventional integrative review, incorporate qualitative and quantitative study designs and data.

4.1.3. Research question

Four types of research questions direct inquiry across the different types of syntheses. The first is a well-developed research question that gives direction to the synthesis (e.g., meta-analysis, systematic review, meta-study, concept analysis, rapid review, realist synthesis). The second begins as a broad general question that evolves and becomes more refined over the course of the synthesis (e.g., meta-ethnography, scoping review, meta-narrative, critical interpretive synthesis). In the third type, the synthesis begins with a phenomenon of interest and the question emerges in the analytic process (e.g., grounded formal theory). Lastly, there is no clear question, but rather a general review purpose (e.g., integrative review). Thus, the requirement for a well-defined question cuts across at least three of the synthesis types (e.g., quantitative, qualitative, and emerging).

4.1.4. Quality appraisal

This is a contested issue within and between the four synthesis categories. There are strong proponents of quality appraisal in the quantitative traditions of systematic review and meta-analysis based on the need for strong studies that will not jeopardize validity of the overall findings. Nonetheless, there is no consensus on pre-defined criteria; many scales exist that vary dramatically in composition. This has methodological implications for the credibility of findings [83] .

Specific methodologies from the conventional, qualitative, and emerging categories support quality appraisal but do so with caveats. In conventional integrative reviews appraisal is recommended, but depends on the sampling frame used in the study [18] . In meta-study, appraisal criteria are explicit but quality criteria are used in different ways depending on the specific requirements of the inquiry [54] . Among the emerging syntheses, meta-narrative review developers support appraisal of a study based on criteria from the research tradition of the primary study [67] , [84] – [85] . Realist synthesis similarly supports the use of high quality evidence, but appraisal checklists are viewed with scepticism and evidence is judged based on relevance to the research question and whether a credible inference may be drawn [69] . Like realist, critical interpretive syntheses do not judge quality using standardized appraisal instruments. They will exclude fatally flawed studies, but there is no consensus on what ‘fatally flawed’ means [49] , [71] . Appraisal is based on relevance to the inquiry, not rigor of the study.

There is no agreement on quality appraisal among qualitative meta-ethnographers with some supporting and others refuting the need for appraisal. [60] , [62] . Opponents of quality appraisal are found among authors of qualitative (grounded formal theory and concept analysis) and emerging syntheses (scoping and rapid reviews) because quality is not deemed relevant to the intention of the synthesis; the studies being reviewed are not effectiveness studies where quality is extremely important. These qualitative synthesis are often reviews of theoretical developments where the concept itself is what is important, or reviews that provide quotations from the raw data so readers can make their own judgements about the relevance and utility of the data. For example, in formal grounded theory, the purpose of theory generation and authenticity of data used to generate the theory is not as important as the conceptual category. Inaccuracies may be corrected in other ways, such as using the constant comparative method, which facilitates development of theoretical concepts that are repeatedly found in the data [86] – [87] . For pragmatic reasons, evidence is not assessed in rapid and scoping reviews, in part to produce a timely product. The issue of quality appraisal is unresolved across the terrain of research synthesis and we consider this further in our discussion.

4.2. Purpose

All research syntheses share a common purpose -- to summarize, synthesize, or integrate research findings from diverse studies. This helps readers stay abreast of the burgeoning literature in a field. Our discussion here is at the level of the four categories of synthesis. Beginning with conventional literature syntheses, the overall purpose is to attend to mature topics for the purpose of re-conceptualization or to new topics requiring preliminary conceptualization [14] . Such syntheses may be helpful to consider contradictory evidence, map shifting trends in the study of a phenomenon, and describe the emergence of research in diverse fields [14] . The purpose here is to set the stage for a study by identifying what has been done, gaps in the literature, important research questions, or to develop a conceptual framework to guide data collection and analysis.

The purpose of quantitative systematic reviews is to combine, aggregate, or integrate empirical research to be able to generalize from a group of studies and determine the limits of generalization [27] . The focus of quantitative systematic reviews has been primarily on aggregating the results of studies evaluating the effectiveness of interventions using experimental, quasi-experimental, and more recently, observational designs. Systematic reviews can be done with or without quantitative meta-analysis but a meta-analysis always takes place within the context of a systematic review. Researchers must consider the review's purpose and the nature of their data in undertaking a quantitative synthesis; this will assist in determining the approach.

The purpose of qualitative syntheses is broadly to synthesize complex health experiences, practices, or concepts arising in healthcare environments. There may be various purposes depending on the qualitative methodology. For example, in hermeneutic studies the aim may be holistic explanation or understanding of a phenomenon [42] , which is deepened by integrating the findings from multiple studies. In grounded formal theory, the aim is to produce a conceptual framework or theory expected to be applicable beyond the original study. Although not able to generalize from qualitative research in the statistical sense [88] , qualitative researchers usually do want to say something about the applicability of their synthesis to other settings or phenomena. This notion of ‘theoretical generalization’ has been referred to as ‘transferability’ [89] – [90] and is an important criterion of rigour in qualitative research. It applies equally to the products of a qualitative synthesis in which the synthesis of multiple studies on the same phenomenon strengthens the ability to draw transferable conclusions.

The overarching purpose of emerging syntheses is challenging the more traditional types of syntheses, in part by using data from both quantitative and qualitative studies with diverse designs for analysis. Beyond this, however, each emerging synthesis methodology has a unique purpose. In meta-narrative review, the purpose is to identify different research traditions in the area, synthesize a complex and diverse body of research. Critical interpretive synthesis shares this characteristic. Although a distinctive approach, critical interpretive synthesis utilizes a modification of the analytic strategies of meta-ethnography [61] (e.g., reciprocal translational analysis, refutational synthesis, and lines of argument synthesis) but goes beyond the use of these to bring a critical perspective to bear in challenging the normative or epistemological assumptions in the primary literature [72] – [73] . The unique purpose of a realist synthesis is to amalgamate complex empirical evidence and theoretical understandings within a diverse body of literature to uncover the operative mechanisms and contexts that affect the outcomes of social interventions. In a scoping review, the intention is to find key concepts, examine the range of research in an area, and identify gaps in the literature. The purpose of a rapid review is comparable to that of a scoping review, but done quickly to meet the time-sensitive information needs of policy makers.

4.3. Method

4.3.1. degree of systematization.

There are varying degrees of systematization across the categories of research synthesis. The most systematized are quantitative systematic reviews and meta-analyses. There are clear processes in each with judgments to be made at each step, although there are no agreed upon guidelines for this. The process is inherently subjective despite attempts to develop objective and systematic processes [91] – [92] . Mullen and Ramirez [27] suggest that there is often a false sense of rigour implied by the terms ‘systematic review’ and ‘meta-analysis’ because of their clearly defined procedures.

In comparison with some types of qualitative synthesis, concept analysis is quite procedural. Qualitative meta-synthesis also has defined procedures and is systematic, yet perhaps less so than concept analysis. Qualitative meta-synthesis starts in an unsystematic way but becomes more systematic as it unfolds. Procedures and frameworks exist for some of the emerging types of synthesis [e.g., [50] , [63] , [71] , [93] ] but are not linear, have considerable flexibility, and are often messy with emergent processes [85] . Conventional literature reviews tend not to be as systematic as the other three types. In fact, the lack of systematization in conventional literature synthesis was the reason for the development of more systematic quantitative [17] , [20] and qualitative [45] – [46] , [61] approaches. Some authors in the field [18] have clarified processes for integrative reviews making them more systematic and rigorous, but most conventional syntheses remain relatively unsystematic in comparison with other types.

4.3.2. Complexity of the process

Some synthesis processes are considerably more complex than others. Methodologies with clearly defined steps are arguably less complex than the more flexible and emergent ones. We know that any study encounters challenges and it is rare that a pre-determined research protocol can be followed exactly as intended. Not even the rigorous methods associated with Cochrane [81] systematic reviews and meta-analyses are always implemented exactly as intended. Even when dealing with numbers rather than words, interpretation is always part of the process. Our collective experience suggests that new methodologies (e.g., meta-narrative synthesis and realist synthesis) that integrate different data types and methods are more complex than conventional reviews or the rapid and scoping reviews.

4.4. Product

The products of research syntheses usually take three distinct formats (see Table 1 and Additional File 1 for further details). The first representation is in tables, charts, graphical displays, diagrams and maps as seen in integrative, scoping and rapid reviews, meta-analyses, and critical interpretive syntheses. The second type of synthesis product is the use of mathematical scores. Summary statements of effectiveness are mathematically displayed in meta-analyses (as an effect size), systematic reviews, and rapid reviews (statistical significance).

The third synthesis product may be a theory or theoretical framework. A mid-range theory can be produced from formal grounded theory, meta-study, meta-ethnography, and realist synthesis. Theoretical/conceptual frameworks or conceptual maps may be created in meta-narrative and critical interpretive syntheses, and integrative reviews. Concepts for use within theories are produced in concept analysis. While these three product types span the categories of research synthesis, narrative description and summary is used to present the products resulting from all methodologies.

4.5. Consideration of context

There are diverse ways that context is considered in the four broad categories of synthesis. Context may be considered to the extent that it features within primary studies for the purpose of the review. Context may also be understood as an integral aspect of both the phenomenon under study and the synthesis methodology (e.g., realist synthesis). Quantitative systematic reviews and meta-analyses have typically been conducted on studies using experimental and quasi-experimental designs and more recently observational studies, which control for contextual features to allow for understanding of the ‘true’ effect of the intervention [94] .

More recently, systematic reviews have included covariates or mediating variables (i.e., contextual factors) to help explain variability in the results across studies [27] . Context, however, is usually handled in the narrative discussion of findings rather than in the synthesis itself. This lack of attention to context has been one criticism leveled against systematic reviews and meta-analyses, which restrict the types of research designs that are considered [e.g., [95] ].

When conventional literature reviews incorporate studies that deal with context, there is a place for considering contextual influences on the intervention or phenomenon. Reviews of quantitative experimental studies tend to be devoid of contextual considerations since the original studies are similarly devoid, but context might figure prominently in a literature review that incorporates both quantitative and qualitative studies.

Qualitative syntheses have been conducted on the contextual features of a particular phenomenon [33] . Paterson et al. [54] advise researchers to attend to how context may have influenced the findings of particular primary studies. In qualitative analysis, contextual features may form categories by which the data can be compared and contrasted to facilitate interpretation. Because qualitative research is often conducted to understand a phenomenon as a whole, context may be a focus, although this varies with the qualitative methodology. At the same time, the findings in a qualitative synthesis are abstracted from the original reports and taken to a higher level of conceptualization, thus removing them from the original context.

Meta-narrative synthesis [67] , [84] , because it draws on diverse research traditions and methodologies, may incorporate context into the analysis and findings. There is not, however, an explicit step in the process that directs the analyst to consider context. Generally, the research question guiding the synthesis is an important factor in whether context will be a focus.

More recent iterations of concept analysis [47] , [96] – [97] explicitly consider context reflecting the assumption that a concept's meaning is determined by its context. Morse [47] points out, however, that Wilson's [98] approach to concept analysis, and those based on Wilson [e.g., [45] ], identify attributes that are devoid of context, while Rodgers' [96] , [99] evolutionary method considers context (e.g., antecedents, consequences, and relationships to other concepts) in concept development.

Realist synthesis [69] considers context as integral to the study. It draws on a critical realist logic of inquiry grounded in the work of Bhaskar [100] , who argues that empirical co-occurrence of events is insufficient for inferring causation. One must identify generative mechanisms whose properties are causal and, depending on the situation, may nor may not be activated [94] . Context interacts with program/intervention elements and thus cannot be differentiated from the phenomenon [69] . This approach synthesizes evidence on generative mechanisms and analyzes contextual features that activate them; the result feeds back into the context. The focus is on what works, for whom, under what conditions, why and how [68] .

4.6. Underlying Philosophical and Theoretical Assumptions

When we began our review, we ‘assumed’ that the assumptions underlying synthesis methodologies would be a distinguishing characteristic of synthesis types, and that we could compare the various types on their assumptions, explicit or implicit. We found, however, that many authors did not explicate the underlying assumptions of their methodologies, and it was difficult to infer them. Kirkevold [101] has argued that integrative reviews need to be carried out from an explicit philosophical or theoretical perspective. We argue this should be true for all types of synthesis.

Authors of some emerging synthesis approaches have been very explicit about their assumptions and philosophical underpinnings. An implicit assumption of most emerging synthesis methodologies is that quantitative systematic reviews and meta-analyses have limited utility in some fields [e.g., in public health – [13] , [102] ] and for some kinds of review questions like those about feasibility and appropriateness versus effectiveness [103] – [104] . They also assume that ontologically and epistemologically, both kinds of data can be combined. This is a significant debate in the literature because it is about the commensurability of overarching paradigms [105] but this is beyond the scope of this review.

Realist synthesis is philosophically grounded in critical realism or, as noted above, a realist logic of inquiry [93] , [99] , [106] – [107] . Key assumptions regarding the nature of interventions that inform critical realism have been described above in the section on context. See Pawson et al. [106] for more information on critical realism, the philosophical basis of realist synthesis.

Meta-narrative synthesis is explicitly rooted in a constructivist philosophy of science [108] in which knowledge is socially constructed rather than discovered, and what we take to be ‘truth’ is a matter of perspective. Reality has a pluralistic and plastic character, and there is no pre-existing ‘real world’ independent of human construction and language [109] . See Greenhalgh et al. [67] , [85] and Greenhalgh & Wong [97] for more discussion of the constructivist basis of meta-narrative synthesis.

In the case of purely quantitative or qualitative syntheses, it may be an easier matter to uncover unstated assumptions because they are likely to be shared with those of the primary studies in the genre. For example, grounded formal theory shares the philosophical and theoretical underpinnings of grounded theory, rooted in the theoretical perspective of symbolic interactionism [110] – [111] and the philosophy of pragmatism [87] , [112] – [114] .

As with meta-narrative synthesis, meta-study developers identify constructivism as their interpretive philosophical foundation [54] , [88] . Epistemologically, constructivism focuses on how people construct and re-construct knowledge about a specific phenomenon, and has three main assumptions: (1) reality is seen as multiple, at times even incompatible with the phenomenon under consideration; (2) just as primary researchers construct interpretations from participants' data, meta-study researchers also construct understandings about the primary researchers' original findings. Thus, meta-synthesis is a construction of a construction, or a meta-construction; and (3) all constructions are shaped by the historical, social and ideological context in which they originated [54] . The key message here is that reports of any synthesis would benefit from an explicit identification of the underlying philosophical perspectives to facilitate a better understanding of the results, how they were derived, and how they are being interpreted.

4.7. Unit of Analysis

The unit of analysis for each category of review is generally distinct. For the emerging synthesis approaches, the unit of analysis is specific to the intention. In meta-narrative synthesis it is the storyline in diverse research traditions; in rapid review or scoping review, it depends on the focus but could be a concept; and in realist synthesis, it is the theories rather than programs that are the units of analysis. The elements of theory that are important in the analysis are mechanisms of action, the context, and the outcome [107] .

For qualitative synthesis, the units of analysis are generally themes, concepts or theories, although in meta-study, the units of analysis can be research findings (“meta-data-analysis”), research methods (“meta-method”) or philosophical/theoretical perspectives (“meta-theory”) [54] . In quantitative synthesis, the units of analysis range from specific statistics for systematic reviews to effect size of the intervention for meta-analysis. More recently, some systematic reviews focus on theories [115] – [116] , therefore it depends on the research question. Similarly, within conventional literature synthesis the units of analysis also depend on the research purpose, focus and question as well as on the type of research methods incorporated into the review. What is important in all research syntheses, however, is that the unit of analysis needs to be made explicit. Unfortunately, this is not always the case.

4.8. Strengths and Limitations

In this section, we discuss the overarching strengths and limitations of synthesis methodologies as a whole and then highlight strengths and weaknesses across each of our four categories of synthesis.

4.8.1. Strengths of Research Syntheses in General

With the vast proliferation of research reports and the increased ease of retrieval, research synthesis has become more accessible providing a way of looking broadly at the current state of research. The availability of syntheses helps researchers, practitioners, and policy makers keep up with the burgeoning literature in their fields without which evidence-informed policy or practice would be difficult. Syntheses explain variation and difference in the data helping us identify the relevance for our own situations; they identify gaps in the literature leading to new research questions and study designs. They help us to know when to replicate a study and when to avoid excessively duplicating research. Syntheses can inform policy and practice in a way that well-designed single studies cannot; they provide building blocks for theory that helps us to understand and explain our phenomena of interest.

4.8.2. Limitations of Research Syntheses in General

The process of selecting, combining, integrating, and synthesizing across diverse study designs and data types can be complex and potentially rife with bias, even with those methodologies that have clearly defined steps. Just because a rigorous and standardized approach has been used does not mean that implicit judgements will not influence the interpretations and choices made at different stages.

In all types of synthesis, the quantity of data can be considerable, requiring difficult decisions about scope, which may affect relevance. The quantity of available data also has implications for the size of the research team. Few reviews these days can be done independently, in particular because decisions about inclusion and exclusion may require the involvement of more than one person to ensure reliability.

For all types of synthesis, it is likely that in areas with large, amorphous, and diverse bodies of literature, even the most sophisticated search strategies will not turn up all the relevant and important texts. This may be more important in some synthesis methodologies than in others, but the omission of key documents can influence the results of all syntheses. This issue can be addressed, at least in part, by including a library scientist on the research team as required by some funding agencies. Even then, it is possible to miss key texts. In this review, for example, because none of us are trained in or conduct meta-analyses, we were not even aware that we had missed some new developments in this field such as meta-regression [117] – [118] , network meta-analysis [119] – [121] , and the use of individual patient data in meta-analyses [122] – [123] .

One limitation of systematic reviews and meta-analyses is that they rapidly go out of date. We thought this might be true for all types of synthesis, although we wondered if those that produce theory might not be somewhat more enduring. We have not answered this question but it is open for debate. For all types of synthesis, the analytic skills and the time required are considerable so it is clear that training is important before embarking on a review, and some types of review may not be appropriate for students or busy practitioners.

Finally, the quality of reporting in primary studies of all genres is variable so it is sometimes difficult to identify aspects of the study essential for the synthesis, or to determine whether the study meets quality criteria. There may be flaws in the original study, or journal page limitations may necessitate omitting important details. Reporting standards have been developed for some types of reviews (e.g., systematic review, meta-analysis, meta-narrative synthesis, realist synthesis); but there are no agreed upon standards for qualitative reviews. This is an important area for development in advancing the science of research synthesis.

4.8.3. Strengths and Limitations of the Four Synthesis Types

The conventional literature review and now the increasingly common integrative review remain important and accessible approaches for students, practitioners, and experienced researchers who want to summarize literature in an area but do not have the expertise to use one of the more complex methodologies. Carefully executed, such reviews are very useful for synthesizing literature in preparation for research grants and practice projects. They can determine the state of knowledge in an area and identify important gaps in the literature to provide a clear rationale or theoretical framework for a study [14] , [18] . There is a demand, however, for more rigour, with more attention to developing comprehensive search strategies and more systematic approaches to combining, integrating, and synthesizing the findings.

Generally, conventional reviews include diverse study designs and data types that facilitate comprehensiveness, which may be a strength on the one hand, but can also present challenges on the other. The complexity inherent in combining results from studies with diverse methodologies can result in bias and inaccuracies. The absence of clear guidelines about how to synthesize across diverse study types and data [18] has been a challenge for novice reviewers.

Quantitative systematic reviews and meta-analyses have been important in launching the field of evidence-based healthcare. They provide a systematic, orderly and auditable process for conducting a review and drawing conclusions [25] . They are arguably the most powerful approaches to understanding the effectiveness of healthcare interventions, especially when intervention studies on the same topic show very different results. When areas of research are dogged by controversy [25] or when study results go against strongly held beliefs, such approaches can reduce the uncertainty and bring strong evidence to bear on the controversy.

Despite their strengths, they also have limitations. Systematic reviews and meta-analyses do not provide a way of including complex literature comprising various types of evidence including qualitative studies, theoretical work, and epidemiological studies. Only certain types of design are considered and qualitative data are used in a limited way. This exclusion limits what can be learned in a topic area.

Meta-analyses are often not possible because of wide variability in study design, population, and interventions so they may have a narrow range of utility. New developments in meta-analysis, however, can be used to address some of these limitations. Network meta-analysis is used to explore relative efficacy of multiple interventions, even those that have never been compared in more conventional pairwise meta-analyses [121] , allowing for improved clinical decision making [120] . The limitation is that network meta-analysis has only been used in medical/clinical applications [119] and not in public health. It has not yet been widely accepted and many methodological challenges remain [120] – [121] . Meta-regression is another development that combines meta-analytic and linear regression principles to address the fact that heterogeneity of results may compromise a meta-analysis [117] – [118] . The disadvantage is that many clinicians are unfamiliar with it and may incorrectly interpret results [117] .

Some have accused meta-analysis of combining apples and oranges [124] raising questions in the field about their meaningfulness [25] , [28] . More recently, the use of individual rather than aggregate data has been useful in facilitating greater comparability among studies [122] . In fact, Tomas et al. [123] argue that meta-analysis using individual data is now the gold standard although access to the raw data from other studies may be a challenge to obtain.

The usefulness of systematic reviews in synthesizing complex health and social interventions has also been challenged [102] . It is often difficult to synthesize their findings because such studies are “epistemologically diverse and methodologically complex” [ [69] , p.21]. Rigid inclusion/exclusion criteria may allow only experimental or quasi-experimental designs into consideration resulting in lost information that may well be useful to policy makers for tailoring an intervention to the context or understanding its acceptance by recipients.

Qualitative syntheses may be the type of review most fraught with controversy and challenge, while also bringing distinct strengths to the enterprise. Although these methodologies provide a comprehensive and systematic review approach, they do not generally provide definitive statements about intervention effectiveness. They do, however, address important questions about the development of theoretical concepts, patient experiences, acceptability of interventions, and an understanding about why interventions might work.

Most qualitative syntheses aim to produce a theoretically generalizable mid-range theory that explains variation across studies. This makes them more useful than single primary studies, which may not be applicable beyond the immediate setting or population. All provide a contextual richness that enhances relevance and understanding. Another benefit of some types of qualitative synthesis (e.g., grounded formal theory) is that the concept of saturation provides a sound rationale for limiting the number of texts to be included thus making reviews potentially more manageable. This contrasts with the requirements of systematic reviews and meta-analyses that require an exhaustive search.

Qualitative researchers debate about whether the findings of ontologically and epistemological diverse qualitative studies can actually be combined or synthesized [125] because methodological diversity raises many challenges for synthesizing findings. The products of different types of qualitative syntheses range from theory and conceptual frameworks, to themes and rich descriptive narratives. Can one combine the findings from a phenomenological study with the theory produced in a grounded theory study? Many argue yes, but many also argue no.

Emerging synthesis methodologies were developed to address some limitations inherent in other types of synthesis but also have their own issues. Because each type is so unique, it is difficult to identify overarching strengths of the entire category. An important strength, however, is that these newer forms of synthesis provide a systematic and rigorous approach to synthesizing a diverse literature base in a topic area that includes a range of data types such as: both quantitative and qualitative studies, theoretical work, case studies, evaluations, epidemiological studies, trials, and policy documents. More than conventional literature reviews and systematic reviews, these approaches provide explicit guidance on analytic methods for integrating different types of data. The assumption is that all forms of data have something to contribute to knowledge and theory in a topic area. All have a defined but flexible process in recognition that the methods may need to shift as knowledge develops through the process.

Many emerging synthesis types are helpful to policy makers and practitioners because they are usually involved as team members in the process to define the research questions, and interpret and disseminate the findings. In fact, engagement of stakeholders is built into the procedures of the methods. This is true for rapid reviews, meta-narrative syntheses, and realist syntheses. It is less likely to be the case for critical interpretive syntheses.

Another strength of some approaches (realist and meta-narrative syntheses) is that quality and publication standards have been developed to guide researchers, reviewers, and funders in judging the quality of the products [108] , [126] – [127] . Training materials and online communities of practice have also been developed to guide users of realist and meta-narrative review methods [107] , [128] . A unique strength of critical interpretive synthesis is that it takes a critical perspective on the process that may help reconceptualize the data in a way not considered by the primary researchers [72] .

There are also challenges of these new approaches. The methods are new and there may be few published applications by researchers other than the developers of the methods, so new users often struggle with the application. The newness of the approaches means that there may not be mentors available to guide those unfamiliar with the methods. This is changing, however, and the number of applications in the literature is growing with publications by new users helping to develop the science of synthesis [e.g., [129] ]. However, the evolving nature of the approaches and their developmental stage present challenges for novice researchers.

4.9. When to Use Each Approach

Choosing an appropriate approach to synthesis will depend on the question you are asking, the purpose of the review, and the outcome or product you want to achieve. In Additional File 1 , we discuss each of these to provide guidance to readers on making a choice about review type. If researchers want to know whether a particular type of intervention is effective in achieving its intended outcomes, then they might choose a quantitative systemic review with or without meta-analysis, possibly buttressed with qualitative studies to provide depth and explanation of the results. Alternately, if the concern is about whether an intervention is effective with different populations under diverse conditions in varying contexts, then a realist synthesis might be the most appropriate.

If researchers' concern is to develop theory, they might consider qualitative syntheses or some of the emerging syntheses that produce theory (e.g., critical interpretive synthesis, realist review, grounded formal theory, qualitative meta-synthesis). If the aim is to track the development and evolution of concepts, theories or ideas, or to determine how an issue or question is addressed across diverse research traditions, then meta-narrative synthesis would be most appropriate.

When the purpose is to review the literature in advance of undertaking a new project, particularly by graduate students, then perhaps an integrative review would be appropriate. Such efforts contribute towards the expansion of theory, identify gaps in the research, establish the rationale for studying particular phenomena, and provide a framework for interpreting results in ways that might be useful for influencing policy and practice.

For researchers keen to bring new insights, interpretations, and critical re-conceptualizations to a body of research, then qualitative or critical interpretive syntheses will provide an inductive product that may offer new understandings or challenges to the status quo. These can inform future theory development, or provide guidance for policy and practice.

5. Discussion

What is the current state of science regarding research synthesis? Public health, health care, and social science researchers or clinicians have previously used all four categories of research synthesis, and all offer a suitable array of approaches for inquiries. New developments in systematic reviews and meta-analysis are providing ways of addressing methodological challenges [117] – [123] . There has also been significant advancement in emerging synthesis methodologies and they are quickly gaining popularity. Qualitative meta-synthesis is still evolving, particularly given how new it is within the terrain of research synthesis. In the midst of this evolution, outstanding issues persist such as grappling with: the quantity of data, quality appraisal, and integration with knowledge translation. These topics have not been thoroughly addressed and need further debate.

5.1. Quantity of Data

We raise the question of whether it is possible or desirable to find all available studies for a synthesis that has this requirement (e.g., meta-analysis, systematic review, scoping, meta-narrative synthesis [25] , [27] , [63] , [67] , [84] – [85] ). Is the synthesis of all available studies a realistic goal in light of the burgeoning literature? And how can this be sustained in the future, particularly as the emerging methodologies continue to develop and as the internet facilitates endless access? There has been surprisingly little discussion on this topic and the answers will have far-reaching implications for searching, sampling, and team formation.

Researchers and graduate students can no longer rely on their own independent literature search. They will likely need to ask librarians for assistance as they navigate multiple sources of literature and learn new search strategies. Although teams now collaborate with library scientists, syntheses are limited in that researchers must make decisions on the boundaries of the review, in turn influencing the study's significance. The size of a team may also be pragmatically determined to manage the search, extraction, and synthesis of the burgeoning data. There is no single answer to our question about the possibility or necessity of finding all available articles for a review. Multiple strategies that are situation specific are likely to be needed.

5.2. Quality Appraisal

While the issue of quality appraisal has received much attention in the synthesis literature, scholars are far from resolution. There may be no agreement about appraisal criteria in a given tradition. For example, the debate rages over the appropriateness of quality appraisal in qualitative synthesis where there are over 100 different sets of criteria and many do not overlap [49] . These differences may reflect disciplinary and methodological orientations, but diverse quality appraisal criteria may privilege particular types of research [49] . The decision to appraise is often grounded in ontological and epistemological assumptions. Nonetheless, diversity within and between categories of synthesis is likely to continue unless debate on the topic of quality appraisal continues and evolves toward consensus.

5.3. Integration with Knowledge Translation

If research syntheses are to make a difference to practice and ultimately to improve health outcomes, then we need to do a better job of knowledge translation. In the Canadian Institutes of Health Research (CIHR) definition of knowledge translation (KT), research or knowledge synthesis is an integral component [130] . Yet, with few exceptions [131] – [132] , very little of the research synthesis literature even mentions the relationship of synthesis to KT nor does it discuss strategies to facilitate the integration of synthesis findings into policy and practice. The exception is in the emerging synthesis methodologies, some of which (e.g., realist and meta-narrative syntheses, scoping reviews) explicitly involve stakeholders or knowledge users. The argument is that engaging them in this way increases the likelihood that the knowledge generated will be translated into policy and practice. We suggest that a more explicit engagement with knowledge users in all types of synthesis would benefit the uptake of the research findings.

Research synthesis neither makes research more applicable to practice nor ensures implementation. Focus must now turn seriously towards translation of synthesis findings into knowledge products that are useful for health care practitioners in multiple areas of practice and develop appropriate strategies to facilitate their use. The burgeoning field of knowledge translation has, to some extent, taken up this challenge; however, the research-practice gap continues to plague us [133] – [134] . It is a particular problem for qualitative syntheses [131] . Although such syntheses have an important place in evidence-informed practice, little effort has gone into the challenge of translating the findings into useful products to guide practice [131] .

5.4. Limitations

Our study took longer than would normally be expected for an integrative review. Each of us were primarily involved in our own dissertations or teaching/research positions, and so this study was conducted ‘off the sides of our desks.’ A limitation was that we searched the literature over the course of 4 years (from 2008–2012), necessitating multiple search updates. Further, we did not do a comprehensive search of the literature after 2012, thus the more recent synthesis literature was not systematically explored. We did, however, perform limited database searches from 2012–2015 to keep abreast of the latest methodological developments. Although we missed some new approaches to meta-analysis in our search, we did not find any new features of the synthesis methodologies covered in our review that would change the analysis or findings of this article. Lastly, we struggled with the labels used for the broad categories of research synthesis methodology because of our hesitancy to reinforce the divide between quantitative and qualitative approaches. However, it was very difficult to find alternative language that represented the types of data used in these methodologies. Despite our hesitancy in creating such an obvious divide, we were left with the challenge of trying to find a way of characterizing these broad types of syntheses.

6. Conclusion

Our findings offer methodological clarity for those wishing to learn about the broad terrain of research synthesis. We believe that our review makes transparent the issues and considerations in choosing from among the four broad categories of research synthesis. In summary, research synthesis has taken its place as a form of research in its own right. The methodological terrain has deep historical roots reaching back over the past 200 years, yet research synthesis remains relatively new to public health, health care, and social sciences in general. This is rapidly changing. New developments in systematic reviews and meta-analysis, and the emergence of new synthesis methodologies provide a vast array of options to review the literature for diverse purposes. New approaches to research synthesis and new analytic methods within existing approaches provide a much broader range of review alternatives for public health, health care, and social science students and researchers.

Acknowledgments

KSM is an assistant professor in the Faculty of Nursing at the University of Alberta. Her work on this article was largely conducted as a Postdoctoral Fellow, funded by KRESCENT (Kidney Research Scientist Core Education and National Training Program, reference #KRES110011R1) and the Faculty of Nursing at the University of Alberta.

MM's work on this study over the period of 2008-2014 was supported by a Canadian Institutes of Health Research Applied Public Health Research Chair Award (grant #92365).

We thank Rachel Spanier who provided support with reference formatting.

Sorry, there was a problem.

Download the free Kindle app and start reading Kindle books instantly on your smartphone, tablet, or computer - no Kindle device required .

Read instantly on your browser with Kindle for Web.

Using your mobile phone camera - scan the code below and download the Kindle app.

Image Unavailable

• To view this video download Flash Player

Follow the author

Meta-Analysis: Quantitative Methods for Research Synthesis (Quantitative Applications in the Social Sciences) 1st Edition

Meta-Analysis shows concisely, yet comprehensively, how to apply statistical methods to achieve a literature review of a common research domain. It demonstrates the use of combined tests and measures of effect size to synthesize quantitatively the results of independent studies for both group differences and correlations. Strengths and weaknesses of alternative approaches, as well as of meta-analysis in general, are presented.

• ISBN-10 0803927568
• ISBN-13 978-0803927568
• Edition 1st
• Publisher SAGE Publications, Inc
• Publication date April 1, 1986
• Part of series Quantitative Applications in the Social Sciences
• Language English
• Dimensions 5.5 x 0.16 x 8.5 inches
• Print length 72 pages
• See all details

Editorial Reviews

About the author, product details.

• Publisher ‏ : ‎ SAGE Publications, Inc; 1st edition (April 1, 1986)
• Language ‏ : ‎ English
• Paperback ‏ : ‎ 72 pages
• ISBN-10 ‏ : ‎ 0803927568
• ISBN-13 ‏ : ‎ 978-0803927568
• Item Weight ‏ : ‎ 3.88 ounces
• Dimensions ‏ : ‎ 5.5 x 0.16 x 8.5 inches
• #641 in Agriculture
• #1,374 in Scientific Research
• #1,591 in General (Books)

About the author

Fredric marc wolf.

Discover more of the author’s books, see similar authors, read author blogs and more

Customer reviews

Customer Reviews, including Product Star Ratings help customers to learn more about the product and decide whether it is the right product for them.

To calculate the overall star rating and percentage breakdown by star, we don’t use a simple average. Instead, our system considers things like how recent a review is and if the reviewer bought the item on Amazon. It also analyzed reviews to verify trustworthiness.

• Sort reviews by Top reviews Most recent Top reviews

Top reviews from the United States

There was a problem filtering reviews right now. please try again later..

Top reviews from other countries

• About Amazon
• Investor Relations
• Amazon Devices
• Amazon Science
• Sell products on Amazon
• Sell on Amazon Business
• Sell apps on Amazon
• Become an Affiliate
• Advertise Your Products
• Self-Publish with Us
• Host an Amazon Hub
• › See More Make Money with Us
• Amazon Business Card
• Shop with Points
• Reload Your Balance
• Amazon Currency Converter
• Amazon and COVID-19
• Your Account
• Your Orders
• Shipping Rates & Policies
• Returns & Replacements
• Manage Your Content and Devices

• Conditions of Use
• Privacy Notice
• Consumer Health Data Privacy Disclosure
• Your Ads Privacy Choices

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed, quantitative synthesis—an update, affiliations.

• 1 Scientific Resource Center for the AHRQ Effective Health Care Program, Portland VA Research Foundation, VA Portland Health Care Systems, Portland, OR
• 2 Mayo Clinic Evidence-based Practice Center, Rochester MN
• 3 Kaiser Permanente Research Affiliates Evidence-based Practice Center, Portland OR
• 4 Southern California Evidence-based Practice Center – RAND Corporation, Santa Monica, CA
• 5 University of Alberta Evidence-based Practice Center, Edmonton, AB
• 6 ECRI Institute – Penn Medicine Evidence-based Practice Center, Plymouth Meeting, PA
• 7 Pacific-Northwest Evidence-based Practice Center – Oregon Health & Sciences University, Portland, OR
• 8 RTI International – University of North Carolina (UNC) Evidence-based Practice Center, Chapel Hill, NC
• 9 Brown University Center for Evidence-based Medicine, Providence, RI
• PMID: 30125065
• Bookshelf ID: NBK519365

Quantitative synthesis, or meta-analysis, is often essential for Comparative Effective Reviews (CERs) to provide scientifically rigorous summary information. Quantitative synthesis should be conducted in a transparent and consistent way with methodologies reported explicitly. This guide provides practical recommendations on conducting synthesis. The guide is not meant to be a textbook on meta-analysis nor is it a comprehensive review of methods, but rather it is intended to provide a consistent approach for situations and decisions that are commonly faced by AHRQ Evidence-based Practice Centers (EPCs). The goal is to describe choices as explicitly as possible, and in the context of EPC requirements, with an appropriate degree of confidence.

This guide addresses issues in the order that they are usually encountered in a synthesis, though we acknowledge that the process is not always linear. We first consider the decision of whether or not to combine studies quantitatively. The next chapter addresses how to extract and utilize data from individual studies to construct effect sizes, followed by a chapter on statistical model choice. The fourth chapter considers quantifying and exploring heterogeneity. The fifth describes an indirect evidence technique that has not been included in previous guidance – network meta-analysis, also known as mixed treatment comparisons. The final section in the report lays out future research suggestions.

PubMed Disclaimer

• Introduction
• Chapter 1. Decision to Combine Trials
• Chapter 2. Optimizing Use of Effect Size Data
• Chapter 3. Choice of Statistical Model for Combining Studies
• Chapter 4. Quantifying, Testing, and Exploring Statistical Heterogeneity
• Chapter 5. Network Meta-Analysis (Mixed Treatment Comparisons/Indirect Comparisons)
• Future Research Suggestions

Similar articles

• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update. Berkman ND, Lohr KN, Ansari M, McDonagh M, Balk E, Whitlock E, Reston J, Bass E, Butler M, Gartlehner G, Hartling L, Kane R, McPheeters M, Morgan L, Morton SC, Viswanathan M, Sista P, Chang S. Berkman ND, et al. 2013 Nov 18. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008–. 2013 Nov 18. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008–. PMID: 24404627 Free Books & Documents. Review.
• Conducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program. Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA. Fu R, et al. J Clin Epidemiol. 2011 Nov;64(11):1187-97. doi: 10.1016/j.jclinepi.2010.08.010. Epub 2011 Apr 7. J Clin Epidemiol. 2011. PMID: 21477993
• Conducting Quantitative Synthesis When Comparing Medical Interventions: AHRQ and the Effective Health Care Program. Fu R, Gartlehner G, Grant M, Shamliyan T, Sedrakyan A, Wilt TJ, Griffith L, Oremus M, Raina P, Ismaila A, Santaguida P, Lau J, Trikalinos TA. Fu R, et al. 2010 Oct 25. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008–. 2010 Oct 25. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008–. PMID: 21433407 Free Books & Documents. Review.
• Fu R, Gartlehner G, Grant M, et al.. Conducting Quantitative Synthesis When Comparing Medical Interventions: AHRQ and the Effective Health Care Program Agency for Healthcare Research and Quality. Rockville, MD: 2010.
• Lau J, Terrin N, Fu R. Expanded Guidance on Selected Quantitative Synthesis Topics Agency for Healthcare Research and Quality. Rockville, MD: 2013.
• Lau J, Chang S, Berkman N, et al.. EPC Response to IOM Standards for Systematic Reviews Agency for Healthcare Research and Quality. Rockville, MD: 2013. - PubMed
• Chou R, Aronson N, Atkins D, et al.. AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol. 2010;63(5):502–12. PMID: 18823754 http://dx.doi.org/10.1016/j.jclinepi.2008.06.007 - DOI - PubMed
• Fu R, Vandermeer BW, Shamliyan TA, et al.. Handling Continuous Outcomes in Quantitative Synthesis Agency for Healthcare Research and Quality. Rockville, MD: 2013.

Publication types

• Search in PubMed
• Search in MeSH
• Add to Search

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• NCBI Bookshelf

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• University of Texas Libraries
• UT Libraries

Systematic Reviews & Evidence Synthesis Methods

• Types of Reviews
• Formulate Question
• Find Existing Reviews & Protocols
• Register a Protocol
• Searching Systematically
• Supplementary Searching
• Managing Results
• Deduplication
• Critical Appraisal
• Glossary of terms
• Librarian Support
• Video tutorials This link opens in a new window
• Systematic Review & Evidence Synthesis Boot Camp

Once you have completed your analysis, you will want to both summarize and synthesize those results. You may have a qualitative synthesis, a quantitative synthesis, or both.

Qualitative Synthesis

In a qualitative synthesis, you describe for readers how the pieces of your work fit together. You will summarize, compare, and contrast the characteristics and findings, exploring the relationships between them. Further, you will discuss the relevance and applicability of the evidence to your research question. You will also analyze the strengths and weaknesses of the body of evidence. Focus on where the gaps are in the evidence and provide recommendations for further research.

Quantitative Synthesis

Whether or not your Systematic Review includes a full meta-analysis, there is typically some element of data analysis. The quantitative synthesis combines and analyzes the evidence using statistical techniques. This includes comparing methodological similarities and differences and potentially the quality of the studies conducted.

Summarizing vs. Synthesizing

In a systematic review, researchers do more than summarize findings from identified articles. You will synthesize the information you want to include.

While a summary is a way of concisely relating important themes and elements from a larger work or works in a condensed form, a synthesis takes the information from a variety of works and combines them together to create something new.

Synthesis :

"The goal of a systematic synthesis of qualitative research is to integrate or compare the results across studies in order to increase understanding of a particular phenomenon, not to add studies together. Typically the aim is to identify broader themes or new theories – qualitative syntheses usually result in a narrative summary of cross-cutting or emerging themes or constructs, and/or conceptual models."

Denner, J., Marsh, E. & Campe, S. (2017). Approaches to reviewing research in education. In D. Wyse, N. Selwyn, & E. Smith (Eds.), The BERA/SAGE Handbook of educational research (Vol. 2, pp. 143-164). doi: 10.4135/9781473983953.n7

• Approaches to Reviewing Research in Education from Sage Knowledge

Data synthesis  (Collaboration for Environmental Evidence Guidebook)

Interpreting findings and and reporting conduct   (Collaboration for Environmental Evidence Guidebook)

Interpreting results and drawing conclusions  (Cochrane Handbook, Chapter 15)

Guidance on the conduct of narrative synthesis in systematic reviews  (ESRC Methods Programme)

• Last Updated: Jun 26, 2024 3:15 PM
• URL: https://guides.lib.utexas.edu/systematicreviews

• Meta-Analysis/Meta-Synthesis

Meta Analysis

Meta-analysis   is a set of statistical techniques for synthesizing data across studies. It is a statistical method for combining the findings from quantitative studies. It evaluates, synthesizes, and summarizes results. It may be conducted independently or as a specialized subset of a systematic review.  A systematic review attempts to collate empirical evidence that fits predefined eligibility criteria to answer a specific research question. Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess the results of previous research to derive conclusions about that body of research (Haidrich, 2010). Rigorously conducted meta-analyses are useful tools in evidence-based medicine . Outcomes from a meta-analysis may include a more precise estimate of the effect of a treatment or risk factor for disease or other outcomes. Not all systematic reviews include meta-analysis , but all meta-analyses are found in systematic reviews (Haidrich, 2010).

A Meta analysis is appropriate when a group of studies report quantitative results rather than qualitative findings or theory, if they examine the same or similar constructs or relationships, if they are derived from similar research designs and report the simple relationships between two variables rather than relationships that have been adjusted for the effect of additional variables (siddaway, et al., 2019).

Haidich A. B. (2010). Meta-analysis in medical research.  Hippokratia ,  14 (Suppl 1), 29–37.

Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.  Annual Review of Psychology, 70 , 747–770.

Meta Synthesis

A meta synthesis is the systematic review and integration of findings from qualitative studies (Lachal et al., 2017). Reviews of qualitative information can be conducted and reported using the same replicable, rigorous, and transparent methodology and presentation. A meta-synthesis can be used when a review aims to integrate qualitative research.  A meta-synthesis attempts to synthesize qualitative studies on a topic to identify key themes, concepts, or theories that provide novel or more powerful explanations for the phenomenon under review (Siddaway et al., 2019).

Lachal, J., Revah-Levy, A., Orri, M., & Moro, M. R. (2017). Metasynthesis: An original method to synthesize qualitative literature in psychiatry.  Frontiers in Psychiatry, 8 , 269 . 

Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.  Annual Review of Psychology, 70 , 747–770 .

• << Previous: Rapid Review
• Next: Selecting a Review Type >>
• Overview of Review Types
• Integrative Review
• Scoping Review
• Rapid Review
• Selecting a Review Type
• Types of Questions
• Key Features and Limitations
• Is a Systematic Review Right for Your Research?
• Reducing Bias
• Guidelines for Student Researchers
• Training Resources
• Register Your Protocol
• Handbooks & Manuals
• Reporting Guidelines
• PRESS 2015 Guidelines
• Search Strategies
• Selected Databases
• Grey Literature
• Handsearching
• Citation Searching
• Study Types & Terminology
• Quantitative vs. Qualitative Research
• Critical Appraisal of Studies
• Broad Functionality Programs & Tools
• Search Strategy Tools
• Deduplication Tools
• CItation Screening
• Critical Appraisal Tools
• Quality Assessment/Risk of Bias Tools
• Data Collection/Extraction
• Meta Analysis Tools
• Books on Systematic Reviews
• Finding Systematic Review Articles in the Databases
• Systematic Review Journals
• More Resources
• Evidence-Based Practice Research in Nursing
• Citation Management Programs
• Last Updated: Jun 28, 2024 5:41 PM
• URL: https://libguides.adelphi.edu/Systematic_Reviews

• Technical Support
• Find My Rep

You are here

Meta-Analysis Quantitative Methods for Research Synthesis

• Fredric M. Wolf - University of Washington, USA
• Description

Author Fredric M. Wolf explains how to use combined statistical tests and measures of effect size to synthesize the results of independent studies of a common research question.

Learn more about "The Little Green Book" - QASS Series! Click Here

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

Select a Purchasing Option

This title is also available on SAGE Research Methods , the ultimate digital methods library. If your library doesn’t have access, ask your librarian to start a trial .

Teaching and Research guides

Systematic reviews.

• Starting the review
• About systematic reviews
• Research question
• Plan your search
• Sources to search
• Search example
• Screen and analyse

What is synthesis?

Quantitative synthesis (meta-analysis), qualitative synthesis.

• Guides and software
• Further help

Synthesis is a stage in the systematic review process where extracted data (findings of individual studies) are combined and evaluated. The synthesis part of a systematic review will determine the outcomes of the review.

There are two commonly accepted methods of synthesis in systematic reviews:

• Quantitative data synthesis
• Qualitative data synthesis

The way the data is extracted from your studies and synthesised and presented depends on the type of data being handled.

If you have quantitative information, some of the more common tools used to summarise data include:

• grouping of similar data, i.e. presenting the results in tables
• charts, e.g. pie-charts
• graphical displays such as forest plots

If you have qualitative information, some of the more common tools used to summarise data include:

• textual descriptions, i.e. written words
• thematic or content analysis

Whatever tool/s you use, the general purpose of extracting and synthesising data is to show the outcomes and effects of various studies and identify issues with methodology and quality. This means that your synthesis might reveal a number of elements, including:

• overall level of evidence
• the degree of consistency in the findings
• what the positive effects of a drug or treatment are, and what these effects are based on
• how many studies found a relationship or association between two things

In a quantitative systematic review, data is presented statistically. Typically, this is referred to as a meta-analysis . 

The usual method is to combine and evaluate data from multiple studies. This is normally done in order to draw conclusions about outcomes, effects, shortcomings of studies and/or applicability of findings.

Remember, the data you synthesise should relate to your research question and protocol (plan). In the case of quantitative analysis, the data extracted and synthesised will relate to whatever method was used to generate the research question (e.g. PICO method), and whatever quality appraisals were undertaken in the analysis stage.

One way of accurately representing all of your data is in the form of a f orest plot . A forest plot is a way of combining results of multiple clinical trials in order to show point estimates arising from different studies of the same condition or treatment. 

It is comprised of a graphical representation and often also a table. The graphical display shows the mean value for each trial and often with a confidence interval (the horizontal bars). Each mean is plotted relative to the vertical line of no difference.

• Forest Plots - Understanding a Meta-Analysis in 5 Minutes or Less (5:38 min) In this video, Dr. Maureen Dobbins, Scientific Director of the National Collaborating Centre for Methods and Tools, uses an example from social health to explain how to construct a forest plot graphic.
• How to interpret a forest plot (5:32 min) In this video, Terry Shaneyfelt, Clinician-educator at UAB School of Medicine, talks about how to interpret information contained in a typical forest plot, including table data.
• An introduction to meta-analysis (13 mins) Dr Christopher J. Carpenter introduces the concept of meta-analysis, a statistical approach to finding patterns and trends among research studies on the same topic. Meta-analysis allows the researcher to weight study results based on size, moderating variables, and other factors.

Journal articles

• Neyeloff, J. L., Fuchs, S. C., & Moreira, L. B. (2012). Meta-analyses and Forest plots using a microsoft excel spreadsheet: step-by-step guide focusing on descriptive data analysis. BMC Research Notes, 5(1), 52-57. https://doi.org/10.1186/1756-0500-5-52 Provides a step-by-step guide on how to use Excel to perform a meta-analysis and generate forest plots.
• Ried, K. (2006). Interpreting and understanding meta-analysis graphs: a practical guide. Australian Family Physician, 35(8), 635- 638. This article provides a practical guide to appraisal of meta-analysis graphs, and has been developed as part of the Primary Health Care Research Evaluation Development (PHCRED) capacity building program for training general practitioners and other primary health care professionals in research methodology.

In a qualitative systematic review, data can be presented in a number of different ways. A typical procedure in the health sciences is  thematic analysis .

As explained by James Thomas and Angela Harden (2008) in an article for  BMC Medical Research Methodology : 

"Thematic synthesis has three stages:

• the coding of text 'line-by-line'
• the development of 'descriptive themes'
• and the generation of 'analytical themes'

While the development of descriptive themes remains 'close' to the primary studies, the analytical themes represent a stage of interpretation whereby the reviewers 'go beyond' the primary studies and generate new interpretive constructs, explanations or hypotheses" (p. 45).

A good example of how to conduct a thematic analysis in a systematic review is the following journal article by Jorgensen et al. (2108) on cancer patients. In it, the authors go through the process of:

(a) identifying and coding information about the selected studies' methodologies and findings on patient care

(b) organising these codes into subheadings and descriptive categories

(c) developing these categories into analytical themes

Jørgensen, C. R., Thomsen, T. G., Ross, L., Dietz, S. M., Therkildsen, S., Groenvold, M., Rasmussen, C. L., & Johnsen, A. T. (2018). What facilitates “patient empowerment” in cancer patients during follow-up: A qualitative systematic review of the literature. Qualitative Health Research, 28(2), 292-304. https://doi.org/10.1177/1049732317721477

Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(1), 45-54. https://doi.org/10.1186/1471-2288-8-45

• << Previous: Screen and analyse
• Next: Write >>

• Last Updated: Jun 27, 2024 2:49 PM
• URL: https://rmit.libguides.com/systematicreviews

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Systematic Review
• Open access
• Published: 24 June 2024

Placebo effects in randomized trials of pharmacological and neurostimulation interventions for mental disorders: An umbrella review

• Nathan T. M. Huneke   ORCID: orcid.org/0000-0001-5981-6707 1 , 2 ,
• Jay Amin   ORCID: orcid.org/0000-0003-3792-0428 1 , 2 ,
• David S. Baldwin 1 , 2 , 3 ,
• Alessio Bellato 4 , 5 ,
• Valerie Brandt   ORCID: orcid.org/0000-0002-3208-2659 5 , 6 ,
• Samuel R. Chamberlain 1 , 2 ,
• Christoph U. Correll   ORCID: orcid.org/0000-0002-7254-5646 7 , 8 , 9 , 10 ,
• Luis Eudave 11 ,
• Matthew Garner 1 , 5 , 12 ,
• Corentin J. Gosling 5 , 13 , 14 ,
• Catherine M. Hill 1 , 15 ,
• Ruihua Hou 1 ,
• Oliver D. Howes   ORCID: orcid.org/0000-0002-2928-1972 16 , 17 , 18 ,
• Konstantinos Ioannidis 1 , 2 ,
• Ole Köhler-Forsberg 19 , 20 ,
• Lucia Marzulli 21 ,
• Claire Reed   ORCID: orcid.org/0000-0003-1385-4729 5 ,
• Julia M. A. Sinclair 1 ,
• Satneet Singh 2 ,
• Marco Solmi   ORCID: orcid.org/0000-0003-4877-7233 5 , 22 , 23 , 24 , 25   na1 &
• Samuele Cortese   ORCID: orcid.org/0000-0001-5877-8075 1 , 5 , 26 , 27 , 28   na1  

Molecular Psychiatry ( 2024 ) Cite this article

420 Accesses

5 Altmetric

Metrics details

• Drug discovery
• Neuroscience
• Psychiatric disorders

There is a growing literature exploring the placebo response within specific mental disorders, but no overarching quantitative synthesis of this research has analyzed evidence across mental disorders. We carried out an umbrella review of meta-analyses of randomized controlled trials (RCTs) of biological treatments (pharmacotherapy or neurostimulation) for mental disorders. We explored whether placebo effect size differs across distinct disorders, and the correlates of increased placebo effects. Based on a pre-registered protocol, we searched Medline, PsycInfo, EMBASE, and Web of Knowledge up to 23.10.2022 for systematic reviews and/or meta-analyses reporting placebo effect sizes in psychopharmacological or neurostimulation RCTs. Twenty meta-analyses, summarising 1,691 RCTs involving 261,730 patients, were included. Placebo effect size varied, and was large in alcohol use disorder ( g  = 0.90, 95% CI [0.70, 1.09]), depression ( g  = 1.10, 95% CI [1.06, 1.15]), restless legs syndrome ( g  = 1.41, 95% CI [1.25, 1.56]), and generalized anxiety disorder ( d  = 1.85, 95% CI [1.61, 2.09]). Placebo effect size was small-to-medium in obsessive-compulsive disorder ( d  = 0.32, 95% CI [0.22, 0.41]), primary insomnia ( g  = 0.35, 95% CI [0.28, 0.42]), and schizophrenia spectrum disorders (standardized mean change = 0.33, 95% CI [0.22, 0.44]). Correlates of larger placebo response in multiple mental disorders included later publication year (opposite finding for ADHD), younger age, more trial sites, larger sample size, increased baseline severity, and larger active treatment effect size. Most (18 of 20) meta-analyses were judged ‘low’ quality as per AMSTAR-2. Placebo effect sizes varied substantially across mental disorders. Future research should explore the sources of this variation. We identified important gaps in the literature, with no eligible systematic reviews/meta-analyses of placebo response in stress-related disorders, eating disorders, behavioural addictions, or bipolar mania.

Similar content being viewed by others

Effects of open-label placebos in clinical trials: a systematic review and meta-analysis

A transdiagnostic meta-analysis of acute augmentations to psychological therapy

Treatment resistance in psychiatry: state of the art and new directions

Introduction.

A placebo is an ‘inactive’ substance or ‘sham’ technique that is used as a control for assessing the efficacy of an active treatment [ 1 ]. However, study participants in a placebo control group may experience considerable symptom improvements - a ‘placebo response’ [ 1 , 2 , 3 ]. Statistical artifacts or non-specific effects account for some of the placebo response. For example, many individuals seek treatment and are enrolled in clinical trials while their symptoms are at their worst. Their symptoms will gradually return to their usual severity (‘regression to the mean’), giving the appearance of a placebo response [ 4 ]. Further, it has been suggested that the placebo response is exacerbated due to unreliable ratings as well as baseline symptom severity inflation if raters are aware of severity criteria for entry to a trial [ 5 , 6 ]. Other potential sources of apparent placebo responses include sampling biases caused by the withdrawal of the least improved patients in the placebo arm, non-specific beneficial effects resulting from interactions with staff delivering the trial, environmental effects due to inpatient care during placebo-controlled trials, or other unaccounted for factors, such as dietary or exercise changes during the trial [ 7 , 8 , 9 ]. Nonetheless, there is evidence that placebo administration results in ‘true’ - or non-artefactual - placebo effects, that is, identifiable changes in biological systems [ 1 , 10 , 11 ]. For example, placebo administration is capable of causing immunosuppression [ 12 , 13 ], placebo effects in Parkinson’s disease are driven by striatal dopamine release [ 10 , 14 ], and placebo analgesia is mediated by endogenous opioid release [ 15 , 16 ]. Furthermore, there is evidence that placebo effects in depressive and anxiety disorders are correlated with altered activity in the ventral striatum, orbitofrontal cortex, rostral anterior cingulate cortex, and the default mode network [ 17 ]. The placebo effect size can be increased through the use of verbal suggestions and conditioning procedures, thus suggesting the underlying role of psychological mechanisms including learning and expectations [ 11 , 18 ].

Across age groups, treatment modalities, and diverse mental disorders, biological treatments (pharmacotherapy or neurostimulation) do reduce symptoms [ 19 , 20 , 21 , 22 ], but only a subgroup of patients experience a clinically significant symptom response or enter remission [ 23 , 24 , 25 ]. Furthermore, current medications may also have unfavourable side effects [ 23 , 26 , 27 , 28 , 29 , 30 , 31 ]. Given the high prevalence of mental disorders and their significant socioeconomic burden [ 32 , 33 , 34 ], there is a need to develop more effective and safer psychopharmacologic and neurostimulation treatments. However, in randomized-controlled trials (RCTs), the magnitude of the placebo response may be considerable, which can affect the interpretation of their results [ 35 , 36 , 37 ]. For example, in antipsychotic trials over the past 40 years, placebo response has increased while medication response has remained consistent [ 38 , 39 ]. Consequently, the trial’s ability to statistically differentiate between an active medication and a placebo is diminished [ 40 ]. Indeed, large placebo response rates have been implicated in hindering psychotropic drug development [ 41 , 42 ]. The increased placebo response can also affect larger data synthesis approaches, such as network meta-analysis, in which assumptions about placebo responses (e.g. stability over time) might affect the validity of results [ 43 ].

Improved understanding of participant, trial, and mental disorder-related factors that contribute to placebo response might allow better clinical trial design to separate active treatment from placebo effects. There is a growing body of research, including individual studies and systematic reviews/meta-analyses, examining the placebo response within specific mental disorders [ 35 ]. However, to date, no overarching synthesis of this literature, to detect any similarities or differences across mental disorders, has been published. We therefore carried out an umbrella review of meta-analyses to address this need. We aimed to assess the placebo effect size in RCTs for a range of mental disorders, whether the effect size differs across distinct mental disorders, and identify any correlates of increased placebo effect size or response rate.

The protocol for this systematic umbrella review was pre-registered on the open science framework ( https://osf.io/fxvn4/ ) and published [ 44 ]. Deviations from this protocol, and additions to it, were: eight authors were involved in record screening rather than two; we reported effect sizes pooled across age groups and analyses comparing placebo effect sizes between age groups; and we included a meta-analysis that incorporated trials of dietary supplements as well as medications in autism. For the rationale behind these decisions, see eMethods.

Eight authors (NH, AB, VB, LE, OKF, LM, CR, SS) carried out the systematic review and data extraction independently in pairs. Discrepancies were resolved through consensus or through arbitration by a third reviewer (NH or SCo). We searched, without date or language restrictions, up to 23.10.2022, Medline, PsycInfo, EMBASE + EMBASE Classic, and Web of Knowledge for systematic reviews with or without meta-analyses of RCTs of biological treatments (psychopharmacotherapy or neurostimulation) compared with a placebo or sham treatment in individuals with mental disorders diagnosed according to standardized criteria. The full search strategy is included in eMethods. We also sought systematic reviews of RCTs conducted in patients with sleep-wake disorders, since these disorders are included in the DSM-5 and their core symptoms overlap with those of mental disorders [ 45 ]. We retained systematic reviews with or without meta-analyses that reported within-group changes in symptoms in the placebo arm.

Next, to prevent duplication of data, a matrix containing all eligible systematic reviews/meta-analyses for each category of mental disorder was created. Where there were multiple eligible systematic reviews/meta-analyses for the same disorder and treatment, we preferentially included meta-analyses, and if multiple eligible meta-analyses remained, then we included the one containing the largest number of studies for the same disorder and treatment, in line with recent umbrella reviews [ 46 , 47 ].

Data were extracted by at least two among six reviewers (AB, VB, LE, OKF, CR, SS) independently in pairs via a piloted form. All extracted data were further checked by a third reviewer (NH). See eMethods for a list of extracted data.

Our primary outcome was the pre-post effect size of the placebo/sham related to the condition-specific primary symptom change for each mental disorder. Secondary outcomes included any other reported clinical outcomes in eligible reviews. We report effect sizes calculated within-group from baseline and post-treatment means by meta-analysis authors, including Cohen’s d and Hedges’ g for repeated measures, which account for both mean difference and correlation between paired observations; and standardized mean change, where the average change score is divided by standard deviation of the change scores. We interpreted the effect size in line with the suggestion by Cohen [ 48 ], i.e. small (~0.2), medium (~0.5), or large (~0.8).

In addition, we extracted data regarding potential correlates of increased placebo effect size or response rate (as defined and assessed by the authors of each meta-analysis) in each mental disorder identified through correlation analyses or meta-regression. Where available, results from multivariate analyses were preferred.

The methodological quality of included reviews was assessed by at least two among six reviewers (AB, VB, LE, OKF, NH, CR) independently and in pairs using the AMSTAR-2 tool, a critical appraisal tool that enables reproducible assessments of the conduct of systematic reviews [ 49 ]. The methodological quality of each included review was rated as high, moderate, low, or critically low.

Our initial search identified 6,108 records. After screening titles and abstracts, we obtained and assessed 115 full-text reports (see eResults for a list of articles excluded following full-text assessment, with reasons). Of these, 20 were deemed eligible, and all were systematic reviews with meta-analysis (Fig.  1 ). In total, the 20 included meta-analyses synthesized data from 1,691 RCTs (median 55) involving 261,730 patients (median 5,365). These meta-analyses were published between 2007 and 2022 and involved individuals with the following mental disorders: major depressive disorder (MDD; n  = 6) [ 50 , 51 , 52 , 53 , 54 , 55 ], anxiety disorders ( n  = 4) [ 55 , 56 , 57 , 58 ], schizophrenia spectrum disorders ( n  = 3) [ 38 , 59 , 60 ], alcohol use disorder (AUD; n  = 1) [ 61 ], attention-deficit/hyperactivity disorder (ADHD; n  = 1) [ 62 ], autism spectrum disorders ( n  = 1) [ 63 ], bipolar depression ( n  = 1) [ 64 ], intellectual disability ( n  = 1) [ 65 ], obsessive-compulsive disorder (OCD; n  = 1) [ 66 ], primary insomnia ( n  = 1) [ 67 ], and restless legs syndrome (RLS; n  = 1) [ 68 ].

Twenty meta-analyses were included.

The methodological quality of the included meta-analyses according to AMSTAR-2 ratings was high in two meta-analyses (ADHD and autism), low in four meta-analyses, and critically low in the remaining 14 meta-analyses (Table  1 ). The most common sources of bias that led to downgrading on the AMSTAR-2 were: no list of excluded full-text articles with reasons ( k  = 14), no explicit statement that the protocol was pre-registered ( k  = 14), and no assessment of the potential impact of risk of bias in individual studies on the results ( k  = 13). The full reasoning behind our AMSTAR-2 ratings is included in eResults.

Our first objective was to determine placebo effect sizes across mental conditions. Data regarding within-group placebo efficacy were reported in sixteen of the included meta-analyses [ 38 , 50 , 52 , 53 , 55 , 56 , 57 , 58 , 60 , 61 , 62 , 63 , 65 , 66 , 67 , 68 ]. Placebo effect sizes for the primary outcomes ranged from 0.23 to 1.85, with a median of 0.64 (Fig.  2 ). Median heterogeneity across meta-analyses was I 2  = 72%, suggesting a generally high percentage of heterogeneity due to true variation across studies.

Dots represent placebo group effect size while triangles represent active effect size. CI confidence interval, MDD major depressive disorder, GAD generalized anxiety disorder, SAD social anxiety disorder, OCD obsessive-compulsive disorder, g Hedges’ g, d Cohen’s d, SMC standardized mean change, NR not reported.

A detailed description of each meta-analysis included for this objective is included in eResults. Here, we report a summary of these results in order of the greatest number of RCT’s and meta-analyses included per disorder. In MDD, a large within-group placebo effect was observed ( g  = 1.10, 95% CI [1.06, 1.15]), although active medication had an even larger effect size ( g  = 1.49, 95% CI [1.44, 1.53]) [ 50 ]. Similarly, in children and adolescents with MDD, placebo effect size was large ( g  = 1.57, 95% CI [1.36, 1.78]), as was serotonergic medication effect size ( g  = 1.85, 95% CI [1.70, 2.00]) [ 55 ]. In treatment-resistant MDD, the within-group placebo effect size was smaller than in non-treatment-resistant MDD ( g  = 0.89, 95% CI [0.81, 0.98]) [ 52 ]. In neuromodulation trials for MDD, the effect size of sham was g  = 0.80 (95% CI [0.65, 0.95]) [ 53 ]. In this meta-analysis, the effect size was larger for non-treatment-resistant ( g  = 1.28, 95% CI [0.47, 2.97]) compared to treatment-resistant participants (g = 0.50 95% CI [0.03, 0.99]) [ 53 ]. In adults with anxiety disorders, placebo effect sizes varied across disorders, with a medium effect size in panic disorder ( d  = 0.57, 95% CI [0.50, 0.64]) [ 56 ] and large effect sizes in generalized anxiety disorder (GAD) ( d  = 1.85, 95% CI [1.61, 2.09]) and social anxiety disorder (SAD) ( d  = 0.94, 95% CI [0.77, 1.12]) [ 57 ]. Other meta-analyses in children and adolescents and older adults pooled RCTs across anxiety disorders, and found large placebo effect sizes ( g  = 1.03, 95% CI [0.84, 1.21] and d  = 1.06, 95% CI [0.71, 1.42], respectively) [ 55 , 58 ]. In ADHD, placebo effect size was medium-to-large for clinician-rated outcomes (SMC = 0.75, 95% CI [0.67, 0.83]) [ 62 ]. There was additionally a significant negative relationship between placebo effect size and drug-placebo difference (−0.56, p  < 0.01) for self-rated outcomes [ 62 ]. In schizophrenia spectrum disorders, placebo effect size was small-to-medium in antipsychotic RCTs (SMC = 0.33, 95% CI [0.22, 0.44]) [ 38 ] and medium in RCTs focusing specifically on negative symptoms ( d  = 0.64, 95% CI [0.46, 0.83]) [ 60 ]. Placebo effect size in RLS was large when measured via rating scales ( g  = 1.41, 95% CI [1.25, 1.56]), but small ( g  = 0.02 to 0.24) in RCTs using objective outcomes [ 68 ]. In autism, placebo effect sizes were small (SMC ranged 0.23 to 0.36) [ 63 ]. Similarly, placebo effect size was small in OCD ( d  = 0.32, 95% CI [0.22, 0.41]), although larger in children and adolescents ( d  = 0.45, 95% CI [0.35, 0.56]) compared with adults ( d  = 0.27, 95% CI [0.15, 0.38]) [ 66 ]. Placebo effect size was large in AUD ( g  = 0.90, 95% CI [0.70, 1.09]) [ 61 ], small in primary insomnia ( g ranged 0.25 to 0.43) [ 67 ], and medium in intellectual disability related to genetic causes ( g  = 0.47, 95% CI [0.18, 0.76]) [ 65 ].

Our second objective was to examine the correlates of increased placebo response. We included 14 meta-analyses that reported correlates of placebo effect size or response rate through correlation analysis or meta-regression [ 38 , 51 , 53 , 54 , 56 , 57 , 59 , 60 , 61 , 62 , 63 , 64 , 66 , 68 ]. The key correlates extracted from these studies are summarized in Table  2 .

Several variables were consistently identified across meta-analyses. Increased number of trial sites was a positive correlate of increased placebo response in MDD [ 51 , 54 ], schizophrenia spectrum disorders [ 59 ], and autism spectrum disorders [ 63 ]. Similarly, increased sample size was positively associated with placebo effect size in schizophrenia spectrum disorders [ 59 ], OCD [ 66 ], and panic disorder [ 56 ]. Later publication or study year was associated with greater placebo response in anxiety disorders [ 56 , 57 ], schizophrenia spectrum disorders [ 38 ], AUD [ 61 ], and OCD [ 66 ] but not in MDD [ 51 ], and with reduced placebo response in ADHD [ 62 ]. Younger age was associated with increased placebo responses in schizophrenia spectrum disorders [ 38 , 59 ] and OCD [ 66 ]. Increased baseline illness severity was associated with increased placebo response in schizophrenia spectrum disorders [ 38 ], ADHD [ 62 ], and AUD [ 61 ]. Increased trial or follow-up duration was positively associated with increased placebo response in MDD [ 51 ], but negatively associated with placebo response in schizophrenia spectrum disorders [ 38 , 60 ] and OCD [ 66 ]. Finally, the effect size of active treatment was positively associated with increased placebo response in neurostimulation trials for MDD [ 53 ], bipolar depression [ 64 ], autistic spectrum disorders [ 63 ], and ADHD [ 62 ].

There were also some variables associated with increased placebo response in single disorders only. Flexible dosing, rather than fixed dosing, was associated with increased placebo response in MDD [ 51 ]. Increased illness duration was associated with reduced placebo response in schizophrenia spectrum disorders [ 38 ]. In RCTs for negative symptoms of schizophrenia, a higher number of active treatment arms was associated with increased placebo response [ 60 ]. A number of treatment administrations was a positive correlate of increased placebo response in patients with AUD [ 61 ]. A low risk of bias in selective reporting was associated with increased placebo response in ADHD [ 62 ]. Finally, a low risk of bias in allocation concealment was associated with increased placebo response in autism [ 63 ].

To our knowledge, this is the first overarching synthesis of the literature exploring the placebo response in RCTs of biological treatments across a broad range of mental disorders. We found that placebo responses were present and detectable across mental disorders. Further, the placebo effect size across these disorders varied between small and large (see Fig.  3 ). Additionally, several variables appeared to be associated with increased placebo effect size or response rate across a number of disorders, while others were reported for individual disorders only.

CI confidence interval, MDD major depressive disorder, GAD generalized anxiety disorder, SAD social anxiety disorder, OCD obsessive-compulsive disorder, g Hedges’ g, d Cohen’s d, SMC standardized mean change.

Our umbrella review distinguishes itself from a recent publication on placebo mechanisms across medical conditions [ 69 ]. Only four systematic reviews of research in mental disorders were included in that recent review [ 69 ], none of which were eligible for inclusion in our umbrella review, as we focus specifically on RCTs in mental disorders. Thus, our current umbrella review synthesizes different literature and is complementary [ 69 ].

We found substantial variation in placebo effect sizes across mental disorders. In GAD, SAD, MDD, AUD, and RLS (for subjective outcomes), placebo effects were large (>0.9), while they were small (approximately 0.3) in OCD, primary insomnia, autism, RLS (for objective outcomes), and schizophrenia spectrum disorders. It is noteworthy that placebo effect size/response rate correlated with active treatment effect size/response rate in many disorders (MDD, bipolar depression, ADHD, and autism). Nonetheless, where reported, active treatment was always superior. This possibly suggests an underlying ‘treatment responsiveness’ of these disorders that can vary in size. Perhaps, the natural history of a disorder is an important factor in ‘responsiveness’, i.e., disorders in which there is greater natural fluctuation in severity will show larger placebo (and active treatment) effect sizes. Supporting this hypothesis, increased trial duration predicted a larger placebo effect size in MDD, a disorder in which the natural course includes improvement [ 31 , 51 , 70 ]. Conversely, in schizophrenia spectrum disorders where improvement (particularly of negative symptoms) is less likely [ 71 ], increased trial and illness duration predicted a smaller placebo effect size [ 38 , 60 ]. However, previous meta-analyses suggest that natural improvement, for example, measured via waiting list control, does not fully account for the placebo effect in depression and anxiety disorders [ 72 , 73 ]. Statistical artifact, therefore, does not seem to fully explain the variation in effect size.

Non-specific treatment mechanisms are likely an additional source of the observed placebo effect. For example, those with treatment-resistant illness might have reduced expectations regarding treatment. This assumption is supported by the subgroup analysis reported by Razza and colleagues showing sham neuromodulation efficacy reduced as the number of previous failed antidepressant trials increased [ 53 ]. Another factor to consider is the outcome measure chosen. For example, the placebo effect size in panic disorder was smaller when calculated with objective or self-report measures compared with clinician-rated measures [ 56 ]. A similar finding was reported in ADHD trials [ 62 ]. Why placebo effect sizes would differ with clinician-rated versus self-rated scales is unclear. This might result from ‘demand characteristics’ (i.e., cues that suggest to a patient how they ‘should’ respond), or unblinding of the rater, or a combination of the two [ 74 , 75 ].

Several correlates of increased placebo response were reported in included meta-analyses. These included a larger sample size, more study sites, a later publication year (but with an opposite finding for ADHD), younger age, and increased baseline illness severity. This might reflect changes in clinical trial methods over time, the potential for increased ‘noise’ in the data with larger samples or more study sites, and, more speculatively, variables associated with increased volatility in symptoms [ 39 , 51 , 76 ]. A more extensive discussion regarding the potential reasons these variables might correlate with, or predict, placebo response is included in the eDiscussion. Although some correlates of increased placebo response were identified, perhaps more pertinently, it is unknown whether these also predict the separation between active treatment and placebo in most mental disorders. Three included meta-analyses did show that as placebo response increases, the likelihood of drug-placebo separation decreases [ 38 , 62 , 64 ]. This suggests correlates of placebo effect size are also correlates of trial success or failure, but this hypothesis needs explicit testing. In addition, few of the meta-analyses we included explored whether correlates of placebo response differed from correlates of active treatment response. For example, in clinical trials for gambling disorder, response to active treatment was predicted by weeks spent in the trial and by baseline severity, while response to placebo was predicted by baseline depressive and anxiety symptoms [ 77 ]. Furthermore, there is evidence that industry sponsorship is a specific correlate of reduced drug-placebo separation in schizophrenia spectrum disorders [ 78 ]. The largest meta-analysis that we included (conducted by Scott et al. [ 50 ]) did not explore correlates of increased placebo response through meta-regression analysis; rather, it was designed specifically to assess the impact of the use of placebo run-in periods in antidepressant trials. The authors found that use of a placebo run-in was associated with reduced placebo response. However, this effect did not enhance sensitivity to detect medication efficacy versus control groups, as trials with placebo run-in periods were also associated with a reduced medication response. Similar effects of placebo run-in were seen in univariate (but not multivariable) models in ADHD, where placebo run-in reduced placebo effect size in youth, but did not affect drug vs placebo difference [ 62 ]. Further work should be undertaken to ascertain whether trial-level correlates (including the use of placebo run-in) differentially explain active treatment or placebo response and whether controlling for these can improve drug-placebo separation.

Our results should be considered in the light of several possible limitations. First, as in any umbrella review, we were limited by the quality of the meta-analyses we included. Our AMSTAR-2 ratings suggest that confidence in the conclusions of most included meta-analyses should be critically low or low. Indeed, several meta-analyses did not assess for publication bias or for bias in included RCTs. This is relevant, as the risk of bias in selective reporting was highlighted as potentially being associated with placebo effect size in ADHD [ 62 ], and might therefore be relevant in other mental disorders. Second, our results are potentially vulnerable to biases or unmeasured confounders present in the included meta-analyses. Third, we attempted to prevent overlap and duplication of information by including only the meta-analyses with the most information. This might, however, have resulted in some data not being included in our synthesis. Fourth, an exploration of the potential clinical relevance of the placebo effect sizes reported here was outside the scope of the current review but should be considered an important question for future research. Finally, the meta-analyses we included encompassed RCTs with different levels of blinding (double-blind, single-blind). Although the majority of trials were likely double-blind, it is possible that different levels of blinding could have influenced placebo effect sizes through effects on expectations. Future analyses of placebo effects and their correlates should either focus on double-blind trials or compare results across levels of blinding. Related to this, the included meta-analyses pooled phase 2 and phase 3 trials (the latter of which will usually follow positive phase 2 trials), which might result in different expectation biases. Therefore, placebo effects should be compared between phase 2 and phase 3 trials in the future.

In this umbrella review, we found placebo effect sizes varied substantially across mental disorders. The sources of this variation remain unknown and require further study. Some variables were correlates of increased placebo response across mental disorders, including larger sample size, higher number of study sites, later publication year (opposite for ADHD), younger age, and increased baseline illness severity. There was also evidence that clinician-rated outcomes were associated with larger placebo effect sizes than self-rated or objective outcomes. We additionally identified important gaps in the literature, with no eligible systematic reviews identified in stress-related disorders, eating disorders, behavioural addictions, or bipolar mania. In relation to these disorders, some analyses have been published but they have not been included in systematic reviews/meta-analyses (e.g. analyses of individual patient data pooled across RCTs in acute mania [ 79 ] or gambling disorder [ 77 , 80 ]) and therefore were not eligible for inclusion here. We also focused on placebo response in RCTs of pharmacotherapies and neurostimulation interventions for mental disorders. We did not include placebo effects in psychosocial interventions, but such an analysis would also be valuable. Future studies should address these gaps in the literature and furthermore should compare findings in placebo arms with active treatment arms, both regarding treatment effect size and its correlates. Gaining additional insights into the placebo response may improve our ability to separate active treatment effects from placebo effects, thus paving the way for potentially effective new treatments for mental disorders.

Data availability

The datasets generated during and/or analysed during the current study are available in the Open Science Framework repository, https://osf.io/fxvn4/ .

Evers AWM, Colloca L, Blease C, Annoni M, Atlas LY, Benedetti F, et al. Implications of placebo and nocebo effects for clinical practice: Expert Consensus. Psychother Psychosom. 2018;87:204–10.

Article   PubMed   Google Scholar  

McQueen D, Cohen S, John-Smith PS, Rampes H. Rethinking placebo in psychiatry: the range of placebo effects. Adv Psychiatr Treat. 2013;19:162–70.

Article   Google Scholar  

Beecher HK. The powerful placebo. J Am Med Assoc. 1955;159:1602–6.

Article   CAS   PubMed   Google Scholar  

Harris I. When the placebo effect is not an effect. Acta Orthop. 2021;92:501–2.

Article   PubMed   PubMed Central   Google Scholar  

Landin R, DeBrota DJ, DeVries TA, Potter WZ, Demitrack MA. The impact of Restrictive Entry Criterion during the placebo lead-in period. Biometrics. 2000;56:271–8.

Jones BDM, Razza LB, Weissman CR, Karbi J, Vine T, Mulsant LS, et al. Magnitude of the Placebo response across treatment modalities used for treatment-resistant depression in adults: a systematic review and meta-analysis. JAMA Netw Open. 2021;4:e2125531.

Miller FG, Rosenstein DL. The nature and power of the placebo effect. J Clin Epidemiol. 2006;59:331–5.

Ashar YK, Chang LJ, Wager TD. Brain mechanisms of the Placebo effect: an affective appraisal account. Annu Rev Clin Psychol. 2017;13:73–98.

Ernst E, Resch KL. Concept of true and perceived placebo effects. BMJ. 1995;311:551–3.

Article   CAS   PubMed   PubMed Central   Google Scholar  

De La Fuente-Fernandez R. Expectation and Dopamine release: mechanism of the Placebo effect in Parkinson’s disease. Science. 2001;293:1164–6.

Benedetti F, Carlino E, Pollo A. How Placebos change the patient’s brain. Neuropsychopharmacology. 2011;36:339–54.

Goebel MU, Trebst AE, Steiner J, Xie YF, Exton MS, Frede S, et al. Behavioral conditioning of immunosuppression is possible in humans. FASEB J. 2002;16:1869–73.

Albring A, Wendt L, Benson S, Witzke O, Kribben A, Engler H, et al. Placebo effects on the immune response in humans: the role of learning and expectation. PloS One. 2012;7:e49477.

Lidstone SC, Schulzer M, Dinelle K, Mak E, Sossi V, Ruth TJ, et al. Effects of expectation on placebo-induced Dopamine release in Parkinson disease. Arch Gen Psychiatry. 2010;67:857–65.

Amanzio M, Benedetti F. Neuropharmacological dissection of placebo analgesia: expectation-activated opioid systems versus conditioning-activated specific subsystems. J Neurosci. 1999;19:484–94.

Amanzio M, Pollo A, Maggi G, Benedetti F. Response variability to analgesics: a role for non-specific activation of endogenous opioids. Pain. 2001;90:205–15.

Huneke NTM, Aslan IH, Fagan H, Phillips N, Tanna R, Cortese S, et al. Functional neuroimaging correlates of placebo response in patients with depressive or anxiety disorders: A systematic review. Int J Neuropsychopharmacol. 2022;25:433–47.

Vase L, Riley JL, Price DD. A comparison of placebo effects in clinical analgesic trials versus studies of placebo analgesia. Pain. 2002;99:443–52.

Solmi M, Croatto G, Piva G, Rosson S, Fusar-Poli P, Rubio JM, et al. Efficacy and acceptability of psychosocial interventions in schizophrenia: systematic overview and quality appraisal of the meta-analytic evidence. Mol Psychiatry. 2023;28:354–68.

Monteleone AM, Pellegrino F, Croatto G, Carfagno M, Hilbert A, Treasure J, et al. Treatment of eating disorders: A systematic meta-review of meta-analyses and network meta-analyses. Neurosci Biobehav Rev. 2022;142:104857.

Rosson S, de Filippis R, Croatto G, Collantoni E, Pallottino S, Guinart D, et al. Brain stimulation and other biological non-pharmacological interventions in mental disorders: An umbrella review. Neurosci Biobehav Rev. 2022;139:104743.

Correll CU, Cortese S, Croatto G, Monaco F, Krinitski D, Arrondo G, et al. Efficacy and acceptability of pharmacological, psychosocial, and brain stimulation interventions in children and adolescents with mental disorders: an umbrella review. World Psychiatry. 2021;20:244–75.

Gaynes BN, Warden D, Trivedi MH, Wisniewski SR, Fava M, Rush AJ. What did STAR*D teach us? Results from a large-scale, practical, clinical trial for patients with depression. Psychiatr Serv. 2009;60:1439–45.

Stone MB, Yaseen ZS, Miller BJ, Richardville K, Kalaria SN, Kirsch I. Response to acute monotherapy for major depressive disorder in randomized, placebo controlled trials submitted to the US Food and Drug Administration: individual participant data analysis. BMJ. 2022;378:e067606.

Hendriks SM, Spijker J, Licht CMM, Hardeveld F, de Graaf R, Batelaan NM, et al. Long-term disability in anxiety disorders. BMC Psychiatry. 2016;16:248.

Dragioti E, Solmi M, Favaro A, Fusar-Poli P, Dazzan P, Thompson T, et al. Association of antidepressant use with adverse health outcomes: a systematic umbrella review. JAMA Psychiatry. 2019;76:1241–55.

Croatto G, Vancampfort D, Miola A, Olivola M, Fiedorowicz JG, Firth J, et al. The impact of pharmacological and non-pharmacological interventions on physical health outcomes in people with mood disorders across the lifespan: An umbrella review of the evidence from randomised controlled trials. Mol Psychiatry. 2023;28:369–90.

Papola D, Ostuzzi G, Gastaldon C, Morgano GP, Dragioti E, Carvalho AF, et al. Antipsychotic use and risk of life-threatening medical events: umbrella review of observational studies. Acta Psychiatr Scand. 2019;140:227–43.

Linden M. How to define, find and classify side effects in psychotherapy: from unwanted events to adverse treatment reactions. Clin Psychol Psychother. 2013;20:286–96.

Reynolds GP, Kirk SL. Metabolic side effects of antipsychotic drug treatment – pharmacological mechanisms. Pharmacol Ther. 2010;125:169–79.

Cuijpers P, Karyotaki E, Weitz E, Andersson G, Hollon SD, van Straten A. The effects of psychotherapies for major depression in adults on remission, recovery and improvement: A meta-analysis. J Affect Disord. 2014;159:118–26.

Bloom DE, Cafiero E, Jané-Llopis E, Abrahams-Gessel S, Bloom LR, Fathima S, et al. The global economic burden of noncommunicable diseases. PGDA Work Pap. (2012).

Vos T, Lim SS, Abbafati C, Abbas KM, Abbasi M, Abbasifard M, et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396:1204–22.

Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. The Lancet. 2013;382:1575–86.

Huneke NTM, van der Wee N, Garner M, Baldwin DS. Why we need more research into the placebo response in psychiatry. Psychol Med. 2020;50:2317–23.

Huneke NTM. Is superiority to placebo the most appropriate measure of efficacy in trials of novel psychotropic medications? Eur Neuropsychopharmacol. 2022;62:7–9.

Khan A, Brown WA. Antidepressants versus placebo in major depression: An overview. World Psychiatry. 2015;14:294–300.

Agid O, Siu CO, Potkin SG, Kapur S, Watsky E, Vanderburg D, et al. Meta-regression analysis of placebo response in antipsychotic trials, 1970–2010. Am J Psychiatry. 2013;170:1335–44.

Leucht S, Leucht C, Huhn M, Chaimani A, Mavridis D, Helfer B, et al. Sixty years of placebo-controlled antipsychotic drug trials in acute schizophrenia: systematic review, bayesian meta-analysis, and meta-regression of efficacy predictors. Am J Psychiatry. 2017;174:927–42.

Enck P, Bingel U, Schedlowski M, Rief W. The placebo response in medicine: minimize, maximize or personalize? Nat Rev Drug Discov. 2013;12:191–204.

Correll CU, Solmi M, Cortese S, Fava M, Højlund M, Kraemer HC, et al. The future of psychopharmacology: a critical appraisal of ongoing phase 2/3 trials, and of some current trends aiming to de-risk trial programmes of novel agents. World Psychiatry. 2023;22:48–74.

Stahl SM, Greenberg GD. Placebo response rate is ruining drug development in psychiatry: why is this happening and what can we do about it? Acta Psychiatr Scand. 2019;139:105–7.

Nikolakopoulou A, Chaimani A, Furukawa TA, Papakonstantinou T, Rücker G, Schwarzer G. When does the placebo effect have an impact on network meta-analysis results? BMJ Evid-Based Med. 2023. https://doi.org/10.1136/bmjebm-2022-112197 .

Huneke NTM, Amin J, Baldwin DS, Chamberlain SR, Correll CU, Garner M, et al. Placebo effects in mental health disorders: protocol for an umbrella review. BMJ Open. 2023;13:e073946.

Gauld C, Lopez R, Morin CM, Maquet J, Mcgonigal A, Geoffroy P-A, et al. Why do sleep disorders belong to mental disorder classifications? A network analysis of the “Sleep-Wake Disorders” section of the DSM-5. J Psychiatr Res. 2021;142:153–9.

Köhler-Forsberg O, Stiglbauer V, Brasanac J, Chae WR, Wagener F, Zimbalski K, et al. Efficacy and safety of antidepressants in patients with comorbid depression and medical diseases: an umbrella systematic review and meta-Analysis. JAMA Psychiatry. 2023. https://doi.org/10.1001/jamapsychiatry.2023.2983 .

Belbasis L, Bellou V, Ioannidis JPA. Conducting umbrella reviews. BMJ Med. 2022;1:e000071.

Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates; (1988).

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Scott AJ, Sharpe L, Quinn V, Colagiuri B. Association of Single-blind Placebo Run-in Periods With the Placebo Response in Randomized Clinical Trials of Antidepressants: A Systematic Review and Meta-analysis. JAMA Psychiatry. 2022;79:42.

Furukawa TA, Cipriani A, Atkinson LZ, Leucht S, Ogawa Y, Takeshima N, et al. Placebo response rates in antidepressant trials: a systematic review of published and unpublished double-blind randomised controlled studies. Lancet Psychiatry. 2016;3:1059–66.

Scott F, Hampsey E, Gnanapragasam S, Carter B, Marwood L, Taylor RW, et al. Systematic review and meta-analysis of augmentation and combination treatments for early-stage treatment-resistant depression. J Psychopharmacol. 2023;37:268–78.

Razza LB, Moffa AH, Moreno ML, Carvalho AF, Padberg F, Fregni F, et al. A systematic review and meta-analysis on placebo response to repetitive transcranial magnetic stimulation for depression trials. Prog Neuropsychopharmacol Biol Psychiatry. 2018;81:105–13.

Meister R, Abbas M, Antel J, Peters T, Pan Y, Bingel U, et al. Placebo response rates and potential modifiers in double-blind randomized controlled trials of second and newer generation antidepressants for major depressive disorder in children and adolescents: a systematic review and meta-regression analysis. Eur Child Adolesc Psychiatry. 2020;29:253–73.

Locher C, Koechlin H, Zion SR, Werner C, Pine DS, Kirsch I, et al. Efficacy and safety of selective Serotonin reuptake inhibitors, Serotonin-Norepinephrine Reuptake inhibitors, and placebo for common psychiatric disorders among children and adolescents: a systematic review and meta-analysis. Jama Psychiatry. 2017;74:1011–20.

Ahmadzad-Asl M, Davoudi F, Mohamadi S, Hadi F, Nejadghaderi SA, Mirbehbahani SH, et al. Systematic review and meta-analysis of the placebo effect in panic disorder: Implications for research and clinical practice. Aust N Z J Psychiatry. 2022;56:1130–41.

Bandelow B, Reitt M, Röver C, Michaelis S, Görlich Y, Wedekind D. Efficacy of treatments for anxiety disorders: a meta-analysis. Int Clin Psychopharmacol. 2015;30:183–92.

Pinquart M, Duberstein PR. Treatment of anxiety disorders in older adults: a meta-analytic comparison of behavioral and pharmacological interventions. Am J Geriatr Psychiatry. 2007;15:639–51.

Leucht S, Chaimani A, Leucht C, Huhn M, Mavridis D, Helfer B, et al. 60 years of placebo-controlled antipsychotic drug trials in acute schizophrenia: Meta-regression of predictors of placebo response. Schizophr Res. 2018;201:315–23.

Czobor P, Kakuszi B, Bitter I. Placebo response in trials of negative symptoms in Schizophrenia: A critical reassessment of the evidence. Schizophr Bull. 2022;48:1228–40.

Del Re AC, Maisel N, Blodgett J, Wilbourne P, Finney J. Placebo group improvement in trials of pharmacotherapies for alcohol use disorders: a multivariate meta-analysis examining change over time. J Clin Psychopharmacol. 2013;33:649.

Faraone SV, Newcorn JH, Cipriani A, Brandeis D, Kaiser A, Hohmann S, et al. Placebo and nocebo responses in randomised, controlled trials of medications for ADHD: a systematic review and meta-analysis. Mol Psychiatry. 2022;27:212–9.

Siafis S, Çıray O, Schneider-Thoma J, Bighelli I, Krause M, Rodolico A, et al. Placebo response in pharmacological and dietary supplement trials of autism spectrum disorder (ASD): systematic review and meta-regression analysis. Mol Autism. 2020;11:66.

Iovieno N, Nierenberg AA, Parkin SR, Hyung Kim DJ, Walker RS, Fava M, et al. Relationship between placebo response rate and clinical trial outcome in bipolar depression. J Psychiatr Res. 2016;74:38–44.

Curie A, Yang K, Kirsch I, Gollub RL, des Portes V, Kaptchuk TJ, et al. Placebo responses in genetically determined intellectual disability: a meta-analysis. PloS One. 2015;10:e0133316.

Mohamadi S, Ahmadzad-Asl M, Nejadghaderi SA, Jabbarinejad R, Mirbehbahani SH, Sinyor M, et al. Systematic review and meta-analysis of the placebo effect and its correlates in obsessive compulsive disorder. Can J Psychiatry. 2023;68:479–94.

Winkler A, Rief W. Effect of placebo conditions on polysomnographic parameters in primary insomnia: a meta-analysis. Sleep. 2015;38:925–31.

PubMed   PubMed Central   Google Scholar  

Silva MA, Duarte GS, Camara R, Rodrigues FB, Fernandes RM, Abreu D, et al. Placebo and nocebo responses in restless legs syndrome: A systematic review and meta-analysis. Neurology. 2017;88:2216–24.

Frisaldi E, Shaibani A, Benedetti F, Pagnini F. Placebo and nocebo effects and mechanisms associated with pharmacological interventions: an umbrella review. BMJ Open. 2023;13:e077243.

Cuijpers P, Stringaris A, Wolpert M. Treatment outcomes for depression: challenges and opportunities. Lancet Psychiatry. 2020;7:925–7.

Bromet EJ, Fennig S. Epidemiology and natural history of schizophrenia. Biol Psychiatry. 1999;46:871–81.

Rutherford BR, Mori S, Sneed JR, Pimontel MA, Roose SP. Contribution of spontaneous improvement to placebo response in depression: A meta-analytic review. J Psychiatr Res. 2012;46:697–702.

Fernández-López R, Riquelme-Gallego B, Bueno-Cavanillas A, Khan KS. Influence of placebo effect in mental disorders research: A systematic review and meta-analysis. Eur J Clin Invest. 2022;52:e13762.

Goodwin GM, Croal M, Marwood L, Malievskaia E. Unblinding and demand characteristics in the treatment of depression. J Affect Disord. 2023;328:1–5.

Coles NA, Gaertner L, Frohlich B, Larsen JT, Basnight-Brown DM. Fact or artifact? Demand characteristics and participants’ beliefs can moderate, but do not fully account for, the effects of facial feedback on emotional experience. J Pers Soc Psychol. 2023;124:287–310.

Weimer K, Colloca L, Enck P. Placebo eff ects in psychiatry: mediators and moderators. Lancet Psychiatry. 2015;2:246–57.

Huneke NTM, Chamberlain SR, Baldwin DS, Grant JE. Diverse predictors of treatment response to active medication and placebo in gambling disorder. J Psychiatr Res. 2021;144:96–101.

Leucht S, Chaimani A, Mavridis D, Leucht C, Huhn M, Helfer B, et al. Disconnection of drug-response and placebo-response in acute-phase antipsychotic drug trials on schizophrenia? Meta-regression analysis. Neuropsychopharmacology. 2019;44:1955–66.

Welten CCM, Koeter MWJ, Wohlfarth T, Storosum JG, van den Brink W, Gispen-de Wied CC, et al. Placebo response in antipsychotic trials of patients with acute mania. Eur Neuropsychopharmacol. 2015;25:1018–26.

Grant JE, Chamberlain SR. The placebo effect and its clinical associations in gambling disorder. Ann Clin Psychiatry. 2017;29:167.

Download references

Acknowledgements

Dr Nathan TM Huneke is an NIHR Academic Clinical Lecturer. The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR, NHS, or the UK Department of Health and Social Care. For the purpose of open access, the author has applied a Creative Commons Attribution License (CC BY) to any Author Accepted Manuscript version arising from this submission.

Author contributors

NTMH, JA, DSB, SRC, CUC, MG, CMH, RH, ODH, JMAS, MS, and SCo conceptualized the study. NTMH, AB, VB, LE, CJG, OKF, LM, CR, SS, and SCo contributed to data collection, data curation, or data analysis. NTMH, MS, and SCo wrote the first draft of the manuscript. All authors had access to the raw data. All authors reviewed and edited the manuscript and had final responsibility for the decision to submit it for publication.

Author information

These authors contributed equally: Marco Solmi, Samuele Cortese.

Authors and Affiliations

Clinical and Experimental Sciences, Faculty of Medicine, University of Southampton, Southampton, UK

Nathan T. M. Huneke, Jay Amin, David S. Baldwin, Samuel R. Chamberlain, Matthew Garner, Catherine M. Hill, Ruihua Hou, Konstantinos Ioannidis, Julia M. A. Sinclair & Samuele Cortese

Southern Health NHS Foundation Trust, Southampton, UK

Nathan T. M. Huneke, Jay Amin, David S. Baldwin, Samuel R. Chamberlain, Konstantinos Ioannidis & Satneet Singh

University Department of Psychiatry and Mental Health, University of Cape Town, Cape Town, South Africa

David S. Baldwin

School of Psychology, University of Nottingham Malaysia, Semenyih, Malaysia

Alessio Bellato

Centre for Innovation in Mental Health, School of Psychology, Faculty of Environmental and Life Sciences, University of Southampton, Southampton, UK

Alessio Bellato, Valerie Brandt, Matthew Garner, Corentin J. Gosling, Claire Reed, Marco Solmi & Samuele Cortese

Clinic of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany

Valerie Brandt

Department of Child and Adolescent Psychiatry, Charité Universitätsmedizin Berlin, Berlin, Germany

Christoph U. Correll

Department of Psychiatry, Zucker Hillside Hospital, Northwell Health, Glen Oaks, NY, USA

Department of Psychiatry and Molecular Medicine, Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY, USA

Center for Psychiatric Neuroscience, Feinstein Institute for Medical Research, Manhasset, NY, USA

Faculty of Education and Psychology, University of Navarra, Pamplona, Spain

Luis Eudave

School of Psychology, Faculty of Environmental and Life Sciences, University of Southampton, Southampton, UK

Matthew Garner

Université Paris Nanterre, DysCo Lab, F-92000, Nanterre, France

Corentin J. Gosling

Université de Paris, Laboratoire de Psychopathologie et Processus de Santé, F-92100, Boulogne-Billancourt, France

Department of Sleep Medicine, Southampton Children’s Hospital, Southampton, UK

Catherine M. Hill

Department of Psychosis Studies, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK

Oliver D. Howes

H Lundbeck A/s, Iveco House, Watford, UK

Institute of Clinical Sciences (ICS), Faculty of Medicine, Imperial College London, London, UK

Department of Clinical Medicine, Aarhus University, Aarhus, Denmark

Ole Köhler-Forsberg

Psychosis Research Unit, Aarhus University Hospital–Psychiatry, Aarhus, Denmark

Department of Translational Biomedicine and Neuroscience (DIBRAIN), University of Studies of Bari “Aldo Moro”, Bari, Italy

Lucia Marzulli

Department of Psychiatry, University of Ottawa, Ottawa, ON, Canada

Marco Solmi

Department of Mental Health, Ottawa Hospital, Ottawa, ON, Canada

Ottawa Hospital Research Institute (OHRI) Clinical Epidemiology Program, University of Ottawa, Ottawa, ON, Canada

School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada

Solent NHS Trust, Southampton, UK

Samuele Cortese

DiMePRe-J-Department of Precision and Regenerative Medicine-Jonic Area, University “Aldo Moro”, Bari, Italy

Hassenfeld Children’s Hospital at NYU Langone, New York University Child Study Center, New York, NY, USA

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Nathan T. M. Huneke .

Ethics declarations

Competing interests.

DSB is President of the British Association for Psychopharmacology, Editor of the Human Psychopharmacology journal (for which he receives an editor’s honorarium), and has received royalties from UpToDate. CMH has acted on an expert advisory board for Neurim Pharmaceuticals. ODH is a part-time employee and stockholder of Lundbeck A/s. He has received investigator-initiated research funding from and/or participated in advisory/speaker meetings organized by Angellini, Autifony, Biogen, Boehringer-Ingelheim, Eli Lilly, Heptares, Global Medical Education, Invicro, Jansenn, Lundbeck, Neurocrine, Otsuka, Sunovion, Recordati, Roche and Viatris/Mylan. ODH has a patent for the use of dopaminergic imaging. All other authors declare no competing interests. MS has received honoraria/has been a consultant for Angelini, Lundbeck, and Otsuka. SCo has received honoraria from non-profit associations (BAP, ACAMH, CADDRA) for educational activities and an honorarium from Medice. KI has received honoraria from Elsevier for editorial work. SRC receives honoraria from Elsevier for associate editor roles at comprehensive psychiatry and NBR journals. CUC has been a consultant and/or advisor to or has received honoraria from: AbbVie, Acadia, Adock Ingram, Alkermes, Allergan, Angelini, Aristo, Biogen, Boehringer-Ingelheim, Bristol-Meyers Squibb, Cardio Diagnostics, Cerevel, CNX Therapeutics, Compass Pathways, Darnitsa, Denovo, Gedeon Richter, Hikma, Holmusk, IntraCellular Therapies, Jamjoom Pharma, Janssen/J&J, Karuna, LB Pharma, Lundbeck, MedAvante-ProPhase, MedInCell, Merck, Mindpax, Mitsubishi Tanabe Pharma, Mylan, Neurocrine, Neurelis, Newron, Noven, Novo Nordisk, Otsuka, Pharmabrain, PPD Biotech, Recordati, Relmada, Reviva, Rovi, Sage, Seqirus, SK Life Science, Sumitomo Pharma America, Sunovion, Sun Pharma, Supernus, Takeda, Teva, Tolmar, Vertex, and Viatris. He provided expert testimony for Janssen and Otsuka. He served on a Data Safety Monitoring Board for Compass Pathways, Denovo, Lundbeck, Relmada, Reviva, Rovi, Supernus, and Teva. He has received grant support from Janssen and Takeda. He received royalties from UpToDate and is also a stock option holder of Cardio Diagnostics, Kuleon Biosciences, LB Pharma, Mindpax, and Quantic.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

41380_2024_2638_moesm1_esm.docx.

PLACEBO EFFECTS IN RANDOMIZED TRIALS OF PHARMACOLOGICAL AND NEUROSTIMULATION INTERVENTIONS FOR MENTAL DISORDERS: AN UMBRELLA REVIEW SUPPLEMENTARY APPENDIX

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Huneke, N.T.M., Amin, J., Baldwin, D.S. et al. Placebo effects in randomized trials of pharmacological and neurostimulation interventions for mental disorders: An umbrella review. Mol Psychiatry (2024). https://doi.org/10.1038/s41380-024-02638-x

Download citation

Received : 01 February 2024

Revised : 17 June 2024

Accepted : 19 June 2024

Published : 24 June 2024

DOI : https://doi.org/10.1038/s41380-024-02638-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Search NORC

Enter Search Value

Student Absenteeism Meta-Analysis

• Client Institute of Education Sciences, Smith Richardson Foundation
• Dates 2024 – 2025

The pandemic exacerbated student absenteeism.

Chronic student absenteeism, characterized as missing at least 10 percent of the school year, has increased markedly in the U.S. since the COVID-19 pandemic. According to the Annie E. Casey Foundation, it doubled from 15% in 2018–19 to 30% in the 2021–22 school year. Chronic absenteeism can have profound effects on student academic performance, social development, and overall well-being. To help students attend school regularly, policymakers and school leaders need to identify and implement evidence-based approaches that work in diverse contexts.  

NORC will synthesize data from multiple studies to estimate the effectiveness of absenteeism interventions. 

NORC will conduct a comprehensive literature review of absenteeism interventions that are used in the U.S. from preschool through high school. It will include studies published in English between 2016 through the present day. That period follows the passage of the Every Student Succeeds Act in 2015, which shone a light on school attendance by requiring states to include an additional “non-achievement” indicator in their accountability reporting. Currently, more than 80 percent of states have chosen an attendance metric as the additional indicator. 

NORC’s review includes the following factors from the studies:

• Setting, such as urban, suburban, or rural
• Grade levels, such as preK, elementary, middle school, high school or mixed 
• Whether the study had been peer-reviewed
• Whether the study was a randomized control trial

We will use meta-analytic methods to quantitatively estimate the effectiveness of different types of absenteeism interventions. 

This study will help identify effective absenteeism interventions.

The findings will identify which types of interventions may work best for varied settings and student populations.

Related Tags

Departments, centers & programs.

• Education & Child Development

Research Divisions

• NORC Education
• Advanced Quantitative Methods
• Elementary & Secondary Education
• Early Care & Education
• Policy, Programs & Evaluation

Project Leads

Vi-nhuan le.

Want to work with us?

Explore NORC Education Projects

A comprehensive analysis and meta-analytic synthesis of student absenteeism interventions

Fulton County Principal Leadership Program Evaluation

A district-level analysis of an effort to train experienced principals to be mentors

Supporting the Illinois Equity in Attainment Initiative

Using a facilitated collaborative to improve racial equity in higher-education organizations

What is the effectiveness of methods for eradicating or controlling abundance and biomass of invasive aquatic plants in Canada? A systematic review protocol

• Harper, Meagan
• Rytwinski, Trina
• Irvine, Robyn
• Cooke, Steven J.

Invasive freshwater aquatic plants can have adverse ecological effects on the systems to which they are introduced, changing ecosystem function, threatening native plant species and causing billions of dollars in damage to infrastructure. Additionally, once established, invasive aquatic plants are often difficult to eradicate or control. Given the importance of managing invasive aquatic plants, and the high associated economic costs of doing so, it is essential to determine the relative effectiveness of different control methods. Here, we present a protocol for a systematic review that will estimate the effectiveness of various biological, chemical, habitat manipulations and/or manual/mechanical methods for eradicating or controlling invasive plant abundance and biomass. This systematic review will use published and grey literature, without date restriction, that determines the effectiveness of invasive plant control methods. English‑language searches will be performed using five bibliographic databases, Google Scholar, and networking tools to find relevant literature. Eligibility screening will be conducted at two stages: (1) title and abstract and (2) full text. Studies that evaluate the effectiveness of methods for controlling the abundance or biomass or eradicating invasive plants will be included. A list of plant species currently, or potentially, in Canadian freshwater systems and of management concern will be considered. Included studies will undergo critical appraisal of internal study validity. We will extract information on study characteristics, intervention and comparator details, measured outcomes (abundance and biomass, broadly defined) and effect modifiers (e.g., plant growth pattern or timing of treatments). A narrative synthesis will be used to describe the quantity and characteristics of the evidence base, while quantitative synthesis (i.e., meta‑analysis) will be conducted to estimate an overall mean and variance of effect when sufficient numbers of similar studies are available.

• Скидки дня
• Справка и помощь
• Адрес доставки Идет загрузка... Ошибка: повторите попытку ОК
• Продажи
• Список отслеживания Развернуть список отслеживаемых товаров Идет загрузка... Войдите в систему , чтобы просмотреть свои сведения о пользователе
• Краткий обзор
• Недавно просмотренные
• Ставки/предложения
• Список отслеживания
• История покупок
• Купить опять
• Объявления о товарах
• Сохраненные запросы поиска
• Сохраненные продавцы
• Сообщения
• Уведомление
• Развернуть корзину Идет загрузка... Произошла ошибка. Чтобы узнать подробнее, посмотрите корзину.

Oops! Looks like we're having trouble connecting to our server.

Refresh your browser window to try again.

Health care accessibility and mobility in breast cancer: a Latin American perspective

• André Ferreira Azeredo-da-Silva 1 ,
• Bruna Stella Zanotto 1 ,
• Flavia Martins 2 ,
• Nádia Navarro 2 ,
• Rafaela Alencar 2 &
• Clarissa Medeiros 2  

BMC Health Services Research volume  24 , Article number:  764 ( 2024 ) Cite this article

62 Accesses

Metrics details

Latin America (LATAM) encompasses a vast region with diverse populations. Despite publicly funded health care systems providing universal coverage, significant socioeconomic and ethno-racial disparities persist in health care access across the region. Breast cancer (BC) incidence and mortality rates in Brazil are comparable to those in other LATAM countries, supporting the relevance of Brazilian data, with Brazil’s health care policies and expenditures often serving as models for neighboring countries. We evaluated the impact of mobility on oncological outcomes in LATAM by analyzing studies of patients with BC reporting commuting routes or travel distances to receive treatment or diagnosis.

We searched MEDLINE (PubMed), Embase, Cochrane CENTRAL, LILACS, and Google Scholar databases. Studies eligible for inclusion were randomized controlled trials and observational studies of patients with BC published in English, Portuguese, or Spanish and conducted in LATAM. The primary outcome was the impact of mobility or travel distance on oncological outcomes. Secondary outcomes included factors related to mobility barriers and access to health services. For studies meeting eligibility, relevant data were extracted using standardized forms. Risk of bias was assessed using the Newcastle-Ottawa Scale. Quantitative and qualitative evidence synthesis focused on estimating travel distances based on available data. Heterogeneity across distance traveled or travel time was addressed by converting reported travel time to kilometers traveled and estimating distances for unspecified locations.

Of 1142 records identified, 14 were included (12 from Brazil, 1 from Mexico, and 1 from Argentina). Meta-analysis revealed an average travel distance of 77.8 km (95% CI, 49.1-106.48) to access BC-related diagnostic or therapeutic resources. Nonetheless, this average fails to precisely encapsulate the distinct characteristics of each region, where notable variations persist in travel distance, ranging from 88 km in the South to 448 km in the North.

The influence of mobility and travel distance on access to BC care is multifaceted and should consider the complex interplay of geographic barriers, sociodemographic factors, health system issues, and policy-related challenges. Further research is needed to comprehensively understand the variables impacting access to health services, particularly in LATAM countries, where the challenges women face during treatment remain understudied.

Trial registration

CRD42023446936.

Peer Review reports

Limited geographic access to health facilities is a major factor contributing to reduced utilization of health services, resulting in poorer health outcomes [ 1 ]. This issue is particularly critical in the context of patients with cancer, as their treatment typically involves a combination of surgery, radiotherapy, and/or chemotherapy, often requiring multiple visits to health facilities. Geographic barriers that impede access may delay treatment initiation, leading to suboptimal outcomes or even premature and preventable deaths. The burden of travel demands on patients with cancer has been linked to more advanced disease at diagnosis, flawed treatment, a grimmer prognosis, and diminished quality of life [ 2 ].

Brazil’s breast cancer (BC) incidence and mortality rates are comparable to those of other Latin American countries. For instance, the age-standardized incidence rates of BC per 100,000 women are 62.9 in Brazil, 61.1 in Argentina, and 49.6 in Mexico, illustrating that Brazil’s epidemiological data are within the regional range [ 3 ]. This epidemiological consistency supports the relevance of Brazilian data to the broader Latin American context. As Latin America’s largest economy, Brazil’s health care policies and expenditures influence regional trends and often serve as models for neighboring countries [ 4 ].

In Brazil, a country of continental dimensions, more than half of patients with cancer are required to travel from their hometown to another city to receive treatment, with persistent disparities in regional accessibility despite the shorter travel distances recently observed in some states [ 2 , 5 , 6 , 7 ]. For example, there are 359 dedicated public treatment centers with asymmetric geographical distribution, where approximately 80% are located in 2 of the 5 Brazilian regions and 20% in the remaining regions [ 8 , 9 ]. Patients with cancer who must commute for treatment face considerable challenges, including fatigue, long waiting times for their return trip, inadequate nourishment, financial constraints due to travel expenses, and disruption to daily life [ 10 ]. Radiotherapy and chemotherapy are of particular concern as they require frequent visits to cancer care facilities.

Low- and middle-income countries find themselves in diverse circumstances with respect to workforce capacity, regulation of private health care, public sector investment, care pathways, and the ineffectiveness of comprehensive national strategies for the development, management, sustainable financing, and accreditation of cancer care centers. Therefore, identifying issues of geographic mobility for patients with cancer in Latin America is important to ensure equitable access to care.

In this systematic review and meta-analysis, we aimed to evaluate the impact of mobility on oncological outcomes in Latin America by analyzing studies of patients with BC reporting their commuting routes or travel distances to receive treatment or diagnosis. We addressed 2 knowledge gaps: (1) whether BC treatment or screening programs have been made geographically accessible to patients in Latin American countries, and (2) whether the existing literature can provide regional estimates of travel distances to health facilities. The paper contributes to the worldwide debate on how to widen access to BC care and may pave the way for further developments and studies on the topic, while providing relevant data to the strategic planning of cancer care services.

We developed this systematic review according to the PRISMA 2020 guidelines [ 11 ] and the recommendations proposed by the Cochrane Collaboration [ 12 ]. A detailed review protocol is available at PROSPERO (CRD42023446936).

Search strategy

We searched MEDLINE (via PubMed), Embase, Cochrane Central Register of Controlled Trials (Cochrane CENTRAL), Latin American and Caribbean Health Sciences Literature (LILACS, via Virtual Health Library), and Google Scholar databases for articles published from inception to June 28, 2023, by entering the following keywords and terms individually, including index terms (MeSH and Emtree terms), subject indexes, and synonyms, or by combining them with Boolean operators (“AND” and “OR”): “Breast cancer,” “Breast neoplasm,” “Mobility,” “Access to healthcare,” and “Latin America.” Terms related to intervention or study design were not used to improve the search sensitivity. Although no language restrictions were imposed, we only considered articles published in English, Portuguese, or Spanish. We hand searched the reference lists of the included studies and of all reviews published to date on the topic to cover potential additional studies within the intended scope. The complete search strategy is provided in Additional Table  1 . A cross-reference check to locate and eliminate duplicates complemented the search strategy.

Eligibility criteria and study selection

Studies eligible for inclusion in this review were published in English, Portuguese, or Spanish and recruited patients with BC in Latin American countries. The study designs considered for inclusion were randomized controlled trials and observational studies (cohort, cross-sectional, case control, case series, or ecological studies) with or without a comparison group, regardless of the intervention used. We excluded conference abstracts, guidelines, editorials, book chapters, commentaries, letters, notes, and study protocols.

We limited the scope of the review to Latin America because we intended to explore mobility-related factors alongside health care resource utilization in Latin American populations. Furthermore, this decision stemmed from the shared health patterns observed in Latin American countries, characterized by popular-collective health care, recurrent discontinuity in public policies—an inherent feature of the region—and a prevailing culture of prioritizing urgency in professional endeavors.

Studies were considered for inclusion if they clearly reported the travel distance (in kilometers or other units) or time (in hours, minutes) required to access BC-related health care. The primary outcome of this review was the impact of mobility or travel distance on oncological outcomes such as mortality, time to treatment initiation, and time to diagnosis. Secondary outcomes included mobility-related factors such as geographic barriers, access to municipal transportation, and travel time.

Data extraction process

After removal of duplicates, 2 reviewers (AFA and BSZ) independently screened titles and abstracts, and then screened potentially eligible or candidate full-text articles for selection based on the inclusion and exclusion criteria. A third independent reviewer was consulted to settle any disagreements between reviewers that had not been resolved by consensus. From studies of overlapping populations, we included only the one with the largest sample size.

The same 2 reviewers (AFA and BSZ) independently extracted data from eligible studies using a standardized form. Disagreements were resolved with discussion and, if required, consensus was reached by consulting a third independent reviewer. The following data were extracted: study characteristics (e.g., author, year, study setting, study design, and study context), sample characteristics (e.g., number of participants, age of participants, and sample size), characteristics of the tools used to measure mobility or access, and comparison groups (if available).

Risk of bias assessment

The same reviewers (AFA and BSZ) independently assessed the risk of bias of each included study. The original Newcastle-Ottawa Scale (NOS) was used to assess cohort studies comparing treatment options. It consists of 8 items that classify methodological quality across 3 categories by a star rating system: participant selection (maximum 4 stars), comparability (maximum 2 stars), and assessment of outcome (maximum 3 stars) [ 13 ]. In the NOS adapted for cross-sectional studies, a maximum of 10 stars can be awarded to each study: selection (maximum 5 stars), comparability (maximum 2 stars), and outcome (maximum 3 stars). Studies reaching 75% or more of the maximum number of stars are considered to be at low risk of bias, while those reaching 50–75% are considered to be at moderate risk of bias.

Data analysis and travel distance estimates

We performed a synthesis of qualitative and quantitative evidence. We collected data on the main findings and consequences of mobility as assessed in each study and the related oncological outcomes. Given the heterogeneity among study results regarding distance traveled or travel time, we decided to use the distance traveled instead of travel time given the relatively deficient and expensive transport systems in Latin American countries. When the outcomes were reported in travel time instead of distance traveled, the study authors were contacted. If there was no response or the data were unavailable, a conversion technique was used. To address this issue, when a study reported data on travel time, we converted the data to kilometers traveled based on the estimates provided by INRIX (vehicle monitoring and software company), which assumes an average city traffic of 30.2 km/h [ 14 , 15 ]. If necessary, we used WebPlotDigitalizer [ 16 ] to extract data from figures and graphs. For studies reporting parameters such as ‘outside the city,’ we estimated the average distance from the capital city to nearby cities using Google Maps.

Based on the data presented in the included studies, we estimated the average distance traveled by people to access BC screening services and by patients with BC to access treatment facilities. Since most studies did not provide sufficient data to estimate the standard error of the distance traveled, we calculated standard errors for the set of average distances estimated for the different studies (imputation-driven meta-analysis). We attributed the standard errors to all studies so that all of them had the same weight in the meta-analysis, as calculated using the inverse variance method. We used R software (meta package v 6.0–0) for data analysis [ 17 ].

Study selection and included studies

The study selection process is shown in Fig.  1 . The database searches provided a total of 1142 records. After adjusting for duplicates, 1117 remained. After title and abstract screening, a total of 36 studies were retrieved for full-text review, 14 of which met the inclusion criteria.

PRISMA flow diagram

Characteristics of included studies

Table  1 provides the main characteristics of each included study, an outline of the individual characteristics of each study population, and the context in which the impact of mobility was studied. Regarding mobility outcomes, 4 studies reported travel distance [ 6 , 7 , 18 , 19 ], 3 studies reported travel time [ 20 , 21 , 22 ], and 7 studies reported any effect measure such as mammograms not performed or effect on access to cancer screening [ 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. To facilitate understanding of the results, the following sections are divided into the impact of mobility on BC screening and the impact of mobility on BC treatment.

Impact of mobility on BC screening

Three studies evaluated BC screening. The study of Agudelo Botero et al. [ 23 ] used secondary data from 3 Mexican databases to explain factors that can impact BC screening for women. In all databases, the sociodemographic variables that together could explain the strongest relationship with breast self-examination were level of education, age group, and type of area (urban vs. rural). The other 2 studies were conducted in Brazil by Rodrigues et al. [ 24 ] and Amaral et al. [ 25 ] and found results similar to those of the Mexican study. Rodrigues et al. [ 24 ] reported that the spatial coverage of mammography machines, using 60 km as a parameter for the maximum distance between an individual’s home and a mammography machine, was fully achieved in the South and Southeast regions and several states in the Northeast but not in the North and Midwest regions.

Impact of mobility on BC treatment

Several studies investigated the association between travel burden and BC treatment. de Almeida et al. [ 20 ] reported an increased likelihood of advanced BC stage at diagnosis in patients who traveled to another city for BC care. Ferreira et al. [ 26 ] evaluated over 150,000 women with BC and concluded that those categorized as non-white with a low level of education living in the North of Brazil had to wait longer from diagnosis to treatment than women in other groups, in addition to being more likely to wait over 60 days to start BC treatment. Oliveira et al. [ 7 ] highlighted that a high percentage of women receiving treatment through the Brazilian publicly funded health care system lived more than 150 km away from the place of care. Knowing that treatment is based on frequent procedures, the authors noted that a large proportion of women receiving care had to face a number of difficulties other than the disease itself due to long travel distances.

Saldanha et al. [ 6 ] reported that commuting can affect 51.34% of BC patients negatively, with over half requiring journeys of more than 3 h in half of the cases. The proportion of patients who need to travel outside their hometown for chemotherapy and radiotherapy is similar to that for hospital admissions. However, given that these therapies require multiple visits to health facilities during the treatment cycle, their potential impact on the quality of life of women undergoing treatment is of particular concern. In a prospective study conducted in Brazil, Medeiros et al. [ 21 ] showed that living outside the city of Rio de Janeiro and older age were associated with a time interval between diagnosis and treatment initiation exceeding 60 days, despite the ‘60-day law’ in place since 2013 establishing that treatment for any type of cancer in the public health system must start within 60 days of the diagnosis. In a study comparing private and public hospitals in the city of Buenos Aires, Argentina, regarding BC treatment, Recondo et al. [ 27 ] found that patients receiving treatment in public hospitals used public transport more often (69.3%) than those treated in private hospitals (29.3%), resulting in significantly longer commutes for those treated in public hospitals. In southern Brazil, Romeiro Lopes et al. [ 28 ] found a mean time to diagnosis of 102.5 (SD 165.5) days, with treatment delay in 63.4% ( n  = 52) of cases, where 60% of patients with a delay in treatment > 30 days lived more than 100 km from the cancer care center. Although without statistical significance, this finding draws attention as a factor influencing treatment adherence over time.

Unlike the previous findings, 2 studies [ 24 , 29 ] did not report geographic distance or commuting as the main access barriers. Evaluating barriers to access to health care as perceived by women with BC in northeastern Brazil, Gonçalves et al. [ 29 ] reported that geographic barriers were rarely mentioned by women during treatment, but this factor requires attention because transfer to another city and difficulty accessing transport provided by the municipal health department were mentioned by the participants, capturing the reality of the Northeast region. Also in the Northeast of Brazil, de Sousa et al. [ 18 ] demonstrated that, despite important data on geographic distance and time to treatment, treatment delay was not linked to geographic barriers but rather to a fragmentation of health services, that is, to a need to shift the points of care from primary to specialized care with a well-defined patient flow. Finally, Aguiar et al. [ 22 ] reported that work commutes of 1 to 2 h were negatively associated with BC mortality in the city of São Paulo, Brazil, and that these findings were important to guide cancer prevention policies.

Quantitative data analysis

Table  2 provides the quantitative results of individual studies and the impact of mobility on the related oncological outcomes. Regarding quantitative data analysis, the heterogeneity was notably high (I 2  = 93%), and the number of studies that provided sufficient information for a meta-analysis was limited to 7, rendering the meta-analysis inadequate for reporting the primary outcome [ 6 , 7 , 19 , 20 , 22 , 26 ]. We performed an exploratory subgroup analysis to investigate the regions of Brazil where the studies had been conducted as a potential source of heterogeneity. Indeed, this analysis revealed that a portion of the observed heterogeneity stemmed from variations in the regions where the studies had been conducted.

The average distances traveled are shown in the forest plot in Fig.  2 . For hypothesis generation purposes only, the average distances traveled to BC-related diagnostic or therapeutic resources in the 5 administrative regions of Brazil were estimated via a random-effects meta-analysis (Additional Fig.  1 ), yielding the following results: 448 km (95% CI, 383.87–512.13) in the North; 323 km (95% CI, 258.87–387.13) in the Midwest; 239.8 km (95% CI, 58.78–419.02) in the Northeast; 104.8 km (95% CI, 70.93–138.82) in the Southeast; and 88 km (95% CI, 23.87–152.13) in the South. Four studies reported results for Brazil as a whole, without specifying a region. For description purposes, these results indicate an average travel distance of 77.8 km (95% CI, 49.1–106.48) to a BC-related diagnostic or therapeutic resource. Even though we acknowledge the limitations and regional disparities both within and between countries, our findings align with the existing literature, indicating an equivalent of 3–4 h of commute on a national average [ 30 ].

Forest plot of average travel distances reported in the studies

Overall, the risk of bias was moderate to low. The 12 cross-sectional non-comparative studies were rated with a median of 7.5 stars on the adapted NOS (maximum 10 stars). The 2 cohort studies were also rated as having a moderate to low risk of bias (7 and 8 out of 9 stars, respectively) on the original NOS (Additional Table  2 ). Particularly in this analysis, the risk of bias had an impact on the interpretability of the studies.

Access to and affordability of appropriate diagnosis and care represent critical limiting factors in health care [ 31 ]. The establishment of national BC plans, whether of a general or specific nature, plays a pivotal role in facilitating organized governance, financing, and health care delivery [ 20 , 31 ]. In this regard, evidence-based treatment guidelines have been disseminated by government authorities, cancer institutes, or scientific associations in numerous countries. Nevertheless, the principal challenge lies in the effective implementation of policies and mechanisms designed to ensure consistent compliance with these guidelines over the entire population.

Consistent with the existing literature, our research findings underscore the presence of regional disparities across the health care landscape of Brazil [ 31 ]. Specifically, our analysis revealed that patients living in the North and Midwest of the country must travel longer distances to access cancer care than their counterparts in the South, Southeast, and Northeast [ 30 ]. It is worth noting that, despite the existence of Law No. 12,732, which mandates a 60-day time frame for initiating cancer treatment after the disease has been diagnosed, there is a lack of empirical evidence to define what constitutes a reasonable travel distance for such treatment [ 32 ], since approximately 40% of patients experience a delay in starting their treatment of more than 60 days, and this delay is longer in the SUS than in the private health insurance system [ 26 , 33 ]. In other words, the law addresses the number of days for initiating treatment but does not establish what distance is considered to be reasonable for patients to obtain such treatment. de Almeida et al. [ 20 ] showed that women traveling to another city to receive BC care were more likely to have advanced disease at the time of diagnosis and that late diagnosis increases the cost of treatment and compromises the patient’s clinical outcome. Despite the ‘60-day law’ and health care policy initiatives in Brazil, there appears to be a gap between policy intentions and their actual implementation, particularly for patients living outside major urban centers [ 22 , 23 , 27 , 29 ]. Some studies have highlighted the underrepresentation of geographic barriers in patients’ perceptions, emphasizing the need for a nuanced contextual understanding.

In the context of BC screening, the health care system should be designed to ensure an adequate number of mammography machines, with due consideration for a maximum distance of 60 km between the machine and the residences of the target population [ 25 , 34 ]. However, although this spatial proximity is deemed essential to facilitate timely and accessible screening services for BC detection, women continue to face difficulties in accessing appropriate screening, and by the time they do, they often present at an advanced disease stage [ 20 ]. Nonetheless, in addition to distance, Bretas et al. [ 35 ] pointed out the lack of a well-defined strategy to receive women with self-detected breast abnormalities in the primary health care unit. Strategies may encompass procedures such as enhancement of clinical breast examination, breast biopsy, and accurate pathology as well as BC surveillance and telehealth. Such actions take place occasionally in one-stop clinics, although patients will often be transferred to different locations [ 35 ]. Therefore, patient navigation programs and integration between primary and tertiary care need to be further improved.

Long travel distances to radiotherapy centers have been associated with diminished utilization of radiotherapy services, elevated mastectomy rates in patients with BC, reduced probability of radiotherapy utilization among individuals with BC and other cancers, and infrequent recourse to palliative radiotherapy [ 30 ]. While it is important to acknowledge that patient travel distance is not the only determinant of access to cancer services, it remains a pivotal factor to be addressed in endeavors to enhance health equity and achieve a broader health coverage [ 7 ].

The concentration of specialized cancer care to centers located in the Southeast of Brazil highlights the need to narrow the gap between supply and demand for this type of care. Providing broad coverage of cancer treatment requires improved planning and regulation, in addition to ensuring the activation of the highly complex infrastructure and qualified human resources that are needed to support treatment [ 30 , 31 ].

The obstacles to mobility and access to BC screening and treatment identified in Brazil, such as geographic barriers, socioeconomic inequalities, and health care infrastructure limitations, resonate with challenges faced by other Latin American countries. Practical solutions to overcome these barriers include implementing telemedicine services and mobile health units [ 36 ] and expanding the role of community health workers to provide education, support, and navigation services [ 37 ]. Policymakers can leverage Brazil’s experiences to inform regional strategies, such as the Brazilian National Policy of Comprehensive Women’s Health Care, which provides a framework for addressing women’s health issues, including BC, adaptable by other Latin American countries to improve outcomes [ 38 ].

Limitations

The heterogeneity of the studies renders the meta-analytic estimates not representative of the overall travel distances observed in the included studies. Even though our objective was to conduct a comprehensive literature review within a Latin American perspective, there were only 2 studies outside Brazil. It was expected that Brazil, Mexico, and Argentina, the largest Latin American countries, would be better represented in the literature, but the lack of studies from other countries in Latin America makes generalization difficult. While concerns about the regional representativeness of Brazilian studies are valid, the similarities in demographics, socioeconomic status, health care structures, and epidemiological trends across Latin America support the relevance of Brazilian data. By addressing practical implications and proposing evidence-based solutions, we aim to improve BC screening and treatment accessibility throughout the region. Besides that, the fact that a sensitive search strategy was unable to retrieve studies from a more diverse group of countries shows a wide gap in the scientific literature on this topic in other countries in Latin America. Furthermore, our analyses and results were limited by the need to convert travel time to travel distance when data were not available even after contacting authors, which could have underestimated or overestimated some results especially in remote areas where the transportation infrastructure is poor. However, this approach allowed the comparison of travel distances in diverse settings.

Conclusions

The collective evidence from these studies underscores the multifaceted and pervasive influence of mobility and travel distance on access to BC care. It also emphasizes the importance of not only addressing geographic barriers but also considering sociodemographic factors, health system issues, and policy-related challenges in the pursuit of equitable BC care. The scarce information on this topic in Latin American countries, especially on the complications and challenges women face before and during treatment, indicates that travel distance alone may not serve as the only determinant of mobility. Therefore, additional research is imperative to comprehensively elucidate the multifaceted variables that underlie the impact of mobility on access to health services.

Data availability

All data analyzed during this study are included in this published article and its supplementary information files.

Abbreviations

breast cancer

Newcastle-Ottawa Scale

Brazilian national public health care system

Hierink F, Okiro EA, Flahault A, Ray N. The winding road to health: a systematic scoping review on the effect of geographical accessibility to health care on infectious diseases in low- and middle-income countries. PLoS ONE. 2021;16:e0244921.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ambroggi M, Biasini C, Giovane CD, Fornari F, Cavanna L. Distance as a barrier to cancer diagnosis and treatment: review of the literature. Oncologist. 2015;20:1378–85.

Article   PubMed   PubMed Central   Google Scholar  

GLOBOCAN. 2020. Global Cancer Observatory: Cancer Today. Lyon, France: International Agency for Research on Cancer. https://gco.iarc.who.int/today . Accessed 15 July 2023.

Nunes L, Silva JD, Vieira MT. Health financing in Brazil: trends and challenges. Saude Debate. 2020;44:105–20.

Google Scholar  

Rocha-Brischiliari SC, Andrade L, Nihei OK, Brischiliari A, Hortelan MDS, Carvalho MDB, et al. Spatial distribution of breast cancer mortality: socioeconomic disparities and access to treatment in the state of Parana, Brazil. PLoS ONE. 2018;13:e0205253.

Saldanha RF, Xavier DR, Carnavalli KM, Lerner K, Barcellos C. [Analytical study of the breast cancer patient flow network in Brazil from 2014 to 2016]. Cad Saude Publica. 2019;35:e00090918.

Article   PubMed   Google Scholar  

Oliveira EX, Melo ECP, Pinheiro RS, Noronha CP, Carvalho MS. [Access to cancer care: mapping hospital admissions and high- complexity outpatient care flows. The case of breast cancer]. Cad Saude Publica. 2011;27:317–26.

Atlas dos Centros de Cuidados do Câncer. https://atlas.oncoguia.org.br/#em_numeros . Accessed 15 July 2023.

Brasil. Saúde e Vigilância Sanitária: Habilitar hospitais em alta complexidade em oncologia. https://www.gov.br/pt-br/servicos/habilitar-hospitais-em-alta-complexidade-em-oncologia . Accessed 15 July 2023.

Teston EF, Fukumori EFC, Benedetti GMS, Spigolon DN, Costa MAR, Marcon SS. Feelings and difficulties experienced by cancer patients along the diagnostic and therapeutic itineraries. Esc Anna Nery. 2018;22:e20180017.

Article   Google Scholar  

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane. 2022. www.training.cochrane.org/handbook . Accessed 20 Aug 2023.

Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta- analyses. Ottawa: The Ottawa Hospital. 2013. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp . Accessed 8 Sept 2020.

INRIX. 2022 INRIX Global Traffic Scorecard. https://inrix.com/scorecard/ . Accessed 15 July 2023.

Companhia de Engenharia de Tráfego do Rio de Janeiro - CET-RIO. https://carioca.rio/orgao/companhia-de-engenharia-de-trafego-do-rio-de-janeiro-cet-rio/#:~:text=Missão%3A,o%20bem%2Destar%20da%20população . Accessed 23 June 2023.

Rohatgi A, WebPlotDigitizer. https://automeris.io/WebPlotDigitizer. 2022. Accessed 13 May 2023.

Balduzzi S, Rücker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health. 2019;22:153–60.

de Souza BC, Dos Santos Figueiredo FW, de Alcantara Sousa LV, da Silva Maciel E, Adami F. Regional disparities in the flow of access to breast cancer hospitalizations in Brazil in 2004 and 2014. BMC Womens Health. 2020;20:37.

Sousa SMMT, Carvalho MGFM, Santos Júnior LA, Mariano SBC. Acesso Ao Tratamento Da mulher com câncer de mama. Saúde Debate. 2019;43:727–41.

de Almeida RJ, de Moraes Luizaga CT, Eluf-Neto J, de Carvalho Nunes HR, Pessoa EC, Murta-Nascimento C. Impact of educational level and travel burden on breast cancer stage at diagnosis in the state of Sao Paulo, Brazil. Sci Rep. 2022;12:8357.

Medeiros GC, Thuler LCS, Bergmann A. Determinants of delay from cancer diagnosis to treatment initiation in a cohort of Brazilian women with breast cancer. Health Soc Care Community. 2021;29:1769–78.

Aguiar BS, Pellini ACG, Rebolledo EAS, Ribeiro AG, Diniz CSG, Bermudi PMM, et al. Intra-urban spatial variability of breast and cervical cancer mortality in the city of São Paulo: analysis of associated factors. Rev Bras Epidemiol. 2023;26:e230008.

PubMed   PubMed Central   Google Scholar  

Agudelo Botero M. [Sociodemographic determinants of access to breast cancer screening in Mexico: a review of national surveys]. Salud Colect. 2013;9:79–90.

Rodrigues DCN, Freitas-Junior R, Rahal RMS, Correa RDS, Peixoto JE, Ribeiro NV, et al. Difficult Access and Poor Productivity: Mammography Screening in Brazil. Asian Pac J Cancer Prev. 2019;20(6):1857–64.

Amaral P, Luz L, Cardoso F, Freitas R. Distribuição Espacial De equipamentos de mamografia no Brasil. RBEUR. 2017;19:326.

Ferreira NAS, Schoueri JHM, Sorpreso ICE, Adami F, Dos Santos Figueiredo FW. Waiting time between breast cancer diagnosis and treatment in Brazilian women: an analysis of cases from 1998 to 2012. Int J Environ Res Public Health. 2020;17:4030.

Recondo G, Cosacow C, Cutuli HJ, Cermignani L, Straminsky S, Naveira M, et al. Access of patients with breast and lung cancer to chemotherapy treatment in public and private hospitals in the city of Buenos Aires. Int J Qual Health Care. 2019;31(9):682–90.

Romeiro Lopes TC, Gravena AAF, Demitto MO, Borghesan DHP, Dell`Agnolo CM, Brischiliari SCR, et al. Delay in diagnosis and treatment of breast Cancer among women attending a Reference Service in Brazil. Asian Pac J Cancer Prev. 2017;18:3017–23.

PubMed   Google Scholar  

Gonçalves LL, Travassos GL, Maria de Almeida A, Guimaraes AM, Gois CF. [Barriers in health care to breast cancer: perception of women]. Rev Esc Enferm USP. 2014;48:394–400.

Fonseca BP, Albuquerque PC, Saldanha RF, Zicker F. Geographic accessibility to cancer treatment in Brazil: a network analysis. Lancet Reg Health Am. 2022;7:100153.

Cazap E. Breast Cancer in Latin America: a map of the Disease in the region. Am Soc Clin Oncol Educ Book. 2018;38:451–6.

Brasil. Lei Nº 12732 de 22 de Novembro de 2012. Dispõe sobre o primeiro tratamento de paciente com neoplasia maligna comprovada e estabelece prazo para seu início (60 dias). Brasília, DF: Diário Oficial da União. 2012. https://www.planalto.gov.br/ccivil_03/_ato2011-2014/2012/lei/l12732.htm . Accessed 10 Sept 2023.

Resende H, Aguiar VQ, Jacob LFP, Renó ALCA, Cunha AP, Assis BR, et al. The journey of breast cancer patient from self-perception of breast abnormalities to first cancer treatment- a sectional study in Sul Fluminense region-RJ-Brazil. Med Res Arch. 2023;11:10.

Brasil. Ministério da Saúde. Gabinete do Ministro. Portaria N.o 1.631 de 1 de outubro de 2015. Aprova critérios e parâmetros para o planejamento e programação de ações e serviços de saúde no âmbito do SUS. DF: Diário Oficial da União, no. 189, Seção 1, p. 38. 2015. https://www.as.saude.ms.gov.br/wp-content/uploads/2016/08/Portaria-parametros-revoga-a-Portaria-1101.pdf . Accessed 12 Apr 2023.

Bretas G, Renna NL, Bines J. Practical considerations for expediting breast cancer treatment in Brazil. Lancet Reg Health Am. 2021;2:100028.

Leite H, Hodgkinson IR, Gruber T. New development: ‘Healing at a distance’—telemedicine and COVID-19. Public Money Manag. 2021;41:65–8.

Nascimento ALB, de Souza Santos I, Andrade FM. The role of community health workers in promoting mental health and connecting with primary care in Brazil. Int J Ment Health. 2018;47:122–34.

Brasil. Ministério Da Saúde. Política Nacional De Atenção Integral à Saúde Da Mulher. Editora MS; 2011.

Download references

Acknowledgements

Not applicable.

This work was supported by Produtos Roche Químicos e Farmacêuticos S/A – Brazil.

Author information

Authors and affiliations.

HTAnalyze Consultoria e Treinamento Ltda, Rua João Abbott, 109, Petrópolis, Porto Alegre, RS, 90460-150, Brazil

André Ferreira Azeredo-da-Silva & Bruna Stella Zanotto

Produtos Roche Químicos e Farmacêuticos S/A, Rua Doutor Rubens Gomes Bueno, 691, Santo Amaro, São Paulo, SP, 04730-000, Brazil

Flavia Martins, Nádia Navarro, Rafaela Alencar & Clarissa Medeiros

You can also search for this author in PubMed   Google Scholar

Contributions

AFA, BSZ, FM, NN, RA, and CM: Concept and design; AFA, BSZ: Acquisition of data; AFA and BSZ: Analysis and interpretation of data; AFA and BSZ: Drafting of the manuscript; AFA, BSZ, FM, NN, RA, and CM: Critical revision of the paper for important intellectual content; CM: Obtaining funding; FM, NN, and RA: Administrative, technical, or logistic support; NN and CM: Supervision. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Clarissa Medeiros .

Ethics declarations

Ethics approval and consent to participate.

We used secondary data already available in literature. Systematic reviews and meta-analyses do not require ethics committee approval. Not applicable.

Consent for publication

Competing interests.

FM, RA, NN, and CM were Roche S/A employees at the time of the analysis. BSZ and AFA were working on behalf of Roche S/A on the project.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

12913_2024_11222_MOESM2_ESM.jpg

Supplementary Material 2. Additional Fig.  1 . Meta-analysis forest plot

Supplementary Material 3

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Azeredo-da-Silva, A.F., Zanotto, B.S., Martins, F. et al. Health care accessibility and mobility in breast cancer: a Latin American perspective. BMC Health Serv Res 24 , 764 (2024). https://doi.org/10.1186/s12913-024-11222-6

Download citation

Received : 19 March 2024

Accepted : 19 June 2024

Published : 25 June 2024

DOI : https://doi.org/10.1186/s12913-024-11222-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Latin America
• Breast cancer

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]