analysis of a systematic literature review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Systematic Reviews and Meta Analysis

• Getting Started
• Guides and Standards
• Review Protocols
• Databases and Sources
• Randomized Controlled Trials
• Controlled Clinical Trials
• Observational Designs
• Tests of Diagnostic Accuracy
• Software and Tools
• Where do I get all those articles?
• Collaborations
• EPI 233/528
• Countway Mediated Search
• Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

• Next: Guides and Standards >>
• Last Updated: Sep 4, 2024 4:04 PM
• URL: https://guides.library.harvard.edu/meta-analysis

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

• A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

• Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
• View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
• Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
• First published as a Review in Advance on August 08, 2018
• Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

• APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
• Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
• Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
• Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
• Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
• Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
• Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
• Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
• Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
• Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
• Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
• Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
• Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
• Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
• Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
• Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
• Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
• Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
• Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
• Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
• Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
• Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
• Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
• John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
• Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
• Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
• Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
• Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
• Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
• Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
• Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
• Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
• Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
• Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
• Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
• Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
• Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
• Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
• Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
• Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
• Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
• Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
• Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
• Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
• Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
• Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
• Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
• Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
• Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
• Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
• Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

• Open access
• Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

• Gehad Mohamed Tawfik 1 , 2 ,
• Kadek Agus Surya Dila 2 , 3 ,
• Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
• Dao Ngoc Hien Tam 2 , 5 ,
• Nguyen Dang Kien 2 , 6 ,
• Ali Mahmoud Ahmed 2 , 7 &
• Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

824k Accesses

303 Citations

94 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Systematic Reviews and Meta-Analysis: A Guide for Beginners

Affiliation.

• 1 Department of Pediatrics, Advanced Pediatrics Centre, PGIMER, Chandigarh. Correspondence to: Prof Joseph L Mathew, Department of Pediatrics, Advanced Pediatrics Centre, PGIMER Chandigarh. [email protected].
• PMID: 34183469
• PMCID: PMC9065227
• DOI: 10.1007/s13312-022-2500-y

Systematic reviews involve the application of scientific methods to reduce bias in review of literature. The key components of a systematic review are a well-defined research question, comprehensive literature search to identify all studies that potentially address the question, systematic assembly of the studies that answer the question, critical appraisal of the methodological quality of the included studies, data extraction and analysis (with and without statistics), and considerations towards applicability of the evidence generated in a systematic review. These key features can be remembered as six 'A'; Ask, Access, Assimilate, Appraise, Analyze and Apply. Meta-analysis is a statistical tool that provides pooled estimates of effect from the data extracted from individual studies in the systematic review. The graphical output of meta-analysis is a forest plot which provides information on individual studies and the pooled effect. Systematic reviews of literature can be undertaken for all types of questions, and all types of study designs. This article highlights the key features of systematic reviews, and is designed to help readers understand and interpret them. It can also help to serve as a beginner's guide for both users and producers of systematic reviews and to appreciate some of the methodological issues.

PubMed Disclaimer

Similar articles

• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• How to review and assess a systematic review and meta-analysis article: a methodological study (secondary publication). Myung SK. Myung SK. J Educ Eval Health Prof. 2023;20:24. doi: 10.3352/jeehp.2023.20.24. Epub 2023 Aug 27. J Educ Eval Health Prof. 2023. PMID: 37619974 Free PMC article.
• How to Critically Appraise and Interpret Systematic Reviews and Meta-Analyses of Diagnostic Accuracy: A User Guide. Frank RA, Salameh JP, Islam N, Yang B, Murad MH, Mustafa R, Leeflang M, Bossuyt PM, Takwoingi Y, Whiting P, Dawit H, Kang SK, Ebrahimzadeh S, Levis B, Hutton B, McInnes MDF. Frank RA, et al. Radiology. 2023 May;307(3):e221437. doi: 10.1148/radiol.221437. Epub 2023 Mar 14. Radiology. 2023. PMID: 36916896 Free PMC article. Review.
• Detecting, quantifying and adjusting for publication bias in meta-analyses: protocol of a systematic review on methods. Mueller KF, Meerpohl JJ, Briel M, Antes G, von Elm E, Lang B, Gloy V, Motschall E, Schwarzer G, Bassler D. Mueller KF, et al. Syst Rev. 2013 Jul 25;2:60. doi: 10.1186/2046-4053-2-60. Syst Rev. 2013. PMID: 23885765 Free PMC article.
• Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. Abbott R, Bethel A, Rogers M, Whear R, Orr N, Shaw L, Stein K, Thompson Coon J. Abbott R, et al. BMJ Evid Based Med. 2022 Jun;27(3):169-177. doi: 10.1136/bmjebm-2021-111710. Epub 2021 Jun 3. BMJ Evid Based Med. 2022. PMID: 34083212 Free PMC article. Review.
• Italian and Middle Eastern adherence to Mediterranean diet in relation to Body Mass Index and non-communicable diseases: nutritional adequacy of simulated weekly food plans. Luconi E, Tosi M, Boracchi P, Colonna I, Rappocciolo E, Ferraretto A, Lorenzini EC. Luconi E, et al. J Transl Med. 2024 Jul 30;22(1):703. doi: 10.1186/s12967-024-05325-1. J Transl Med. 2024. PMID: 39080668 Free PMC article.
• A commentary on 'Efficacy of electroacupuncture in improving postoperative ileus in patients receiving colorectal surgery: a systematic review and meta-analysis'. Zhou M, Ye H, Ma R. Zhou M, et al. Int J Surg. 2024 Aug 1;110(8):5290-5291. doi: 10.1097/JS9.0000000000001564. Int J Surg. 2024. PMID: 38716889 Free PMC article. No abstract available.
• Evidence, Experience, and Eminence: Building Blocks for Pediatric Pulmonology Practice and Research in India. Mathew JL. Mathew JL. Indian J Pediatr. 2023 Jul;90(7):690-692. doi: 10.1007/s12098-023-04649-y. Epub 2023 Jun 2. Indian J Pediatr. 2023. PMID: 37264278 Free PMC article. No abstract available.
• Efficiency, Economy and Excellence: Experimental Exploration of Evidence-based Guideline Development in Resource-Constrained Settings. Mathew JL. Mathew JL. Indian J Pediatr. 2023 Jul;90(7):700-707. doi: 10.1007/s12098-023-04575-z. Epub 2023 Apr 28. Indian J Pediatr. 2023. PMID: 37106227 Free PMC article. Review.
• Sackett D, Strauss S, Richardson W, et al. Evidence-Based Medicine: How to practice and teach EBM. 2nd ed. Churchill Livingstone: 2000.
• Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: Synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376–80. doi: 10.7326/0003-4819-126-5-199703010-00006. - DOI - PubMed
• PennState Eberley College of Science. Lesson 4: Bias and Random Error. Accessed October 01, 2020. Available from: https://online.stat.psu.edu/stat509/node/26/
• Comprehensive Meta-analysis. Accessed October 01, 2020. Available from: https://www.meta-analysis.com/pages/why_do.php?cart=
• National Institute for Health Research. PROSPERO International prospective register of systematic reviews. Accessed October 01, 2020. Available from: https://utas.libguides.com/SystematicReviews/Protocol

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Europe PubMed Central
• Indian Pediatrics
• PubMed Central

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Jul 23, 2024 3:40 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 07 September 2024

Public support for carbon pricing policies and revenue recycling options: a systematic review and meta-analysis of the survey literature

• Farah Mohammadzadeh Valencia   ORCID: orcid.org/0009-0008-8832-2798 1 , 2 ,
• Cornelia Mohren 1 , 3 ,
• Anjali Ramakrishnan   ORCID: orcid.org/0000-0002-6557-5206 4 ,
• Marlene Merchert 1 , 5 ,
• Jan C. Minx   ORCID: orcid.org/0000-0002-2862-0178 1 , 6 &
• Jan Christoph Steckel   ORCID: orcid.org/0000-0002-5325-9214 1 , 2  

npj Climate Action volume  3 , Article number:  74 ( 2024 ) Cite this article

Metrics details

• Climate-change mitigation
• Climate-change policy

Since public support is critical for implementing carbon pricing policies, we conduct a systematic review and meta-analysis to examine the survey-based literature on change in public support for direct and indirect carbon pricing policies with and without revenue recycling options. Following a comprehensive and transparent machine-learning assisted screening of the literature, our dataset comprises 35 studies containing 70 surveys across 26 countries with over 100,000 respondents. We find that the introduction of any type of revenue recycling option increases public support for carbon pricing. Results from our meta-regression indicate that green spending (i.e. using revenues for climate-friendly projects) is the only revenue recycling option associated with a statistically significant increase in public support. Our findings moreover suggest that the effects may depend on which region the survey was carried out, highlighting the need for additional research in countries in the regions of Africa and Latin America and the Caribbean.

Similar content being viewed by others

Systematic review and meta-analysis of ex-post evaluations on the effectiveness of carbon pricing

Limited impacts of carbon tax rebate programmes on public support for carbon pricing

Pervasive over-crediting from cookstove offset methodologies

Introduction.

To achieve ambitious climate change mitigation, it is crucial to address the current mispricing of fossil fuel based goods. This can be achieved by removing fossil fuel subsidies (i.e. indirect carbon pricing policy abolishing negative carbon prices) and by putting a price on carbon (i.e. direct carbon pricing policy introducing a positive price associated with carbon emissions) 1 , 2 , 3 , 4 . Such policies result in salient price increases passed on to consumers which, without additional compensatory policies, could lead to strong public opposition. Recent examples of protests against fuel price increases include the ‘yellow vests’ in France in 2018, as well as protests in Nigeria in 2020, and in India in 2021. To address some of the public’s concerns and to increase support for carbon pricing, studies in economics, behavioural psychology and political science recommend policymakers to introduce complementary policies with clear benefits intended for their constituents 3 , 5 , 6 . Such complementary policies are often referred to as revenue recycling schemes and can vary in design. For example, revenue recycling could reduce the financial impact of increased fossil fuel prices by redistributing some of the revenues back to all or a targeted segment of the population. Other types of revenue recycling include investing in low-carbon infrastructure projects, or agreeing to reduce other types of taxes.

Current survey-based evidence on the effect of revenue recycling on public support for direct and indirect carbon pricing policies is mostly comprised of ex-ante studies with scattered and at times contradictory results. These studies elicit attitudes for different types of direct and indirect carbon pricing policies (e.g. carbon tax, fossil fuel taxes, fossil fuel subsidy reform, environmental taxes, and congestion charges), vary by the type of revenue recycling presented to respondents, and are conducted in countries with diverse socio-economic and political contexts. Several studies find an overall tendency for respondents to prefer investing revenues towards climate-friendly activities, such as renewable energy projects, expanding public infrastructure, or further climate research 7 , 8 , 9 . Some detect favourable responses for applying tax cuts for income or labour taxes to counterbalance the increase in costs 10 , 11 . Yet other survey studies reveal that respondents prefer to compensate all households equally or specifically target low-income households and highly affected population groups, such as workers or the elderly, for financial compensation 12 , 13 . Real-world examples of carbon pricing with complementary revenue recycling are limited 14 . Moreover, the ex-post survey-based literature is even smaller and also reveals some scepticism on the effects of revenue recycling to increase public support 15 , 16 .

Given the varying conclusions from the existing studies, this study aims to synthesise and reconcile the available knowledge from primary ex-ante survey literature in a systematic, transparent and rigorous way. Our aim is to understand the change in public attitude – and the determinants of this change – towards a direct or indirect carbon pricing policy after revenue recycling schemes are presented to survey respondents. By conducting a machine-learning assisted systematic review and meta-analysis, we provide robust evidence on how the levels of support vary systematically for different pricing policies, revenue recycling options, and sampling regions of the existing survey literature.

The results from our study indicate that the introduction of any type of revenue recycling scheme increases public support for carbon pricing. When comparing the different schemes to each other using a logistic meta-regression we find that only green spending increases public support at a statistically significant level for carbon pricing when compared to using revenues for public finance . The effect of other revenue recycling schemes on public support, including uniform cash transfer s, remain statistically insignificant. Our meta-regression findings moreover suggest that the effect of revenue recycling on public support depends on the region in which the survey is conducted, and is stronger in Africa, Asia and Pacific, and Latin America and Caribbean than in the baseline region North America. However, the small sample size for studies on this topic conducted in regions from low- and middle-income countries urgently beckons for further research.

Systematic review – data and sample

We perform a comprehensive machine-learning assisted search and selection strategy and identify a final data sample of 35 primary studies from a total of 3542 retrieved search records (Web of Science, Scopus, snowballing), which comprise 70 surveys representative at the general population level, with 113,356 respondents spanning 26 countries 6 (see Supplementary Table 3 for an overview of all included studies). Surveys were conducted between 2005 and 2022 and 87% of the sample comes from surveys conducted after 2016. We note that 20 surveys (close to a third of our sample) come from one working paper published in 2022, which allowed to expand regional variations in Latin America and the Caribbean as well as in Africa 17 . Our complete review methodology is described in the Methods section.

We categorise the different revenue recycling schemes presented in the 70 surveys into seven categories (see Fig. 1 ), namely (1) uniform cash transfers; (2) targeted cash transfers; (3) green spending; (4) tax cuts; (5); corporate tax cuts; (6) public finance; and (7) miscellaneous. We classify any scheme as revenue recycling which specifies how revenues generated from a direct carbon price or saved from a fossil fuel subsidy reform would be explicitly used (or recycled) to benefit households or society as a whole. Although using revenue towards public finance does not directly benefit households, often surveys explicitly ask respondents their support for a carbon pricing policy when revenues would be used to manage government expenditures and debts.

The category green spending captures all schemes that reference infrastructure, energy efficiency, research and development, and other types of environmental projects. Cash transfers – both uniform and targeted – refer to schemes that explicitly state a redistribution scheme to all or some portion of the population. Tax cuts refer to schemes whereby individual people would benefit, either through income or labour tax cuts, consumption tax cuts, or a tax rebate. Corporate tax cuts refer to tax cuts that benefit companies. Public finance refers to schemes that explicitly state that revenues would go towards a general government budget or explicitly to reduce government debt/budget deficit. The category miscellaneous refers to revenue recycling schemes that were presented as a combination of two categories, or that were did not fit the other categories.

In terms of regional distribution, 44% of surveys were sampled in Europe and West Asia, 24% in North America, 23% in Asia and Pacific, 6% in Latin America and the Caribbean, and 3% in Africa. Regarding country distribution, the majority of surveys were conducted in the USA (19%), followed by Switzerland (7%), Canada (6%), China (6%), and Spain (6%) (Fig. 2 ). The survey evidence for different revenue recycling schemes is scattered unevenly across surveys. 83% of surveys in our sample elicit responses for green spending , 68% for tax cuts , 58% for targeted cash transfers , 47% for uniform cash transfers , 35% for corporate tax cuts , 11% for public finance, and 16% for miscellaneous .

This figure indicates which revenue recycling options are elicited by country and the country as well as regional shares represented in the final sample. In each country, one or more surveys elicited preferences for green spending, and targeted cash transfers . Surveys in all countries except for Norway and Ecuador elicited for uniform cash transfers . Surveys in all countries except for Norway elicit for tax cuts . Surveys elicit corporate tax cuts in all countries except for Egypt, Norway, Sweden, and Ecuador. Miscellaneous types of revenue recycling options are elicited only in Egypt, India, Indonesia, Spain, Sweden Switzerland, Ecuador, Mexico, and USA. Countries marked in bold and with the check mark are represented in the quantitative analysis. This is because for those countries, our sample includes surveys that provide descriptive statistics and use an unspecified baseline before presenting respondents with revenue recycling options.

The type of carbon pricing policy elicited also varies across the surveys, whereby 67% of surveys elicit support for carbon taxes, 12% for fossil fuel taxes, 10% for fossil fuel subsidy reforms, and less than 10% for environmental taxes or congestion charges. Furthermore, we find that the majority of surveys (77%) specify that a carbon pricing policy is intended towards a specific fuel, specifically gasoline, whereas the rest did not specify.

Regarding the measure of public attitudes, 66% of papers focus on support while the rest use acceptance , acceptability , willingness to pay , or other measures, such as preferences . However, a clear definition of the attitude measurement is often not provided in the primary studies. Conceptually, the different attitude measures appear to be used interchangeably or as potentially equivalent terms in the identified studies. While the exact definitions of these public attitude terms have potential implications for decision-making based on their different meanings, we interpret them as an overarching measurement concept to reveal respondents’ favour or disfavour for a certain policy in the primary study 18 .

Quantitative results

To ensure comparability between effect sizes, in our quantitative analysis we only include papers that provide effect sizes when the change of public support with a revenue recycling options is compared to a baseline of “unspecified revenue use” (i.e. when surveys do not initially specify how generated revenues would be used). Moreover, this allows us to rule out potential effects from respondents being primed or prompted to compare revenue recycling to different baselines. Due to wider availability and to ensure comparability of effect sizes, we use summary statistics rather than results from econometric analyses in the primary studies. For the quantitative analysis, since very few studies conducted econometric analyses on the impact of revenue recycling, we expanded our dataset by including those that provide summary statistics. This also reduces the risk of comparing results from econometric analyses that are of different quality and potentially subject to biases. Our sample for the meta-regression thus includes 259 effect sizes from nine studies, 36 surveys, 68,043 respondents, spanning 22 countries. The surveys in our sample use different support measures, such as Likert-scales (with different levels across the studies), binary scales (support-not support), or stated preferences among policy alternatives. Although we may lose some important information from binning scaled responses into a binary system, to harmonise the different scales across the primary studies we standardise the effects by testing for the direction of the aggregated effect found for each revenue recycling scheme (that is, increase vs. decrease in public support), similar to comparable meta-analyses 19 , 20 .

Our descriptive analysis reveals that presenting revenue recycling to respondents increase support for a given carbon pricing policy when compared to ”unspecified revenue use” (see Fig. 3a ). Revenue recycling options for which at least 80% of the available effect sizes report an increase in support for carbon pricing policies include green spending (97%), targeted cash transfers (86%), and corporate tax cuts (80%). In contrast, for uniform cash transfers only about two thirds of available effect sizes reported an increase in support, and labour and consumptions tax cuts and public finance about 80% of the effect sizes report an increase in support when compared to our baseline.

This figure shows the share of positive and negative effect sizes for all regions and individually by each region. a For all surveys included in the quantitative analysis, the share of positive effect sizes is largest for green spending (97%) and lowest for uniform cash (67%). b In the Africa region, all effect sizes for all revenue recycling options are positive, but the sample is small. c In the Asia and Pacific region, effect sizes are positive for public finance (100%) followed by targeted cash transfers (95%). d In the Europe and West Asia region, only for green spending (100%) show over 90% of effect sizes are positive. The lowest positive share is found for uniform cash transfers (64%). e In the Latin America and Caribbean region, all effect sizes are positive for uniform cash transfers , green spending , corporate tax cuts and public finance . The share of positive effect sizes is lowest for t ax cuts (75%). f In the North America region, the effect of targeted cash transfers is always positive. For uniform cash transfers, corporate tax cuts and public finance half of the effect sizes are positive (50%), and more than 80% of the effect sizes are positive for green spending and tax cuts.

Regionally (see Fig. 3 b–f), we find that in North America the reported effect sizes for at least three out of six types of revenue recycling schemes result in an increase in public support for a given carbon pricing policy. With regards to Africa, we find that all effect sizes reported for revenue recycling options are positive. For Asia and Pacific, we find that the reported effect sizes for public finance, targeted cash transfers , and green spending are over 90% positive. The revenue recycling option with the lowest share of positive effects in the Asia and Pacific region is uniform cash transfers (67%). For Europe and West Asia, we find that all effect sizes for green spending are positive; uniform cash transfers seem to have the least positive impact with only 64% positive effect sizes. In Latin America and the Caribbean, we find that effect sizes for uniform cash transfers, green spending , and corporate tax cuts , and public finance are all positive. However, for targeted cash transfers to low-income households or disproportionally affected population segments shows lower positive effects sizes (88%) as well as for income and consumption tax cuts (75%), which indicate potentially lowest impact on public support in that region. The evidence base, however, is very limited for some regions such as Africa and Latin America and the Caribbean.

Next, we run a logistic meta-regression on the same sample to compare the different revenue recycling schemes with each other, taking the allocation of revenues to public finance as a baseline (see Table 1 ). We find that targeted cash transfers and green spending increase public support for carbon pricing, but only the latter is statistically significant at the 1% level (3.09; standard error 1.08; p  < 1%, two-tailed test). Our results suggest that redistributing revenues equally to all households via uniform cash transfers (−0.60; standard error 0.86; p  > 10%, two-tailed test) is associated with a decrease in public support. We also find that corporate tax cuts (−0.51; standards error 0.47; p  > 10%, two-tailed test) are associated with a decrease in support, but none of the effects is significant at conventional level.

We control for the type of policy, whether a direct carbon pricing policy was in place at the time of the survey, regional variation and study design, but find only few significant effects – probably related to the limited sample size, which is a common problem in meta-regression 21 (see Supplementary Table 4 for details on variance inflation factors). Due to insufficient variation across observations, we did not control for other potentially interesting features, such as the year the sample was collected or the article was published, the gender of the lead author, the timing of the survey in relation to upcoming elections or a forthcoming introduction of a carbon pricing policy, the type of rating scales used to measure support, or other elements of the survey, such as checking for communication effects.

Our regional dummies suggest that – compared to North America – increases in public support through revenue recycling are significantly higher in regions mainly composed out of developing countries, i.e. Africa (significant at the 1% level, two-tailed test), East Asia and Pacific (significant at the 5% level, two-tailed test), and Latin America and Caribbean (significant at the 1% level, two-tailed test). The reported effect is also nominally positive for Europe and West Asia and Asia and Pacific, but comparatively small and not statistically significant. Non-experimental surveys tend to inflate estimates when compared to surveys with an experimental component (significant at the 1% level, two-tailed test), highlighting the importance of the choice of survey design 22 . Applying revenue recycling to carbon and environmental taxes compared to fossil fuel subsidy removal policies shows to increase support, but the effect is not statistically significant at conventional level.

Since all surveys underlying our analysis are representative for the general population, we decided against the use of sample weights in our main regression. However, we provide results from a weighted regression in the Supplementary Information (see Supplementary Table 5 ). Additional robustness checks are also included in the Supplementary, such as an additional indicator variable (see Supplementary Table 6 ) to control for the influence of the working paper which provides our dataset with 20 surveys 17 in our logit results, publication bias (see Supplementary Table 7 ) and a logit specification by OLS regressions (see Supplementary Table 8 ).

The existing literature has identified various channels on how the use of revenues generated from indirect and direct carbon pricing policies could positively change public perception by highlighting the role of ‘perceived fairness’ and ‘effectiveness’ 6 , 20 . Specifically, questions of personal and distributional effects have shown to influence public perception of fairness and acceptability of carbon pricing policies. This is also why in theoretical analyses, uniform cash transfers are often touted as the optimal revenue recycling option with regards to the impact on acceptability, equity, and efficiency 3 . Regarding effectiveness, specifically effectiveness in reducing greenhouse gas emissions, using revenues to go towards climate-friendly and environmental purposes can also have positive effects on policy acceptability. Building on our systematic review and meta-analysis of the survey based literature, our regression results indicate that compared to using revenue towards public finance , only those schemes for environmental purposes ( green spending ) are associated with a statistically significant positive effect on public support.

While traditional economics as well as behavioural and political science suggest uniform cash transfer to increase public acceptability, our analysis of empirical studies indicate a decrease in support for this revenue recycling scheme when compared to public finance . One of the reasons why uniform cash transfers decrease support (if they have any effect) may be that respondents are averse to inequity and more concerned with distributional fairness 23 , 24 . Thus, rather than providing all households with equal amount of compensation, respondents may prefer to target and compensate narrowly defined groups. Indeed, our analysis suggests that respondents systematically prefer targeted cash transfers when compared to revenue use for public finance , though none of the effects show statistical significance. This is well in-line with another recent meta-analysis, which finds that distributional fairness of climate change taxes and laws is the most important factor influencing public attitudes 20 .

One way to interpret our findings for the substantial positive effect of green spending is that survey participants may not fully understand the incentive effects of Pigouvian taxes to reduce actions that have negative externalities, in this case change behaviour away from using fossil fuels to reduce greenhouse gas emissions 25 . The concept of linking the ineffectiveness of carbon pricing policies with a preference for using revenues explicitly towards environmental purposes is known as “issue-linkage” 8 . Thus, the perceived ineffectiveness of such pricing policies might be a driver in our findings of why the general public prefers revenues to be used for programs to improve the environment 1 , 26 . The concept of issue-linkage also applies to other Pigouvian taxes, such as red meat taxes. For example, a survey in Norway found that earmarking revenues from a red meat tax for environmentally friendly technology increased public acceptability more than using revenues for income tax 27 . Similarly for tobacco taxes, a study conducted in the US showed that public support for the tobacco increased when revenues were earmarked for health purposes 28 . Both in the case of red meat and tobacco taxes, respondents favoured revenue recycling options which aimed to reduce the negative externality targeted by the policy to begin with.

Our analysis generally reveals a low amount of primary studies that analyse the relationship between public support for carbon pricing schemes and revenue recycling options. Despite statistically significant results that certain revenue recycling options increase public support for carbon pricing policies when compared to unspecified revenue use, the relatively small sample size makes it difficult to draw definitive conclusions for policy-making. For instance, uniform cash transfers has one of the lowest number of observations in our meta-analysis and we observe a high variability for support in the primary studies. To further illustrate, uniform cash transfers are found to be the most popular revenue option in the studies by Nowlin et al. 7 and Beiser-McGrath & Bernauer 29 , while it is comparatively the least popular option identified in studies by Douenne & Fabre 30 and Ewald et al. 31 . This might underline the contextual importance of such research – depending on a country’s trust in institutions, recent economic history, regulatory traditions, political messaging and many other factors which we cannot control due to limited information in primary studies, certain revenue schemes might be more popular in one country and highly contested somewhere else. Moreover, unlike observational data or revealed preferences, a survey is an inherently controlled environment that identifies only the specified variations but cannot control for all potential biases and factors that may influence public opinion. This could also explain the perceived popularity of green spending as a potential artifact of surveys. When a survey intends to elicit support for green spending, the questions and information provided to respondents may create a more obvious link between carbon pricing policies and their environmental purpose, thus influencing “issue-linkage”.

We tried to increase the sample size in several ways. First, we looked at other regional databases in other languages. For example, given by the large number of Spanish-speaking countries and their low representation in our data base, we conducted a search for relevant studies in a major Spanish-language database called “La Referencia”. We used the same policies and public attitude keywords as we did in the English-language query and translated it into Spanish (see Supplementary Table 2 ). We retrieved 535 articles; however, none fit our inclusion criteria. We would not expect that including search queries in additional languages would significantly increase our sample. Another way we increased our sample size is by including results from 20 surveys (close to a third of our sample) from one working paper published in 2022 17 . Our main results remain unaltered when including an additional indicator variable to control for the influence of this paper in our meta-regression (see Supplementary Table 6 ). Although we could have expanded our sample size for the meta-regression by assuming that respondents perceive “unspecified revenue use” to automatically imply public finance , we decided to treat those two concepts differently. We know from studies that wording and additional information can influence public support, for example when calling a policy a “carbon tax” or a “carbon contribution” 1 . Since we did not find any study that analyses what respondents imagine revenues are used for when the survey does not specify its usage, and to rule out any potential effects priming might have on respondents who receive information on how revenues are used (e.g., public finance) versus those that do not (e.g. “unspecified revenue use”), we decided to only included studies in our meta-regression which use a baseline without revenue recycling. This reveals a research gap on the cognitive-based effects of framing, additional information, political messaging and other priming methods on the responses from respondents.

Ultimately, our systematic review and meta-analysis show that the existing evidence on how revenue recycling influences public support for carbon pricing is relatively scarce. This points to a general gap in the empirical research of revenue recycling and public support for direct and indirect pricing policies. For example, the overwhelming majority of the surveys we identified are conducted ex-ante to introducing any type of carbon pricing policy. Although there is a limited number of countries and jurisdictions that have carbon pricing policies with revenue recycling, even fewer studies analyse ex-post the public support for these policies 14 , 15 , 16 . Moreover, our overall sample elicits support for a carbon tax (67%) and only few analyse public attitudes for fossil fuel subsidy reform (10%). Since fossil fuel subsidy reforms also lead to highly visible price increases for consumers, our study reveals a gap in empirical understanding how revenue recycling could increase support for this indirect carbon pricing policies. Moreover, most empirical studies in our dataset have been conducted in North America, Europe and West Asia, and Asia and Pacific, revealing a lack of research on how to increase support for carbon pricing policies with revenue recycling schemes in low-and middle-income countries in regions such as Latin America and the Caribbean and Africa. Given the number of countries in Latin America and the Caribbean which are implementing a direct and indirect carbon pricing policy and considering that protest responses are a form of active civic engagement in the region, additional empirical evidence could help inform policy-makers 32 , 33 . Moreover, very few of the revenue recycling schemes in our dataset consider other types of socio-economic and welfare policies beyond uniform- or targeted cash transfers . This points to another gap in the empirical literature to consider what types of revenue recycling policies could increase public support specifically in low- and middle-income countries.

Literature search and data extraction

There is a large literature that discusses systematic reviews and meta-analyses and how to conduct them in a methodologically sound way in social sciences. This systematic review broadly follows the guidelines by the Collaboration for Environmental Evidence 34 . We make use of machine-learning at the title and abstract screening stage to enable a comprehensive study identification. In particular, after screening a random sample of 100 publications, we train a support vector machine to rank the studies according to their probability of being relevant for our study. This is commonly known as “prioritised screening” that has been shown to substantially reduce workload without significant loss of recall 35 , 36 . In fact, we argue that it enables the design of comprehensive search string with high-levels of recall that would not be feasible otherwise.

For the literature identification, we conducted a query search in February 2022 and 2023 in the Web of Science and the Scopus literature databases. We connect three groups of keywords with boolean operators filtering for research on (1) explicit carbon pricing policies ( carbon, CO2, GHG, ecological, environmental AND pricing, tax, trading, market, certificate, levy, allowance ) and (2) implicit carbon pricing policies ( fuel, diesel, gas, oil, kerosene, energy, LPG AND subsidy reform, tax ) investigating (3) impacts on the general public attitudes ( people, public, social AND attitudes, response, perception, resistance, support, fair, fairness, acceptance, acceptability, opinion, belief, willingness, willing, opposition, willingness to pay ) (see Supplementary Table 1 ).

In order to ensure that our search query has a high recall of relevant research, we assembled a benchmark list of 53 studies of known relevance from a previous review which focuses more broadly on factors influencing public support of carbon pricing policies but does not focus on revenue recycling 37 . Our final search string located 51 out 53 benchmark studies. The searches were performed in Web of Science and Scopus identifying 3721 articles after correcting for duplicates. We also included all the references from the paper which were included by using citation chaser, which added an additional 1137 papers. All the results were exported and combined in the NACSOS review management tool 38 for screening. The NACSOS review tool automatically removes duplicate records using trigram similarity-based fuzzy title matching. The meta-data is automatically recorded from the bibliographic databases using the NACSOS software 38 . A coding tool was used to systematically extract data from all eligible studies (codebook available upon request).

Records were screened at two levels (1) title and abstract, and (2) full-text. At an abstract-level, we screened all abstracts jointly by two people to minimise bias. When reviewing abstracts, all discrepancies between the two people were discussed and inclusion/exclusion criteria were further clarified prior to completing the screening. For the meta-analysis, we added additional inclusion criteria, that is we only included studies that also provided descriptive statistics and used the “unspecified” baseline before presenting respondents with revenue recycling options.

To screen the articles at an abstract level, the following eligibility criteria were followed by all reviewers:

Inclusion criteria

Studies that conduct a public opinion survey towards different types of direct and indirect carbon pricing policies, e.g. Carbon -pricing, -tax, -trading, -levy, -allowance; or fuel –subsidy reform, -tax (including, diesel, gas, oil, kerosene, energy, or LPG). Congestion and pollution pricing policies are also included if the study reference carbon emissions reductions.

Studies that are empirical, conducting surveys on a randomised set and representative sample of the general population.

Studies that include a specified a measure of public opinion toward accepting carbon pricing policies, such as acceptance, acceptability, support, and willingness to pay.

Studies that analyse some form of revenue recycling option as part of survey analysis using quantitative methods.

Studies that collect data from the population through primary survey or representative sample through secondary data.

Studies need either regression analyses and/or summary statistics indicating a change in public attitude.

Exclusion criteria

Studies that assess unspecified tax or pricing policies (e.g. “Pigouvian taxes”) which do not reference carbon emissions. Studies were also excluded if they analysed public opinion on waste or water management, sulphur dioxide related policies, food, attitude toward specific technologies, housing, plastic tax, corporate social responsibility, reforestation, geoengineering, or personal and voluntary carbon trading.

Studies that present no revenue use options to survey respondents.

Studies using focus groups, in a laboratory setting, collecting responses from specific groups, such as farmers, policy makers, industry representatives, or students rather than the general population.

After the first screening of 200 random studies, we included 20 studies. These 20 studies plus the benchmark studies from prior literature searches are then used to train a machine learning classifier using terms from the titles, abstracts, and keywords. As a result, the remaining documents in the NACSOS management tool are ordered according to their predicted relevance based on the previous inclusion decisions. After each subsequent round of abstract screening, the machine learning classifier gets re-trained. As more documents are manually screened, the machine learning algorithm is updated using these additional labels, and the remaining documents are reordered by predicted relevance score. See Fig. 4 for an overview of the search and screening process.

Adapted from the ROSES flow diagram for systematic reviews. 40

Methods for meta-analysis

Dependent variable.

The dependent variable of our econometric analysis is the change in public support of a carbon pricing policy following the introduction of a revenue recycling option. One main obstacle for the comparison of surveys is that they use different measures of public support. This results from the variety of rating scales such as Likert- scales (with different levels across the studies), binary scales (support-not support), or stated preferences among policy alternatives makes meaningful comparisons of the effect sizes found in different studies difficult. We therefore take the direction of the effect (that is, increase vs. decrease in public support) as the dependent variable in our regression analysis, similar to meta-analyses conducted by refs. 19 , 20 .

Moderator variables

Different moderator variables in our model account for the variation in the effects of revenue uses reported in our sample of studies. First, we create indicator variables, which capture the different survey designs that were presented to the respondents. For this, we include variables for the different revenue use options. We distinguish between redistribution schemes to the whole population or targeted to parts of it ( uniform cash transfers and targeted cash transfers ), infrastructure and other environmental and climate projects ( green spending ), tax-related relief mechanisms that could directly benefit populations (income and consumption captured in tax cuts) , and relief mechanisms for firms ( corporate tax cuts ). We use public finance as a reference category for the regression. Hence, this meta-regression allows us to make statements about the impact of different revenue use options on the observed changes in public support relative to the effect of transferring it to the government budget.

In addition to variables that account for the revenue recycling schemes tested, our specifications account for the design of the survey type (survey questionnaire or choice experiment). It also accounts for the type of policy respondents support (carbon tax, fuel tax, or fossil fuel subsidy reform policy). We also include a dummy that indicates whether a carbon policy was already in place in the country under study at the time of the survey, and region fixed effects. We assume fixed effects for country region groups instead of single countries to decrease the risk of multi-collinearity and imprecise results in our regression analysis. Other interesting aspects of the survey designs such as the mode of data collection or the different measures of acceptance do not show enough variation to be included in our analysis.

Logistic regression for meta-analysis

We run a logit regression to determine the relationship between the dependent variable and the moderator variables. Suppressing the observation-specific index, the logit model assumes a continuous latent variable \({y}^{* }\) to measure the propensity for a study to find an increase in support caused by the introduction of a revenue recycling scheme. \({y}^{* }\) is supposed to react continuously to the characteristics gathered in the vector of moderator variables \({\bf{x}}\) . However, we cannot observe \({y}^{* }\) , but a binary variable \(y\) equal to 1 if the unobserved latent variable \({y}^{* }\) breaks a threshold theta, and 0 otherwise. Denoting \(\alpha\) the constant and \(\varepsilon\) the normally distributed error term, the model we want to estimate can be expressed as follows:

We can set \(\theta =0\) without a loss of generality. The resulting probability of a survey indicating a decrease in public support caused by the introduction of a revenue recycling scheme can be written as follows:

Using straightforward algebra, this equation can be transformed to:

We cluster standard errors on survey level to account for non-independence of observations. We then estimate our model with the maximum likelihood method. The \(\beta\) -coefficients and their p -values provide the direction and the significance of the effects of the moderator variables: A positive \(\beta\) suggests that a moderator variable increases the probability of obtaining an increase in public support when introducing a revenue recycling mechanism \(\Pr (y=1)\) . We provide marginal effects at mean, which show the magnitude of the probability change for the two possible outcomes induced by the moderator variables. We report the Pseudo R 2 as a measure of fit 39 . We also test the validity of the logit specification by OLS regressions (see Supplementary Table 8 ).

Data availability

The data and codebook of this study are made available in a public repository and can be accessed here: https://github.com/farahmkv/PublicSupport_RR_CP .

Code availability

The code used for the meta-analysis can be accessed here: https://github.com/farahmkv/PublicSupport_RR_CP .

Baranzini, A. & Carattini, S. Effectiveness, earmarking and labeling: testing the acceptability of carbon taxes with survey data. Environ. Econ. Policy Stud. 19 , 197–227 (2017).

Article   Google Scholar  

High-Level Commission on Carbon Prices. Report of the High-Level Commission on Carbon Prices . (2017).

Klenert, D. et al. Making carbon pricing work for citizens. Nat. Clim. Change 8 , 669–677 (2018).

Pigou, A. The Economics of Welfare . (Macmillan, 1920).

Baranzini, A., Goldemberg, J. & Speck, S. A future for carbon taxes. Ecol. Econ. 32 , 395–412 (2000).

Maestre-Andrés, S., Drews, S. & van den Bergh, J. Perceived fairness and public acceptability of carbon pricing: a review of the literature. Clim. Policy 19 , 1186–1204 (2019).

Nowlin, M. C., Gupta, K. & Ripberger, J. T. Revenue use and public support for a carbon tax. Environ. Res. Lett. 15 , 084032 (2020).

Sælen, H. & Kallbekken, S. A choice experiment on fuel taxation and earmarking in Norway. Ecol. Econ. 70 , 2181–2190 (2011).

Sommer, S., Mattauch, L. & Pahle, M. Supporting carbon taxes: The role of fairness. Ecol. Econ. 195 , 107359 (2022).

Jagers, S. C., Lachapelle, E., Martinsson, J. & Matti, S. Bridging the ideological gap? How fairness perceptions mediate the effect of revenue recycling on public support for carbon taxes in the United States. Canada and Germany. Rev. Policy Res. 38 , 529–554 (2021).

Kaplowitz, S. A. & McCright, A. M. Effects of policy characteristics and justifications on acceptance of a gasoline tax increase. Energy Policy 87 , 370–381 (2015).

Dolšak, N., Adolph, C. & Prakash, A. Policy design and public support for carbon tax: Evidence from a 2018 US national online survey experiment. Public Adm. 98 , 905–921 (2020).

Maestre-Andrés, S., Drews, S., Savin, I. & van den Bergh, J. Carbon tax acceptability with information provision and mixed revenue uses. Nat. Commun. 12 , 7017 (2021).

Carl, J. & Fedor, D. Tracking global carbon revenues: A survey of carbon taxes versus cap-and-trade in the real world. Energy Policy 96 , 50–77 (2016).

Mildenberger, M., Lachapelle, E., Harrison, K. & Stadelmann-Steffen, I. Limited impacts of carbon tax rebate programmes on public support for carbon pricing. Nat. Clim. Change 12 , 141–147 (2022).

Fremstad, A., Mildenberger, M., Paul, M. & Stadelmann-Steffen, I. The role of rebates in public support for carbon taxes. Environ. Res. Lett. 17 , 084040 (2022).

Dechezleprêtre, A. et al. Fighting Climate Change: International Attitudes Toward Climate Policies . w30265 http://www.nber.org/papers/w30265.pdf https://doi.org/10.3386/w30265 . (2022).

Kyselá, E., Ščasný, M. & Zvěřinová, I. Attitudes toward climate change mitigation policies: a review of measures and a construct of policy attitudes. Clim. Policy 19 , 878–892 (2019).

Ohlendorf, N., Jakob, M., Minx, J. C., Schröder, C. & Steckel, J. C. Distributional Impacts of Carbon Pricing: A Meta-Analysis. Environ. Resour. Econ. 78 , 1–42 (2021).

Bergquist, M., Nilsson, A., Harring, N. & Jagers, S. C. Meta-analyses of fifteen determinants of public opinion about climate change taxes and laws. Nat. Clim. Change 12 , 235–240 (2022).

Higgins, J. et al. Cochrane Handbook for Systematic Reviews of Interventions . (John Wilsey & Sons, 2019).

Stantcheva, S. How to Run Surveys: A guide to creating your own identifying variation and revealing the invisible.

Kallbekken, S. & Sælen, H. Public acceptance for environmental taxes: Self-interest, environmental and distributional concerns. Energy Policy 39 , 2966–2973 (2011).

Douenne, T. & Fabre, A. French attitudes on climate change, carbon taxation and other climate policies. Ecol. Econ. 169 , 106496 (2020).

Kallbekken, S., Kroll, S. & Cherry, T. L. Do you not like Pigou, or do you not understand him? Tax aversion and revenue recycling in the lab . J. Environ. Econ. Manag. 62 , 53–64 (2011).

Google Scholar  

Dresner, S., Dunne, L., Clinch, P. & Beuermann, C. Social and political responses to ecological tax reform in Europe: an introduction to the special issue. Energy Policy 34 , 895–904 (2006).

Grimsrud, K. M., Lindhjem, H., Sem, I. V. & Rosendahl, K. E. Public acceptance and willingness to pay cost-effective taxes on red meat and city traffic in Norway. J. Environ. Econ. Policy 9 , 251–268 (2020).

Hamilton, W. L., Biener, L. & Rodger, C. N. Who Supports Tobacco Excise Taxes? Factors Associated With Towns’ and Individuals’ Support in Massachusetts. J. Public Health Manag. Pract. 11 , 333 (2005).

Beiser-McGrath, L. F. & Bernauer, T. Could revenue recycling make effective carbon taxation politically feasible? Science Advances . https://doi.org/10.1126/sciadv.aax3323 (2019).

Douenne, T. & Fabre, A. Yellow Vests, Pessimistic Beliefs, and Carbon Tax Aversion. American Economic Journal: Economic Policy , 14 , 81–110 (2022). https://doi.org/10.1257/pol.20200092 .

Ewald, J., Sterner, T. & Sterner, E. Understanding the resistance to carbon taxes: Drivers and barriers among the general public and fuel-tax protesters. Resource and Energy Economics , 70 , 101331 (2022). https://doi.org/10.1016/j.reseneeco.2022.101331 .

Moseley, M. W. Contentious Engagement: Understanding Protest Participation in Latin American Democracies. J. Polit. Lat. Am. 7 , 3–48 (2015).

Bank, W. State and Trends of Carbon Pricing 2022 . (Washington, DC: World Bank, 2022). https://doi.org/10.1596/978-1-4648-1895-0 .

Pullin, A., Frampton, G., Livoreil, B. & Petrokofsky, G. Guidelines and Standards for Evidence Synthesis in Environmental Management . Version 5.0 . (2018).

Callaghan, M. W. & Müller-Hansen, F. Statistical stopping criteria for automated screening in systematic reviews. Syst. Rev. 9 , 273 (2020).

van de Schoot, R. et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat. Mach. Intell. 3 , 125–133 (2021).

Muhammad, I., Mohd Hasnu, N. N. & Ekins, P. Empirical Research of Public Acceptance on Environmental Tax: A Systematic Literature Review. Environments 8 , 109 (2021).

Callaghan, M., Müller-Hansen, F., Hilaire, J. & Lee, Y. T. NACSOS: NLP Assisted Classification. Synthesis and Online Screening. Zenodo https://doi.org/10.5281/ZENODO.4121526 (2020).

McFadden, D. Frontiers in Econometrics, Chapter Conditional Logit Analysis of Qualitative Choice Behavior . (Academic Press New York, NY, USA, 1974).

Haddaway, N., Macura, B., Whaley, P. & Pullin, A. ROSES flow diagram for systematic reviews. Version 1.0 https://figshare.com/articles/ROSES_Flow_Diagram_Version_1_0/5897389 (2018).

Download references

Acknowledgements

F.M.V. is supported by a PhD stipend from the Heinrich Böll Stiftung. We further acknowledge funding from the European Union’s Horizon Europe Research and Innovation Programme (ELEVATE project – Grant No. 101056873 and CAPABLE project – Grant No. 101056891) as well as the International Development Research Centre (Supporting Low-Carbon Transition and Gender Equity in the Global South project). We also thank the participants of the workshop “Environmental policy attitudes in an era of crises” held during the 2022 Nordic Environmental Social Science conference for their helpful comments and suggestions. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. Views and opinions expressed are only those of the authors.

Author information

Authors and affiliations.

Mercator Research Institute on Global Commons and Climate Change, Berlin, Germany

Farah Mohammadzadeh Valencia, Cornelia Mohren, Marlene Merchert, Jan C. Minx & Jan Christoph Steckel

Brandenburg University of Technology, Cottbus, Germany

Farah Mohammadzadeh Valencia & Jan Christoph Steckel

Paris School of Economics, Paris, France

Cornelia Mohren

Environment for Development, Gothenburg University, Gothenburg, Sweden

Anjali Ramakrishnan

Technische Universität Berlin, Berlin, Germany

Marlene Merchert

University of Leeds, Leeds, United Kingdom

Jan C. Minx

You can also search for this author in PubMed   Google Scholar

Contributions

F.M.V., J.C.S., J.C.M., and A.R. designed the research. F.M.V., A.R., and C.M. developed the literature screening strategy. F.M.V., C.M., A.R., and M.M. manually screened the literature and extracted the data. F.M.V. and A.R. performed the machine learning-enabled screening. C.M. and A.R. performed the meta-analysis. F.M.V., C.M., and J.C.S. analysed the results. F.M.V., C.M., and J.C.S. wrote the manuscript with contributions from all authors.

Corresponding authors

Correspondence to Farah Mohammadzadeh Valencia or Jan Christoph Steckel .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Mohammadzadeh Valencia, F., Mohren, C., Ramakrishnan, A. et al. Public support for carbon pricing policies and revenue recycling options: a systematic review and meta-analysis of the survey literature. npj Clim. Action 3 , 74 (2024). https://doi.org/10.1038/s44168-024-00153-x

Download citation

Received : 31 October 2023

Accepted : 30 July 2024

Published : 07 September 2024

DOI : https://doi.org/10.1038/s44168-024-00153-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Open access
• Published: 31 August 2024

Impaired glucose metabolism and the risk of vascular events and mortality after ischemic stroke: A systematic review and meta-analysis

• Nurcennet Kaynak   ORCID: orcid.org/0000-0002-0637-8421 1 , 2 , 3 , 4 , 5 ,
• Valentin Kennel   ORCID: orcid.org/0009-0000-0354-4167 1 , 2 ,
• Torsten Rackoll   ORCID: orcid.org/0000-0003-2170-5803 2 , 6 ,
• Daniel Schulze   ORCID: orcid.org/0000-0001-9415-2555 7 ,
• Matthias Endres   ORCID: orcid.org/0000-0001-6520-3720 1 , 2 , 4 , 5 , 8 &
• Alexander H. Nave   ORCID: orcid.org/0000-0002-0101-4557 1 , 2 , 3 , 5  

Cardiovascular Diabetology volume  23 , Article number:  323 ( 2024 ) Cite this article

1 Altmetric

Metrics details

Diabetes mellitus (DM), prediabetes, and insulin resistance are highly prevalent in patients with ischemic stroke (IS). DM is associated with higher risk for poor outcomes after IS.

Investigate the risk of recurrent vascular events and mortality associated with impaired glucose metabolism compared to normoglycemia in patients with IS and transient ischemic attack (TIA).

Systematic literature search was performed in PubMed, Embase, Cochrane Library on 21st March 2024 and via citation searching. Studies that comprised IS or TIA patients and exposures of impaired glucose metabolism were eligible. Study Quality Assessment Tool was used for risk of bias assessment. Covariate adjusted outcomes were pooled using random-effects meta-analysis.

Main outcomes

Recurrent stroke, cardiac events, cardiovascular and all-cause mortality and composite of vascular outcomes.

Of 10,974 identified studies 159 were eligible. 67% had low risk of bias. DM was associated with an increased risk for composite events (pooled HR (pHR) including 445,808 patients: 1.58, 95% CI 1.34–1.85, I 2  = 88%), recurrent stroke (pHR including 1.161.527 patients: 1.42 (1.29–1.56, I 2  = 92%), cardiac events (pHR including 443,863 patients: 1.55, 1.50–1.61, I 2  = 0%), and all-cause mortality (pHR including 1.031.472 patients: 1.56, 1.34–1.82, I 2  = 99%). Prediabetes was associated with an increased risk for composite events (pHR including 8,262 patients: 1.50, 1.15–1.96, I 2  = 0%) and recurrent stroke (pHR including 10,429 patients: 1.50, 1.18–1.91, I 2  = 0), however, not with mortality (pHR including 9,378 patients, 1.82, 0.73–4.57, I 2  = 78%). Insulin resistance was associated with recurrent stroke (pHR including 21,363 patients: 1.56, 1.19–2.05, I 2  = 55%), but not with mortality (pHR including 21,363 patients: 1.31, 0.66–2.59, I 2  = 85%).

DM is associated with a 56% increased relative risk of death after IS and TIA. Risk estimates regarding recurrent events are similarly high between prediabetes and DM, indicating high cardiovascular risk burden already in precursor stages of DM. There was a high heterogeneity across most outcomes.

Introduction

Ischemic stroke (IS) is associated with high mortality and high risk of recurrent vascular events worldwide [ 1 , 2 , 3 ]. Despite adequate secondary prevention, about 11% of patients suffer a recurrent stroke within the first year [ 4 ]. Diabetes mellitus (DM) is a highly prevalent cardiovascular risk factor and is present in about one-third of IS patients [ 5 , 6 ]. Stroke prevention guidelines recommend screening for unrecognized DM after IS [ 7 ]. Besides DM, other forms of impaired glucose metabolism (IGM), such as prediabetes and insulin resistance (IR) have been gaining importance over the last decades in terms of their association with increased cardiovascular risk [ 8 ]. Prediabetes, comprising impaired fasting glucose and impaired glucose tolerance, represents a hyperglycemic condition of patients not yet within the diabetic range [ 9 ]. In comparison, IR constitutes a pathophysiological mechanism, which usually precedes and coexists with both DM and prediabetes [ 10 ]. Observational studies report that 70% of the patients with IS have either DM (46%) or prediabetes (24%), and 50% of those who have no DM at baseline have IR [ 11 , 12 ].

Considering that the majority of patients with stroke have some form of IGM, it represents an important aspect of secondary stroke prevention. Numerous studies, including systematic reviews, have shown the association between DM and prediabetes and stroke recurrence [ 13 , 14 , 15 ]. However, only few studies have looked at composite vascular events as an outcome. Furthermore, mortality risk associated with DM after stroke has not been addressed in previous meta-analyses. A comprehensive systematic approach is needed to identify and compare risks associated with composite vascular events and mortality after IS and TIA between different forms of IGM.

Stroke prevention guidelines recommend the use of new generation antidiabetics based on the finding that these agents demonstrated cardiovascular protective effects in patients with previous cardiovascular disease including stroke [ 7 ]. However, only the minority of patients had a history of stroke and subgroup analyses of patients with a previous IS or TIA remained mostly inconclusive [ 16 , 17 ]. In contrast, in the IRIS Trial only patients with IR and a recent IS or TIA were included [ 18 ]. Despite the lower risk of cardiovascular events associated with pioglitazone, the high risk of adverse events restricted the clinical implication of the drug. Currently, it remains unclear which pharmacological treatments are beneficial in terms of secondary stroke prevention in patients with acute or subacute IS or TIA and different forms of IGM.

Identifying increased cardiovascular risk not only in DM but also other forms of IGM would capture a greater population at risk and eventually prompt implementation of secondary preventive measures. We conducted a systematic literature review and meta-analysis to extend our knowledge on the burden of IGM in patients with IS and TIA in the context of cardiovascular events and mortality.

This manuscript adheres to the PRISMA guideline [ 19 ]. Study protocol was pre-registered in open science framework in 2021 [ 20 ].

Information sources

We conducted a systematic literature search on Medline via Pubmed, Ovid via Embase, and Cochrane Library that was last updated on March 21, 2024. Search terms included “diabetes”, “prediabetes”, “insulin resistance”, “stroke” and “transient ischemic attack”, restricted to English language. See full search strategy in supplementary material methods. Reference lists of previous systematic reviews and of studies included in our review were searched manually.

Study selection and data extraction

Screening was performed by two reviewers independently (NK and VK) and consensus was reached with two additional reviewers (TR and AHN) in case of disagreement. Eligible studies were observational studies that included patients within 3 months after an IS or TIA and reported at least one of the following outcomes: composite vascular events, recurrent stroke, cardiovascular and all-cause mortality, cardiac events including but not limited to myocardial infarction, all regardless of follow-up duration (see supplementary Table 1 for the eligibility criteria). Composite events comprised at least stroke, cardiac events, and cardiovascular death. Studies were required to report hazard ratios (HR), odds ratios (OR), or risk ratios using a multivariable model. Exposures of interest were DM, prediabetes and IR, which were included independently of the definition used in the respective study. Additionally, we screened for studies that compared the use of an antidiabetic therapy to placebo or another antidiabetic therapy within the same population and outcomes mentioned above, regardless of study design.

Data extraction and assessment of risk of bias were performed by one reviewer (NK) and the internal validity was checked with a second reviewer (VK) for a random sample of 10% of studies. Interrater reliability was calculated. Authors were contacted via email if substantial outcome data were lacking, unclear or discrepant. Risk of bias assessment was made using the Study Quality Assessment Tool of National Heart, Lung, and Blood Institute [ 21 ]. A detailed methodological description can be found in the methods section of the supplementary material.

Data synthesis

We performed random effects meta-analyses with the restricted maximum likelihood estimator method after grouping studies into outcome measures HR for each study outcome. OR were pooled using meta-regression with follow-up duration as moderator and with random effects meta-analysis if moderator showed no significant effect (p < 0.05). Studies used different sets of covariates that included sociodemographic and clinical characteristics. We included the effect size from the models with the most adjusting factors available. We calculated the 95% confidence interval (CI) and prediction intervals. Prediction intervals describe the expected range of future study results, while confidence intervals relate to the precision of the aggregated effect. Multi-level meta-analysis was performed if multiple subgroups from a single study were included in the analysis. Furthermore, we performed meta-analyses of absolute risks derived from event numbers for each outcome and exposure group, whenever such data were reported. Heterogeneity was assessed using Cochran’s Q and I 2 and was assumed present when p < 0.05 or I 2  > 50% [ 22 ]. Results of meta-analyses were visualized using forest plots. Subgroup analyses were conducted based on history of previous stroke (first-ever event, yes/no) and type of ischemic event (IS/TIA/both). Subgroup analyses based on sex were not conducted because the studies included both sexes in their analyses, and individual patient data were not available. As a sensitivity analysis, we conducted meta-analyses using unadjusted odds ratios. Publication bias was assessed by funnel plots and Egger´s regression. Statistical calculations were performed using the Software R Version 4.0.2 with the package “Metafor” [ 23 ]. Studies investigating the association between antidiabetic therapies and recurrent cardiovascular events after IS or TIA were summarized narratively.

Systematic literature search

The systematic literature search yielded 10,974 records. After screening titles and abstracts, 8,219 records were excluded, and 1,717 records were further screened based on full texts (Fig.  1 ). Finally, 159 studies met the eligibility criteria (supplementary references). Of those, 26 reported data for composite outcome, 71 for recurrent stroke, 10 for cardiac events, 104 for all-cause mortality, and five for cardiovascular mortality (Table  1 ). During data extraction an inter-rater reliability of 90% was reached. Authors of twenty-six studies were contacted for missing information, and seven of them provided the requested data. Most studies were observational studies (n = 146), and others were post-hoc analyses of randomized trials (n = 13). Follow-up duration ranged from end-of-hospital-stay to longer than 20 years. The diagnostic criteria used for DM varied highly including based on medical records or medication history only (n = 61), laboratory biomarkers only (n = 14) and both (n = 50). Twenty-one studies did not report the definition used. Prediabetes was defined either according to American Diabetes Association [ 24 ] or World Health Organization criteria [ 25 ], whereas one study defined prediabetes as a non-fasting glucose level of 140–198 mg/dL. IR was quantified using: HOMA-IR, Triglyceride-Glucose Index, Matsuda Insulin Sensitivity Index, Glucose/Insulin Ratio, QUICKI Index, and estimated glucose disposal rate. Overall, 67% (n = 107) of the included studies were rated as having good quality of evidence, 27% (n = 43) as fair and 6% (n = 9) as poor (supplementary Fig. 1). Study characteristics are presented in supplementary Table 2.

Flowchart of the screening and selection process of the systematic review

Association of IGM with cardiovascular events

Composite vascular events.

Twenty-four studies were eligible for the exposure DM, three studies for prediabetes and two studies for IR. Five studies reporting data from the same cohort were excluded, resulting in 19 eligible studies for the exposure DM (16 reported HR, three reported OR; see supplementary Table 3). Except for one study reporting a 3-month follow-up period, all studies reported at least 1-year follow-up. One study that assessed incident DM during follow-up opposed to pre-existing DM as an exposure was not included in the analysis [ 26 ].

Presence of DM was statistically significantly associated with an increased risk of composite vascular events with a pooled HR (pHR) of 1.58 (95% confidence interval (CI) 1.34 to 1.85, I 2  = 88%) including 445,808 patients (Fig.  2 A) and a pooled OR (pOR) of 1.87 (95% CI 0.76 to 4.60, I 2  = 64%) including 1,609 patients. No publication bias was observed (supplementary Fig. 2). The meta-analysis of absolute risks reported in seven studies revealed that during a mean follow-up of three years, 43% (95% CI 23% to 64%) of stroke patients with DM reached a composite endpoint of a recurrent cardiovascular event or death. This rate was 17% (95% CI 3% to 31%) in patients without DM (supplementary Table 4).

a Forest plot for the meta-analysis of studies that reported the association of diabetes with composite outcome. b Forest plot for the meta-analysis of studies that reported the association of prediabetes with composite outcome

Meta-analysis of two studies showed an increased risk of composite events associated with prediabetes with a pHR of 1.50 (95% CI 1.15 to 1.96, I 2  = 0%; Fig.  2 B) in 8,262 patients. An absolute risk of 31% (95% CI 12% to 50%) and 7% (95% CI 5% to 10%) was observed in the group of patients with and without prediabetes, respectively. IR was reported in two studies, which were derived from the same cohort. One of the studies demonstrated no association between high IR and composite vascular events [ 27 ]. In the other study, which only encompassed patients without DM, increased IR based on HOMA-IR was statistically significantly associated with an increased risk for vascular events [ 28 ].

Recurrent stroke

Sixty-three studies reported recurrent stroke outcome data in patients with DM, see supplementary Table 5. Follow-up duration ranged from discharge from hospital to a mean follow-up time of 12.3 years. Studies encompassing the same population were excluded from the analysis. Finally, 40 studies reporting HR and 12 studies reporting OR were eligible for analysis, respectively. The pHR was 1.42 (95% CI 1.29 to 1.56, I 2  = 92%; Fig.  3 A) involving 1.161.527 patients. There was evidence for possible publication bias (supplementary Fig. 3). Studies that reported OR involving 47,629 patients showed a similar increase of risk (pOR 1.33, 95% CI 1.13 to 1.56, I 2  = 48%; supplementary Fig. 4). Follow-up duration was not a statistically significant moderator for the outcome (p = 0.40). Neither the type of baseline event (IS or TIA), nor previous stroke was a statistically significant moderator (p = 0.08 and p = 0.90, respectively, see supplementary Fig. 5) in subgroup analyses. Baujat plots revealed that the studies contributing most to heterogeneity had a design of post-hoc analysis of randomized trials. Meta-analysis of absolute risks extracted from 23 studies resulted in 13% (95% CI 10% to 16%) for patients with diabetes vs. 9% (95% CI 6% to 11%) without, within a follow-up period of more than a year.

a Forest plot for the meta-analysis of studies that reported the association of diabetes with recurrent stroke. b Forest plot for the meta-analysis of studies that reported the association of prediabetes with recurrent stroke. c Forest plot for the meta-analysis of studies that reported the association of insulin resistance with recurrent stroke

Patients with prediabetes had an increased risk for recurrent stroke compared to patients with normoglycemia (pHR in 10,429 patients 1.50, 95% CI 1.18 to 1.91, I 2  = 0%, see Fig.  3 B). This was also the case in terms of absolute risk 10% (95% CI 8% to 12%) and 7% (95% CI 7% to 8%), respectively. Of five studies eligible for IR, only three could be included in the meta-analysis, because multiple studies were conducted in the same cohort. The pHR for recurrent stroke associated with IR in 21,363 patients was 1.56, 95% CI 1.19 to 2.05, I 2  = 55% (Fig.  3 C). Absolute risks associated with IR during 10.4 months follow-up was 10% (95% CI 5% to 15%) vs. 7% (95% CI 6% to 7%) in patients without increased IR.

Cardiac events

All studies eligible for cardiac events comprised DM as the exposure, see supplementary Table 6. The shortest follow-up time was three months, all other studies followed patients for at least one year. One study that investigated new DM during follow-up was not included in the meta-analysis [ 26 ]. Presence of DM was associated with an increased risk of cardiac events with a pHR of 1.55 (95% CI 1.50 to 1.61, I 2  = 0%) involving 443,863 patients. The pOR of two studies with 839,029 patients was 1.47 (95% CI 0.48 to 4.44), I 2  = 89% (supplementary Fig. 6). Meta-analysis of three studies reporting data revealed an absolute risk of 5% (95% CI − 1% to 11%) in patients with DM and 3% (95% CI 0% to 6%) without DM. One study that investigated prediabetes reported a HR of 2.0 (95% CI 1.30 to 3.20) for cardiac events. No study reported IR as an exposure.

Association between IGM and mortality

Cardiovascular mortality.

Five studies reported data of cardiovascular mortality in patients with DM (supplementary Table 7). Meta-analysis involving 127,445 patients showed a statistically significant association between DM and cardiovascular mortality (pHR 1.65, 95% CI 1.41 to 1.93, I 2  = 50%, see supplementary Fig. 7). Pooling available data of absolute risks from three studies, resulted in a pooled risk of 18% (95% CI −10% to 47%) in patients with DM vs. 16% (95% CI −9% to 41%) in patients without DM, during 1 year of follow-up.

All-cause mortality

Ninety-four studies investigated associations between all-cause mortality and DM, see supplementary Table 8. Studies that included patients from the same population were excluded from the analysis (n = 10). Presence of DM was associated with an increased risk for all-cause mortality (pHR 1.56, 95% CI 1.34 to 1.82, I 2  = 99%, see Fig.  4 A) summarizing 42 studies including 1.031.472 patients. Subgroup analyses based on follow-up duration resulted in a pHR of 1.10 (95% CI 0.72 to1.68) during hospitalization (n = 3 studies), pHR of 1.35 (95% CI 1.18 to 1.56) up to one year (n = 12 studies), and pHR of 1.74 (95% CI 1.40 to 2.17) longer than one year (n = 27 studies). However, follow-up duration was not revealed as a statistically significant moderator (p = 0.15, see supplementary Fig. 8). The Galbraith plot revealed the most influential studies to be the subgroups of the study from Zamir et al. (supplementary Fig. 9). The meta-analysis of forty-two studies involving 3.290.353 patients reporting OR showed a risk estimate of 1.30 (95% CI 1.21 to 1.41, see supplementary Fig. 10). Subgroup analyses based on first-ever vs. recurrent event at baseline and the type of ischemic event revealed no statistically significant difference between groups. Funnel plots suggested existence of publication bias (supplementary Fig. 11). During a mean follow-up of 1.8 months, the absolute risk of all-cause mortality was 23% (95% CI 14% to 31%) for patients with DM vs. 17% (95% CI 11% to 23%) without DM.

a Forest plot for the meta-analysis of studies that reported the association of diabetes with all-cause mortality. b Forest plot for the meta-analysis of studies that reported the association of prediabetes with all-cause mortality. c Forest plot for the meta-analysis of studies that reported the association of insulin resistance with all-cause mortality

Six studies were eligible for prediabetes and all-cause mortality (3 HR, 3 OR). Prediabetes was not statistically significantly associated with an increased risk for mortality after IS (pHR 1.82, 95% CI 0.73 to 4.57, I 2  = 78% in 9,378 patients, and pOR 1.37, 95% CI 0.54 to 3.43, I 2  = 71% in 1,969 patients, see Fig.  4 B & supplementary Fig. 12). Meta-analysis of absolute risks during a mean follow-up of seven months was 8% (95% CI 2% to 15%) for patients with prediabetes vs. 9% (95% CI 0% to 18%) with normoglycemia.

Nine studies reported IR as an exposure. The meta-analyses could not demonstrate an association between increased IR and mortality (pHR 1.31, 95% CI 0.66 to 2.59, I 2  = 85%, including 21,363 patients across three studies and pOR 1.05, 95% CI 0.76 to 1.45, I 2  = 16%, including 6,434 patients across 2 studies). Absolute risks were 6% (95% CI -1% to 12%) for patients with increased IR and 4% (95% CI 2% to 6%) without.

Sensitivity analyses with crude odds ratios

Sensitivity analyses using unadjusted odds ratios, to accommodate the variation in adjustment factors used across studies, revealed similar risk estimates, though often slightly higher than the respective adjusted pooled outcomes (supplementary Fig. 13 and 14).

Antidiabetic therapy and recurrent vascular events

Nine observational studies investigated the association between antidiabetic therapies and cardiovascular events after an IS or TIA in the preceding three months, see Table  2 . The drug classes investigated were metformin, sulfonylurea, thiazolidinedione, and incretin-mimetics. We did not identify and studies with SGLT-2 Inhibitors or alfa glucosidase inhibitors. Due to the differences in the exposure and comparator groups, we did not perform a meta-analysis. Studies showed a risk reduction for recurrent stroke, mortality and composite vascular events associated with the use of pioglitazone and lobeglitazone as well as a lower risk of mortality associated with metformin use [ 29 , 30 , 31 , 32 ]. There were no clear benefits in terms of decreased risk of cardiovascular events associated with sulfonylurea or incretin-mimetics [ 33 , 34 , 35 , 36 , 37 ].

In this systematic review and meta-analysis, we provide a comprehensive and up-to-date summary of previous studies investigating the association between IGM and residual cardiovascular risk following IS and TIA. To our knowledge, this is the first meta-analysis to investigate the risk of composite vascular events associated with IGM as well as the risk of mortality associated with DM in this population. The results of the presented meta-analysis indicate that (1) patients with DM have an approximately 1.6-fold (60%) increased risk of both death and recurrent vascular events after IS and TIA, (2) the risk of recurrent vascular events after stroke is already increased in the prediabetic stage and appears just as high as in patients with DM, and (3) presence of IR is associated with recurrent stroke risk. In contrast, this meta-analysis was unable to demonstrate an increased mortality risk after stroke associated with prediabetes or IR. Overall, there were significantly fewer eligible studies on prediabetes and IR compared to DM (Table  1 ).

DM is a well-known risk factor for cardiovascular disease. The results of our study confirm a robust association between DM and risk of composite recurrent vascular events after IS and TIA. We could confirm the risk of recurrent stroke associated with DM that was previously reported in a meta-analysis by Zhang et al . [ 14 ] The risk of mortality in patients with DM is observed to be 56% higher compared to patients without DM. Although mortality risk estimates were greater for diabetic patients with increasing mean follow-up durations of studies, we could not observe a statistically significant interaction between mortality risk and follow-up duration. This could be due to the fact that there were only a few studies with short-term follow-up in studies that reported HR (supplementary Fig. 8) and only a few studies with long-term follow-up in studies that reported OR (supplementary Fig. 10). Still, inferring from this finding, DM likely remains a relevant risk factor over time and an important target for secondary prevention strategies, given the high prevalence of DM in this population [ 6 ].

Our analyses demonstrated a positive relationship between prediabetes and recurrent vascular events as well as between IR and stroke recurrence. However, there was no association detected between the two conditions and mortality. This difference could have several reasons: First, patients with prediabetes or IR are less likely to have been exposed to deleterious effects of a dysregulated glucose metabolism for a longer time, compared to patients with DM. Second, the shorter follow-up duration of studies investigating prediabetes and IR generally limits the probability to detect difference in mortality risk. The risk associated with prediabetes and recurrent stroke is in line with a previous meta-analysis conducted by Pan et al . in 2019 [ 15 ]. Despite substantial methodological differences such as avoiding pooling ORs and HRs together and excluding studies with hemorrhagic stroke in our study, also having identified two more studies, similar to Pan et al ., we also could not demonstrate a relationship between prediabetes and mortality.

Contrary to DM, prediabetes has rather recently been regarded as a cardiovascular risk factor [ 39 ]. The meta-analysis conducted by Cai et al. showed a risk increase in all-cause mortality and vascular events associated with prediabetes in population-based cohorts as well as in patients with previous atherosclerotic disease [ 40 ]. Further, a recent analysis of the UK Biobank cohort including more than 400 thousand individuals confirmed the excess risk for any cardiovascular disease in patients with IGM compared to normoglycemia [ 41 ]. The risk was higher for DM than for prediabetes. Still, after accounting for obesity and use of antihypertensive and statins both risks were attenuated, lending support to the modifiability of the excess risk. Together with these previous findings, our results strongly support considering prediabetes as a continuous entity with DM on the spectrum of IGM, with a relevant increase in cardiovascular and mortality risk.

There was a statistically significant association between increased IR and stroke recurrence. However, it should be noted that, there were only three studies eligible for the analysis and the parameters used to define an increased IR as well as the timing of measurement after stroke (7 days and 14 days) was heterogeneous between studies. IR can be increased during the acute phase of the stroke due to the stress reaction and show changes during this time [ 42 ]. The increased relative risk for recurrent stroke observed in patients with IR compared to patients without IR was higher than the relative risk in diabetics compared to non-diabetics. This might be explained by the differences in the patient groups. Patients with DM are more likely to receive antidiabetic treatment and have a higher risk of dying before suffering a recurrent stroke. Another difference could be in the comparator groups, namely that the patients without IR could be generally healthier than patients without DM.

Despite the association between increased IR and stroke recurrence, we could not identify many studies with other cardiovascular outcomes. Furthermore, we encountered different parameters and criteria to define IR across studies. Thus, prognostic value of increased IR in terms of composite cardiovascular risk as well as the best biomarker to predict the said risk remains speculative in patients with IS or TIA. Further research is needed to investigate this conundrum.

We observed a significant research gap in the number of large studies with congruent definitions of prediabetes and IR. Uncertainty remains about the different diagnostic criteria for both prediabetes and IR [ 24 , 25 , 43 , 44 ], leading to the lack of adequate implementation of preventive strategies [ 45 ]. As the prevalence of prediabetes expected to rise, the whole spectrum of IGM rather than DM alone is assumed to gain more significance in terms of primary and secondary stroke prevention [ 46 ]. Consistent diagnostic criteria would facilitate a reliable data synthesis and the development of prevention strategies.

Until the advent of the GLP1 and SGLT2 therapies, no antidiabetic therapy has improved cardiovascular risk or death despite improvements in glucose control [ 47 ]. Both classes of drugs revolutionized the field after randomized controlled trials showed cardiovascular risk reduction in patients with DM [ 48 , 49 , 50 , 51 ]. However, until now, it is unclear if these drugs are equally effective at reducing cardiovascular risk in patients with IS [ 33 , 34 , 52 ]. As our systematic review indicates, to date, only few studies exist that investigated the effectiveness of antidiabetic therapy in preventing recurrent vascular events after an acute or subacute IS. Even though the promising results related to pioglitazone use in patients with IR from the IRIS trial unfortunately faced a limitation due to side effects [ 18 ], recent cohort studies shown beneficial effects associated with thiazolidinediones [ 29 , 30 ]. Clinical trial investigating secondary stroke prevention in patients with prediabetes are yet to been undertaken.

Strengths and limitations

The most important strength of our study lies in the comprehensiveness, encompassing over 10.000 records and having included more than seven million patients over all exposures and outcomes. This enabled us to investigate all three entities of IGM together. Another strength constitutes the methodology. We included studies with both outcome measures HR and OR, which led us to identify more studies. We also used multi-level meta-analysis to account for multiple subgroups of the same cohorts and used meta-regression to account for moderators.

There are limitations to this study. Firstly, as in every meta-analysis, the quality of synthesized evidence depends on the quality of evidence of the individual studies. We assessed the risk of bias of the included studies and could not identify an influence of studies with high risk of bias on the effect estimates. Secondly, we encountered high heterogeneity between studies. As this systematic review included observational studies, the high variability across study populations and diagnostic criteria used was expected. Further, the fact that studies used different adjustment factors in their multivariable analyses most likely contributed substantially to the high heterogeneity. To alleviate the difference in the adjustment factors, we have conducted sensitivity analyses. Both crude odds ratios and absolute risks indicated a similar change of risk estimates to the per protocol analyses, strengthening our primary results. Another factor contributing to heterogeneity could be methodological differences between studies, such as how competing events were treated. This could not be taken into consideration when determining eligibility, since the information was mostly not available. Finally, severity and duration of DM could not be taken into consideration.

Different types of IGM are associated with increased cardiovascular risk and mortality after IS and TIA. The entities of IGM should be considered as a continuous spectrum with increased cardiovascular risk that represent an important target for early cardiovascular prevention programs.

Availability of data and materials

The extracted data from the involved studies in this systematic review have been made available in supplementary material.

Abbreviations

• Ischemic stroke

Diabetes mellitus

Impaired glucose metabolism

• Insulin resistance

Hazard ratios

Odds ratios

Confidence interval

GBD 2019 Stroke Collaborators. Global, regional, and national burden of stroke and its risk factors, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Neurol. 2021;20(10):795–820. https://doi.org/10.1016/S1474-4422(21)00252-0 .

Article   Google Scholar  

Chen Y, Wright N, Guo Y, et al. Mortality and recurrent vascular events after first incident stroke: a 9-year community-based study of 0·5 million Chinese adults. Lancet Glob Health. 2020;8(4):e580–90. https://doi.org/10.1016/S2214-109X(20)30069-3 .

Article   PubMed   PubMed Central   Google Scholar  

Carlsson A, Irewall AL, Graipe A, Ulvenstam A, Mooe T, Ögren J. Long-term risk of major adverse cardiovascular events following ischemic stroke or TIA. Sci Rep. 2023;13(1):8333. https://doi.org/10.1038/s41598-023-35601-x .

Article   PubMed   PubMed Central   CAS   Google Scholar  

Mohan KM, Wolfe CDA, Rudd AG, Heuschmann PU, Kolominsky-Rabas PL, Grieve AP. Risk and cumulative risk of stroke recurrence: a systematic review and meta-analysis. Stroke. 2011;42(5):1489–94. https://doi.org/10.1161/STROKEAHA.110.602615 .

Article   PubMed   Google Scholar  

Sarwar N, Gao P, Kondapally Seshasai SR, et al. Diabetes mellitus, fasting blood glucose concentration, and risk of vascular disease: a collaborative meta-analysis of 102 prospective studies. The Lancet. 2010;375(9733):2215–22. https://doi.org/10.1016/S0140-6736(10)60484-9 .

Article   CAS   Google Scholar  

Lau LH, Lew J, Borschmann K, Thijs V, Ekinci EI. Prevalence of diabetes and its effects on stroke outcomes: a meta-analysis and literature review. J Diabetes Investig. 2019;10(3):780–92. https://doi.org/10.1111/jdi.12932 .

Kleindorfer DO, Towfighi A, Chaturvedi S, et al. Guideline for the Prevention of Stroke in Patients With Stroke and Transient Ischemic Attack: A Guideline From the American Heart Association/American Stroke Association. Stroke. 2021. https://doi.org/10.1161/str.0000000000000375 .

Schlesinger S, Neuenschwander M, Barbaresko J, et al. Prediabetes and risk of mortality, diabetes-related complications and comorbidities: umbrella review of meta-analyses of prospective studies. Diabetologia. 2022;65(2):275–85. https://doi.org/10.1007/s00125-021-05592-3 .

Article   PubMed   CAS   Google Scholar  

Nathan DM, Davidson MB, DeFronzo RA, et al. Impaired fasting glucose and impaired glucose tolerance: implications for care. Diabetes Care. 2007;30(3):753–9. https://doi.org/10.2337/dc07-9920 .

Abdul-Ghani MA, Tripathy D, DeFronzo RA. Contributions of β-cell dysfunction and insulin resistance to the pathogenesis of impaired glucose tolerance and impaired fasting glucose. Diabetes Care. 2006;29(5):1130–9. https://doi.org/10.2337/dc05-2179 .

Jia Q, Zheng H, Zhao X, et al. Abnormal glucose regulation in patients with acute stroke across China: prevalence and baseline patient characteristics. Stroke. 2012;43(3):650–7. https://doi.org/10.1161/STROKEAHA.111.633784 .

Kernan WN, Inzucchi SE, Viscoli CM, et al. Impaired insulin sensitivity among nondiabetic patients with a recent TIA or ischemic stroke. Neurology. 2003;60(9):1447–51. https://doi.org/10.1212/01.WNL.0000063318.66140.A3 .

Echouffo-Tcheugui JB, Xu H, Matsouaka RA, et al. Diabetes and long-term outcomes of ischaemic stroke: findings from Get With The Guidelines-Stroke. Eur Heart J. 2018;39(25):2376–86. https://doi.org/10.1093/eurheartj/ehy036 .

Zhang L, Li X, Wolfe CDA, O’Connell MDL, Wang Y. Diabetes as an independent risk factor for stroke recurrence in ischemic stroke patients: an updated meta-analysis. Neuroepidemiology. 2021;55(6):427–35. https://doi.org/10.1159/000519327 .

Pan Y, Chen W, Wang Y. Prediabetes and outcome of ischemic stroke or transient ischemic attack: a systematic review and meta-analysis. J Stroke Cerebrovasc Dis. 2019;28(3):683–92. https://doi.org/10.1016/j.jstrokecerebrovasdis.2018.11.008 .

Strain WD, Frenkel O, James MA, et al. Effects of semaglutide on stroke subtypes in type 2 diabetes: post hoc analysis of the randomized SUSTAIN 6 and PIONEER 6. Stroke. 2022;53(9):2749–57. https://doi.org/10.1161/STROKEAHA.121.037775 .

Zhou Z, Lindley RI, Rådholm K, et al. Canagliflozin and stroke in type 2 diabetes mellitus. Stroke. 2019;50(2):396–404. https://doi.org/10.1161/STROKEAHA.118.023009 .

Kernan WN, Viscoli CM, Furie KL, et al. Pioglitazone after ischemic stroke or transient ischemic attack. N Engl J Med. 2016;374(14):1321–31. https://doi.org/10.1056/NEJMoa1506930 .

Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021. https://doi.org/10.1136/bmj.n71 .

Kaynak N, Rackoll T, Endres M, Nave AH. The residual risk of impaired glucose metabolism on vascular events and mortality after ischemic stroke and the effect of antidiabetic therapy on reducing this risk: A systematic review and Meta-analysis. 2021. https://doi.org/10.17605/OSF.IO/JVYHW .

National Heart Lung and Blood Institute. Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies. 2013. Accessed December 18, 2020. https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools .

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60. https://doi.org/10.1136/bmj.327.7414.557 .

Viechtbauer W. metafor: Meta-analysis package for R. R package version 2.4–0. R package version 24–0. 2020;(1):1–275.

American Diabetes Association. 2. Classification and Diagnosis of Diabetes: Standards of Medical Care in Diabetes-2020. Diabetes Care. 2020;43(January):S14-S31. https://doi.org/10.2337/dc20-S002 .

Geneva: World Health Organization. Classification of Diabetes Mellitus. 2019.

Rutten-Jacobs LCA, Keurlings PAJ, Arntz RM, et al. High incidence of diabetes after stroke in young adults and risk of recurrent vascular events: The FUTURE study. PLoS ONE. 2014. https://doi.org/10.1371/journal.pone.0087171 .

Lu Z, Xiong Y, Feng X, et al. Insulin resistance estimated by estimated glucose disposal rate predicts outcomes in acute ischemic stroke patients. Cardiovasc Diabetol. 2023;22(1):225. https://doi.org/10.1186/s12933-023-01925-1 .

Jin A, Wang S, Li J, et al. Mediation of systemic inflammation on insulin resistance and prognosis of nondiabetic patients with ischemic stroke. Stroke. 2023;54(3):759–69. https://doi.org/10.1161/STROKEAHA.122.039542 .

Yoo J, Jeon J, Baik M, Kim J. Lobeglitazone, a novel thiazolidinedione, for secondary prevention in patients with ischemic stroke: a nationwide nested case-control study. Cardiovasc Diabetol. 2023;22(1):106. https://doi.org/10.1186/s12933-023-01841-4 .

Woo MH, Lee HS, Kim J. Effect of pioglitazone in acute ischemic stroke patients with diabetes mellitus: a nested case-control study. Cardiovasc Diabetol. 2019;18(1):67. https://doi.org/10.1186/s12933-019-0874-5 .

Morgan CL, Inzucchi SE, Puelles J, Jenkins-Jones S, Currie CJ. Impact of treatment with pioglitazone on stroke outcomes: a real-world database analysis. Diabetes Obes Metab. 2018;20(9):2140–7. https://doi.org/10.1111/dom.13344 .

Tu WJ, Liu Z, Chao BH, et al. Metformin use is associated with low risk of case fatality and disability rates in first-ever stroke patients with type 2 diabetes. Ther Adv Chronic Dis. 2022;13:20406223221076896. https://doi.org/10.1177/20406223221076894 .

Chen DY, Wang SH, Mao CT, et al. Sitagliptin after ischemic stroke in type 2 diabetic patients: a nationwide cohort study. Medicine. 2015;94(28): e1128. https://doi.org/10.1097/MD.0000000000001128 .

Chen DY, Li YR, Mao CT, et al. Cardiovascular outcomes of vildagliptin in patients with type 2 diabetes mellitus after acute coronary syndrome or acute ischemic stroke. J Diabetes Investig. 2020;11(1):110–24. https://doi.org/10.1111/jdi.13078 .

Li YR, Tsai SS, Chen DY, et al. Linagliptin and cardiovascular outcomes in type 2 diabetes after acute coronary syndrome or acute ischemic stroke. Cardiovasc Diabetol. 2018;17(1):2. https://doi.org/10.1186/s12933-017-0655-y .

Favilla CG, Mullen MT, Ali M, Higgins P, Kasner SE. Sulfonylurea use before stroke does not influence outcome. Stroke. 2011;42(3):710–5. https://doi.org/10.1161/STROKEAHA.110.599274 .

Tsivgoulis G, Goyal N, Iftikhar S, et al. Sulfonylurea Pretreatment and In-Hospital Use Does Not Impact Acute Ischemic Strokes (AIS) Outcomes Following Intravenous Thrombolysis. J Stroke Cerebrovasc Dis. 2017;26(4):795–800. https://doi.org/10.1016/j.jstrokecerebrovasdis.2016.10.019 .

Horsdal HT, Mehnert F, Rungby J, Johnsen SP. Type of preadmission antidiabetic treatment and outcome among patients with ischemic stroke: a nationwide follow-up study. J Stroke Cerebrovasc Dis. 2012;21(8):717–25. https://doi.org/10.1016/j.jstrokecerebrovasdis.2011.03.007 .

Richter B, Hemmingsen B, Metzendorf MI, Takwoingi Y. Development of type 2 diabetes mellitus in people with intermediate hyperglycaemia. Cochrane Database Syst Rev. 2018;10(10): CD012661. https://doi.org/10.1002/14651858.CD012661.pub2 .

Cai X, Zhang Y, Li M, et al. Association between prediabetes and risk of all cause mortality and cardiovascular disease: updated meta-analysis. BMJ. 2020;370: m2297. https://doi.org/10.1136/bmj.m2297 .

Rentsch CT, Garfield V, Mathur R, et al. Sex-specific risks for cardiovascular disease across the glycaemic spectrum: a population-based cohort study using the UK Biobank. The Lancet Regional Health - Europe. 2023;32:1–14. https://doi.org/10.1016/j.lanepe.2023.100693 .

Huff TA, Lebovitz HE, Heyman A, Davis L. Serial changes in glucose utilization and insulin and growth hormone secretion in acute cerebrovascular disease. Stroke. 1972;3(5):543–52. https://doi.org/10.1161/01.STR.3.5.543 .

Cleeman JI. Executive summary of the third report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (adult treatment panel III). J Am Med Assoc. 2001;285(19):2486–97. https://doi.org/10.1001/jama.285.19.2486 .

Kahn R, Buse J, Ferrannini E, Stern M. The metabolic syndrome: time for a critical appraisal: joint statement from the American Diabetes Association and the European Association for the Study of Diabetes. Diabetes Care. 2005;28(9):2289–304. https://doi.org/10.2337/diacare.28.9.2289 .

Echouffo-Tcheugui JB, Selvin E. Prediabetes and what it means: the epidemiological evidence. Annu Rev Public Health. 2020;42:59–77. https://doi.org/10.1146/annurev-publhealth-090419-102644 .

Lee M, Saver JL, Hong KS, Song S, Chang KH, Ovbiagele B. Effect of pre-diabetes on future risk of stroke: meta-analysis. BMJ. 2012;344: e3564. https://doi.org/10.1136/bmj.e3564 .

Gerstein HC, Miller ME, Byington RP, et al. Effects of intensive glucose lowering in type 2 diabetes. N Engl J Med. 2008;358(24):2545–59.

Marso SP, Bain SC, Consoli A, et al. Semaglutide and cardiovascular outcomes in patients with type 2 diabetes. N Engl J Med. 2016;375(19):1834–44. https://doi.org/10.1056/NEJMoa1607141 .

Zinman B, Wanner C, Lachin JM, et al. Empagliflozin, cardiovascular outcomes, and mortality in type 2 diabetes. N Engl J Med. 2015;373(22):2117–28. https://doi.org/10.1056/NEJMoa1504720 .

Neal B, Perkovic V, Mahaffey KW, et al. Canagliflozin and cardiovascular and renal events in type 2 diabetes. N Engl J Med. 2017;377(7):644–57. https://doi.org/10.1056/NEJMoa1611925 .

Dawson J, Béjot Y, Christensen LM, et al. European Stroke Organisation (ESO) guideline on pharmacological interventions for long-term secondary prevention after ischaemic stroke or transient ischaemic attack. Eur Stroke J. 2022;7(3):I–II. https://doi.org/10.1177/23969873221100032 .

Gerstein HC, Hart R, Colhoun HM, et al. The effect of dulaglutide on stroke: an exploratory analysis of the REWIND trial. Lancet Diabetes Endocrinol. 2020;8(2):106–14. https://doi.org/10.1016/S2213-8587(19)30423-1 .

Download references

This study was partially funded by the Corona Foundation. Protocol https://osf.io/jvyhw . The funder had no role in the conceptualization, design, data collection, analysis, decision to publish, or preparation of the manuscript.

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and affiliations.

Center for Stroke Research Berlin (CSB), Charité– Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany

Nurcennet Kaynak, Valentin Kennel, Matthias Endres & Alexander H. Nave

Department of Neurology with Experimental Neurology, Charité– Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany

Nurcennet Kaynak, Valentin Kennel, Torsten Rackoll, Matthias Endres & Alexander H. Nave

Berlin Institute of Health at Charité, Charité– Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany

Nurcennet Kaynak & Alexander H. Nave

German Center for Neurodegenerative Diseases (DZNE), partner site Berlin, Berlin, Germany

Nurcennet Kaynak & Matthias Endres

German Center for Cardiovascular Research (DZHK), partner site Berlin, Berlin, Germany

Nurcennet Kaynak, Matthias Endres & Alexander H. Nave

Berlin Institute of Health (BIH) QUEST Center for Responsible Research, Charité– Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany

Torsten Rackoll

Department of Biometry and Clinical Epidemiology, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin und Humboldt-Universität zu Berlin, Berlin, Germany

Daniel Schulze

German Center for Mental Health (DZPG), partner site Berlin, Berlin, Germany

Matthias Endres

You can also search for this author in PubMed   Google Scholar

Contributions

NK had full access to study data and is the guarantor of the study, taking full responsibility for the conduct of the study. NK, AHN, TR and ME conceived the study design and contributed to study protocol. NK, VK, and TR acquired data and performed the analysis. DS contributed to statistical methods and analyses. NK drafted the manuscript and all authors contributed to interpretation of the data and critical appraisal of the final work. AHN supervised the study. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Corresponding author

Correspondence to Alexander H. Nave .

Ethics declarations

Ethics approval.

Ethics approval was not required.

Consent for publication

Not applicable.

Competing interests

NK, VK, DS report no conflicts of interest. ME reports grants from Bayer and fees paid to the Charité from Amgen, AstraZeneca, Bayer Healthcare, Boehringer Ingelheim, BMS, Daiichi Sankyo, Sanofi, Pfizer, all outside the submitted work. AHN receives funding from the Corona foundation and the German Center for cardiovascular research (DZHK), no conflict of interest. TR receives funding from the European Commission, no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file1 (docx 3212 kb), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kaynak, N., Kennel, V., Rackoll, T. et al. Impaired glucose metabolism and the risk of vascular events and mortality after ischemic stroke: A systematic review and meta-analysis. Cardiovasc Diabetol 23 , 323 (2024). https://doi.org/10.1186/s12933-024-02413-w

Download citation

Received : 30 June 2024

Accepted : 19 August 2024

Published : 31 August 2024

DOI : https://doi.org/10.1186/s12933-024-02413-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Prediabetes
• Vascular events

Cardiovascular Diabetology

ISSN: 1475-2840

• Submission enquiries: [email protected]

• Systematic Review
• Open access
• Published: 04 September 2024

Artificial intelligence-assisted interventions for perioperative anesthetic management: a systematic review and meta-analysis

• Kensuke Shimada 1 , 2 , 3 ,
• Ryota Inokuchi 4 , 5 , 6 ,
• Tomohiro Ohigashi 7 , 8 ,
• Masao Iwagami 4 ,
• Makoto Tanaka 9 ,
• Masahiko Gosho 8 &
• Nanako Tamiya 4 , 10 , 11 , 12  

BMC Anesthesiology volume  24 , Article number:  306 ( 2024 ) Cite this article

Metrics details

Integration of artificial intelligence (AI) into medical practice has increased recently. Numerous AI models have been developed in the field of anesthesiology; however, their use in clinical settings remains limited. This study aimed to identify the gap between AI research and its implementation in anesthesiology via a systematic review of randomized controlled trials with meta-analysis (CRD42022353727).

We searched the databases of Medical Literature Analysis and Retrieval System Online (MEDLINE), Excerpta Medica Database (Embase), Web of Science, Cochrane Central Register of Controlled Trials (CENTRAL), Institute of Electrical and Electronics Engineers Xplore (IEEE), and Google Scholar and retrieved randomized controlled trials comparing conventional and AI-assisted anesthetic management published between the date of inception of the database and August 31, 2023.

Eight randomized controlled trials were included in this systematic review ( n  = 568 patients), including 286 and 282 patients who underwent anesthetic management with and without AI-assisted interventions, respectively. AI-assisted interventions used in the studies included fuzzy logic control for gas concentrations (one study) and the Hypotension Prediction Index (seven studies; adding only one indicator). Seven studies had small sample sizes ( n  = 30 to 68, except for the largest), and meta-analysis including the study with the largest sample size ( n  = 213) showed no difference in a hypotension-related outcome (mean difference of the time-weighted average of the area under the threshold 0.22, 95% confidence interval -0.03 to 0.48, P  = 0.215, I 2 93.8%).

Conclusions

This systematic review and meta-analysis revealed that randomized controlled trials on AI-assisted interventions in anesthesiology are in their infancy, and approaches that take into account complex clinical practice should be investigated in the future.

Trial registration

This study was registered with the International Prospective Register of Systematic Reviews (PROSPERO ID: CRD42022353727).

Peer Review reports

There has been a recent surge in studies on the application of artificial intelligence (AI) in medicine [ 1 ]. A similar trend has also been observed in the field of anesthesia, as evidenced by the numerous predictive models that have been proposed [ 2 , 3 , 4 ]. The number of publications is expected to increase with advances in technology [ 5 ].

However, to date, in the conventional operating room setting, AI models have not been comprehensively employed to replace clinical judgement in patient care. Typically, anesthesiologists continue to rely on own clinical judgment, often without AI support. Thus, there is a gap between current practices and the growing body of research on AI applications in this area.

To the best of our knowledge, no systematic review of randomized controlled trials (RCTs) has covered AI-assisted interventions and their outcomes in anesthesiology. Gaining an in-depth understanding of the characteristics and results of studies on AI can help clarify the delay in the widespread adoption of AI in the field of anesthesiology and guide future research. Therefore, this systematic review aimed to summarize the findings of RCTs that compared interventions with and without AI assistance in anesthesiology.

This study was registered with the International Prospective Register of Systematic Reviews (CRD42022353727). This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [ 6 ].

Eligibility criteria

RCTs that met the following criteria were included in this review: (i) published in peer-reviewed journals, (ii) included patients who underwent surgery under anesthesia as the study population, (iii) included patients who underwent surgery without AI-assisted interventions for perioperative anesthetic management as the control group, and (iv) investigated any anesthesia-related outcomes (e.g. vital signs, indicators in perioperative managements, and complications). Observational studies, reviews, editorials, conference articles, comments, standalone abstracts, and nonhuman studies were excluded. We defined AI as computer or model-based methods including neural network, machine learning, prediction network (in the context of machine learning), regression, and fuzzy logic (which allows computers to make flexible, human-like decisions by representing vague situations with numerical values). We used a broad definition of AI because a comprehensive review is more useful than a narrow definition that results in the exclusion of important studies from the review. However, we did not include studies for which the algorithm was not specified.

Search strategy

Medical Literature Analysis and Retrieval System Online (MEDLINE), Excerpta Medica Database (Embase), Web of Science, Cochrane Central Register of Controlled Trials (CENTRAL), Institute of Electrical and Electronics Engineers Xplore (IEEE), and Google Scholar were searched to retrieve relevant articles published between the date of database inception and August 31, 2023, without language restrictions. The search terms included: (“artificial intelligence” OR “machine learning” OR “supervised learning” OR “unsupervised learning” OR “reinforcement learning” OR “neural network*” OR “support vector machine” OR “fuzzy logic” OR “data mining” OR “pattern recognition*” OR “deep learning” OR “prediction network*”) AND (anesthes* OR anaesthes*) AND ("randomized controlled trial"). Additional file 1 summarizes the search strategies used for each database. In addition, the reference lists of all included studies and recent relevant reports or reviews were manually searched.

Study selection

Two authors (K.S. and R.I.) independently screened the literature using Covidence. The reference lists of the included articles were also screened to identify additional eligible studies. In the case of disagreements between the two authors, a consensus was reached via discussion with a third reviewer (M.I.). The corresponding authors of studies under evaluation were contacted for clarification if it was unclear whether the study was eligible for inclusion in the present review or if the study did not report sufficient data.

Data extraction

Data regarding the following characteristics were extracted: study characteristics (publication year and country), participant characteristics (age, sex, and type of surgery and anesthesia), interventions (type and details of AI), and anesthesia-related outcomes.

Statistical analyses

If the same outcome was measured for the same intervention in ≥ 3 studies, random-effects meta-analysis was conducted to estimate the outcome effect size and 95% confidence intervals (CI) using the quantile matching estimation method as implemented in the “metamedian” package (version 1.1.1) in R (version 4.2.2) [ 7 , 8 ]. Heterogeneity was assessed using the I 2 statistic (low, I 2  < 25%; moderate, 25% ≤  I 2  ≤ 50%; I 2  > 50%) [ 9 , 10 ]. Small study effects were assessed using funnel plot asymmetry and Egger’s regression test [ 11 ]. P  < 0.05 was considered to be statistically significant.

Risk of bias assessment

Two authors (K.S. and R.I.) independently assessed the risk of bias of the included RCTs using the Cochrane risk of bias tool. The RCTs were rated as having a low risk of bias, some concerns regarding bias, or a high risk of bias across the following domains: randomization process, changes in the intended intervention, missing outcome data, outcome measurements, and selection of the reported results. The overall risk of bias was rated as high if one or more of the evaluated domains were rated as high risk. The overall risk of bias was rated as low if all domains were rated as low risk. In the case of disagreements between the two authors, a consensus was reached via discussion. Risk of bias plots were created using robvis [ 12 ].

Figure  1 illustrates the study selection process. Nineteen of the 176 identified studies were considered potential candidates for inclusion in this systematic review [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 ]. Thirteen studies were excluded after the application of the inclusion and exclusion criteria: three studies were excluded based on the publication type [ 18 , 19 , 20 ], four studies were excluded based on the study design [ 13 , 15 , 16 , 17 ], one study was excluded based on the setting [ 27 ], four studies were excluded based on the intervention [ 21 , 26 , 28 , 30 ], and one study was excluded based on the comparator [ 23 ]. Two RCTs identified manually were added subsequently [ 32 , 33 ]; thus, a total of eight RCTs were included in this systematic review [ 14 , 22 , 24 , 25 , 29 , 31 , 32 , 33 ].

Flow chart of the study selection process. Abbreviations: CENTRAL, Cochrane Central Register of Controlled Trials; Embase, Excerpta Medica Database; IEEE, Institute of Electrical and Electronics Engineers Xplore; MEDLINE, Medical Literature Analysis and Retrieval System Online

Study characteristics

A total of 568 patients were included in the eight studies. Among them, 286 and 282 patients underwent AI-assisted and non-AI-assisted interventions, respectively (Table  1 ). The study by Schenk et al. [ 24 ] ( n  = 54) was a sub-study of the study by Wijnberge et al. [ 22 ] ( n  = 68). Thus, the number of patients included in each of these studies was counted twice. All studies were conducted in an operating room setting under general anesthesia. Only adult participants were included. One study included patients who underwent elective lumbar spinal surgery under induced hypotension anesthesia to reduce blood loss (Koo et al. [ 25 ], n  = 68). One study included patients who underwent major elective thoracic surgeries under general anesthesia with one-lung ventilation (Šribar et al. [ 29 ], n  = 34).

Interventions

Table 1 summarizes the interventions used in the included studies. Fuzzy logic control for gas concentrations was used during general anesthesia in one study (Curatolo et al. [ 14 ]). Fuzzy logic was used for control of inspired oxygen and isoflurane concentration. The Hypotension Prediction Index (Edwards Life Sciences Corporation, California, USA) was used for intraoperative monitoring in seven studies (Wijnberge et al. [ 22 ], Maheshwari et al. [ 32 ], Schneck et al. [ 33 ], Schenk et al. [ 24 ], Koo et al. [ 25 ], Šribar et al. [ 29 ], and Frassanito et al. [ 31 ]). The Hypotension Prediction Index was only used in addition to usual care in each study, with no specific instructions on how to respond to the prediction. All interventions, with the exception of fuzzy logic control, simply added one indicator to routine clinical practice.

Table 2 presents the outcomes.

The time-weighted average of the area under the threshold intra- or postoperatively

The time-weighted average of the area under the threshold (TWA-AUT) was calculated in five studies that used the Hypotension Prediction Index [ 22 , 24 , 29 , 31 , 32 ]. TWA-AUT is defined as the area under the threshold divided by the total duration of surgery: TWA-AUT = (depth of hypotension in mmHg below a mean arterial pressure [MAP] of 65 mmHg × time in minutes below a MAP of 65 mmHg) / (total duration of operation in minutes) [ 22 , 34 ]. However, Schenk et al. used “total observed time” rather than “total duration of surgery” for the calculation of TWA-AUT in the post-anesthesia care unit (PACU) [ 24 ]. TWA-AUT was found to be significantly lower in the Hypotension Prediction Index group than in the control group in the three studies that assessed TWA-AUT in the operating room (0.10 mmHg vs 0.44 mmHg by Wijnberge et al., 0.01 mmHg vs 0.08 mmHg by Šribar et al., and 0.14 mmHg vs 0.77 mmHg by Frassanito et al., the Hypotension Prediction Index vs control, respectively) [ 22 , 29 , 31 ]. However, the study by Maheshwari et al. [ 32 ], which had a larger sample size ( n  = 213), revealed no differences between the groups (0.14 mmHg vs 0.14 mmHg, P  = 0.757). Similarly, the study by Schenk et al. [ 24 ], which was conducted in the PACU, revealed no significant differences (0.07 mmHg vs 0.23 mmHg, P  = 0.295, the Hypotension Prediction Index vs control, respectively). Note that all values in this section are listed as medians.

Number of intraoperative hypotensive events per patient

Two studies (Šribar et al. [ 29 ] and Frassanito et al. [ 31 ]) that used the Hypotension Prediction Index evaluated the number of intraoperative hypotensive events per patient. The number of hypotensive events was significantly lower in the Hypotension Prediction Index group than that in the control group.

Duration of intraoperative hypotension

Four studies, comprising one study (Curatolo et al. [ 14 ]) that used fuzzy logic and three studies (Wijnberge et al. [ 22 ], Schneck et al. [ 33 ], and Frassanito et al. [ 31 ]) that used the Hypotension Prediction Index, evaluated the duration of intraoperative hypotension. The duration of intraoperative hypotension time was significantly lower in the Hypotension Prediction Index group than in the control group in the three studies that used the Hypotension Prediction Index. However, almost no differences were observed in the study that used fuzzy logic (durations of period of systolic blood pressure were 0% [0–0%] vs 0% [0–0%] for systolic blood pressure under 90 mmHg and 2% [0–9%] vs 1% [0–7%] for systolic blood pressure over 140 mmHg, fuzzy group vs control, respectively).

Volume of intraoperative blood loss

Schneck et al. [ 33 ] and Koo et al. [ 25 ] evaluated the volume of intraoperative blood loss. Koo et al. [ 25 ] compared the volume of blood loss under induced hypotension anesthesia with and without the Hypotension Prediction Index and revealed that the volume of blood loss was lower in the Hypotension Prediction Index group than control group (299.3 ± 219.8 mL vs 532.0 ± 232.7 mL). Note that Koo et al. did not evaluate hypotension, which is supposed to be the original purpose of the Hypotension Prediction Index. However, by Schneck et al. [ 33 ], no significant difference was observed between the groups.

Percentage of oxygen concentration within the target range

Curatolo et al. [ 14 ] compared the percentage of oxygen concentration within the target range with and without the use of fuzzy logic and revealed that fuzzy logic control of oxygen concentration is superior to manual control. In this study, isoflurane concentrations were also controlled by fuzzy logic, but there was no comparison with the control group with respect to isoflurane.

• Meta-analysis

For intraoperative TWA-AUT and duration of intraoperative hypotension, meta-analyses were conducted because 3 or more studies measured the same outcome for the same intervention (Fig.  2 ).

Forest plot and funnel plot of meta-analyses of each outcome. Abbreviations: AI, artificial intelligence; HPI, Hypotension Prediction Index. a , b Forest plot and funnel plot of meta-analysis of intraoperative time-weighted average of the area under the threshold with vs without the Hypotension Prediction Index. c , d Forest plot and funnel plot of meta-analysis of duration of intraoperative hypotension with vs without the Hypotension Prediction Index

Intraoperative TWA-AUT

Wijnberge et al. [ 22 ], Maheshwari et al. [ 32 ], Šribar et al. [ 29 ], and Frassanito et al. [ 31 ] used the Hypotension Prediction Index as interventions and measured the intraoperative TWA-AUT. There was no significant difference between the Hypotension Prediction Index group and the control group (mean difference 0.22, 95% CI -0.03 to 0.48, P  = 0.086; mean difference > 0 indicates the superiority of the Hypotension Prediction Index; Fig.  2 a). The heterogeneity between the studies was high, with I 2 of 93.8%. Figure  2 b shows the funnel plot. Egger’s regression test did not show small study effects but the point estimate of intercept was large (intercept 29.15, 95% CI -40.75 to 99.04, P  = 0.215).

Wijnberge et al. [ 22 ], Schneck et al. [ 33 ], and Frassanito et al. [ 31 ] used the Hypotension Prediction Index as an intervention and measured the duration of intraoperative hypotension. The mean difference was 7.41%, which indicates the superiority of the Hypotension Prediction Index (95% CI 4.95 to 9.86, P  < 0.001; Fig.  2 c). The heterogeneity between the studies was low, with I 2 of 0.0%. The funnel plot was shown in Fig.  2 d. Egger’s regression test showed no publication bias (intercept 0.92, 95% CI -0.95 to 2.78, P  = 0.101).

Risk of bias

Figure  3 presents the results of the risk of bias assessment. Two studies (Curatolo et al. [ 14 ] and Šribar et al. [ 29 ]) were categorized as having a high risk of bias owing to inappropriate randomization processes or the presence of multiple primary outcomes. The remaining six studies were considered to have a low risk of bias.

Risk of bias. A Summary plot of risk of bias. B Risk of bias of each study

This systematic review explored the impact of AI-assisted interventions, such as fuzzy logic and Hypotension Prediction Index, on anesthesia-related outcomes (hypotension, blood loss, and the accuracy of oxygen concentration). The findings of this review suggest that some small studies reported promising results, whereas the results of the meta-analysis with the largest sample study showed no significant differences in hypotension-related outcomes.

The interventions used in this systematic review involved the addition of a single measure to routine anesthetic management, with the exception of the fuzzy logic study by Curatolo et al. [ 14 ]. Anesthesiologists make decisions based on many considerations in routine clinical practice, including the patient characteristics, multiple vital signs and their trends, medication status, and surgical progress (e.g. appropriate management goals vary according to patient characteristics, comorbidities, and surgical procedure and its progress; depth of anesthesia may be intentionally deepened before a highly invasive procedure, or conversely, the depth may be decreased toward awakening). Thus, a new single metric would have a limited impact in the case of anesthesiologists managing their day-to-day clinical practice without AI assistance.

The following is a summary of the characteristics of each intervention.

Fuzzy logic

Fuzzy logic was mainly studied in the 1990s in the field of anesthesiology and is widely used in household appliances, such as washing machines and microwave ovens [ 13 , 35 ]. Rather than regulating whether the cut-off value is on or off, fuzzy logic defines an intermediate state. For instance, fuzzy logic does not interpret 99 mmHg as “low” or 100 mmHg as “normal.” The values are divided into fuzzy sets, and each value can be categorized into one or more sets. For instance, 85 mmHg can be categorized as 75% to “low” as well as 25% to “normal” [ 35 ]. This enables computers to understand and respond to imprecise information.

Curatolo et al. [ 14 ] used fuzzy logic to regulate the oxygen and isoflurane concentrations intraoperatively and reported that the concentration control in the fuzzy logic group was superior to that in the manual control group. Some studies that attempted to apply fuzzy logic to the management of intraoperative blood pressure were identified during the search [ 13 , 15 ]. However, they were excluded as they did not meet the inclusion criteria of this systematic review. Thus, only one RCT using fuzzy logic was included. The concept of fuzzy logic is compatible with the intraoperative management of anesthesia; therefore, although it is an old method, it may be worth revisiting with an appropriate study design.

Hypotension prediction index

The Hypotension Prediction Index is a machine learning-based technology [ 36 ] trained using the arterial pressure waveform data of 1334 of the 1684 patients included in the historical database that comprised intensive care unit and operating room data. It was internally validated using the data of the remaining 350 patients. External validation was performed using the data of 204 patients in the operating room. The machine learning mechanism was a logistic regression analysis. The objective variables were hypotensive event and non-event samples: the event sample included data from 5, 10, 15, and 20 min before the hypotensive episode (MAP < 65 mmHg for at least 1 min), whereas the non-event sample included data at least 20 min away from the hypotensive episode (MAP > 75 mmHg). The non-event sample is the midpoint of a 30-min hypotensive episode. Multiple data points extracted from the arterial pressure waveform data at the corresponding time points were used as explanatory variables. The 0–1 prediction obtained from the logistic regression analysis was multiplied by 100 for scaling [ 36 ].

Regarding the Hypotension Prediction Index, differences in results were observed between the included studies. Four small studies with the Hypotension Prediction Index reported improved intraoperative hypotension-related outcomes in the Hypotension Prediction Index groups [ 22 , 29 , 31 , 33 ]. The meta-analysis of these studies also showed that the duration of intraoperative hypotension was significantly lower in the Hypotension Prediction Index group than that in the control group. These findings suggest that the Hypotension Prediction Index could be useful for intraoperative anesthetic management. However, no significant difference was observed in intraoperative TWA-AUT in the study by Maheshwari et al. [ 32 ], which had the largest sample size. The authors considered that this result could be attributed to clinicians largely ignoring this unfamiliar alert [ 37 ]. The meta-analysis including this largest study also showed no significant difference in intraoperative TWA-AUT. Although the Egger's regression test of the meta-analysis showed that the intercept was not significantly larger than zero (intercept 29.15, 95% CI -40.75 to 99.04, P  = 0.215), the small study effects could not be excluded, and the presence of publication bias was suspected because (i) the intercept of Egger’s regression model was large, (ii) only four studies were included in the analysis, and (iii) the funnel plot was asymmetric. Therefore, taking these findings together, the usefulness of the Hypotension Prediction Index may not yet be reliable.

There may be several possible avenues for improvement in this index: (i) this index does not suggest further measures to be taken if the index increases; (ii) the algorithm is based on a simple logistic regression model for complex situations; (iii) data regarding blood pressure and arterial pressure waveform are used only at a few selected time points; and (iv) blood pressure data are replaced by binary variables in the analysis phase, and the degree of hypotension and time information are not used effectively [ 36 ]. It should be also noted that there are concerns in terms of evaluating the performance of the Hypotension Prediction Index [ 38 ]. Correcting these problems may facilitate the construction of a model that has improved predictive performance and might be able to indicate what interventions should be used [ 4 , 39 , 40 ].

Studies on the use of AI in the field of anesthesiology have increased recently [ 41 , 42 ]. However, only eight RCTs on AI interventions were included in this systematic review. A PubMed search found 64 results using our search formula again (7 August, 2024), while replacing "AND ("Randomized Controlled Trial"[Publication Type])" at the end of the search formula with "AND ("observational" OR "retrospective" OR "simulation")" at the end of the search formula found 1,009 results. Thus, it is clear that perioperative studies using AI are biased toward non-interventional studies. This situation may indicate that the use of AI has been limited to data analysis and model building, and its usefulness in actual clinical practice has not yet been evaluated. Although in the field of basic research, systems have been established to introduce new drugs in clinical practice [ 43 ], there are a few clinicians who can translate between the computational aspects of model building and the clinical insights around the problem to be solved and then integrate the model into the clinical workflow [ 44 ]. Thus, the RCTs are less likely to be conducted in the field of AI at this point. However, AI could potentially be applied at any point in the perioperative period, as it has been widely studied in pre-operative, intra-operative, post-operative, and operating room managements [ 45 ]. Therefore, it is necessary to create a system in the field of data science to verify model building in real-world settings.

This study has certain limitations. First, two high risk of bias studies were included. Second, arbitrary elements might influence the outcomes of some studies that were categorized into the “low-risk” group, as the intervention was not blinded [ 46 ]. Lastly, meta-analyses were performed, but caution should be exercised in interpreting the results because of the high heterogeneity among the studies. There were also some differences in the participants, interventions, and controls. However, due to the small number of included studies, no additional subgroup analyses were performed. At this stage, the results from our meta-analyses should be used as reference values, due to the insufficient number of studies evaluated.

This systematic review and meta-analysis found that only a few high-quality RCTs comparing interventions with and without AI assistance in anesthetic management have been conducted. Future RCTs of AI-assisted anesthesia interventions that take into account complex clinical situations are necessary.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

• Artificial intelligence

Cochrane central register of controlled trials

Confidence interval

Combined index of stimulus and analgesia

Excerpta medica database

Institute of electrical and electronics engineers xplore

Mean arterial pressure

Medical literature analysis and retrieval system online

Numerical rating scale

Post-anesthesia care unit

Preferred reporting items for systematic reviews and meta-analyses

International prospective register of systematic reviews

Randomized controlled trial

Standard deviation

The time-weighted average of the area under the threshold

Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44–56.

Article   CAS   PubMed   Google Scholar  

Palla K, Hyland SL, Posner K, Ghosh P, Nair B, Bristow M, Paleva Y, Williams B, Fong C, Van Cleve W, et al. Intraoperative prediction of postanaesthesia care unit hypotension. Br J Anaesth. 2022;128(4):623–35.

Article   PubMed   Google Scholar  

Xue B, Li D, Lu C, King CR, Wildes T, Avidan MS, Kannampallil T, Abraham J. Use of machine learning to develop and evaluate models using preoperative and intraoperative data to identify risks of postoperative complications. JAMA Netw Open. 2021;4(3):e212240.

Article   PubMed   PubMed Central   Google Scholar  

Nolde JM, Schlaich MP, Sessler DI, Mian A, Corcoran TB, Chow CK, Chan MTV, Borges FK, McGillion MH, Myles PS, et al. Machine learning to predict myocardial injury and death after non-cardiac surgery. Anaesthesia. 2023;78(7):853–60.

Lee CK, Hofer I, Gabel E, Baldi P, Cannesson M. Development and validation of a deep neural network model for prediction of postoperative in-hospital mortality. Anesthesiology. 2018;129(4):649–62.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

McGrath S, Sohn H, Steele R, Benedetti A. Meta-analysis of the difference of medians. Biom J. 2020;62(1):69–98.

McGrath S, Zhao X, Ozturk O, Katzenschlager S, Steele R, Benedetti A. metamedian: An R package for meta-analyzing studies reporting medians. Res Synth Methods. 2024;15(2):332–46.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

Xie W, Huang H, Zhan S, Zhang Z. Risk of psychiatric disorders and all-cause mortality with belimumab therapy in patients with systemic lupus erythematosus: a meta-analysis of randomised controlled trials. Lupus Sci Med. 2021;8(1):e000534.

Harrer M, Cuijpers P, Furukawa T, Ebert D: Doing Meta-Analysis with R: A Hands-On Guide. Florida, USA: CRC Press, Taylor & Francis Group, A Chapman & Hall Book, 2021:227–284.

McGuinness LA, Higgins JPT. Risk-of-bias VISualization (robvis): An R package and Shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. 2020;12(1):55–61.

Zbinden AM, Feigenwinter P, Petersen-Felix S, Hacisalihzade S. Arterial pressure control with isoflurane using fuzzy logic. Br J Anaesth. 1995;74(1):66–72.

Curatolo M, Derighetti M, Petersen-Felix S, Feigenwinter P, Fischer M, Zbinden AM. Fuzzy logic control of inspired isoflurane and oxygen concentrations using minimal flow anaesthesia. Br J Anaesth. 1996;76(2):245–50.

Schäublin J, Derighetti M, Feigenwinter P, Petersen-Felix S, Zbinden AM. Fuzzy logic control of mechanical ventilation during anaesthesia. Br J Anaesth. 1996;77(5):636–41.

Shieh JS, Kao MH, Liu CC. Genetic fuzzy modelling and control of bispectral index (BIS) for general intravenous anaesthesia. Med Eng Phys. 2006;28(2):134–48.

Padmanabhan R, Meskin N, Haddad WM. Closed-loop control of anesthesia and mean arterial pressure using reinforcement learning. Biomed Signal Process Control. 2015;22:54–64.

Article   Google Scholar  

Casas FD, Fernandez JM. Abstract PR603: evaluation of a closed loop total intravenous anesthesia system with bis compared to an open loop target controlled infusion system (Tci): Randomized controlled clinical trial. Anesth Analg. 2016;123(3S):766–7.

Pouska J, Cerveny V, Zatloukal J, Kletecka J, Bene J. The use of HPI (Hypotension probability indicator) during major intracranial surgery; preliminary results of a prospective randomized trial. Intens Care Med Exp. 2019;7(3):55.

Google Scholar  

Wijnberge M, Geerts B, Lemmers N, Hol L, Hollmann M, Vlaar A, Veelo D. The use of a machine-learning based algorithm to reduce hypotension during surgery: A randomised clinical trial. Intens Care Med Exp. 2019;7(3):55.

Meijer F, Honing M, Roor T, Toet S, Calis P, Olofsen E, Martini C, van Velzen M, Aarts L, Niesters M, et al. Reduced postoperative pain using Nociception Level-guided fentanyl dosing during sevoflurane anaesthesia: a randomised controlled trial. Br J Anaesth. 2020;125(6):1070–8.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wijnberge M, Geerts BF, Hol L, Lemmers N, Mulder MP, Berge P, Schenk J, Terwindt LE, Hollmann MW, Vlaar AP, et al. Effect of a machine learning-derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: The HYPE randomized clinical trial. JAMA. 2020;323(11):1052–60.

Liu Y, Cheng L. Ultrasound images guided under deep learning in the anesthesia effect of the regional nerve block on scapular fracture surgery. J Healthc Eng. 2021;2021:6231116.

PubMed   PubMed Central   Google Scholar  

Schenk J, Wijnberge M, Maaskant JM, Hollmann MW, Hol L, Immink RV, Vlaar AP, van der Ster BJP, Geerts BF, Veelo DP. Effect of Hypotension Prediction Index-guided intraoperative haemodynamic care on depth and duration of postoperative hypotension: a sub-study of the Hypotension Prediction trial. Br J Anaesth. 2021;127(5):681–8.

Koo JM, Choi H, Hwang W, Hong SH, Kim SI, Kim YH, Choi S, Kim CJ, Chae MS. Clinical implication of the acumen hypotension prediction index for reducing intraoperative haemorrhage in patients undergoing lumbar spinal fusion surgery: a prospective randomised controlled single-blinded trial. J Clin Med. 2022;11(16):4646.

Xu C, Zhu Y, Wu L, Yu H, Liu J, Zhou F, Xiong Q, Wang S, Cui S, Huang X, et al. Evaluating the effect of an artificial intelligence system on the anesthesia quality control during gastrointestinal endoscopy with sedation: a randomized controlled trial. BMC Anesthesiol. 2022;22(1):313.

Zhang F, Wu S, Qu M, Zhou L. Application of a remotely controlled artificial intelligence analgesic pump device in painless treatment of children. Contrast Media Mol Imaging. 2022;2022:1013241.

Wang LT, Zhang AR, Wang QQ, Bai B. Auto interpretable depth learning model to analyze the hemodynamic changes and pulmonary complications in laparoscopic gynecologic tumor surgery with nalmefene hydrochloride combined with general anesthesia. Eur Rev Med Pharmacol Sci. 2023;27(14):6510–22.

PubMed   Google Scholar  

Šribar A, Jurinjak IS, Almahariq H, Bandić I, Matošević J, Pejić J, Peršec J. Hypotension prediction index guided versus conventional goal directed therapy to reduce intraoperative hypotension during thoracic surgery: a randomized trial. BMC Anesthesiol. 2023;23(1):101.

Fuica R, Krochek C, Weissbrod R, Greenman D, Freundlich A, Gozal Y. Reduced postoperative pain in patients receiving nociception monitor guided analgesia during elective major abdominal surgery: a randomized, controlled trial. J Clin Monit Comput. 2023;37(2):481–91.

Frassanito L, Giuri PP, Vassalli F, Piersanti A, Garcia MIM, Sonnino C, Zanfini BA, Catarci S, Antonelli M, Draisci G. Hypotension prediction index guided goal directed therapy and the amount of hypotension during major gynaecologic oncologic surgery: a randomized controlled clinical trial. J Clin Monit Comput. 2023;37(4):1081–93.

Maheshwari K, Shimada T, Yang D, Khanna S, Cywinski JB, Irefin SA, Ayad S, Turan A, Ruetzler K, Qiu Y, et al. Hypotension prediction index for prevention of hypotension during moderate- to high-risk noncardiac surgery. Anesthesiology. 2020;133(6):1214–22.

Schneck E, Schulte D, Habig L, Ruhrmann S, Edinger F, Markmann M, Habicher M, Rickert M, Koch C, Sander M. Hypotension Prediction Index based protocolized haemodynamic management reduces the incidence and duration of intraoperative hypotension in primary total hip arthroplasty: a single centre feasibility randomised blinded prospective interventional trial. J Clin Monit Comput. 2020;34(6):1149–58.

Wijnberge M, Schenk J, Terwindt LE, Mulder MP, Hollmann MW, Vlaar AP, Veelo DP, Geerts BF. The use of a machine-learning algorithm that predicts hypotension during surgery in combination with personalized treatment guidance: study protocol for a randomized clinical trial. Trials. 2019;20(1):582.

Asbury AJ, Tzabar Y. Fuzzy logic: new ways of thinking for anaesthesia. Br J Anaesth. 1995;75(1):1–2.

Hatib F, Jian Z, Buddi S, Lee C, Settels J, Sibert K, Rinehart J, Cannesson M. Machine-learning algorithm to predict hypotension based on high-fidelity arterial pressure waveform analysis. Anesthesiology. 2018;129(4):663–74.

Sessler DI, Turan A, Stapelfeldt WH, Mascha EJ, Yang D, Farag E, Cywinski J, Vlah C, Kopyeva T, Keebler AL, et al. Triple-low alerts do not reduce mortality: a real-time randomized trial. Anesthesiology. 2019;130(1):72–82.

Enevoldsen J, Vistisen ST. Performance of the hypotension prediction index may be overestimated due to selection bias. Anesthesiology. 2022;137(3):283–9.

Kouz K, Brockmann L, Timmermann LM, Bergholz A, Flick M, Maheshwari K, Sessler DI, Krause L, Saugel B. Endotypes of intraoperative hypotension during major abdominal surgery: a retrospective machine learning analysis of an observational cohort study. Br J Anaesth. 2023;130(3):253–61.

Solomon SC, Saxena RC, Neradilek MB, Hau V, Fong CT, Lang JD, Posner KL, Nair BG. Forecasting a crisis: machine-learning models predict occurrence of intraoperative bradycardia associated with hypotension. Anesth Analg. 2020;130(5):1201–10.

Wingert T, Lee C, Cannesson M. Machine learning, deep learning, and closed loop devices-anesthesia delivery. Anesthesiol Clin. 2021;39(3):565–81.

Hashimoto DA, Witkowski E, Gao L, Meireles O, Rosman G. Artificial intelligence in anesthesiology: current techniques, clinical applications, and limitations. Anesthesiology. 2020;132(2):379–94.

Woolf SH. The meaning of translational research and why it matters. JAMA. 2008;299(2):211–3.

Lonsdale H. The challenges of AI in anesthesiology. ASA Monitor. 2023;87(11):22–3.

Bellini V, Rafano Carnà E, Russo M, Di Vincenzo F, Berghenti M, Baciarello M, Bignami E. Artificial intelligence and anesthesia: a narrative review. Ann Transl Med. 2022;10(9):528.

Nickerson RS. Confirmation bias: A ubiquitous phenomenon in many guises. Rev of Gen Psychol. 1998;2(2):175–220.

Download references

Acknowledgements

Not applicable.

Author information

Authors and affiliations.

Graduate School of Comprehensive Human Sciences, University of Tsukuba, Ibaraki, Japan

Kensuke Shimada

Translational Research Promotion Center, Tsukuba Clinical Research & Development Organization, University of Tsukuba, Ibaraki, Japan

Department of Anesthesiology, University of Tsukuba Hospital, Ibaraki, Japan

Department of Health Services Research, Institute of Medicine, University of Tsukuba, Ibaraki, Japan

Ryota Inokuchi, Masao Iwagami & Nanako Tamiya

Department of Emergency and Critical Care Medicine, The University of Tokyo Hospital, Tokyo, Japan

Ryota Inokuchi

Department of Clinical Engineering, The University of Tokyo Hospital, Tokyo, Japan

Department of Information and Computer Technology, Faculty of Engineering, Tokyo University of Science, Tokyo, Japan

Tomohiro Ohigashi

Department of Biostatistics, Institute of Medicine, University of Tsukuba, Ibaraki, Japan

Tomohiro Ohigashi & Masahiko Gosho

Department of Anesthesiology, Institute of Medicine, University of Tsukuba, Ibaraki, Japan

Makoto Tanaka

Health Services Research and Development Center, University of Tsukuba, Ibaraki, Japan

Nanako Tamiya

Center for Artificial Intelligence Research, University of Tsukuba, Ibaraki, Japan

Cybermedicine Research Center, University of Tsukuba, Ibaraki, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

KS helped handle this manuscript, designing the study, and conducting the review and meta-analysis. RI helped in designing the study, conducting the review, and drafting the manuscript. TO helped in designing the study, conducting the meta-analysis, and drafting the manuscript. MI helped design the study and write the manuscript. MG helped in designing the study, conducting the meta-analysis, and drafting the manuscript. MT helped design the study and write the manuscript. NT helped design the study and write the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ryota Inokuchi .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. step-wise approach to the search strategy development., additional file 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Shimada, K., Inokuchi, R., Ohigashi, T. et al. Artificial intelligence-assisted interventions for perioperative anesthetic management: a systematic review and meta-analysis. BMC Anesthesiol 24 , 306 (2024). https://doi.org/10.1186/s12871-024-02699-z

Download citation

Received : 25 June 2024

Accepted : 26 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1186/s12871-024-02699-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic review
• Rondomized controlled trial

BMC Anesthesiology

ISSN: 1471-2253

• General enquiries: [email protected]

Risk of incident venous thromboembolism in patients with atopic dermatitis: systematic analysis of the literature and meta-analysis

• Published: 06 September 2024

Cite this article

• Yifei Wang 1 , 2   na1 ,
• Zhiqiang Chen 3   na1 ,
• Ting He 1 , 2 ,
• Changzheng Huang 1 , 2 &
• Chen Shen   ORCID: orcid.org/0000-0001-8231-9303 1 , 2  

Previous studies have showed conflicting results regarding the association between atopic dermatitis (AD) and venous thromboembolism (VTE) events.

Based on our systematic review of 6 cohort studies, patients with AD have an increased risk of VTE events.

These findings provide key evidence-based estimates to inform decision-making that VTE is a comorbidity of AD.

Atopic dermatitis (AD) is a prevalent chronic inflammatory skin disease. While various inflammatory conditions have been linked to venous thromboembolism (VTE), the risk of VTE among patients with AD remains unclear. We sought to systematically review and meta-analyze population-based studies to determine the association between AD and incident VTE. A systematic review was performed of published studies in PubMed, Web of Science, Embase and Cochrane library from their inception to 27 May 2024. At least two reviewers conducted title/abstract, full-text review and data extraction. Cohort studies examining the association of AD with incident VTE were included. Quality of evidence was assessed using the Newcastle-Ottawa Scale. Six cohort studies, encompassing a total of 10,186,861 participants, were included. The meta-analysis revealed a significantly increased risk for incident VTE among AD patients (pooled hazard ratio (HR), 1.10; 95% CI, 1.00–1.21), with an incidence rate of VTE at 3.35 events per 1000 patient-years. Individual outcome analyses suggested that AD was associated with higher risks of deep vein thrombosis (pooled HR, 1.15; 95% CI, 1.04–1.27) but not pulmonary embolism (pooled HR, 0.99; 95% CI, 0.87–1.13). This systematic review and meta-analysis indicated an increased risk of incident VTE among patients with AD. Future studies are necessary to elucidate the underlying pathophysiology of the association between AD and VTE.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data availability

The data of this study are available from the corresponding author upon reasonable request.

Langan SM, Irvine AD, Weidinger S (2020) Atopic dermatitis. Lancet 396(10247):345–360. https://doi.org/10.1016/s0140-6736(20)31286-1

Article   CAS   PubMed   Google Scholar  

Weidinger S et al (2018) Atopic dermatitis. Nat Rev Dis Primers 4(1). https://doi.org/10.1038/s41572-018-0001-z

Eichenfield LF et al (2014) Guidelines of care for the management of atopic dermatitis: Sect. 1. Diagnosis and assessment of atopic dermatitis. J Am Acad Dermatol 70(2):338–351. https://doi.org/10.1016/j.jaad.2013.10.010

Article   PubMed   Google Scholar  

Ascott A et al (2019) Atopic eczema and major cardiovascular outcomes: a systematic review and meta-analysis of population-based studies. J Allergy Clin Immunol 143(5):1821–1829. https://doi.org/10.1016/j.jaci.2018.11.030

Article   PubMed   PubMed Central   Google Scholar  

Silverwood RJ et al (2018) Severe and predominantly active atopic eczema in adulthood and long term risk of cardiovascular disease: population based cohort study. BMJ (clinical research ed. 361. https://doi.org/10.1136/bmj.k1786

Arshad N et al (2017) Recurrence and mortality after first venous thromboembolism in a large population-based cohort. J Thromb Haemostasis: JTH 15(2):295–303. https://doi.org/10.1111/jth.13587

Article   CAS   Google Scholar  

Cohen AT et al (2017) Epidemiology of first and recurrent venous thromboembolism in patients with active cancer. A population-based cohort study. Thromb Haemost 117(1):57–65. https://doi.org/10.1160/TH15-08-0686

Riva N, Donadini MP, Ageno W (2015) Epidemiology and pathophysiology of venous thromboembolism: similarities with atherothrombosis and the role of inflammation. Thromb Haemost 113(6):1176–1183. https://doi.org/10.1160/TH14-06-0563

Tamagawa-Mineoka R et al (2008) Elevated platelet activation in patients with atopic dermatitis and psoriasis: increased plasma levels of beta-thromboglobulin and platelet factor 4. Allergology International: Official J Japanese Soc Allergology 57(4):391–396. https://doi.org/10.2332/allergolint.O-08-537

Piazza G, Ridker PM (2015) Is venous thromboembolism a chronic inflammatory disease? Clin Chem 61(2):313–316. https://doi.org/10.1373/clinchem.2014.234088

Shaheen MS, Silverberg JI (2021) Association of inflammatory skin diseases with venous thromboembolism in US adults. Arch Dermatol Res 313(4):281–289. https://doi.org/10.1007/s00403-020-02099-6

Chen T-L et al (2022) Association of risk of incident venous thromboembolism with atopic dermatitis and treatment with Janus kinase inhibitors: a systematic review and Meta-analysis. JAMA Dermatology 158(11):1254–1261. https://doi.org/10.1001/jamadermatol.2022.3516

Merola JF et al (2024) Venous thromboembolism risk is lower in patients with atopic dermatitis than other immune-mediated inflammatory diseases: a retrospective, observational, comparative cohort study using US claims data. J Am Acad Dermatol 90(5):935–944. https://doi.org/10.1016/j.jaad.2023.12.027

Afonso J et al (2023) The perils of Misinterpreting and Misusing Publication Bias in Meta-analyses: an Education Review on Funnel plot-based methods. Sports Medicine (Auckland, N.Z.) https://doi.org/10.1007/s40279-023-01927-9

Meyers KJ et al (2021) Risk of venous thromboembolism among patients with atopic dermatitis: a Cohort Study in a US administrative claims database. Dermatology Therapy 11(3):1041–1052. https://doi.org/10.1007/s13555-021-00538-4

Zirpel H et al (2024) Atopic dermatitis is associated with an increased risk of cardiovascular diseases: a large-scale, propensity-matched US-based retrospective study. Clin Exp Dermatol. https://doi.org/10.1093/ced/llae164

Wan J et al (2023) Incidence of Cardiovascular Disease and venous thromboembolism in patients with atopic dermatitis. The Journal of Allergy and Clinical Immunology. Practice 11(10). https://doi.org/10.1016/j.jaip.2023.08.007

Warren RB et al (2023) The risk of venous thromboembolism in atopic dermatitis: a matched cohort analysis in UK primary care. Br J Dermatol 189(4):427–436. https://doi.org/10.1093/bjd/ljad212

Chen T-L et al (2023) Risk of venous thromboembolism among adults with atopic dermatitis. JAMA Dermatology 159(7):720–727. https://doi.org/10.1001/jamadermatol.2023.1300

Schneeweiss MC et al (2021) Incidence of venous thromboembolism in patients with dermatologist-diagnosed chronic inflammatory skin diseases. JAMA Dermatology 157(7):805–816. https://doi.org/10.1001/jamadermatol.2021.1570

Bello N et al (2023) Systematic Literature Review and Meta-analysis of venous thromboembolism events in systemic Lupus Erythematosus. Rheumatol Therapy 10(1). https://doi.org/10.1007/s40744-022-00513-1

Grainge MJ, West J, Card TR (2010) Venous thromboembolism during active disease and remission in inflammatory bowel disease: a cohort study. Lancet (London England) 375(9715):657–663. https://doi.org/10.1016/S0140-6736(09)61963-2

Chen T-L et al (2022) Association of Psoriasis With Incident venous thromboembolism and peripheral vascular disease: a systematic review and Meta-analysis. JAMA Dermatology 158(1):59–67. https://doi.org/10.1001/jamadermatol.2021.4918

Davis DMR et al (2022) American Academy of Dermatology Guidelines: awareness of comorbidities associated with atopic dermatitis in adults. J Am Acad Dermatol 86(6). https://www.jaad.org/article/S0190-9622(22)00080-9/fulltext

Brunner PM et al (2017) The atopic dermatitis blood signature is characterized by increases in inflammatory and cardiovascular risk proteins. Sci Rep 7(1):8707. https://doi.org/10.1038/s41598-017-09207-z

Article   CAS   PubMed   PubMed Central   Google Scholar  

Réau V et al (2021) Venous thrombosis and predictors of relapse in eosinophil-related diseases. Sci Rep 11(1):6388. https://doi.org/10.1038/s41598-021-85852-9

Nastałek M, Wojas-Pelc A, Undas A (2010) Plasma fibrin clot properties in atopic dermatitis: links between thrombosis and atopy. J Thromb Thrombolysis 30(2):121–126. https://doi.org/10.1007/s11239-010-0478-0

Wollenberg A et al (2020) Summary of efficacy, impact on work productivity and activity, and safety of baricitinib in moderateto-severe atopic dermatitis from two monotherapy phase III trials. British Journal of Dermatology, 183(SUPPL 1): p. 66. https://doi.org/10.1111/bjd.18968

Del Duca E et al (2023) Proteomic characterization of atopic dermatitis blood from infancy to adulthood. J Am Acad Dermatol 88(5):1083–1093. https://pubmed.ncbi.nlm.nih.gov/36773824/

Pavel AB et al (2020) The proteomic skin profile of moderate-to-severe atopic dermatitis patients shows an inflammatory signature. J Am Acad Dermatol 82(3):690–699. https://doi.org/10.1016/j.jaad.2019.10.039

He H et al (2021) Mild atopic dermatitis lacks systemic inflammation and shows reduced nonlesional skin abnormalities. J Allergy Clin Immunol 147(4):1369–1380. https://doi.org/10.1016/j.jaci.2020.08.041

Hedderson MM et al (2022) Rates of cardiovascular events among patients with moderate-to-severe atopic dermatitis in an integrated health care system: a retrospective cohort study. PLoS ONE 17(11):e. https://doi.org/10.1371/journal.pone.0277469

Silverberg JI et al (2020) Validation of five patient-reported outcomes for atopic dermatitis severity in adults. Br J Dermatol 182(1):104–111. https://doi.org/10.1111/bjd.18002

Download references

Acknowledgements

We thank Professor Adam J Adler for critically reading the manuscript.

This work was supported by the Fundamental Research Funds for the Central Universities, Grant/Award Numbers: YCJJ20230213.

Author information

Yifei Wang and Zhiqiang Chen contributed equally to this work.

Authors and Affiliations

Department of Dermatology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430022, China

Yifei Wang, Ting He, Changzheng Huang & Chen Shen

Hubei Engineering Research Center for Skin Repair and Theranostics, Wuhan, 430022, China

Department of Vascular Surgery, Fuyang Hospital, Anhui Medical University, Fuyang, 236000, China

Zhiqiang Chen

You can also search for this author in PubMed   Google Scholar

Contributions

Data curation: Yifei Wang, Zhiqiang Chen, Ting He, Changzheng Huang, Chen Shen. Methodology: Yifei Wang, Zhiqiang Chen, Ting He, Changzheng Huang, Chen Shen. Formal analysis and investigation: Yifei Wang, Zhiqiang Chen, Ting He, Changzheng Huang, Chen Shen. Writing – original draft preparation: Yifei Wang, Zhiqiang Chen, Chen Shen, Changzheng Huang. Writing – review and editing: Yifei Wang, Chen Shen, Changzheng Huang. Funding acquisition: Yifei Wang. All authors read and approved the final version of the manuscript.

Corresponding authors

Correspondence to Changzheng Huang or Chen Shen .

Ethics declarations

Ethical approval.

The manuscript has been read and approved by all the authors. The requirements for authorship as stated earlier in this document have been met. All the authors believe that the manuscript represents honest.

Conflict of interest

The authors declare no conflict of interest regard- ing the publication of this paper.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Wang, Y., Chen, Z., He, T. et al. Risk of incident venous thromboembolism in patients with atopic dermatitis: systematic analysis of the literature and meta-analysis. J Thromb Thrombolysis (2024). https://doi.org/10.1007/s11239-024-03038-2

Download citation

Accepted : 21 August 2024

Published : 06 September 2024

DOI : https://doi.org/10.1007/s11239-024-03038-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Atopic dermatitis
• Atopic eczema
• Venous thromboembolism
• Deep vein thrombosis
• Pulmonary embolism
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley Open Access Collection

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]