literature review critical analysis

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 27, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

What is the purpose of literature review , a. habitat loss and species extinction: , b. range shifts and phenological changes: , c. ocean acidification and coral reefs: , d. adaptive strategies and conservation efforts: .

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 

Frequently asked questions 

What is a literature review .

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field.

Find academic papers related to your research topic faster. Try Research on Paperpal

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example 

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:  

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!

How to write a good literature review 

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as yo u go with Paperpal Research. Start now for free!

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review 

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:  

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:  

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:  

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:  

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:  

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:  

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?  

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research | Cite feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface. It also allows you auto-cite references in 10,000+ styles and save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research | Cite” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 

• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references in 10,000+ styles into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 22+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, machine translation vs human translation: which is reliable..., what is academic integrity, and why is it..., how to make a graphical abstract, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa....

Library Guides

Literature reviews: criticality.

• Criticality

Express Critical Analysis

The literature review of a dissertation should include critical analysis. You cannot simply juxtapose the literature you find: you have to  evaluate and draw conclusions from it.  

Paragraph level  

Try expressing your voice in each paragraph of your literature review. Write strong paragraphs. In strong paragraphs your voice can be heard in the topic sentence, development (where you analyse and compare/contrast the sources, sometimes as individual pieces, sometimes in a synthesis) and, even more easily, in the concluding sentence, where you present the "therefore" of the paragraph. 

How to express criticality at the paragraph level:  

Identify the significance of the sources, and why the points they are making are relevant  

Make connections between the sources 

Compare and contrast sources, literatures  

Accept/adopt points made by the sources, with reasons  

Reject the points made by the sources, with reasons (e.g., limitations in the methodology; out of date; limited scope; geographical delimitation) 

Indicate the position you are taking in your own work on the theories and concepts presented by the sources 

Show how limitations in the existing literature create a research gap for you 

Organise the materials, synthesising them in an original way, that sheds new light on the topic.  

To find out more about paragraph writing, check out the Assignment Writing Guides.

Literature review level 

Try to take ownership of the literature review. Remember the purposes of the review (providing background on the subject you are researching and identifying a gap in the existing literature on this subject). Thus, throughout the review:   

Identify the key themes relevant to your subject matter  

Identify the most logical and effective order for your themes 

Relate the sources back to the dissertation's research question 

Shed new light on the topic 

Draw conclusions on the existing literature  

Identify gaps in the literature  

Your literature review should present an argument (which you can recap in the concluding paragraph of the literature review). For instance, 

"The literature says/illustrates/reveals that... there are debates in the literature as of... it can be understood from the literature that... however, there are gaps in the literature... the literature does not specifically address (specific sector/location/population)... there is a lack of independent/recent studies on...  therefore in order to answer the research question(s) (you can repeat the question) this dissertation uses method xyz, as illustrated in the next section (if applicable)". 

Manchester University’s  academic phrase bank  is a great resource for learning new words and phrases. 

Extra Resources

For extra help with all aspects of study skills including how to undertake literature reviews, appointments are available with learning advisors on Engage. 

Appointments are also available with an Academic Engagement Librarian to discuss any issues you might be having with research.

• << Previous: Synthesis
• Last Updated: Nov 18, 2023 10:56 PM
• URL: https://libguides.westminster.ac.uk/literature-reviews

CONNECT WITH US

• Help and Support
• Research Guides

Literature reviews - research guide

• Critical reading and analysis
• Literature reviews home
• Planning your Review & EBP
• Searching for literature
• Managing your results
• Writing your review
• Systematic literature reviews

Critical reading & analysis

• Critical reading
• Analysing sources

Author analysis

• Journal analysis
• Note taking

A critical reader:​ ​

• Does not believe everything they read​ ​
• Questions what they read​ ​
• Rereads if necessary​ ​
• Understands the influence of style​ ​
• Analyses arguments​ ​
• Discounts arguments that are unsupported or based on faulty reasoning

When reading critically, focus on the purpose of your literature review:

• Think about what you expect from the article or chapter, before reading it
• Skim the abstract, headings, conclusion, and the first sentence of each paragraph
• Focus on the arguments presented rather than facts
• Take notes as you read and start to organise your review around themes and ideas
• Consider using a table, matrix or concept map to identify how the different sources relate to each other
• Note four to six points for each study that summarises the main points and conclusions
• Be as objective as possible

Critical appraisal

Critical appraisal is the process of carefully and syst ematically examining research to judge its credibility, its value and its relevance in a specific context.

The aim of critical appraisal is to understand the strengths, weaknesses, and potential for bias in the research. Validity, applicability, and clinical importance should be considered during critical appraisal to ensure that research evidence is used reliably and efficiently and false conclusions are not drawn.

Why do we need to critically appraise the literature?

Critical appraisal is necessary to:

• Assess benefits and strengths for research against flaws and weaknesses
• Decide whether studies have been undertaken in a way that makes their findings reliable
• Make sense of the result
• Know what these results mean in the context of the clinical decision being made
• Assess the usefulness of  the evidence for clinical decisions

Elements of sources

• Abstract: this is what the author wants the reader to take away from their article - what is the starting point? ​ ​
• Introduction:  provides background and a starting point - how does it guide the reader?​ ​
• Materials and methods:  often overlooked but very important - is the methodology understandable, reproducible, direct and robust?
• What do the tables, figures and legends actually report? ​ ​
• What do you think the data means? Decide before reading the discussion​.
• Discussion:  author draws conclusions – how does this correlate with your conclusions?

Evaluation of sources

Consider the following criteria:

• Is the source up-to-date?​
• Does it consider the latest research on your topic?​
• Is the article relevant to your topic?​
• Is the research methodology comprehensively described?​
• Does the work ultimately contribute in any significant way to an understanding of the subject?
• How reputable is the source?​
• Is the source peer-reviewed?​
• What is the source's impact factor ?
• Is the author from a reputable institution?​
• Have you seen the author cited in other sources?​
• Does the data support the conclusions drawn?​
• Are the author's opinions and conclusions convincing? 
• Are the author's arguments supported by evidence (primary material, case studies, narratives, statistics, recent findings)?​ ​
• Is the article properly referenced?​
• What is the purpose of the article and its intended audience?​
• Can you detect any bias in the content?​
• Is the reporting objective?​
• Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Which of the author's arguments are most/least convincing?​ ​
• Were the objectives achieved? ​ ​
• Hypotheses tested? ​ ​
• How do these results relate to other studies you have found?​ ​
• Do the authors openly discuss any limitations of their study?​ ​
• What else needs be studied in the future?

Interpretation

• Read critically​ ​
• Note 2-4 bullet points for each study that summarises the main points and conclusions​ ​
• Use matrix to analyse findings, relevance and importance of each text​ ​
• Draw attention to studies that are important, influential or that bring a new understanding or method of studying your area of research
• Literature Analysis Worksheet
• Literature Review Matrix

Databases such as Scopus and Web of Science can be used to:

• Locate the papers of a specific author
• Compare the research output of more than one author

The h -index

The h -index is a metric that allows you to compare the publications or research output of authors. This metric is calculated by determining the number of articles (n) written by an author, in the database, that have received the same number or more (n) citations over time. The h -index is a useful metric for comparing rates of publication, as the value is not skewed by a single highly cited paper, nor by a large number of poorly cited papers.

• The h -index is not a static value – if discussing an author’s h-index, you need to specify the date on which the h -index was calculated.
• The h -index is also calculated by other databases/resources and may vary from the h -index given by Scopus – if discussing an author’s h -index, you need to specify the source of the h-index.

See the example below of how an author's  h -index may appear in Scopus.

To locate papers of an author in  Scopus :

• Go to the default Scopus search screen and select Authors tab.
• Enter the author details and affiliation (university). Only include author surname for a comprehensive search. If the author has a common surname, include the first name's initial only.
• The author’s details and the documents that they have written, and which are indexed by Scopus, will be retrieved. Click on the author’s name to see a full list of their publications.

The information about the author will also tell you:

• How many of their publications have been indexed by Scopus
• How many times their publications have been cited
• Which of their publications are most highly cited
• Who they have co-authored papers with
• Their publication and citation trends for the past nine years
• Their h -index

See the Scopus resources below for more help:

• How to search for authors by topic
• How to assess an author's impact
• How to keep track of an author
• How to create citation overviews in Scopus

Web of Science Core Collection

To locate papers of an author in Web of Science Core Collection (WoS CC):

• Go to the default WoS CC search screen and select Researchers tab.
• Enter the author's surname and first name's initial.

The author’s details and the papers that they have written, and which are indexed by WoS CC, will be retrieved. Click on the Publications tab to see a full list of their publications.

• How many of their publications have been indexed by WoS CC

The author's citation report will tell you:

The Measuring research quality and impact guide has more detailed information on author analytics:

• Measuring research quality and impact

Source/journal analysis

Databases such as Scopus and Web of Science  (including CAB Abstracts ) can be used to determine the quality of journals in a discipline or field of research. For more information on journal analytics, please see the relevant section of our guide on Measuring Research Quality & Impact:

• Journal quality & impact (in Measuring research quality & impact guide)

Taking clear, legible notes will help to focus your critical reading and analysis of your literature review sources. When taking notes, avoid plagiarism by:

•  Keeping track of the difference between information from your sources and from your own ideas
•  Providing clear references, including page numbers

Note taking methods

Some effective methods of note-taking include:

• Outlining method. Use headings, sub-headings and bullet points to organize topics
• Cornell method. Use two columns - in one column write your summary of the authors' conclusions and evidence, and in the other column write down your own analysis and other comments
• Charting method. Create a list of topics or points you want to write about - use a column for each one. As you read, add references and make notes in the appropriate column
• Sentence method. Simply write down new ideas and bits of information as a numbered  sentence
• Mapping method. Write down key concepts and terms, with related ideas radiating out from these

You may consider using the matrix below for your note taking and analysis:

Critical reading & analysis checklist

• Does your literature review highlight flaws, gaps, or shortcomings of specific texts or groups of texts?
• Have you identified areas that have not yet been researched or have not yet been researched sufficiently?
• Does the literature demonstrate a change over time or recent developments that make your research relevant now?
• Are you able to discuss research methods used to study this topic and/or related topics?
• Can you clearly state why your research is necessary?
• << Previous: Managing your results
• Next: Writing your review >>
• Last Updated: Mar 8, 2024 1:16 PM
• URL: https://libguides.murdoch.edu.au/LitReview

Conduct a literature review

What is a literature review.

A literature review is a summary of the published work in a field of study. This can be a section of a larger paper or article, or can be the focus of an entire paper. Literature reviews show that you have examined the breadth of knowledge and can justify your thesis or research questions. They are also valuable tools for other researchers who need to find a summary of that field of knowledge.

Unlike an annotated bibliography, which is a list of sources with short descriptions, a literature review synthesizes sources into a summary that has a thesis or statement of purpose—stated or implied—at its core.

How do I write a literature review?

Step 1: define your research scope.

• What is the specific research question that your literature review helps to define?
• Are there a maximum or minimum number of sources that your review should include?

Ask us if you have questions about refining your topic, search methods, writing tips, or citation management.

Step 2: Identify the literature

Start by searching broadly. Literature for your review will typically be acquired through scholarly books, journal articles, and/or dissertations. Develop an understanding of what is out there, what terms are accurate and helpful, etc., and keep track of all of it with citation management tools . If you need help figuring out key terms and where to search, ask us .

Use citation searching to track how scholars interact with, and build upon, previous research:

• Mine the references cited section of each relevant source for additional key sources
• Use Google Scholar or Scopus to find other sources that have cited a particular work

Step 3: Critically analyze the literature

Key to your literature review is a critical analysis of the literature collected around your topic. The analysis will explore relationships, major themes, and any critical gaps in the research expressed in the work. Read and summarize each source with an eye toward analyzing authority, currency, coverage, methodology, and relationship to other works. The University of Toronto's Writing Center provides a comprehensive list of questions you can use to analyze your sources.

Step 4: Categorize your resources

Divide the available resources that pertain to your research into categories reflecting their roles in addressing your research question. Possible ways to categorize resources include organization by:

• methodology
• theoretical/philosophical approach

Regardless of the division, each category should be accompanied by thorough discussions and explanations of strengths and weaknesses, value to the overall survey, and comparisons with similar sources. You may have enough resources when:

• You've used multiple databases and other resources (web portals, repositories, etc.) to get a variety of perspectives on the research topic.
• The same citations are showing up in a variety of databases.

Additional resources

Undergraduate student resources.

• Literature Review Handout (University of North Carolina at Chapel Hill)
• Learn how to write a review of literature (University of Wisconsin-Madison)

Graduate student and faculty resources

• Information Research Strategies (University of Arizona)
• Literature Reviews: An Overview for Graduate Students (NC State University)
• Oliver, P. (2012). Succeeding with Your Literature Review: A Handbook for Students [ebook]
• Machi, L. A. & McEvoy, B. T. (2016). The Literature Review: Six Steps to Success [ebook]
• Graustein, J. S. (2012). How to Write an Exceptional Thesis or Dissertation: A Step-by-Step Guide from Proposal to Successful Defense [ebook]
• Thomas, R. M. & Brubaker, D. L. (2008). Theses and Dissertations: A Guide to Planning, Research, and Writing

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Jul 23, 2024 3:40 PM
• URL: https://guides.library.ucla.edu/systematicreviews

• Library Guides
• Literature Reviews
• Getting Started

Literature Reviews: Getting Started

What is a literature review.

A literature review is an overview of the available research for a specific scholarly topic. Literature reviews summarize existing research to answer a review question, provide context for new research, or identify important gaps in the existing body of literature.

An incredible amount of academic literature is published each year; by some estimates nearly three million articles .

Sorting through and reviewing that literature can be complicated, so this Research Guide provides a structured approach to make the process more manageable.

THIS GUIDE IS AN OVERVIEW OF THE LITERATURE REVIEW PROCESS:

• Getting Started (asking a research question | defining scope)
• Choosing a Type of Review
• Searching the Literature
• Organizing the Literature
• Writing the Literature Review (analyzing | synthesizing)

A  literature search  is a systematic search of the scholarly sources in a particular discipline. A  literature review   is the analysis, critical evaluation and synthesis of the results of that search. During this process you will move from a review  of  the literature to a review  for   your research.   Your synthesis of the literature is your unique contribution to research.

WHO IS THIS RESEARCH GUIDE FOR?

— those new to reviewing the literature

— those that need a refresher or a deeper understanding of writing literature reviews

You may need to do a literature review as a part of a course assignment, a capstone project, a master's thesis, a dissertation, or as part of a journal article. No matter the context, a literature review is an essential part of the research process. 

Literature Review Process

Purpose of a Literature Review

What is the purpose of a literature review.

A literature review is typically performed for a specific reason. Even when assigned as an assignment, the goal of the literature review will be one or more of the following:

• To communicate a project's novelty by identifying a research gap

• An overview of research issues , methodologies or results relevant to field
• To explore the  volume and types of available studies
• To establish familiarity with current research before carrying out a new project
• To resolve conflicts amongst contradictory previous studies

Reviewing the literature helps you understand a research topic and develop your own perspective.

A LITERATURE REVIEW IS NOT :

• An annotated bibliography – which is a list of annotated citations to books, articles and documents that includes a brief description and evaluation for each entry
• A literary review – which is a critical discussion of the merits and weaknesses of a literary work
• A book review – which is a critical discussion of the merits and weaknesses of a particular book

Attribution

Thanks to Librarian Jamie Niehof at the University of Michigan for providing permission to reuse and remix this Literature Reviews guide.

Workshop Materials

• Materials from OSU Literature Review Workshop
• Last Updated: Sep 23, 2024 11:11 AM
• URL: https://info.library.okstate.edu/literaturereviews

• Privacy Policy

Home » Literature Review – Types Writing Guide and Examples

Literature Review – Types Writing Guide and Examples

Table of Contents

Literature Review

Definition:

A literature review is a comprehensive and critical analysis of the existing literature on a particular topic or research question. It involves identifying, evaluating, and synthesizing relevant literature, including scholarly articles, books, and other sources, to provide a summary and critical assessment of what is known about the topic.

Types of Literature Review

Types of Literature Review are as follows:

• Narrative literature review : This type of review involves a comprehensive summary and critical analysis of the available literature on a particular topic or research question. It is often used as an introductory section of a research paper.
• Systematic literature review: This is a rigorous and structured review that follows a pre-defined protocol to identify, evaluate, and synthesize all relevant studies on a specific research question. It is often used in evidence-based practice and systematic reviews.
• Meta-analysis: This is a quantitative review that uses statistical methods to combine data from multiple studies to derive a summary effect size. It provides a more precise estimate of the overall effect than any individual study.
• Scoping review: This is a preliminary review that aims to map the existing literature on a broad topic area to identify research gaps and areas for further investigation.
• Critical literature review : This type of review evaluates the strengths and weaknesses of the existing literature on a particular topic or research question. It aims to provide a critical analysis of the literature and identify areas where further research is needed.
• Conceptual literature review: This review synthesizes and integrates theories and concepts from multiple sources to provide a new perspective on a particular topic. It aims to provide a theoretical framework for understanding a particular research question.
• Rapid literature review: This is a quick review that provides a snapshot of the current state of knowledge on a specific research question or topic. It is often used when time and resources are limited.
• Thematic literature review : This review identifies and analyzes common themes and patterns across a body of literature on a particular topic. It aims to provide a comprehensive overview of the literature and identify key themes and concepts.
• Realist literature review: This review is often used in social science research and aims to identify how and why certain interventions work in certain contexts. It takes into account the context and complexities of real-world situations.
• State-of-the-art literature review : This type of review provides an overview of the current state of knowledge in a particular field, highlighting the most recent and relevant research. It is often used in fields where knowledge is rapidly evolving, such as technology or medicine.
• Integrative literature review: This type of review synthesizes and integrates findings from multiple studies on a particular topic to identify patterns, themes, and gaps in the literature. It aims to provide a comprehensive understanding of the current state of knowledge on a particular topic.
• Umbrella literature review : This review is used to provide a broad overview of a large and diverse body of literature on a particular topic. It aims to identify common themes and patterns across different areas of research.
• Historical literature review: This type of review examines the historical development of research on a particular topic or research question. It aims to provide a historical context for understanding the current state of knowledge on a particular topic.
• Problem-oriented literature review : This review focuses on a specific problem or issue and examines the literature to identify potential solutions or interventions. It aims to provide practical recommendations for addressing a particular problem or issue.
• Mixed-methods literature review : This type of review combines quantitative and qualitative methods to synthesize and analyze the available literature on a particular topic. It aims to provide a more comprehensive understanding of the research question by combining different types of evidence.

Parts of Literature Review

Parts of a literature review are as follows:

Introduction

The introduction of a literature review typically provides background information on the research topic and why it is important. It outlines the objectives of the review, the research question or hypothesis, and the scope of the review.

Literature Search

This section outlines the search strategy and databases used to identify relevant literature. The search terms used, inclusion and exclusion criteria, and any limitations of the search are described.

Literature Analysis

The literature analysis is the main body of the literature review. This section summarizes and synthesizes the literature that is relevant to the research question or hypothesis. The review should be organized thematically, chronologically, or by methodology, depending on the research objectives.

Critical Evaluation

Critical evaluation involves assessing the quality and validity of the literature. This includes evaluating the reliability and validity of the studies reviewed, the methodology used, and the strength of the evidence.

The conclusion of the literature review should summarize the main findings, identify any gaps in the literature, and suggest areas for future research. It should also reiterate the importance of the research question or hypothesis and the contribution of the literature review to the overall research project.

The references list includes all the sources cited in the literature review, and follows a specific referencing style (e.g., APA, MLA, Harvard).

How to write Literature Review

Here are some steps to follow when writing a literature review:

• Define your research question or topic : Before starting your literature review, it is essential to define your research question or topic. This will help you identify relevant literature and determine the scope of your review.
• Conduct a comprehensive search: Use databases and search engines to find relevant literature. Look for peer-reviewed articles, books, and other academic sources that are relevant to your research question or topic.
• Evaluate the sources: Once you have found potential sources, evaluate them critically to determine their relevance, credibility, and quality. Look for recent publications, reputable authors, and reliable sources of data and evidence.
• Organize your sources: Group the sources by theme, method, or research question. This will help you identify similarities and differences among the literature, and provide a structure for your literature review.
• Analyze and synthesize the literature : Analyze each source in depth, identifying the key findings, methodologies, and conclusions. Then, synthesize the information from the sources, identifying patterns and themes in the literature.
• Write the literature review : Start with an introduction that provides an overview of the topic and the purpose of the literature review. Then, organize the literature according to your chosen structure, and analyze and synthesize the sources. Finally, provide a conclusion that summarizes the key findings of the literature review, identifies gaps in knowledge, and suggests areas for future research.
• Edit and proofread: Once you have written your literature review, edit and proofread it carefully to ensure that it is well-organized, clear, and concise.

Examples of Literature Review

Here’s an example of how a literature review can be conducted for a thesis on the topic of “ The Impact of Social Media on Teenagers’ Mental Health”:

• Start by identifying the key terms related to your research topic. In this case, the key terms are “social media,” “teenagers,” and “mental health.”
• Use academic databases like Google Scholar, JSTOR, or PubMed to search for relevant articles, books, and other publications. Use these keywords in your search to narrow down your results.
• Evaluate the sources you find to determine if they are relevant to your research question. You may want to consider the publication date, author’s credentials, and the journal or book publisher.
• Begin reading and taking notes on each source, paying attention to key findings, methodologies used, and any gaps in the research.
• Organize your findings into themes or categories. For example, you might categorize your sources into those that examine the impact of social media on self-esteem, those that explore the effects of cyberbullying, and those that investigate the relationship between social media use and depression.
• Synthesize your findings by summarizing the key themes and highlighting any gaps or inconsistencies in the research. Identify areas where further research is needed.
• Use your literature review to inform your research questions and hypotheses for your thesis.

For example, after conducting a literature review on the impact of social media on teenagers’ mental health, a thesis might look like this:

“Using a mixed-methods approach, this study aims to investigate the relationship between social media use and mental health outcomes in teenagers. Specifically, the study will examine the effects of cyberbullying, social comparison, and excessive social media use on self-esteem, anxiety, and depression. Through an analysis of survey data and qualitative interviews with teenagers, the study will provide insight into the complex relationship between social media use and mental health outcomes, and identify strategies for promoting positive mental health outcomes in young people.”

Reference: Smith, J., Jones, M., & Lee, S. (2019). The effects of social media use on adolescent mental health: A systematic review. Journal of Adolescent Health, 65(2), 154-165. doi:10.1016/j.jadohealth.2019.03.024

Reference Example: Author, A. A., Author, B. B., & Author, C. C. (Year). Title of article. Title of Journal, volume number(issue number), page range. doi:0000000/000000000000 or URL

Applications of Literature Review

some applications of literature review in different fields:

• Social Sciences: In social sciences, literature reviews are used to identify gaps in existing research, to develop research questions, and to provide a theoretical framework for research. Literature reviews are commonly used in fields such as sociology, psychology, anthropology, and political science.
• Natural Sciences: In natural sciences, literature reviews are used to summarize and evaluate the current state of knowledge in a particular field or subfield. Literature reviews can help researchers identify areas where more research is needed and provide insights into the latest developments in a particular field. Fields such as biology, chemistry, and physics commonly use literature reviews.
• Health Sciences: In health sciences, literature reviews are used to evaluate the effectiveness of treatments, identify best practices, and determine areas where more research is needed. Literature reviews are commonly used in fields such as medicine, nursing, and public health.
• Humanities: In humanities, literature reviews are used to identify gaps in existing knowledge, develop new interpretations of texts or cultural artifacts, and provide a theoretical framework for research. Literature reviews are commonly used in fields such as history, literary studies, and philosophy.

Role of Literature Review in Research

Here are some applications of literature review in research:

• Identifying Research Gaps : Literature review helps researchers identify gaps in existing research and literature related to their research question. This allows them to develop new research questions and hypotheses to fill those gaps.
• Developing Theoretical Framework: Literature review helps researchers develop a theoretical framework for their research. By analyzing and synthesizing existing literature, researchers can identify the key concepts, theories, and models that are relevant to their research.
• Selecting Research Methods : Literature review helps researchers select appropriate research methods and techniques based on previous research. It also helps researchers to identify potential biases or limitations of certain methods and techniques.
• Data Collection and Analysis: Literature review helps researchers in data collection and analysis by providing a foundation for the development of data collection instruments and methods. It also helps researchers to identify relevant data sources and identify potential data analysis techniques.
• Communicating Results: Literature review helps researchers to communicate their results effectively by providing a context for their research. It also helps to justify the significance of their findings in relation to existing research and literature.

Purpose of Literature Review

Some of the specific purposes of a literature review are as follows:

• To provide context: A literature review helps to provide context for your research by situating it within the broader body of literature on the topic.
• To identify gaps and inconsistencies: A literature review helps to identify areas where further research is needed or where there are inconsistencies in the existing literature.
• To synthesize information: A literature review helps to synthesize the information from multiple sources and present a coherent and comprehensive picture of the current state of knowledge on the topic.
• To identify key concepts and theories : A literature review helps to identify key concepts and theories that are relevant to your research question and provide a theoretical framework for your study.
• To inform research design: A literature review can inform the design of your research study by identifying appropriate research methods, data sources, and research questions.

Characteristics of Literature Review

Some Characteristics of Literature Review are as follows:

• Identifying gaps in knowledge: A literature review helps to identify gaps in the existing knowledge and research on a specific topic or research question. By analyzing and synthesizing the literature, you can identify areas where further research is needed and where new insights can be gained.
• Establishing the significance of your research: A literature review helps to establish the significance of your own research by placing it in the context of existing research. By demonstrating the relevance of your research to the existing literature, you can establish its importance and value.
• Informing research design and methodology : A literature review helps to inform research design and methodology by identifying the most appropriate research methods, techniques, and instruments. By reviewing the literature, you can identify the strengths and limitations of different research methods and techniques, and select the most appropriate ones for your own research.
• Supporting arguments and claims: A literature review provides evidence to support arguments and claims made in academic writing. By citing and analyzing the literature, you can provide a solid foundation for your own arguments and claims.
• I dentifying potential collaborators and mentors: A literature review can help identify potential collaborators and mentors by identifying researchers and practitioners who are working on related topics or using similar methods. By building relationships with these individuals, you can gain valuable insights and support for your own research and practice.
• Keeping up-to-date with the latest research : A literature review helps to keep you up-to-date with the latest research on a specific topic or research question. By regularly reviewing the literature, you can stay informed about the latest findings and developments in your field.

Advantages of Literature Review

There are several advantages to conducting a literature review as part of a research project, including:

• Establishing the significance of the research : A literature review helps to establish the significance of the research by demonstrating the gap or problem in the existing literature that the study aims to address.
• Identifying key concepts and theories: A literature review can help to identify key concepts and theories that are relevant to the research question, and provide a theoretical framework for the study.
• Supporting the research methodology : A literature review can inform the research methodology by identifying appropriate research methods, data sources, and research questions.
• Providing a comprehensive overview of the literature : A literature review provides a comprehensive overview of the current state of knowledge on a topic, allowing the researcher to identify key themes, debates, and areas of agreement or disagreement.
• Identifying potential research questions: A literature review can help to identify potential research questions and areas for further investigation.
• Avoiding duplication of research: A literature review can help to avoid duplication of research by identifying what has already been done on a topic, and what remains to be done.
• Enhancing the credibility of the research : A literature review helps to enhance the credibility of the research by demonstrating the researcher’s knowledge of the existing literature and their ability to situate their research within a broader context.

Limitations of Literature Review

Limitations of Literature Review are as follows:

• Limited scope : Literature reviews can only cover the existing literature on a particular topic, which may be limited in scope or depth.
• Publication bias : Literature reviews may be influenced by publication bias, which occurs when researchers are more likely to publish positive results than negative ones. This can lead to an incomplete or biased picture of the literature.
• Quality of sources : The quality of the literature reviewed can vary widely, and not all sources may be reliable or valid.
• Time-limited: Literature reviews can become quickly outdated as new research is published, making it difficult to keep up with the latest developments in a field.
• Subjective interpretation : Literature reviews can be subjective, and the interpretation of the findings can vary depending on the researcher’s perspective or bias.
• Lack of original data : Literature reviews do not generate new data, but rather rely on the analysis of existing studies.
• Risk of plagiarism: It is important to ensure that literature reviews do not inadvertently contain plagiarism, which can occur when researchers use the work of others without proper attribution.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Evaluating Research – Process, Examples and...

Critical Analysis – Types, Examples and Writing...

Table of Contents – Types, Formats, Examples

Purpose of Research – Objectives and Applications

Tables in Research Paper – Types, Creating Guide...

Research Topics – Ideas and Examples

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

Critical Analysis in a Literature Review

• 3-minute read
• 29th June 2015

A literature review is vital to any in-depth research , providing a foundation your work will build upon. Familiarizing yourself with the existing literature allows you to identify current debates in the field, ensuring that your work is up-to-date and addresses significant questions.

But a good literature review will require reading critically. This means deciding whether you agree or disagree with certain viewpoints, arguments and theories, rather than simply describing them.

It also requires being able to spot the flaws and strengths of particular studies, which can in turn help when developing your own ideas. To make sure you do this effectively, it’s worth looking for the following things.

1. Overgeneralizations

One common issue in research is the scope of its application, especially when dealing with limited sample sizes or when a study is generalized too broadly.

The conclusions of a psychological study conducted with all male participants, for instance, may not be applicable in the same way to female subjects.

2. Methodological Limitations

When writing a literature review, ask yourself whether the methods used for particular studies were appropriate.

For example, whether the study used a quantitative, qualitative or mixed-methods research design can make a big difference to the conclusions reached.

3. How Well Explained is the Research?

When reading for a critical literature review, it is important to consider how well written the studies you examine are.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

Does the author explain their methods? Is enough detail provided for any experiments to be replicated? Are sampling, data collection and analysis techniques clearly identified? Does the conclusion follow from the results?

Asking these and similar questions will help you discern between good and bad research.

4. Identify Biases

Another important factor is to consider whether implicit biases might have influenced the research.

The term “confirmation bias,” for example, refers to the tendency to focus on evidence which supports one’s existing beliefs, which can lead to overlooking alternative hypotheses.

5. Challenge Your Own Assumptions

If you come across a study which seems to oppose your hypothesis, consider whether it presents good counterarguments to your own position. If it does, ask yourself whether this affects how you conduct the rest of your research.

The final point here is important because conducting a literature review serves two purposes . The finished literature review will help your reader to understand the background of your research, so critical analysis helps to clarify what your work contributes to the debate.

But comparing different studies and theories for a literature review will also help you to develop a research approach. The better your critical analysis, then, the better prepared you’ll be.

Share this article:

Post A New Comment

Got content that needs a quick turnaround? Let us polish your work. Explore our editorial business services.

5-minute read

Free Email Newsletter Template

Promoting a brand means sharing valuable insights to connect more deeply with your audience, and...

6-minute read

How to Write a Nonprofit Grant Proposal

If you’re seeking funding to support your charitable endeavors as a nonprofit organization, you’ll need...

9-minute read

How to Use Infographics to Boost Your Presentation

Is your content getting noticed? Capturing and maintaining an audience’s attention is a challenge when...

8-minute read

Why Interactive PDFs Are Better for Engagement

Are you looking to enhance engagement and captivate your audience through your professional documents? Interactive...

7-minute read

Seven Key Strategies for Voice Search Optimization

Voice search optimization is rapidly shaping the digital landscape, requiring content professionals to adapt their...

4-minute read

Five Creative Ways to Showcase Your Digital Portfolio

Are you a creative freelancer looking to make a lasting impression on potential clients or...

Make sure your writing is the best it can be with our expert English proofreading and editing.

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health (m-health) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Literature reviews

• Introduction
• Conduct your search
• Store and organise the literature

Evaluate the information you have found

Critique the literature.

• Different subject areas
• Find literature reviews

When conducting your searches you may find many references that will not be suitable to use in your literature review.

• Skim through the resource. A quick read through the table of contents, the introductory paragraph or the abstract should indicate whether you need to read further or whether you can immediately discard the result.
• Evaluate the quality and reliability of the references you find. Our evaluating information  page outlines what you need to consider when evaluating the books, journal articles, news and websites you find to ensure they are suitable for use in your literature review.

Critiquing the literature involves looking at the strength and weaknesses of the paper and evaluating the statements made by the author/s.

Books and resources on reading critically

• CASP Checklists Critical appraisal tools designed to be used when reading research. Includes tools for Qualitative studies, Systematic Reviews, Randomised Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies and Clinical Prediction Rule.
• How to read critically - business and management From Postgraduate research in business - the aim of this chapter is to show you how to become a critical reader of typical academic literature in business and management.
• Learning to read critically in language and literacy Aims to develop skills of critical analysis and research design. It presents a series of examples of `best practice' in language and literacy education research.
• Critical appraisal in health sciences See tools for critically appraising health science research.

• << Previous: Store and organise the literature
• Next: Different subject areas >>
• Last Updated: Aug 9, 2024 8:34 AM
• URL: https://guides.library.uq.edu.au/research-techniques/literature-reviews

Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.

We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.

Brief introduction to this section that descibes Open Access especially from an IntechOpen perspective

Want to get in touch? Contact our London head office or media team here

Our team is growing all the time, so we’re always on the lookout for smart people who want to help us reshape the world of scientific publishing.

Home > Books > National Security in the Digital and Information Age

The Front End of Innovation in Defense: A Comprehensive Literature Review

Submitted: 18 January 2024 Reviewed: 02 March 2024 Published: 30 April 2024

DOI: 10.5772/intechopen.1005191

Cite this chapter

There are two ways to cite this chapter:

From the Edited Volume

National Security in the Digital and Information Age

To purchase hard copies of this book, please contact the representative in India: CBS Publishers & Distributors Pvt. Ltd. www.cbspd.com | [email protected]

Chapter metrics overview

38 Chapter Downloads

Impact of this chapter

Total Chapter Downloads on intechopen.com

Total Chapter Views on intechopen.com

Innovation management, a multifaceted organizational process encompassing opportunities and ideas from inception to implementation, demands a systematic approach, particularly in the critical initial phase known as the Front End of Innovation (FEI). This pivotal phase significantly influences the entire innovation management chain. Despite its recognized importance, FEI in the defense sector has yet to be systematically addressed in the academic literature. Recognizing the vital role of FEI in the defense sector, this study addresses this deficiency through a systematic review, scrutinizing 24 documents from the scientific literature (Scopus and Web of Science databases) and gray literature (government defense documents). This research systematically maps key activities identified in seminal FEI models. These activities include the identification and analysis of opportunities; generation, enrichment, and screening of ideas; product concept definition; and consideration of influencing factors. Concurrently, this work aligns defense practices with established innovation models and provides valuable insights for optimizing the management dynamics of the military innovation process. Through this systematic inquiry, this study contributes to a nuanced understanding of the FEI in the defense sector, offering practical implications for enhancing defense innovation development.

• front end of innovation
• literature review

Author Information

Romullo girardi *.

• Military Institute of Engineering, Rio de Janeiro, Brazil
• INESC TEC and Faculty of Engineering, University of Porto, Porto, Portugal

Juraci Ferreira Galdino

Paulo césar pellanda.

*Address all correspondence to: [email protected]

1. Introduction

Innovation management, a complex and broad organizational process covering the entire spectrum, from identifying new opportunities and ideas to their practical implementation, poses significant challenges for managers across all organizational levels [ 1 ].

Notably, innovation seldom fails due to a lack of creativity; instead, it is the absence of discipline that plays a pivotal role in innovation failures [ 2 ]. From this perspective, Boeddrich [ 3 ] contends that systematic and structured procedures in the early phase, known as the Front End of Innovation (FEI), are imperative to avert adverse effects throughout the innovation management chain.

Multiple researchers emphasize that enhancing FEI activities contributes positively to organizational outcomes, bolstering the likelihood of successful innovation development [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ]. Yet, the successful adoption of a FEI model requires considering some factors like organizational size and culture, as well as decision-making styles [ 10 , 11 ].

Despite the increasing attention to FEI as a complex and multidisciplinary field [ 12 ], the defense context of FEI has not been sufficiently addressed in the academic literature, a gap this study endeavors to address. Therefore, this research aims to unravel the dynamics of FEI in the military sector through a systematic literature review, focusing on the research question: How can the current literature on the early phase of the innovation process in defense be mapped within seminal FEI models?

This question is pertinent given the Armed Forces’ distinct organizational culture, demanding innovation to sustain high-tech operational capabilities and mainly requiring innovations capable of inducing technological surprise in the theater of operations. Aligning defense practices with established models in innovation literature can furnish invaluable insights for improving the management dynamics of the military’s initial innovation phase. Moreover, by reviewing approaches used by different countries, the study recognizes that the suitability and significance of FEI management practices can differ across national defense contexts. It emphasizes how cultural and procedural nuances impact the development and adoption of new technologies in military settings.

Structured around the research question, this paper is organized as follows: Section 2 provides a theoretical foundation on FEI, seminal FEI models, and defense peculiarities. Section 3 outlines the research methodology. Section 4 delineates the mapping of the FEI in the defense sector within seminal FEI models. Section 5 discusses salient aspects identified throughout the study. Finally, Section 6 highlights the concluding remarks, outlining directions for future research.

2. Theoretical foundation

Before exploring the current literature on the early stage of the innovation process in defense, it is essential to understand the foundational topics involved: the FEI concept, the seminal FEI models, and the unique aspects of the military sector.

2.1 Front end of innovation (FEI)

The Fuzzy Front End (FFE) refers to the earliest stage in the New Product Development (NPD) process. This term was popularized by Smith and Reinertsen [ 13 ], as pointed out by Khurana and Rosenthal [ 10 ].

In 2002, Koen et al. [ 14 ] proposed the term Front End of Innovation (FEI), considering that the adjective “fuzzy” is “mysterious, lacks accountability, and cannot be critically evaluated” ([ 14 ], p. 30). The new term dissociated the idea that the initial phase of the innovation process was nebulous and uncontrollable. In this approach, the FEI is described as “those activities that come before the formal and well-structured NPD process” ([ 14 ], p. 30).

Figure 1 illustrates the breakdown of the product innovation management process into three phases: FEI, NPD, and implementation (the commercialization of the product in the market). The circular shape of the FEI suggests that ideas should flow and iterate until the formal definition of products is developed. In contrast, the NPD phase is depicted as a series of sequential, well-structured, and chronologically ordered steps [ 14 , 15 ].

Breakdown of the product innovation management process. Source: Adapted from Koen et al. [ 14 , 15 ].

The FEI emerges as a crucial driver of positive outcomes for new products and, consequently, the overall success of the business [ 16 ]. Markham [ 6 ] underscores the profound impact of early-phase activities on product performance, emphasizing that the success of the front end stands as the strongest independent predictor of all NPD performance variables.

Selecting an appropriate FEI model requires careful consideration of various factors, including organizational size and culture, as well as decision-making style [ 10 , 11 ]. As a response to these diverse organizational needs, numerous FEI models have been developed over time, offering distinct approaches to navigate their complexities, as detailed in the next section.

2.2 Seminal FEI models

In an integrative literature review, Pereira et al. [ 12 ] found that 26% of articles related to FEI contributed in terms of frameworks, models, processes, tools, and methodologies, exemplifying endeavors to structure the early phase of the innovation process in specific contexts.

While recent contributions are significant, seminal works have produced models that served as reference points for structuring the FEI. Table 1 provides an overview of the four seminal models identified by Pereira et al. [ 12 , 20 ].

Seminal FEI models.

Source: Adapted from Pereira et al. [ 12 , 20 ].

Stage 0 (Discovery) : In this inaugural stage, the organization actively generates ideas for new products.

Gate 1 (Idea screen) : Ideas undergo a concise evaluation based on strategic, feasibility, and market criteria. Financial considerations are deferred at this point. Accepted ideas proceed to the next phase.

Stage 1 (Scoping) : The accepted idea transitions into a project, initiating a dual evaluation process:

Market evaluation : Involves research, user contact, and conceptual testing to determine market size and acceptance.

Technical evaluation : Encompasses feasibility, costs, and development timelines.

Gate 2 (Second screen) : Comprehensive information from market and technical evaluations prompts a reassessment of the project’s viability. If approved, the project advances to the next stage.

Stage 2 (Build business case) : Positioned just before product development, this stage involves:

Assessing the project’s attractiveness.

Defining clear objectives.

Conducting market, technical, operational, and financial evaluations.

Gate 3 (Development) : A pivotal decision point where the organization determines resource allocation for project development [ 17 , 18 ].

The FEI within the stage-gate model. Source: Adapted from Cooper [ 17 ].

Pre-Phase Zero : This initial phase concentrates on the continuous identification of opportunities within the organization. It involves generating ideas and conducting technological and market analyzes. When a promising opportunity is identified, it triggers the transition to Phase Zero. The authors emphasize that this phase should occur continuously within the organization.

Phase Zero : This phase is initiated when Pre-Phase Zero identifies a promising opportunity. Its primary objective is to define the concept of a new product.

Phase One : Following the conceptualization of the new product, Phase One focuses on analyzing feasibility and planning the project to initiate the NPD process formally. It is crucial to maintain a constant interface with the organization’s product and portfolio strategy throughout the entire process.

Three Phase Front End model. Source: Khurana and Rosenthal [ 11 ].

The New Concept Development (NCD) model, proposed by Koen et al. [ 14 , 15 ], is a theoretical construction composed of the three fundamental concepts: controllable activities, “engine” and influencing factors. Controllable activities represent the elements that the organization can control. The “engine” encompasses the controllable aspects of the organization that are responsible for driving the activities of the FEI. Finally, the influencing factors are variables that have an impact on the FEI and are relatively outside of the organization’s control [ 14 , 15 ]. Table 2 details the structure of the NCD model. In the structure of the NCD model, organizational capabilities are classified as an influencing factor because they usually change very slowly and are therefore uncontrollable. Alternatively, organizational capabilities can be incorporated into the “engine” to the extent that the organization can modify and control them [ 14 ].

Structure of the NCD model.

Source: Koen et al. [ 14 ].

The model proposed by Reid and De Brentani [ 19 ] provides a unique focus on disruptive innovations, highlighting their distinct entry into the organization compared to incremental innovations. According to this model, disruptive innovations typically originate from the external environment. Figure 4 illustrates how the front end of the innovation process initiates its flow based on information from the external environment, involving the identification of unstructured problems and the recognition of opportunities. This model emphasizes that disruptive innovations follow a distinctive path, with the FEI process being strongly influenced by external inputs. The opportunities identified undergo thorough analyzes and decisions at various organizational levels before being formally integrated into an NPD project [ 19 ].

The Fuzzy Front End of new product development for discontinuous innovations. Source: Reid and De Brentani [ 19 ].

The seminal models presented – Cooper’s Stage-Gate Model, Khurana, and Rosenthal’s Three Phase Front End Model, Koen et al.’s New Concept Development (NCD) Model, and Reid and De Brentani’s Model for Discontinuous Innovations – vary in focus, approach, depth, and structuring of activities. Despite these differences, a common thread emerges as they collectively address the FEI through key activities: identification and analysis of opportunities, generation, enrichment, and screening of ideas, product concept definition, and consideration of influencing factors (encompassing the outside world, market and technology information, scenario planning, competitive analysis, and organizational issues such as culture, leadership, strategy, portfolio, and capabilities).

The ultimate objective of these FEI activities is to formulate a well-defined product concept before entering the formal NPD stage. Table 3 establishes the correspondence between FEI activities and the structures of the seminal models.

FEI activities and their relationship with the seminal models.

To mitigate potential ambiguities in the interpretation of certain terms related to the FEI, Table 4 provides standardized reference definitions. This table serves as a reference guide, providing clear and standardized definitions for key terms associated with the FEI, enhancing clarity and consistency in their interpretation.

Definitions for terms related to the FEI.

2.3 Defense sector

High technological level : The defense industry requires substantial investments in research, development, and innovation to create sophisticated products such as aircraft, ships, weapons, and systems. These must operate safely and reliably under severe conditions [ 25 ], often characterized as Complex Products and Systems (CoPS). CoPS involve customization, production in small quantities by a few companies, integration of diverse knowledge areas, and a lifecycle spanning decades [ 26 ].

Technological duality : Innovations developed for military purposes may have civilian applications (spin-off) and vice versa (spin-in) [ 27 , 28 ]. Dual-use technologies, like GPS and the Internet, initially developed for defense, now find widespread civilian applications.

Governmental dependence : The defense market is highly regulated and relies on government contracts, resulting in a strong dependence on public resources. From a demand perspective, the defense market operates as a monopsony, with the State being the primary purchaser of goods and services [ 29 , 30 , 31 ].

High market concentration : Global defense market dominance by a few companies leads to limited competition and protectionist practices. Oligopolies in the defense market can collude, manipulate prices, limit competition, or engage in practices like dumping [ 32 ] to control strategic interests [ 31 , 33 ].

Vulnerability to geopolitical issues : The demand for defense equipment is influenced by geopolitical conflicts and international relations, resulting in a volatile market subject to sudden changes. Companies may exploit geopolitical issues for financial or strategic reasons, impacting commitments during times of national crisis [ 31 , 34 ].

In summary, the defense sector is characterized by its strategic importance, technological advancement, dual-use nature, governmental dependence, market concentration, and vulnerability to geopolitical issues. This situation is a combination of a monopoly/oligopoly, where a few major global players dominate the supply, and a monopsony, where the State centralizes the demand. Recognizing these peculiarities highlights the need for a comprehensive exploration of FEI dynamics within the defense sector.

3. Methodology

Databases : Scopus and Web of Science (WoS), aligning with the methodology outlined by Ferreira et al. [ 36 ].

Search string : The search string was crafted by combining key terms related to the FEI and the defense sector. The FEI-related terms were derived from a frequency analysis of authors’ keywords, following the methodology of Ferreira et al. [ 36 ]. To capture the comprehensive scope of the defense sector, terms related to aerospace were also incorporated, acknowledging that certain countries treat both topics as a unified strategic theme. For instance, the United States has the U.S. Space Force under its Department of Defense [ 37 ]. The search was conducted on September 21, 2023, focusing on terms found in the title, abstract, and keywords:

Scopus database: TITLE-ABS-KEY((“front end of innovation” OR “front-end of innovation” OR “front end innovation” OR “front-end innovation” OR “fuzzy front end” OR “fuzzy front-end”) AND (“military” OR “defense” OR “defense” OR “navy” OR “army” OR “air force” OR “aerospace” OR “aeronautic*” OR “astronautic*” OR “avionics”)).

WoS database: TS = ((“front end of innovation” OR “front-end of innovation” OR “front end innovation” OR “front-end innovation” OR “fuzzy front end” OR “fuzzy front-end”) AND (“military” OR “defense” OR “defense” OR “navy” OR “army” OR “air force” OR “aerospace” OR “aeronautic*” OR “astronautic*” OR “avionics”)).

Inclusion criteria : publications in English, Portuguese, or Spanish were considered, and the accessibility of the entire document was taken into account. Additionally, the publication had to address explicitly the FEI in the defense context.

The academic literature search yielded five documents from the Scopus database and two from WoS. Upon analysis, one redundancy was identified, resulting in six unique documents. Subsequently, it was observed that two articles did not meet the inclusion criteria, leaving four documents within the review scope. Following this initial search, the snowball technique was applied to identify relevant documents citing or cited by the selected publications. Additional efforts were made to explore works authored by the selected publications’ authors, uncovering three more documents.

In parallel with the academic literature search, the exploration of government defense management agencies’ websites led to the identification of 17 more documents. Therefore, while the academic literature contributed documents presenting general aspects of FEI in the defense sector, the gray literature addressed defense management in specific nations, ensuring representation across continents and encompassing both developed and developing countries. The nations (or alliances) covered included Australia, Brazil, China, India, NATO (North Atlantic Treaty Organization), the United Kingdom, the United States, and South Africa.

Thus, a total of 24 documents were selected for review. It is noteworthy that, during the research, no review works similar to this article were found. The steps of the review are represented in summary in Figure 5 , using the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) diagram, a tool for presenting the flow of information through the different phases of a systematic literature review [ 38 ].

Stages of the systematic literature review. Source: Adapted from Page et al. [ 38 ].

4. Front end of innovation in the defense sector

The investigation of FEI in the defense sector is structured based on content mapping of the selected review documents, focusing on key FEI activities outlined in Section 2.2. These activities include the identification and analysis of opportunities; generation, enrichment, and screening of ideas; product concept definition; and consideration of influencing factors.

4.1 Identification and analysis of opportunities

The identification and analysis of opportunities serve as the primary catalyst for the FEI process. It occurs when an organization recognizes a gap, whether related to business or technology issues. This gap represents the difference between the current state and a desired future state, presenting an opportunity that can be exploited to gain a competitive advantage, address a threat, solve a problem, or enhance a situation [ 14 ]. As stressed by Khurana and Rosenthal [ 11 ], the phase of identifying and analyzing opportunities should be an ongoing and continuous process within an organization.

In the defense sector, the identification and analysis of opportunities are intricately tied to Capability-Based Planning (CBP), a central process in strategic defense management [ 39 , 40 ]. This strategic planning paradigm, initially utilized in the United States Nuclear Program during the 1960s, saw broader adoption by the U.S. Department of Defense in 2001, becoming a reference for armed forces worldwide [ 41 ]. From this standpoint, Table 5 provides a mapping of the phase of identification and analysis of opportunities within the documents reviewed.

Approaches to identification and analysis of opportunities within the review documents.

4.2 Generation, enrichment, and screening of ideas

According to Koen et al. [ 14 ], the generation and enrichment of ideas follow the identification and analysis of opportunities. An idea, as conceptualized by the authors, represents the most preliminary form of a new product or service, typically outlining a high-level vision for the planned solution related to the identified opportunity [ 14 ]. Cooper [ 17 ], in the stage-gate model, underscores that ideas for new products must undergo initial screening, known as gate 1, before being integrated into an organization’s project [ 17 ]. Reid and De Brentani [ 19 ] also stress the importance of a formal analysis of opportunities/ideas at the corporate level before progressing to the project level [ 19 ].

In the defense sector, as outlined in Table 6 , the identification of the need for a new product occurs when a capability gap analysis indicates the necessity for a new materiel solution. This triggers the formal process of acquiring a defense product. Table 6 provides a mapping of the phase of generation, enrichment, and screening of ideas within the reviewed documents.

Approaches to idea generation, enrichment, and screening within the review documents.

4.3 Product concept definition

According to Koen et al. [ 14 ], a well-defined product concept should entail a comprehensive configuration, offering both written and visual descriptions that encapsulate the primary features, customer benefits, and a broad understanding of the required technologies. This stage in product development represents the final step preceding the formal NPD process [ 14 ]. The models proposed by Khurana and Rosenthal [ 11 ], Reid and De Brentani [ 19 ], and Cooper [ 17 ] also emphasize the significance of product concept development, feasibility analysis, project planning, and decision-making as crucial prerequisites before formally entering the NPD phase.

In the defense sector, the Armed Forces commonly adopt the systems engineering lifecycle concept to structure their acquisition processes [ 56 ]. Within this framework, the product concept undergoes development through a top-down approach, comprising two distinct phases: logical description (problem domain) and physical description (solution domain). The logical or functional description essentially outlines the intended functionalities of the new product from the user’s perspective, providing an operational view. Building upon the logical description, the physical description then defines the high-level architecture of the product elements, encompassing systems, subsystems, assemblies, and/or components, from a technical perspective [ 57 , 58 , 59 ]. Table 7 provides a mapping of the product concept definition phase within the reviewed documents.

Approaches to product concept definition within the review documents.

4.4 Consideration of influencing factors

According to Koen et al. [ 14 ], influencing factors are variables that impact FEI and are relatively outside the organization’s control. Table 8 maps the influencing factors considered in the review documents.

Influencing factors within the review documents.

5. Discussion

After presenting the review results, it is essential to delve deeper into key findings and considerations identified throughout the study.

5.1 Synthetic diagnosis of the results

Table 9 provides a condensed overview, offering a synthetic diagnosis of the results derived from the review. It succinctly outlines FEI activities in the defense sector and establishes connections with the influencing factors under consideration.

FEI activities in the defense sector.

5.2 Peculiarities of FEI in the defense sector

After systematically mapping the FEI in defense sector against established FEI models, several distinct aspects specific to the military context have emerged, as shown in Table 10 .

Peculiarities of FEI in the defense sector.

5.3 Contributions to the FEI literature

Use of systems engineering approach : The defense sector prominently employs systems engineering activities during the early phase of military innovation. This approach encompasses requirements engineering and systems lifecycle management, aspects not explicitly emphasized in traditional FEI models.

Relevance of technological duality, criticality, and maturity : Concepts such as technological duality, criticality, and maturity play a crucial role in military FEI. These factors, while not extensively covered in established FEI models, are instrumental in decision-making processes, risk mitigation, and the strategic development of defense capabilities.

Organizational capabilities as the “engine” of FEI : In contrast to seminal NPD models that classify organizational capabilities as influencing factors, the defense sector integrates organizational capabilities as a fundamental component of the FEI “engine.” Capability-based planning is a central element in identifying and analyzing opportunities, as well as in generating, enriching, and screening ideas.

“Implementation” of military innovation : The implementation of military innovation necessitates a broader interpretation compared to traditional FEI models. In defense, implementation occurs when a new or improved product is seamlessly integrated into the capability’s portfolio of an Armed Force. This integration involves adjustments in various non-technological aspects, emphasizing the symbiosis between technological and doctrinal advancement.

Continuity in project leadership : Recognizing the high turnover of military leaders and the extended durations of defense projects, the continuity of leadership emerges as a critical consideration. Seminal FEI models do not explicitly address managing leadership continuity in NPD project planning.

Moreover, it is noteworthy that recent contributions in the FEI literature have started to delve deeper into the alignment between organizational strategy and FEI activities. Unlike seminal models that treat this issue generically, recent works, such as the integrative ontologies developed by Pereira et al. [ 20 ] and Castro and Ferreira [ 72 , 73 ], provide management artifacts designed to align organizational strategic vision with FEI activities. Employing the design science paradigm, these artifacts integrate constructs, models, methods, and instantiations, thereby enriching the strategic dimension of FEI literature.

5.4 Contributions to the defense literature

The defense sector, encompassing products ranging from CoPS to mass-produced items, presents a unique challenge due to its diverse complexity and production volume [ 74 ]. While the CoPS research area has an established connection with systems engineering literature, the realm of mass-produced products aligns more closely with the theoretical foundations of the FEI literature. Notably, defense documents predominantly draw from the CoPS approach, sparingly incorporating principles from mass production. However, recognizing that the military context spans both worlds, the integration of these approaches becomes crucial, and mapping established FEI models within the dynamics of the initial phase of the military innovation process serves as a valuable step in achieving this harmonization.

Moreover, the FEI literature, characterized by well-defined seminal models and recent integrative ontologies (as discussed in the previous section), contrasts with the more heterogeneous nature of the systems engineering literature. The latter encompasses diverse authors, countries, organizations, and standardization bodies, each adhering to distinct management models with unique nomenclatures and structures [ 43 , 44 , 45 , 47 , 48 , 54 , 55 , 57 , 58 , 59 ]. In this context, the FEI literature emerges as a unifying force, facilitating the creation of a common representation of knowledge related to the early stage of the military innovation process. This not only streamlines communication among specialists, decision-makers, managers, researchers, entrepreneurs, and other stakeholders in the defense field but also promotes greater efficiency in navigating the diverse landscape of defense innovation.

5.5 Limitations

Selection of seminal FEI models : The identification of seminal FEI models relied on findings from Pereira et al. [ 12 , 20 ] and co-citation analysis of FEI-related works available in the Scopus and WoS databases. Alternative criteria for model selection might yield a different set of seminal documents, potentially influencing the analysis.

Data collection in gray literature : The exploration of gray literature related to FEI in the military sector was constrained by the availability of documents on government websites of defense management agencies. This limitation could result in an incomplete representation of the landscape.

Scope of mapping : The review presented an initial mapping of FEI in the defense sector within seminal models. A more comprehensive and structured mapping could be achieved through the adoption of more robust methodological approaches, such as the design science paradigm [ 75 , 76 ]. This suggests that there is potential for a more in-depth and detailed examination of FEI activities in the defense sector.

Acknowledging these limitations is essential for a nuanced understanding of the scope and implications of the study, guiding future research endeavors in this domain.

6. Conclusion

This study aimed to comprehensively explore the dynamics of FEI in the defense sector through a systematic review encompassing 24 documents from both academic and gray literature. By analyzing seminal FEI models, the research mapped key activities within the defense context, including the identification and analysis of opportunities, generation, enrichment, and screening of ideas, product concept definition, and consideration of influencing factors.

The study’s contributions extend to both FEI and defense literature, introducing original perspectives. Notably, it emphasized the systems engineering approach, national strategic focus, technological duality, technological criticality, technological maturity, organizational capabilities as the “engine”, the unique concept of “implementation” in military innovation, and the importance of continuity in project leadership.

Acknowledging limitations, such as the criteria for selecting seminal FEI models, constraints in accessing gray literature, and the preliminary nature of the mapping, the study calls for future research to employ more robust methodologies, like the design science paradigm [ 75 , 76 ], for an in-depth understanding of the initial phase of the military innovation process.

In conclusion, this research lays a foundation for further exploration and synthesis of knowledge, contributing to the advancement of both FEI theory and its application in the defense sector.

Acknowledgments

This work was supported by the Brazilian Army (Atv PCENA V23-011).

We thank Dr. João José Pinto Ferreira (INESC TEC and Faculty of Engineering, University of Porto) for his expertise and help in writing the manuscript.

• 1. Tidd J, Bessant J. Managing Innovation: Integrating Technological, Market and Organizational Change. 7th ed. Hoboken, USA: Wiley; 2020
• 2. Keeley L, Walters H, Pikkel R, et al. Ten Types of Innovation: The Discipline of Building Breakthroughs. 1st ed. Hoboken, USA: Wiley; 2013
• 3. Boeddrich H-J. Ideas in the workplace: A new approach towards organizing the fuzzy front end of the innovation process. Creativity and Innovation Management. 2004; 13 :274-285
• 4. Koen P, Bertels H, Kleinschmidt E. Managing the front end of innovation - Part I: Results from a three-year study. Research-Technology Management. 2014; 57 :34-43
• 5. Koen P, Bertels H, Kleinschmidt E. Managing the front end of innovation - Part II: Results from a three-year study. Research-Technology Management. 2014; 57 :25-35
• 6. Markham SK. The impact of front-end innovation activities on product performance. Journal of Product Innovation Management. 2013; 30 :77-92
• 7. Stevens GA, Burley J. Piloting the rocket of radical innovation. IEEE Engineering Management Review. 2004; 32 :16-25. DOI: 10.1109/EMR.2004.25114 [Epub ahead of print]
• 8. Verworn B, Herstatt C, Nagahira A. The fuzzy front end of Japanese new product development projects: Impact on success and differences between incremental and radical projects. R&D Management. 2008; 38 :1-19
• 9. Williams MA, Kochhar AK, Tennant C. An object-oriented reference model of the fuzzy front end of the new product introduction process. International Journal of Advanced Manufacturing Technology. 2007; 34 :826-841
• 10. Khurana A, Rosenthal SR. Integrating the Fuzzy Front End of New Product Development. Cambridge, USA: MIT Sloan Management Review; 1997
• 11. Khurana A, Rosenthal SR. Towards holistic ‘front ends’ in new product development. Journal of Product Innovation Management. 1998; 15 :57-74
• 12. Pereira AR, Ferreira JJP, Lopes A. Front end of innovation: An integrative literature review. Journal of Innovation Management. 2017; 5 :22-39
• 13. Smith PG, Reinertsen DG. Developing Products in Half The Time. New York: Van Nostrand Reinhold; 1991
• 14. Koen P, Ajamian G, Boyce S, et al. Fuzzy front end: Effective methods, tools, and techniques. Industrial Research. 2002. pp. 5-35
• 15. Koen P, Ajamian G, Burkart R, et al. Providing clarity and a common language to the ‘fuzzy front end’. Research Management. 2001; 44 :46-55
• 16. Kock A, Heising W, Gemünden HG. How ideation portfolio management influences front-end success. Journal of Product Innovation Management. 2015; 32 :539-555
• 17. Cooper RG. Perspective: The stage-gates idea-to-launch process-update, what’s new, and NexGen systems. Journal of Product Innovation Management. 2008; 25 :213-232
• 18. Cooper RG. Stage-gate systems: A new tool for managing new products. Business Horizons. 1990; 33 :44-54
• 19. Reid SE, De Brentani U. The fuzzy front end of new product development for discontinuous innovations: A theoretical model. Journal of Product Innovation Management. 2004; 21 :170-184
• 20. Pereira AR, Ferreira JJP, Lopes A. A knowledge representation of the beginning of the innovation process: The front end of innovation integrative ontology (FEI2O). Data & Knowledge Engineering. 2020; 125 :1-20
• 21. Schein EH. Organizational Culture and Leadership. 3rd ed. San Francisco, CA: Jossey-Bass; 2004
• 22. PMI. The Standard for Portfolio Management. 2017. Available from: https://www.pmi.org/pmbok-guide-standards/foundational/standard-for-portfolio-management [Accessed: March 5, 2024]
• 23. Teece DJ, Pisano G, Shuen A. Dynamic capabilities and strategic management. Strategic Management Journal. 1997; 18 :509-533. DOI: 10.1002/(SICI)1097-0266(199708)18:7<509::AID-SMJ882>3.0.CO;2-Z [Epub ahead of print]
• 24. Brasil. Política Nacional de Defesa e Estratégia Nacional de Defesa. 2016
• 25. Bitzinger RA. The Modern Defense Industry: Political, Economic, and Technological Issues. Westport, USA: Praeger Publishers; 2009
• 26. Hobday M. Product complexity, innovation and industrial organisation. Research Policy. 1998; 26 :689-710
• 27. Amarante JCA d. Processos de obtenção de tecnologia militar. In: Texto para discussão. Rio de Janeiro, Brasil: Instituto de Pesquisa Econômica Aplicada (Ipea); 2013. p. 1877
• 28. Brustolin VM. Inovação e Desenvolvimento via Defesa Nacional nos EUA e no Brasil. Rio de Janeiro, Brasil: Universidade Federal do Rio de Janeiro e Universidade de Harvard; 2014
• 29. Urbano EP. A contribuição dos offsets em defesa para a inovação e transferência de tecnologia para a base industrial de defesa. Brasília, Brasil: Universidade de Brasília; 2019
• 30. Araujo BC, De Negri F, De Negri JA, et al. Base industrial de defesa. In: de Negri JA, Lemos MB, editors. O Núcleo Tecnológico da Indústria Brasileira. Brasília, DF: Ipea, FINEP e ABDI; 2011. pp. 595-653.
• 31. Galdino JF, Schons DL. Maquiavel e a importância do poder militar nacional. Coleção Meira Mattos: revista das ciências militares. 2022; 16 :369-384
• 32. Ethier WJ. Dumping. Journal of Political Economy. 1982; 90 :487-506
• 33. Anderton CH. Economics of arms trade. In: Handbook of Defense Economics. Amsterdam, Netherlands: Elsevier; 1995. pp. 523-561
• 34. da Silva CD. Planejamento Baseado em Capacidades e suas perspectivas para o Exército Brasileiro. Centro de Estudos Estratégicos do Exército - Artigos Estratégicos. 2019; 7 :21-29
• 35. Thomé AMT, Scavarda LF, Scavarda AJ. Conducting systematic literature review in operations management. Production Planning and Control. 2016; 27 :408-420
• 36. Ferreira JJP, Mention AL, Torkkeli M. Phrasing the giant: On the importance of rigour in literature search process. Journal of Innovation Management. 2020; 8 :1-10. DOI: 10.24840/2183-0606_008.002_0001 [Epub ahead of print]
• 37. United States. Defense Space Strategy Summary. 2020.
• 38. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. The BMJ. 2021; 372 :1-9. DOI: 10.1136/bmj.n71 [Epub ahead of print]
• 39. Najgebauer A, Antkiewicz R, Chmielewski M, et al. The qualitative and quantitative support method for capability based planning. In: Intelligent Information and Database Systems. Bali, Indonesia: Springer; 2015. pp. 212-223
• 40. Tagarev T. Capabilities-based planning for security sector transformation. Information & Security An International Journal. 2009; 24 :27-35
• 41. United States. Analytic Architecture for Capabilities-Based Planning, Mission-System Analysis, and Transformation. 2002
• 42. United States. 5123.01H-Charter of the Joint Requirements Oversight Council (JROC) and Implementation of the Joint Capabilities Integration and Development System (JCIDS). 2018
• 43. NATO. Joint Analysis Handbook. Lisbon, Portugal: NATO; 2016
• 44. United Kingdom. Knowledge in Defence (KiD). 2019
• 45. Australia. Defence Capability Manual. 2022
• 46. Barton J. China’s PLA Modernization through the DOTMLPF-P Lens. TRADOC Mad Scientist Laboratory; 2021. Available from: https://madsciblog.tradoc.army.mil/330-chinas-pla-modernization-through-the-dotmlpf-p-lens/
• 47. India. Defence Acquisition Procedure 2020. 2020
• 48. South Africa. South African Defence Review. 2015
• 49. Brasil. EB20-MF-10.102 - Manual de Fundamentos da Doutrina Militar Terrestre. 2019
• 50. Brasil. EB20-C-07.001 - Catálogo de Capacidades do Exército. 2014
• 51. Helfat CE, Peteraf MA. The dynamic resource-based view: Capability lifecycles. Strategic Management Journal. 2003; 24 :997-1010
• 52. Salvato C, Rerup C. Beyond collective entities: Multilevel research on organizational routines and capabilities. Journal of Management. 2010; 37 :468-490
• 53. Wallin J, Parida V, Isaksson O. Understanding product-service system innovation capabilities development for manufacturing companies. Journal of Manufacturing Technology Management. 2015; 26 :763-787. DOI: 10.1108/JMTM-05-2013-0055 [Epub ahead of print]
• 54. United States. 5000.85 - Major Capability Acquisition. 2020
• 55. Brasil. EB10-IG-01.018 - Instruções Gerais para a Gestão do Ciclo de Vida dos Sistemas e Materiais de Emprego Militar. 2022
• 56. Faulconbridge R, Ryan M. Introduction to Systems Engineering. Canberra: Argos Press; 2015
• 57. Blanchard B, Fabrycky W. Systems Engineering and Analysis. 5th ed. Upper Saddle River, NJ: Prentice Hall; 2011
• 58. International Council on Systems Engineering (INCOSE). INCOSE Systems Engineering Handbook. San Diego, USA: INCOSE; 2023
• 59. International Organization for Standardization (ISO). ISO/IEC/IEEE 15288 - Systems and Software Engineering - System Life Cycle Processes. Geneva, Switzerland: ISO; 2023
• 60. Innovations. Competing for defence ideas: Looking wider for innovation. Strategic Direction. 2008; 24 :35-37. DOI: 10.1108/02580540810839359 [Epub ahead of print]
• 61. Clegg B, Alexander I, Wingrove S, et al. Tool support for integrating extended enterprises. IEE Proceedings - Software. 2000; 147 :101-108. DOI: 10.1049/ip-sen:20000907 [Epub ahead of print]
• 62. Larsson A, Larsson T, Bylund N, et al. Rethinking virtual teams for streamlined development. In: Torres-Coronas T, Macgregor SP, editors. Higher Creativity for Virtual Teams - Developing Platforms for Co-Creation. Hershey, USA: Information Science Reference; 2007. DOI: 10.4018/978-1-59904-129-2.ch007 [Epub ahead of print]
• 63. Johansson C, Hicks B, Larsson AC, et al. Knowledge maturity as a means to support decision making during product-service systems development projects in the aerospace sector. Project Management Journal. 2011; 42 :32-50
• 64. United States. Critical and Emerging Technologies List Update. 2022
• 65. United States. Technology Readiness Assessment (TRA) Deskbook. 2009
• 66. United Kingdom. The Defence Capability Framework. 2022
• 67. Australia. Blueprint for Critical Technologies. 2021
• 68. Australia. The Action Plan for Critical Technologies. 2021
• 69. IEDI. O 14 o Plano Quinquenal Chinês: Transformando a China em potência industrial e tecnológica. Carta do Instituto de Estudos para o Desenvolvimento Industrial-Edição 1094; 2021. Available from: https://iedi.org.br/cartas/carta_iedi_n_1094.html [Accessed: July 2, 2023]
• 70. Brasil. Plano Estratégico do Exército 2020-2023. 2020
• 71. Organização para a Cooperação e o Desenvolvimento Econômico (OCDE). Manual de Oslo: Diretrizes para a Coleta e Interpretação de dados sobre Inovação Tecnológica. Paris: OCDE; 2005
• 72. Castro RN, Ferreira JJP. Project portfolio management in the front-end of innovation of research centers: A literature review. Technology Innovation Management Review. 2020; 10 :46-59
• 73. Castro RN, Ferreira JJP. The front-end of R&D at non-profit research centers: How does research produce impact? International Journal of Innovation and Technology Management. 2023; 20 :1-22
• 74. Girardi R, França Junior JA, Galdino JF. A customização de processos de avaliação de prontidão tecnológica baseados na escala TRL: Desenvolvimento de uma metodologia para o Exército Brasileiro. Coleção Meira Mattos: Revista das ciências militares. 2022; 16 :491-527. DOI: 10.52781/cmm.a084
• 75. Hevner AR, March ST, Park J, et al. Design science in information systems research. MIS Quarterly: Management Information Systems. 2004; 28 :75-105
• 76. Vaishnavi V, Kuechler B, Stacey P. Design Science Research in Information Systems. desrist.org - Design science research in information systems and technology; 2021. Available from: http://www.desrist.org/design-research-in-information-systems [Accessed: September 28, 2023]

© The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution 3.0 License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Continue reading from the same book

Edited by Sally Burt

Published: 25 September 2024

By Yuval Karniel and Amit Lavie-Dinur

88 downloads

By Haki Demolli

26 downloads

By Ulpia-Elena Botezatu and Adrian-Victor Vevera

40 downloads

IntechOpen Author/Editor? To get your discount, log in .

Discounts available on purchase of multiple copies. View rates

Local taxes (VAT) are calculated in later steps, if applicable.

Support: [email protected]

• Systematic Review
• Open access
• Published: 27 September 2024

Using best-worst scaling to inform policy decisions in Africa: a literature review

• Laura K. Beres 1 ,
• Nicola B. Campoamor 2 ,
• Rachael Hawthorn 3 ,
• Melissa L. Mugambi 4 ,
• Musunge Mulabe 5 ,
• Natlie Vhlakis 5 ,
• Michael Kabongo 5 ,
• Anne Schuster 2 &
• John F. P. Bridges 2  

BMC Public Health volume  24 , Article number:  2607 ( 2024 ) Cite this article

Metrics details

Stakeholder engagement in policy decision-making is critical to inform required trade-offs, especially in low-and-middle income settings, such as many African countries. Discrete-choice experiments are now commonly used to engage stakeholders in policy decisions, but other methods such as best-worst scaling (BWS), a theory-driven prioritization technique, could be equally important. We sought to document and explore applications of BWS to assess stakeholder priorities in the African context to bring attention to BWS as a method and to assess how and why it is being used to inform policy.

We conducted a literature review of published applications of BWS for prioritization in Africa.

Our study identified 35 studies, with the majority published in the past four years. BWS has most commonly been used in agriculture (43%) and health (34%), although its broad applicability is demonstrated through use in fields influencing social and economic determinants of health, including business, environment, and transportation. Published studies from eastern, western, southern, and northern Africa include a broad range of sample sizes, design choices, and analytical approaches. Most studies are of high quality and high policy relevance. Several studies cited benefits of using BWS, with many of those citing potential limitations rather than observed limitations in their study.

Conclusions

Growing use of the method across the African continent demonstrates its feasibility and utility, recommending it for consideration among researchers, program implementers, policy makers, and funders when conducting preference research to influence policy and improve health systems.

Registration

The review was registered on PROSPERO (CRD42020209745).

Peer Review reports

Introduction

Health policies govern both health systems hardware (e.g., human resources, finance, medicines and technologies) and software (e.g., values, norms, power dynamics) by constraining or facilitating individual, organizational, and community actions and experiences [ 1 , 2 , 3 ]. Additionally, healthcare workers take numerous discretionary decisions each day to translate policy into practice and to fill gaps between policy guidance and implementation realities [ 4 , 5 , 6 ]. When guided by evidence, health policies and clinical decision-making facilitate optimized health practices and outcomes [ 6 , 7 , 8 ]. However, policy and practice decisions are often made in an evidence void due to a lack of data or poor evidence access and translation, resulting in inefficient, ineffective, or harmful health system outcomes [ 9 , 10 , 11 ].

Evidence about the preferences of those affected by health decision-making is particularly limited, but greatly needed for policy and practice. Internationally recognized processes for developing health guidelines and recommendations include incorporating the values and preferences of affected parties, such as patients and healthcare workers, into decision-making [ 12 , 13 ]. Limited availability of preference-based evidence downgrades the strength of recommendations [ 13 , 14 ]. Additionally, the welcomed and growing call for person-centered healthcare explicitly requires integration of patient preferences and perspectives into health practice [ 15 , 16 ]. Across all settings, policy makers must trade off services implemented with available resources. Required trade-offs are often more common and more challenging in more resource-limited settings, such as low-and-middle-income countries. Expanded use of tools to understand and systematically incorporate evidence on the preferences of patients, healthcare workers, and other stakeholders into policies and practices is needed to improve health systems at every level [ 17 , 18 , 19 , 20 ].

Researchers and practitioners use multiple methods to understand preferences and priorities, including deliberative processes such as testimony or community meetings, qualitative processes such as focus group discussions and interviews, and mixed methods approaches such as human-centered participatory design processes, surveys, Likert scales, and community scorecards. Developing an even broader methodological toolkit allows for more influential data to facilitate policy and practice changes, as teams will be equipped to optimize the methods selected for the target audience and research question. Stated preference methods offer a theory-driven, structured approach to understanding preferences and priorities. They produce interpretable outcomes with clear relevance to the questions of interest. They have been used across various industries, including healthcare, transportation services, and grocery retailing demonstrating their versatility and potential for suitability. However, while a range of preference elicitation methods exist [ 21 , 22 , 23 ], discrete choice experiments (DCEs) predominate in published health literature [ 24 ]. Recent studies internationally have shown the potential utility and appropriateness of other, lesser known but valuable quantitative stated preference methods, such as best-worst scaling (BWS) [ 25 ]. Studies from eastern, western, southern, and northern Africa have demonstrated interesting advances [ 26 , 27 , 28 ], but have received less attention than BWS in other regions.

The goal of this study was to document and explore applications of BWS to assess stakeholder priorities in eastern, western, southern, and northern Africa to inform future preference assessment implementation. Such documentation of current BWS in the African setting is an important step in bringing attention to this increasingly important method and to stress that there are other theory-driven alternatives to DCEs – that are now commonly applied in Africa [ 24 ]. The review presents study design, methods, quality, and policy relevance from extant studies to enable preference researchers to consider the appropriateness of similar BWS applications in their work. While several international reviews have been conducted on BWS in health [ 25 , 29 ] and more generally [ 30 ], it is important to document how this method is being used in the African context, and what specific role it might have in informing policy there. Furthermore, there has been increased use of these methods in Africa since these previous reviews. It is important that contributions of African preference researchers are well-document to ensure their inclusion in international efforts around preference methods and the presented strengths and weaknesses of these methods are well understood [ 31 ].

Best-worst scaling (BWS) is a choice experiment that is aimed to assess how individiuals or groups prioritize concepts. It offers a theory-driven alternative to descriptive rating, ranking or Likert scale preference measurement, leveraging relative participant ease of selecting extremes compared to mid-range rankings. Several types of BWS exist; however, they all share the same underlying concept. BWS relies on the concept of individuals choosing the ‘best’ and ‘worst’ (or ‘most’ and ‘least’ important) items from a given sub-set of three or more items. Sub-sets of the items are shown repeatedly in different combinations requiring choices of ‘best’ and ‘worst’ within each sub-set. This results in a prioritization of the items. Even if all options are preferred, participants are forced to prioritize among the choices. The purpose is to determine the most and least preferred options of items existing on a subjective, latent value continuum. The application of BWS for prioritizing objects has also been referred to as MaxDiff, object scaling, BWS case 1, or BWS object case. Best-worst responses can also be used in other choice formats that are more similar to DCEs, which are not the focus of this paper [ 32 ].

BWS draws on random utility theory to identify perceived importance and priorities among a set of items (known as attributes) of a scenario based on repeat choices. It can estimate the likelihood of preference selection and heterogeneity of preferences between groups. BWS can be used with a relatively small sample size and analyzed with a range of methods including more simple count analyses or more complex probablistic models. The range of analysis approaches makes them particularly useful when working across a broad range of stakeholders, including policy makers, who would want and need to understand how conclusions were drawn. Compared to DCEs, where participants select which of two or more presented profiles (specific item combinations) are preferred, BWS offers more information per choice task (i.e., best and worst choices instead of only best), allowing for a more efficient design with either a smaller sample size or more information per task. BWS may have a lower cognitive burden for participants than DCEs [ 33 , 34 ]. We refer the reader to additional resources for more detail on the theory, methods, and application of BWS [ 35 , 36 ].

Our review of BWS for prioritization in published research from southern, eastern, western, and northern Africa drew from a broader database of BWS studies identified in previous reviews. While an earlier literature review on all types of BWS choice formats had previously been published [ 29 ], our team completed the first systematic review of BWS applications in health published prior to 2022 [ 25 ]. We then extended our review (PROSPERO: CRD42020209745) to include publications from any field (e.g., health, business, agriculture, etc.) published prior to 2023 [ 30 ]. We have continued to improve this database of articles, utilizing additional search strategies and including previously unidentified relevant articles directly sent to our team. Our database currently has 623 published studies from which we systematically extract data application, development, design, administration/analysis, quality, and policy relevance. The study reported in this paper leverages the most expanded database. Review methods were detailed in prior publications [ 25 ].

We included all studies from the database that focused exclusively on, or incorporated into a multi-country study, participants from eastern, western, southern, or northern Africa. We then accessed extracted data in the database for each study to characterize BWS application types, study methods, context, quality, and policy relevance. Specific fields included: study year, country, topic, objective, sample size, perspective (i.e., whose preferences are measured), terminology used to describe BWS type (e.g., object case, MaxDiff), mode of survey administration (e.g., in-person), time frame of prioritization scenario (i.e., past, present, or future), methods of instrument development (e.g., formative research, literature review to determine attributes and survey tool), time frame of prioritization scenario (i.e., past, present, or future), measurement scale, experimental design type, BWS anchor description (i.e., most/least, best/worst), directionality, total number of objects, number of objects per task, number of tasks in the experiment, number of tasks per respondent, analysis approach, and theoretical assumptions [ 25 ]. We re-classified database ‘topic’ for three studies from ‘agriculture’ to ‘business’ after reviewing study journal and conclusions. To understand study quality, we utilized the PREFS checklist quality assessment which measures quality and validity of preference studies on a 0–5 point scale, assigning one point for each of the following: p urpose of the study clearly defined; r espondents similar to non-respondents (sampling); e xplanation of preference assessment methods clear; f indings reported for all respondents; and significance testing done [ 37 ]. We also present the validated subjective quality (range: 1–10) and policy relevance (range: 1–10) scores adjudicated by the prior review.

Identified strengths and weaknesses of BWS were extracted from the studies. This information primarily came from the background, methods, and discussion portions of the studies, specifically when justifying the use of BWS and highlighting any study limitations. Seven domains of strengths and weaknesses were dervied and modified from exisiting best-practice documents for preference research [ 38 , 39 ].

To highlight the application of BWS and improve understanding of the method, we include a narrative case study description of two of the included manuscripts. We chose health-related studies selected for their high quality (≥ 4 PREFS score) and policy relevance (≥ 7) with illustrative diversity across other factors including country, perspective, design, instrument development, and analysis approach. While diversity in other points such as assumptions, directionality, and heterogeneity analysis are interesting, (See Tables  1 , 2 and 3 ) the selected articles offered rich contrast among the articles in this review.

The review identified 35 published studies using best-worst scaling for prioritization focused on (in full or in part) participants from Africa. As seen in Table  1 , studies originated from northern Africa ( N  = 1), eastern Africa ( N  = 7), southern Africa ( N  = 13), and western Africa ( N  = 14). This included 7 studies not identified by previous reviews. The majority of the papers (72%) from Africa were published between 2019 and 2023 [ 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 ] and referred to ‘Best Worst Scaling’ in their write-up (85%) [ 40 , 41 , 43 , 44 , 45 , 46 , 47 , 49 , 51 , 52 , 54 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 ]. Of the 14 countries where research was conducted, South Africa produced the most studies (34%) [ 41 , 42 , 46 , 51 , 52 , 55 , 56 , 60 , 61 , 66 , 67 , 70 , 74 ]. Most papers presented results from empirical research (97%) [ 40 , 41 , 42 , 44 , 45 , 46 , 47 , 48 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 70 , 71 , 72 , 73 , 74 ]. Agriculture (43%) [ 41 , 43 , 44 , 45 , 47 , 49 , 50 , 51 , 54 , 59 , 60 , 63 , 64 , 65 , 69 , 70 , 71 ] and health (34%) [ 40 , 46 , 48 , 52 , 55 , 56 , 57 , 62 , 67 , 72 , 73 , 74 ] were the most common research topics. Most studies (51%) measured preferences from the perspective of the patient / consumer [ 41 , 42 , 46 , 49 , 51 , 54 , 55 , 57 , 60 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 72 ], with nearly a third measuring provider / producer preferences (37%) [ 40 , 43 , 44 , 45 , 47 , 48 , 50 , 56 , 58 , 59 , 62 , 71 , 74 ].

Study design

All identified studies articulated their approach to developing their BWS instrument. The vast majority, (74%), utilized a literature review [ 41 , 42 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 52 , 53 , 54 , 55 , 56 , 57 , 60 , 61 , 64 , 65 , 66 , 67 , 68 , 69 , 71 , 72 , 74 ] while less than a quarter conducted piloting (17%) [ 42 , 46 , 50 , 53 , 56 , 73 ] or pretesting (20%) [ 47 , 50 , 52 , 54 , 56 , 66 , 71 ] of the instrument prior to administration. Half of the studies reported utilizing formal qualitative methods during instrument development [ 40 , 42 , 43 , 46 , 48 , 50 , 53 , 56 , 57 , 60 , 62 , 67 , 68 , 69 , 73 , 74 ]. In-person, surveyor-administered was the most common mode of survey administration (66%) [ 40 , 43 , 44 , 45 , 46 , 47 , 48 , 50 , 52 , 53 , 54 , 55 , 58 , 59 , 61 , 62 , 64 , 65 , 66 , 67 , 68 , 69 , 72 ] with online (9%) [ 51 , 56 , 74 ] and self-administered (11%) [ 41 , 57 , 70 , 73 ] less frequently utilized. The time horizon used to contextualize the survey was present tense most frequently (89%) [ 40 , 41 , 42 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 74 ] with only four studies asking about the future (11%) [ 43 , 54 , 55 , 73 ] and no studies asking about the past. BWS most frequently measured importance (69%) [ 41 , 45 , 46 , 49 , 51 , 52 , 53 , 55 , 56 , 57 , 59 , 60 , 61 , 63 , 64 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 ], followed by preferences motivating responses (17%) [ 40 , 43 , 44 , 48 , 54 , 65 ]. The most common phrasing to anchor the experiment was asking participants to choose the “most” and “least” [important/preferred/concerning] (86%) [ 40 , 41 , 42 , 44 , 45 , 46 , 47 , 48 , 49 , 51 , 52 , 54 , 55 , 56 , 57 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 68 , 69 , 70 , 71 , 72 , 73 , 74 ], followed by asking participants to choose the “best” and “worst” (11%) [ 43 , 53 , 58 , 67 ]. The most common experimental design used was a Balanced Incomplete Block Design (BIBD) (69%) [ 40 , 41 , 42 , 43 , 44 , 45 , 46 , 48 , 49 , 50 , 51 , 53 , 54 , 56 , 58 , 59 , 60 , 64 , 65 , 67 , 70 , 71 , 72 , 73 ] with 14% using a design from Sawtooth software [ 47 , 57 , 66 , 68 , 69 ]. The mean total objects per experiment was 15.9 (standard deviation (sd): 9.1, min 6: max: 48). Experiments had a mean of 5.2 objects per task (sd: 3.5, min: 3 max: 24), 22.4 (sd: 38.4, min: 1 max: 210) choice tasks per experiment and a mean of 13.7 (sd: 8.2, min: 1 max: 51) choice tasks per respondent during their participation in the experiment (Table  2 ).

BWS administration and analysis

Median sample size was 282 participants (IQR: 150–451, min: 28, max: 1002) but only 17% of papers gave a formal sample size justification (Table  3 ) [ 40 , 47 , 48 , 57 , 62 , 66 ]. Stata was the most reported data analysis program utilized (14%) [ 46 , 50 , 53 , 67 , 73 ], followed by Excel [ 42 , 61 , 66 ], SAS [ 43 , 48 , 62 ], or SPSS (9% each) [ 47 , 66 , 69 ]. The remaining studies (60%) did not specify which statistical analysis program they used. Probability / ratio rescaling (43%) [ 40 , 42 , 43 , 45 , 48 , 49 , 51 , 54 , 59 , 60 , 64 , 65 , 67 , 68 , 71 ], counts (49%) [ 40 , 41 , 45 , 46 , 50 , 52 , 53 , 54 , 56 , 58 , 61 , 63 , 66 , 67 , 70 , 71 , 73 ], regression coefficients (43%) [ 43 , 44 , 48 , 49 , 52 , 53 , 55 , 63 , 64 , 65 , 68 , 69 , 71 , 72 , 73 ], and best-worst scores (46%) [ 41 , 42 , 45 , 46 , 47 , 52 , 54 , 58 , 59 , 60 , 61 , 64 , 66 , 70 , 71 , 73 ] were common analysis approaches. Approximately half of the studies (40%) effects coded their data [ 40 , 42 , 49 , 50 , 52 , 55 , 64 , 65 , 69 , 71 , 72 ] while fewer (17%) used an omitted variable [ 43 , 44 , 48 , 53 , 54 , 62 ]. Heterogeneity analyses were conducted by most studies (63%) [ 42 , 43 , 44 , 45 , 46 , 49 , 50 , 51 , 52 , 54 , 55 , 57 , 58 , 59 , 64 , 66 , 67 , 68 , 69 , 72 ], with stratification being the most common heterogeneity analysis approach (40%) [ 42 , 44 , 49 , 51 , 52 , 55 , 57 , 63 , 64 , 65 , 66 , 67 , 68 , 72 ] followed next by latent class analysis (20%) [ 45 , 46 , 50 , 54 , 58 , 59 , 69 ].

Study quality and policy relevance

Policy relevance of studies was high with 66% scoring 7 or above on the 10-point scale [ 40 , 41 , 42 , 44 , 46 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 57 , 59 , 64 , 65 , 67 , 68 , 70 , 71 , 72 , 73 ]. Most studies scored in the upper half of the PREFS scale [ 3 , 4 , 5 , 40 , 42 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 57 , 58 , 59 , 60 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 ].

Strengths and weaknesses of BWS

Strengths and weaknesses of using BWS were identified in most of the published studies from Africa. Most of these studies focused solely on strengths of using BWS, though some identified both strengths and weaknesses and a few focused solely on weaknesses. These strengths and weaknesses were categorized into seven domains: context, purpose, method, burden, results, comparisons, and bias (Table  4 ).

Strengths were noted across all seven domains. In the context domain, BWS was noted for eliciting priorities and preferences from populations with lower education levels [ 52 , 72 ] and lower income [ 72 , 73 ]. The purpose domain highlighted strengths such as engaging communities [ 42 ] and informing decision-making [ 55 ]. The method domain emphasized that BWS overcomes limitations of other ranking methods [ 41 , 42 , 59 , 61 , 71 ], including variations in interpretation related to Likert-type scales. The burden domain commonly cited that BWS reduces respondent burden [ 41 , 45 , 49 , 50 , 52 , 55 , 59 , 61 , 66 , 68 , 72 ]. The results domain stressed that BWS captures more information [ 49 , 59 , 64 , 66 , 68 , 72 ] and produces higher quality and more precise results than other methods [ 41 , 45 , 49 , 58 , 71 ]. The comparisons domain focused on the ability to discriminate between objects [ 45 , 49 , 59 , 65 , 68 , 73 ]. The bias domain noted a reduction of general bias [ 41 , 51 , 53 , 59 , 66 , 70 ].

Weaknesses or limitations of using BWS were provided for only five of the seven domains. In the context domain, it was suggested that BWS might be challenging to use in clinical practice as a decision-making tool [ 57 ] and, contrary to studies noting it as a strength, some identified it as, challenging for populations with lower education levels [ 52 ]. The method domain pointed out that BWS involves hypothetical scenarios that may not be realistic [ 64 ]. The burden domain cited possible respondent burden associated with completing a series of BWS tasks [ 52 , 59 , 66 ]. The comparisons domain highlighted potential difficulties in making comparisons between populations within the sample [ 68 , 69 ]. The bias domain noted the possibility of desirability bias, where respondents report socially acceptable factors rather than genuine preferences [ 72 ]. Most of these weaknesses were posed as possibilities, rather than observed limitations.

Case studies

Policy relevance : Ozawa et al. [ 72 ] used BWS scaling to inform message development and effective delivery strategies with the goal of improving childhood vaccination awareness and demand in Nahuche, Zamfara State northern Nigeria, a region with low vaccination uptake. Instrument development : The survey items were developed through a review of published literature from Nigeria and other low-and-middle income countries on factors that affect childhood vaccine demand and uptake. Identified factors were categorized into four groups and each written out as a negative or positive influence based on the literature (e.g., vaccines may harm a child (negative), trust the views of leaders about vaccines (positive)), balancing equal numbers of positive and negative statements. A local study advisory board reviewed the items. Population : The survey was administered in-person to parents with children under 5 years old during a household survey from a representative sample of households. Administration : The survey was translated into Hausa and presented as a pictorial questionnaire with both photographs and words used to represent each factor. Perspective and time scale : Participants were asked to select the most and least important factors to them (consumer/patient) when deciding whether to vaccinate a 1-year-old child (present). Design : The study utilized a BIBD where every participant was presented with 16 choice sets of 6 factors. The survey took approximately 1 h to complete. Analysis : They assumed sequential BWS and used conditional logistic regression with effects coding to determine factor rankings. Heterogeneity : They examined heterogeneity in the results by looking at different strata: male/female parent and by previous diphtheria, tetanus, and pertussis (DTaP) vaccination status (yes/no). Participation : 198 parents participated. Results : The most important motivating factor for vaccinating children was the perception that vaccination makes one a good parent. Trust and norms were found to be more important than benefits and risks in vaccination decisions. They identified differences in rankings between fathers and mothers and in families with and without prior DTaP vaccination.

Policy relevance : Yemeke et al. [ 52 ] compared the Uganda national budget resource allocations across 16 sectors to citizen preferences for such allocations. A particular emphasis was placed on understanding the health care sector as a funding priority. Instrument development : The sixteen survey factors represented each of the sixteen sector allocations within the Uganda national government’s budget. Results from a pre-test of the survey to assess respondent understanding were used to improve the instrument. Population : The survey was administered in-person to the head of household or spouse, of at least 18 years of age, in both rural and urban areas in the Mukono district in central Uganda. Administration : The survey was translated into Luganda and displayed accompanying pictorial representations (such as photographs or graphics) of the sectors. Descriptions of the sectors and their functions were also read aloud. Perspective and time scale : Respondents were asked to select the most and least important sectors for resource allocation for their community (present) in each choice task, offering societal perspective prioritization. Design : A main effects orthogonal design was used to generate 16 choice tasks, each with 4 sectors (factors). Analysis : Count analysis: Relative mean best-worse scores were calculated for each sector. Scores were transformed to a positive scale, anchored at zero, to calculate percentage preference relative to estimated cumulative sums. The preferred percentages were compared to the actual percentages allocated in the national budget. Assuming sequential BWS, the authors used McFadden’s conditional logistic regression with effects coding to regress a single dichotomous choice variable on all sectors to assess ranking of preferred sectors. Heterogeneity : They examined heterogeneity in the results across two strata (urban/rural). Participation : There were 432 respondents across two settings (217 urban respondents, 215 rural respondents). Results : The health sector was the highest ranked sector by a significant margin amongst both rural and urban respondents. This result was consistent in both the relative best-minus-worst score method and the regression analysis. This highlighted a clear disparity between citizen preferences and national budget al.location, where the health sector was ranked sixth.

Policy relevance : Nyarko et al. [ 48 ] used BWS to quantify the antimicrobial dispensing practices of medicine sales outlet staff. Understanding these practices can help to improve patient safety and care quality, as well as to serve as a guide for decision-making in the pharmaceutical sector. Instrument development : The initial list of survey items was identified through informant interviews with medicine sales experts and an extensive literature review. The list of items was condensed into eight objects through focus group discussions with medicine sales outlet staff. Population : The survey was conducted in-person through interviewer-questionnaire administration with medicine sales outlet staff over a two-month period. Staff were eligible for the study if they had dispensed microbials within the past year. Administration : Demographic information was collected at the start of the questionnaire, followed by questions regarding the staff’s prescribing and dispensing practices of antimicrobial medications. Perspective and time scale : Participants were asked to indicate which object regarding antimicrobial dispensing practices concerned them the most and least. Design : A BIBD was used, generating 8 tasks, each with 7 objects. Analysis : Assuming the respondent chose the items they most liked and disliked, a maximum difference model with effects coding was used to determine parameter estimates. The relative importance of each object was determined based on the parameter estimates, allowing the objects to be ranked by level of importance. Heterogeneity : Heterogeneity was examined by comparing antimicrobial dispensing practices with their associated objects. Participation : 200 staff participated. Results : The antimicrobial dispensing practice that concerned respondents most was the need to follow the drug act and avoid dispensing antimicrobials without a prescription. Dispensing antibiotics to poor patients who may not be able to afford medical bills was not a concern for respondents. Overall, the study suggests that staff are careful when dispensing antimicrobials.

Our study identified 35 studies from across Africa, with the majority published in the past four years. BWS has most commonly been used in agriculture and health, although its broad applicability is demonstrated through its use in fields including business, environment, and transportation. Published studies from eastern, western, southern, and northern Africa include a broad range of sample sizes, design choices, and analytical approaches. Consistent with other BWS reviews [ 25 ], the majority of studies are both of high quality and of high policy relevance. It is interesting to highlight that among articles classified as ‘multi-country,’ two articles included participants from at least one African country in their sample. However, we considered them ‘near misses’ and excluded them as neither disaggregated data by country to ensure review data came from Africa. Both had few participants from African countries relative to the overall sample.

The application of BWS for prioritization in the Africa context is an emerging practice as demonstrated by our findings that its use has increased dramatically over time. The quality of studies, as measured by PREFS, has remained consistently high over time [ 24 ]. As the method continues to be applied, guidelines exist that could further ensure researchers conduct high-quality studies and publish high-quality papers about them [ 38 , 39 , 75 ]. This includes the increased use of instrument development methods, especially formal qualitative work, pretesting with cognitive interviewing, and piloting [ 76 ]. Importantly, this also requires attentiveness to ensuring accuracy of conceptual translation which is often missed if focusing exclusively on linguistic translation for studies working across multiple language groups [ 77 ].

The use of BWS also draws attention to the notion of prioritization itself. Our findings highlight the relevance of the method to policy making; over three-quarters of the included studies received a policy relevance score of seven or more. Clearly expressed priorities may allow policy makers to shift away from informal decision-making heuristics to more formal, principled decision making practices. Certainly, the more recently observed integration of BWS into deliberative processes [ 78 ] such as the modified policy Delphi [ 79 ] and deliberative democracy [ 80 ] exemplifies other ways that priority and preference elicitation can help inform group deliberation to achieve consensus [ 81 , 82 ]. That said, there remain questions about aggregating individual priorities in group decision making [ 36 ] and is a topic that others have grappled with, including in health state valuation [ 83 ]. Finally, it is important to note that BWS is only one method to assess priorities, where other methods include deliberation [ 84 ], simple rating or ranking approaches alone or as a part of a Delphi approach [ 85 ], pick n of m tasks [ 86 ], or stated preference methods such as willingness to pay [ 87 ], DCEs, or conjoint analyses [ 88 ].

With only 34% of included studies in health, this shows an opportunity for greater application in the health field. The range of health-specific topics to which BWS was applied in this study demonstrates versatility across health areas. Various types of preference facilitation have proven successful in health-specific areas [ 25 , 89 ]. Additionally, as health systems are conceptualized more broadly to include social determinants of health such as transportation options, food systems, and the environment the other topical applications demonstrate direct relevance to health systems and decision-making. Similarly, its use in business may be applicable to business-based approaches to health such as social marketing strategies for behavior change. While patient-centered care may involve individual-level preference accommodation of individual patients (e.g., choosing a community-adherence group over fast-track appointments among HIV differentiated service delivery options), systematic understanding of trends in patient preferences at a broader level can inform efficient health system decision-making (e.g., the creation of differentiated service delivery options for HIV). The two studies including data from an African country that we nearly included but did not due to lack of geographic specificity both show successful implementation of multi-country BWS.

Our study is subject to publication bias, as the systematic review searched published literature. The review targeted object case BWS (also known as case 1, MaxDiff, and object scaling). Further research into other types of BWS would contribute to a more comprehensive understanding of BWS in Africa. While we introduce BWS, we do not offer guidance on methods or analysis. Multiple other peer reviewed papers, including some included in this review [ 42 ], and texts offer clear instruction on BWS implementation to aid researchers wishing to apply this method [ 36 ]. Further, we do not include assessments of BWS participation from the participant perspective, as literature on this topic is very limited [ 90 ]. The field would benefit from frameworks for participant assessment of BWS participation to better incorporate this into BWS findings, as have been developed for instrument development [ 76 ].

We need effective tools to measure preferences and priorities, including tools that suit those whose input is more traditionally sought in health decision making (e.g., providers) and those whose voice is critical, but often unheard (e.g., patients, consumers, and community members). BWS is one of those tools. BWS offers a versatile alternative to DCEs and other better-known methods of measuring preferences. Researchers can successfully employ BWS across a range of sample sizes, and using various analysis approaches and programs. Growing use of the method across the African continent demonstrates its feasibility and utility, recommending it for consideration among researchers, program implementers, policy makers, and funders when conducting preference research to influence policy and improve health systems. Further research can help to recommend specific applications, including further work to understand context-specific implications of the strengths and limitations of the methods alongside cognitive burden and population-specific recommendations.

Data availability

All study data are available upon request from the authors for the review registered on PROSPERO (CRD42020209745).

Abbreviations

Balanced incomplete block design

• Best-worst scaling

Discrete choice experiments

Diphtheria, tetanus, and pertussis

Purpose, responses, explanations, findings

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Sheikh K, Gilson L, Agyepong IA, Hanson K, Ssengooba F, Bennett S. Building the field of health policy and systems research: framing the questions. PLoS Med. 2011;8(8):e1001073.

Article   PubMed   PubMed Central   Google Scholar  

Waitzberg R, Quentin W, Webb E, Glied S. The structure and financing of Health Care systems affected how providers coped with COVID-19. Milbank Q. 2021;99(2):542.

Maeda M, Muraki Y, Kosaka T, Yamada T, Aoki Y, Kaku M, et al. Impact of health policy on structural requisites for antimicrobial stewardship: a nationwide survey conducted in Japanese hospitals after enforcing the revised reimbursement system for antimicrobial stewardship programs. J Infect Chemother. 2021;27(1):1–6.

Article   PubMed   Google Scholar  

Lipsky M. Street-level bureaucracy: dilemmas of the individual in public service. Russell Sage Foundation; 2010.

Mwamba C, Mukamba N, Sharma A, Lumbo K, Foloko M, Nyirenda H, et al. Provider discretionary power practices to support implementation of patient-centered HIV care in Lusaka, Zambia. Front Health Serv. 2022;2:918874.

Bylund S, Målqvist M, Peter N, van Herzig S. Negotiating social norms, the legacy of vertical health initiatives and contradicting health policies: a qualitative study of health professionals’ perceptions and attitudes of providing adolescent sexual and reproductive health care in Arusha and Kilimanjaro region, Tanzania. Global Health Action. 2020;13(1):1775992.

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health. 2004;58(10):811–6.

Gagliardi AR, Brouwers MC. Do guidelines offer implementation advice to target users? A systematic review of guideline applicability. BMJ open. 2015;5(2):e007047.

Cairney P, Oliver K. Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy? Health Res Policy Syst. 2017;15:1–11.

Article   Google Scholar  

Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011;6(7):e21704.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.

Organization WH. WHO handbook for guideline development. 2nd ed. Geneva: World Health Organization; 2014.

Google Scholar  

Neumann I, Brignardello-Petersen R, Wiercioch W, Carrasco-Labra A, Cuello C, Akl E, et al. The GRADE evidence-to-decision framework: a report of its testing and application in 15 international guideline panels. Implement Sci. 2015;11:1–8.

De Man J, Roy WM, Sarkar N, Waweru E, Leys M, Van Olmen J, et al. Patient-centered care and people-centered health systems in sub-saharan Africa: why so little of something so badly needed? Int J Person Centered Med. 2016;6(3):162.

Scholl I, Zill JM, Härter M, Dirmaier J. An integrative model of patient-centeredness–a systematic review and concept analysis. PLoS ONE. 2014;9(9):e107828.

Armstrong MJ, Bloom JA. Patient involvement in guidelines is poor five years after institute of medicine standards: review of guideline methodologies. Res Involv Engagem. 2017;3:1–11.

Erismann S, Pesantes MA, Beran D, Leuenberger A, Farnham A, Berger Gonzalez de White M, et al. How to bring research evidence into policy? Synthesizing strategies of five research projects in low-and middle-income countries. Health Res Policy Syst. 2021;19:1–13.

McHugh N, Baker R, Bambra C. Policy actors’ perceptions of public participation to tackle health inequalities in Scotland: a paradox? Int J Equity Health. 2023;22(1):57.

Kuipers SJ, Cramm JM, Nieboer AP. The importance of patient-centered care and co-creation of care for satisfaction with care and physical and social well-being of patients with multi-morbidity in the primary care setting. BMC Health Serv Res. 2019;19:1–9.

Soekhai V, Whichello C, Levitan B, Veldwijk J, Pinto CA, Donkers B, et al. Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review. Drug Discovery Today. 2019;24(7):1324–31.

Meara A, Crossnohere NL, Bridges JF. Methods for measuring patient preferences: an update and future directions. Curr Opin Rheumatol. 2019;31(2):125–31.

Bridges JFW, Albert W, Segal J, Bandeen-Roche K, Bone LR, Purnell T. Stated-preference methods. Center for Health Services and Outcome Research - The Johns Hopkins Bloomberg School of Public Health; 2014.

Brown L, Lee TH, De Allegri M, Rao K, Bridges JFP. Applying stated-preference methods to improve health systems in sub-saharan Africa: a systematic review. Expert Rev PharmacoEcon Outcomes Res. 2017;17(5):441–58.

Hollin IL, Paskett J, Schuster ALR, Crossnohere NL, Bridges JFP. Best-worst scaling and the prioritization of objects in Health: a systematic review. PharmacoEconomics. 2022;40(9):883–99.

Karuga R, Kok M, Luitjens M, Mbindyo P, Broerse JE, Dieleman M. Participation in primary health care through community-level health committees in Sub-saharan Africa: a qualitative synthesis. BMC Public Health. 2022;22(1):359.

Laterra A, Callahan T, Msiska T, Woelk G, Chowdhary P, Gullo S, et al. Bringing women’s voices to PMTCT CARE: adapting CARE’s community score Card© to engage women living with HIV to build quality health systems in Malawi. BMC Health Serv Res. 2020;20:1–14.

Rosenberg NE, Obiezu-Umeh C, Gbaja-Biamila T, Tahlil KM, Nwaozuru U, Oladele D, et al. Strategies for enhancing uptake of HIV self-testing among Nigerian youths: a descriptive analysis of the 4YouthByYouth crowdsourcing contest. BMJ Innovations. 2021;7(3):590.

Cheung KL, Wijnen BFM, Hollin IL, Janssen EM, Bridges JFP, Evers SMAA, et al. Using Best–Worst Scaling Investigate Preferences Health Care PharmacoEconomics. 2016;34(12):1195–209.

PubMed   Google Scholar  

Schuster ALR, Crossnohere NL, Campoamor NB, Hollin IL, Bridges JFP. The rise of best-worst scaling for prioritization: a transdisciplinary literature review. J Choice Modelling. 2024;50:100466.

DiSantostefano RL, Smith IP, Falahee M, Jiménez-Moreno AC, Oliveri S, Veldwijk J, et al. Research priorities to increase confidence in and Acceptance of Health Preference Research. What Questions Should be Prioritized Now? The Patient - Patient-Centered Outcomes Research; 2023.

Mühlbacher AC, Kaczynski A, Zweifel P, Johnson FR. Experimental measurement of preferences in health and healthcare using best-worst scaling: an overview. Health Econ Rev. 2016;6(1):2.

Potoglou D, Burge P, Flynn T, Netten A, Malley J, Forder J, et al. Best–worst scaling vs. discrete choice experiments: an empirical comparison using social care data. Soc Sci Med. 2011;72(10):1717–27.

Flynn TN, Louviere JJ, Peters TJ, Coast J. Best–worst scaling: what it can do for health care research and how to do it. J Health Econ. 2007;26(1):171–89.

Aizaki H, Fogarty J. R packages and tutorial for case 1 best–worst scaling. J Choice Modelling. 2023;46:100394.

Louviere JJ, Flynn TN, Marley AAJ. Best-worst scaling: theory, methods and applications. Cambridge: Cambridge University Press; 2015.

Book   Google Scholar  

Joy SM, Little E, Maruthur NM, Purnell TS, Bridges JFP. Patient preferences for the treatment of type 2 diabetes: a scoping review. PharmacoEconomics. 2013;31(10):877–92.

Bridges JFP, de Bekker-Grob EW, Hauber B, Heidenreich S, Janssen E, Bast A, et al. A Roadmap for increasing the usefulness and impact of patient-preference studies in decision making in Health: a good practices Report of an ISPOR Task Force. Value Health. 2023;26(2):153–62.

Bridges JFP, Hauber AB, Marshall D, Lloyd A, Prosser LA, Regier DA, et al. Conjoint Analysis Applications in Health—a Checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011;14(4):403–13.

Honda A, Krucien N, Ryan M, Diouf ISN, Salla M, Nagai M, et al. For more than money: willingness of health professionals to stay in remote Senegal. Hum Resour Health. 2019;17(1):28.

Pentz C, Filter M, editors. From Generation Y to Generation Wine? A Best-Worst scaling study of wine attribute importance. Proceedings of the 52nd International Academic Conferences; 2019; Barcelona: International Institute of Social and Economic Sciences.

Teffo M, Earl A, Zuidgeest M. Understanding public transport needs in Cape Town’s informal settlements: a best-worst-scaling approach. J South Afr Institution Civil Eng. 2019;61(2):39–50.

Amadou Z. Agropastoralists’ Climate Change Adaptation Strategy Modeling: Software and Coding Method Accuracies for Best-Worst Scaling Data. African Handbook of Climate Change Adaptation2020. pp. 1–10.

Amadou Z. Which Sustainable Development Goals and Eco-challenges Matter Most to Niger’s Farmers and Herdsmen? A Best Worst Scaling Approach. Agricultural Research & Technology Open Access Journal; 2020.

Jin S, Mansaray B, Jin X, Li H. Farmers’ preferences for attributes of rice varieties in Sierra Leone. Food Secur. 2020;12(5):1185–97.

Kim H-Y, Hanrahan CF, Dowdy DW, Martinson NA, Golub JE, Bridges JF. Priorities among HIV-positive individuals for tuberculosis preventive therapies. Int J Tuberculosis Lung Disease. 2020;24(4):396–402.

Nyabinwa P, Kashongwe OB, Hirwa CD, Bebe BO. Perception of farmers about endometritis prevention and control measures for zero-grazed dairy cows on smallholder farms in Rwanda. BMC Vet Res. 2020;16(1):175.

Nyarko E, Akoto FM, Doku-Amponsah K. Perceived antimicrobial dispensing practices in medicine outlets in Ghana: a maximum difference experiment design. PLoS ONE. 2023;18(7):e0288519.

Okpiaifo G, Durand-Morat A, West GH, Nalley LL, Nayga RM, Wailes EJ. Consumers’ preferences for sustainable rice practices in Nigeria. Global Food Secur. 2020;24.

Ola O, Menapace L. Revisiting constraints to smallholder participation in high-value markets: a best‐worst scaling approach. Agric Econ. 2020;51(4):595–608.

Pentz C, Forrester A. The importance of wine attributes in an emerging wine-producing country. South Afr J Bus Manage. 2020;51(1).

Yemeke TT, Kiracho EE, Mutebi A, Apolot RR, Ssebagereka A, Evans DR, et al. Health versus other sectors: multisectoral resource allocation preferences in Mukono district, Uganda. PLoS ONE. 2020;15(7):e0235250.

Lewis-Brown E, Beatty H, Davis K, Rabearisoa A, Ramiaramanana J, Mascia MB et al. The importance of future generations and conflict management in conservation. Conserv Sci Pract. 2021;3(9).

Muunda E, Mtimet N, Schneider F, Wanyoike F, Dominguez-Salas P, Alonso S. Could the new dairy policy affect milk allocation to infants in Kenya? A best-worst scaling approach. Food Policy. 2021;101:102043.

Seymour ZA, Cloete E, McCurdy M, Olson M, Hughes J. Understanding values of sanitation users: examining preferences and behaviors for sanitation systems. J Water Sanitation Hygiene Dev. 2021;11(2):195–207.

van Niekerk K, Dada S, Tönsing K. Perspectives of rehabilitation professionals on assistive technology provision to young children in South Africa: a national survey. Disabil Rehabilitation: Assist Technol. 2023;18(5):588–95.

Gertz AM, Soffi ASM, Mompe A, Sickboy O, Gaines AN, Ryan R et al. Developing an Assessment of Contraceptive preferences in Botswana: piloting a Novel Approach using best-worst scaling of attributes. Front Global Women’s Health. 2022;3.

Maruyama Y, Ujiie K, Ahmed C, Diagne M, Irie M. Farmers’ preferences of the agricultural inputs for rice farming in Senegal River Basin, Mauritania: a best-worst scaling approach. J Arid Land Stud. 2022;32(S):61–5.

Ahoudou I, Sogbohossou DE, Hotegni NVF, Adjé CO, Komlan FA, Moumouni-Moussa I, et al. Farmers’ selection criteria for sweet potato varieties in Benin: an application of best-worst scaling. Exp Agric. 2023;59:e25.

Filter M, Pentz CD. Dealcoholised wine: exploring the purchasing considerations of South African generation Y consumers. Br Food J. 2023;125(13):205–19.

Walaza S, Onderwater P, Zuidgeest M, editors. Best-worst scaling approach to measure public transport user quality perceptions and preferences in cape town2023: Southern African Transport Conference.

Nyarko E, Arku D, Duah G. Best-worst scaling in studying the impact of the coronavirus pandemic on health professionals in Ghana. Model Assist Stat Appl. 2023;18(3):227–36.

Amadou Z, Which agricultural innovations matter most to Niger’s farmers?, Count-based and best-worst scaling approaches. Russian J Agricultural Socio-Economic Sci. 2023;140(8):3–12.

Abubakar M, Amadou Z, Daniel K. Best-worst scaling approach in predicting seed attribute preferences among resource poor farmers in Northern Nigeria. Int J Humanit Social Sci. 2014;2(9):304–10.

Amadou Z, Baky AD. Consumers’ preferences for quality and safety attributes of milk products in Niger: a best-worst scaling approach. J Agric Sci Technol. 2015;5(9):635–42.

Irlam JH, Zuidgeest M. Barriers to cycling mobility in a low-income community in Cape Town: a best-worst scaling approach. Case Stud Transp Policy. 2018;6(4):815–23.

Kim HY, Dowdy DW, Martinson NA, Golub E, Bridges J, Hanrahan JF. Maternal priorities for preventive therapy among HIV-positive pregnant women before and after delivery in South Africa: a best–worst scaling survey. J Int AIDS Soc. 2018;21(7):e25143.

Lagerkvist CJ, Kokko S, Karanja N. Health in perspective: framing motivational factors for personal sanitation in urban slums in Nairobi, Kenya, using anchored best–worst scaling. J Water Sanitation Hygiene Dev. 2014;4(1):108–19.

Lagerkvist CJ, Okello J, Karanja N. Anchored vs. relative best–worst scaling and latent class vs. hierarchical bayesian analysis of best–worst choice data: investigating the importance of food quality attributes in a developing country. Food Qual Prefer. 2012;25(1):29–40.

Lategan BW, Pentz CD, du Preez R. Importance of wine attributes: a South African generation Y perspective. Br Food J. 2017;119(7):1536–46.

Mansaray B, Jin S, Yuan R, Li H, editors. Farmers Preferences for Attributes of Seed Rice in Sierra Leone: A Best-Worst Scaling Approach. International Conference of Agricultural Economists; 2018.

Ozawa S, Wonodi C, Babalola O, Ismail T, Bridges J. Using best-worst scaling to rank factors affecting vaccination demand in northern Nigeria. Vaccine. 2017;35(47):6429–37.

Torbica A, De Allegri M, Belemsaga D, Medina-Lara A, Ridde V. What criteria guide national entrepreneurs’ policy decisions on user fee removal for maternal health care services? Use of a best–worst scaling choice experiment in West Africa. J Health Serv Res Policy. 2014;19(4):208–15.

van Niekerk K, Dada S, Tönsing K. Perspectives of rehabilitation professionals on assistive technology provision to young children in South Africa: a national survey. Disability and Rehabilitation: Assistive Technology; 2021.

Hollin IL, Craig BM, Coast J, Beusterien K, Vass C, DiSantostefano R, et al. Reporting formative qualitative research to support the development of quantitative preference study protocols and corresponding Survey instruments: guidelines for authors and reviewers. Patient - Patient-Centered Outcomes Res. 2020;13(1):121–36.

Janssen EM, Segal JB, Bridges JFP. A Framework for Instrument Development of a choice experiment: an application to type 2 diabetes. Patient - Patient-Centered Outcomes Res. 2016;9(5):465–79.

Herdman M, Fox-Rushby J, Badia X. ‘Equivalence’and the translation and adaptation of health-related quality of life questionnaires. Qual Life Res. 1997;6.

Oortwijn W, Husereau D, Abelson J, Barasa E, Bayani DD, Santos VC, et al. Designing and implementing deliberative processes for health technology assessment: a good practices report of a joint HTAi/ISPOR task force. Int J Technol Assess Health Care. 2022;38(1):e37.

Majumder MA, Blank ML, Geary J, Bollinger JM, Guerrini CJ, Robinson JO, et al. Challenges to building a gene variant commons to assess hereditary cancer risk: results of a modified policy Delphi panel deliberation. J Personalized Med. 2021;11(7):646.

Boyd JL, Sugarman J. Toward responsible public engagement in neuroethics. AJOB Neurosci. 2022;13(2):103–6.

Manera KE, Tong A, Craig JC, Shen J, Jesudason S, Cho Y, et al. An international Delphi survey helped develop consensus-based core outcome domains for trials in peritoneal dialysis. Kidney Int. 2019;96(3):699–710.

Van Schoubroeck S, Springael J, Van Dael M, Malina R, Van Passel S. Sustainability indicators for biobased chemicals: a Delphi study using Multi-criteria decision analysis. Resour Conserv Recycl. 2019;144:198–208.

Akunne AF, Bridges JF, Sanon M, Sauerborn R. Comparison of individual and group valuation of health state scenarios across communities in West Africa. Appl Health Econ Health Policy. 2006;5:261–8.

Raj M, Ryan K, Nong P, Calhoun K, Trinidad MG, De Vries R, et al. Public deliberation process on patient perspectives on health information sharing: evaluative descriptive study. JMIR cancer. 2022;8(3):e37793.

Voehler D, Neumann PJ, Ollendorf DA. Patient and caregiver views on measures of the value of health interventions. Patient Prefer Adherence. 2022:3383–92.

Kinter ET, Schmeding A, Rudolph I, dosReis S, Bridges JFP. Identifying patient-relevant endpoints among individuals with schizophrenia: an application of patient-centered health technology assessment. Int J Technol Assess Health Care. 2009;25(1):35–41.

Heinzen RR, Bridges JF. Comparison of four contingent valuation methods to estimate the economic value of a pneumococcal vaccine in Bangladesh. Int J Technol Assess Health Care. 2008;24(4):481–7.

Bridges JFP, Selck FW, Gray GE, McIntyre JA, Martinson NA. Condom avoidance and determinants of demand for male circumcision in Johannesburg, South Africa. Health Policy Plann. 2011;26(4):298–306.

Beckham SW, Crossnohere NL, Gross M, Bridges JFP. Eliciting preferences for HIV Prevention technologies: a systematic review. Patient - Patient-Centered Outcomes Res. 2021;14(2):151–74.

Rogers HJ, Marshman Z, Rodd H, Rowen D. Discrete choice experiments or best-worst scaling? A qualitative study to determine the suitability of preference elicitation tasks in research with children and young people. J patient-reported Outcomes. 2021;5:1–11.

Download references

Acknowledgements

Not applicable.

John F.P. Bridges holds an Innovation in Regulatory Science Award from the Burroughs Wellcome Fund. Laura K Beres’ contributions were supported by National Institute of Mental Health 1K01MH130244-01A1. The contents included here are the responsibility of the authors and do not represent the official views of the National Institute of Mental Health.

Author information

Authors and affiliations.

Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe Street, Office, Baltimore, MD, 5032, 21205, USA

Laura K. Beres

Department of Biomedical Informatics, The Ohio State University College of Medicine, 220 Lincoln Tower, 1800 Cannon Drive, Columbus, OH, 43210, USA

Nicola B. Campoamor, Anne Schuster & John F. P. Bridges

Center for the Advancement of Team Science, Analytics, and Systems Thinking in Health Services and Implementation Science Research (CATALYST), The Ohio State University, 700 Ackerman Road, Columbus, OH, 43202, USA

Rachael Hawthorn

Department of Global Health, University of Washington, UW Box #351620, Seattle, WA, 98195, USA

Melissa L. Mugambi

Centre for Infectious Disease Research in Zambia, Stand 378A / 15, Main Street, P.O. Box 34681, Ibex, Lusaka, Zambia

Musunge Mulabe, Natlie Vhlakis & Michael Kabongo

You can also search for this author in PubMed   Google Scholar

Contributions

LKB, JFPB: Conceptualization; LKB, RH, AS: Formal analysis; LKB, JFPB: Funding acquisition; LKB, RH, AS, JFPB: Wrote original draft; MLM, MM, NK, MK, NC: Review and editing.

Corresponding author

Correspondence to John F. P. Bridges .

Ethics declarations

Ethics approval and consent to participate.

The study did not include human subjects research. The review was registered on PROSPERO (CRD42020209745).

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Beres, L.K., Campoamor, N.B., Hawthorn, R. et al. Using best-worst scaling to inform policy decisions in Africa: a literature review. BMC Public Health 24 , 2607 (2024). https://doi.org/10.1186/s12889-024-20068-w

Download citation

Received : 10 February 2024

Accepted : 12 September 2024

Published : 27 September 2024

DOI : https://doi.org/10.1186/s12889-024-20068-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Measuring priorities
• Stakeholder engagement

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

A Systematic Literature Review on Vulnerabilities, Mitigation Techniques, and Attacks in Field-Programmable Gate Arrays

• Review Article--Computer Engineering and Computer Science
• Published: 23 September 2024

Cite this article

• Ali Alsuwaiyan   ORCID: orcid.org/0000-0002-2317-026X 1 , 2 ,
• Aliyu Abubakar Habib 1 ,
• Ali Bello Imoukhuede 1 ,
• Mohamed Osman Omar 1 ,
• Md Al Maruf 3 ,
• Mansour Alqarni 3 ,
• Aiman El-Maleh 1 , 2 ,
• Abdulaziz Tabbakh 1 , 2 ,
• Muhamad Felemban 1 , 2 &
• Akramul Azim 3  

44 Accesses

1 Altmetric

Explore all metrics

This paper presents a systematic literature review (SLR) of the vulnerabilities of field-programmable gate arrays (FPGAs), based on 51 carefully selected articles from a pool of 271 unique publications sourced from various databases, including IEEE, ACM, and SpringerLink, covering the period from January 2012 to September 2022. The study identifies 22 distinct vulnerabilities and examines 27 types of attacks that exploit these vulnerabilities. We provide detailed descriptions of each vulnerability, including associated threat models, and review various mitigation techniques while highlighting the application of machine learning (ML) in both detection and defense whenever applicable. The significance of this work stems from its focus on a vulnerability-centric approach, where mitigation strategies directly target vulnerabilities rather than simply countering attacks. This approach highlights the critical role of safeguarding bitstream access, which can prevent a broad spectrum of attacks. Additionally, the study notes the absence of common vulnerabilities and exposures (CVE) datasets for the selected period, suggesting a gap in the documentation of FPGA-related vulnerabilities. The findings of this review have important implications for mitigating economic and security risks in FPGA applications, and we suggest potential future research directions, including further integration of ML techniques to enhance FPGA security.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Turning the Table: Using Bitstream Reverse Engineering to Detect FPGA Trojans

The Detection of Malicious Modifications in the FPGA

Malicious and Deliberate Attacks and Power System Resiliency

Explore related subjects.

• Artificial Intelligence

Coughlin, A.; Cusack, G.; Wampler, J.; Keller, E.; Wustrow, E.: Breaking the trust dependence on third party processes for reconfigurable secure hardware. In: Proceedings of the 2019 ACM/SIGDA International symposium on field-programmable gate arrays. FPGA ’19, pp. 282–291. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3289602.3293895 Accessed 2022-09-29

Fu, H.; Gan, L.; Clapp, R.G.; Ruan, H.; Pell, O.; Mencer, O.; Flynn, M.; Huang, X.; Yang, G.: Scaling reverse time migration performance through reconfigurable dataflow engines. IEEE Micro 34 (1), 30–40 (2013)

Article   Google Scholar  

Huang, Z.; Wang, Q.: Enhancing architecture-level security of SoC designs via the distributed security IPs deployment methodology. J. Info. Sci. Eng. 36 (2), 387–421 (2020)

Google Scholar  

Neshatpour, K.; Makrani, H.M.; Sasan, A.; Ghasemzadeh, H.; Rafatirad, S.; Homayoun, H.: Design space exploration for hardware acceleration of machine learning applications in MapReduce. In: 2018 IEEE 26th Annual international symposium on field-programmable custom computing machines (FCCM), pp. 221–221 (2018). IEEE

Malik, A.; Ullah, A.; Zahir, A.; Qamar, A.; Khattak, S.; Reviriego, P.: Isolation design flow effectiveness evaluation methodology for Zynq SoCs. Electronics 9 , 814 (2020). https://doi.org/10.3390/electronics9050814

Geier, M.; Faller, D.; Brändle, M.; Chakraborty, S.: Cost-effective energy monitoring of a Zynq-based real-time system including dual gigabit ethernet. In: 2019 IEEE 27th Annual international symposium on field-programmable custom computing machines (FCCM), pp. 327–327 (2019). https://doi.org/10.1109/FCCM.2019.00068

Jarrah, A.; Amri, S.: Optimized FPGA-based implementation of brain tumor detection by combining \(k\) -means and Grey Wolf optimization algorithms. Traitement Du Signal 39 , 1879–1891 (2022). https://doi.org/10.18280/ts.390601

Fujii, N.; Koike, N.: IoT Remote group experiments in the cyber laboratory: a FPGA-based remote laboratory in the hybrid cloud. In: 2017 International conference on cyberworlds (CW), pp. 162–165 (2017). IEEE

Asadi, H.; Tahoori, M.B.: Analytical techniques for soft error rate modeling and mitigation of FPGA-based designs. IEEE Trans. Very Large Scale Integr. (VLSI) Syst. 15 (12), 1320–1331 (2007)

Tajik, S.; Lohrke, H.; Seifert, J.-P.; Boit, C.: On the power of optical contactless probing: attacking bitstream encryption of FPGAs. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. CCS ’17, pp. 1661–1674. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3133956.3134039

She, X.; Li, N.: Reducing critical configuration bits via partial TMR for SEU mitigation in FPGAs. IEEE Trans. Nuclear Sci. 64 (10), 2626–2632 (2017)

Gnad, D.R.E.; Oboril, F.; Tahoori, M.B.: Voltage drop-based fault attacks on FPGAs using valid bitstreams. In: 2017 27th International conference on field programmable logic and applications (FPL), pp. 1–7 (2017). https://doi.org/10.23919/FPL.2017.8056840

Rakin, A.S.; Luo, Y.; Xu, X.; Fan, D.: Deep-dup: an adversarial weight duplication attack framework to crush deep neural network in multi-tenant FPGA. In: 30th USENIX security symposium (2021)

Luo, Y.; Gongye, C.; Ren, S.; Fei, Y.; Xu, X.: Stealthy-shutdown: practical remote power attacks in multi-tenant FPGAs. In: 2020 IEEE 38th International conference on computer design (ICCD), pp. 545–552. IEEE, Hartford, CT, USA (2020). https://doi.org/10.1109/ICCD50377.2020.00097

Luo, Y.; Xu, X.: A Quantitative defense framework against power attacks on multi-tenant FPGA. In: Proceedings of the 39th International conference on computer-aided design, pp. 1–9 (2020)

Duan, S.; Wang, W.; Luo, Y.; Xu, X.: A Survey of recent attacks and mitigation on FPGA systems. In: 2021 IEEE Computer society annual symposium on VLSI (ISVLSI), pp. 284–289 (2021). IEEE

Lohrke, H.; Tajik, S.; Boit, C.; Seifert, J.-P.: No place to hide: contactless probing of secret data on FPGAs. In: Cryptographic hardware and embedded systems —CHES 2016: 18th International Conference 2016, pp. 147–167. Springer, Berlin, Heidelberg (2016)

Sari, A.; Psarakis, M.: Scrubbing-aware placement for reliable FPGA systems. IEEE Trans. Emerg. Topics Comput. 8 (3), 564–576 (2017)

Wei, H.; Yueke, W.; Kefei, X.; Wei, D.: Single event effect vulnerability analysis and on-orbit error rate prediction. In: 2016 IEEE International conference on signal and image processing (ICSIP), pp. 471–477. IEEE, New York (2016). https://doi.org/10.1109/SIPROCESS.2016.7888307

Hinderks, A.; Mayo, F.J.D.; Thomaschewski, J.; Escalona, M.J.: An SLR-tool: search process in practice: a tool to conduct and manage systematic literature review (SLR). In: Proceedings of the ACM/IEEE 42nd international conference on software engineering: companion proceedings, pp. 81–84 (2020)

Zhang, J.; Qu, G.: Recent attacks and defenses on FPGA-based systems. ACM Trans. Reconfig. Technol. Syst. (TRETS) 12 (3), 1–24 (2019)

Article   MathSciNet   Google Scholar  

Mahmoud, D.G.; Lenders, V.; Stojilović, M.: Electrical-level attacks on CPUs, FPGAs, and GPUs: survey and implications in the heterogeneous era. ACM Comput. Surv. (CSUR) 55 (3), 1–40 (2022)

Venn, M.: Subject guides: systematic reviews for health: 1. Formulate the research question. Accessed: Nov 2022 (2022). https://utas.libguides.com/SystematicReviews/FormulateQuestion

Xu, X.; Zhang, J.: Rethinking FPGA security in the new era of artificial intelligence. In: 2020 21st International symposium on quality electronic design (ISQED), pp. 46–51 (2020). IEEE

He, W.; Torre, E.; Riesgo, T.: A precharge-absorbed DPL logic for reducing early propagation effects on FPGA implementations. In: 2011 International conference on reconfigurable computing and FPGAs, pp. 217–222 (2011). IEEE

Jacobs, A.; Cieslewski, G.; George, A.D.: Overhead and reliability analysis of algorithm-based fault tolerance in FPGA systems. In: 22nd International conference on field programmable logic and applications (FPL), pp. 300–306 (2012). IEEE

Zick, K.M.; Srivastav, M.; Zhang, W.; French, M.: Sensing nanosecond-scale voltage attacks and natural transients in FPGAs. In: Proceedings of the ACM/SIGDA international symposium on field programmable gate arrays. FPGA ’13, pp. 101–104. ACM, New York, NY, USA (2013). https://doi.org/10.1145/2435264.2435283

Hoang, A.-T.; Fujino, T.: Intra-masking dual-rail memory on LUT implementation for SCA-resistant AES on FPGA. ACM Trans. Reconfig. Technol. Syst. 7 (2), 10–11019 (2014). https://doi.org/10.1145/2617595

Duncan, A.; Rahman, F.; Lukefahr, A.; Farahmandi, F.; Tehranipoor, M.: FPGA bitstream security: a day in the life. In: 2019 IEEE International test conference (ITC), pp. 1–10 (2019). IEEE

Zhao, M.; Suh, G.E.: FPGA-based remote power side-channel attacks. In: 2018 IEEE symposium on security and privacy (SP), pp. 229–244 (2018). IEEE

Zhang, Z.; Njilla, L.; Kamhoua, C.A.; Yu, Q.: Thwarting security threats from malicious FPGA tools with novel FPGA oriented moving target defense. IEEE Trans. Very Large Scale Integr. Syst. (VLSI) 27 , 665–678 (2018)

Ender, M.; Moradi, A.; Paar, C.: The unpatchable silicon: a full break of the bitstream encryption of Xilinx 7-Series FPGAs, pp. 1803–1819 (2020). https://www.usenix.org/conference/usenixsecurity20/presentation/ender Accessed Oct 2022

Hoque, T.; Yang, K.; Karam, R.; Tajik, S.; Forte, D.; Tehranipoor, M.; Bhunia, S.: Hidden in plaintext: an obfuscation-based countermeasure against FPGA bitstream tampering attacks. ACM Trans. Design Autom. Electron. Syst. (TODAES) 25 (1), 1–32 (2019)

Huang, Z.; Wang, Q.: MSIPS: Multi-tiered security IPs architecture for secure SoC design. In: 2017 International conference on networking and network applications (NaNA), pp. 203–208 (2017). https://doi.org/10.1109/NaNA.2017.43

Zhang, J.; Lin, Y.; Qu, G.: Reconfigurable binding against FPGA replay attacks. ACM Trans. Design Autom. Electron. Syst. (TODAES) 20 (2), 1–20 (2015)

Mal-Sarkar, S.; Krishna, A.; Ghosh, A.; Bhunia, S.: Hardware Trojan attacks in FPGA devices: threat analysis and effective counter measures. In: Proceedings of the 24th edition of the Great Lakes symposium on VLSI, pp. 287–292 (2014)

Zhang, J.-L.; Wang, W.-Z.; Wang, X.-W.; Xia, Z.-H.: Enhancing security of FPGA-based embedded systems with combinational logic binding. J. Comput. Sci. Technol 32 (2), 329–339 (2017). https://doi.org/10.1007/s11390-017-1700-8

ZamanZadeh, S.; Shahabi, S.; Jahanian, A.: Security improvement of FPGA configuration file against the reverse engineering attack. In: 2016 13th International Iranian society of cryptology conference on information security and cryptology (ISCISC), pp. 101–105 (2016). IEEE

Kroeger, T.; Cheng, W.; Danger, J.-L.; Guilley, S.; Karimi, N.: Cross-PUF attacks: targeting FPGA implementation of arbiter-PUFs. J. Electron. Test. 38 (3), 261–277 (2022)

Dombrowski, J.; Andel, T.R.; McDonald, J.T.: The application of moving target defense to field programmable gate arrays. In: Proceedings of the 11th Annual cyber and information security research conference, pp. 1–4 (2016)

Olney, B.; Karam, R.: Tunable FPGA bitstream obfuscation with Boolean satisfiability attack countermeasure. ACM Trans. Design Autom. Electron. Syst. (TODAES) 25 (2), 1–22 (2020)

Zahid, K.: The detection of malicious modifications in the FPGA. J. Electron. Test. 38 (3), 247–260 (2022)

Mahmud, S.; Olney, B.; Karam, R.: Architectural diversity: bio-inspired hardware security for FPGAs. In: 2018 IEEE 3rd International verification and security workshop (IVSW), pp. 48–51 (2018). IEEE

Sozio, C.; Jordan, Z.; Skipper, G.; Lukefahr, A.; Duncan, A.: Patchable hardware security module (PHaSM) for extending FPGA root-of-trust capabilities. In: 2021 IEEE physical assurance and inspection of electronics (PAINE), pp. 1–8 (2021). IEEE

Sun, P.; Cui, A.: A new pay-per-use scheme for the protection of FPGA IP. In: 2019 IEEE International symposium on circuits and systems (ISCAS), pp. 1–5 (2019). IEEE

Wang, Z.; Chen, W.; Yao, Z.; Zhang, F.; Luo, Y.; Tang, X.; Guo, X.; Ding, L.; Peng, C.: Proton-induced single-event effects on 28 nm Kintex-7 FPGA. Microelectron. Reliabil. 107 , 113594 (2020)

Zhang, J.; Lin, Y.; Lyu, Y.; Qu, G.: A PUF-FSM binding scheme for FPGA IP protection and pay-per-device licensing. IEEE Trans. Info. Forensics Secur. 10 (6), 1137–1150 (2015)

Spenke, A.; Breithaupt, R.; Plaga, R.: An arbiter PUF secured by remote random reconfigurations of an FPGA. In: Trust and trustworthy computing: 9th international conference, TRUST 2016, Vienna, Austria, August 29-30, 2016, Proceedings 9, pp. 140–158 (2016). Springer

Zhang, J.; Wu, Q.; Lyu, Y.; Zhou, Q.; Cai, Y.; Lin, Y.; Qu, G.: Design and implementation of a delay-based PUF for FPGA IP protection. In: 2013 international conference on computer-aided design and computer graphics, pp. 107–114 (2013). IEEE

Barbareschi, M.; Bagnasco, P.: Implementation of a reliable mechanism for protecting IP cores on low-end FPGA devices. Int. J. Embed. Syst. 9 (4), 337–352 (2017)

Duncan, A.; Skipper, G.; Stern, A.; Nahiyan, A.; Rahman, F.; Lukefahr, A.; Tehranipoor, M.; Swany, M.: FLATS: filling logic and testing spatially for FPGA authentication and tamper detection. In: 2019 IEEE international symposium on hardware oriented security and trust (HOST), pp. 81–90 (2019). IEEE

Labafniya, M.; Saeidi, R.: Secure FPGA design by filling unused spaces. ISeCure 11 (1), 47–55 (2019)

Khaleghi, B.; Ahari, A.; Asadi, H.; Bayat-Sarmadi, S.: FPGA-based protection scheme against hardware Trojan horse insertion using dummy logic. IEEE Embed. Syst. Lett. 7 (2), 46–50 (2015)

Ye, M.; Feng, X.; Wei, S.: HISA: Hardware isolation-based secure architecture for CPU-FPGA embedded systems. In: 2018 IEEE/ACM international conference on computer-aided design (ICCAD), pp. 1–8 (2018). ACM

Sayeeshwari, S.; Prabhu, E.: A simple countermeasure to mitigate buffer overflow attack using minimalistic hardware-integrated software simulation for FPGA. In: 2022 IEEE international conference on electronics, computing and communication technologies (CONECCT), pp. 1–4 (2022). IEEE

Weissman, Z.; Tiemann, T.; Moghimi, D.; Custodio, E.; Eisenbarth, T.; Sunar, B.: JackHammer: efficient Rowhammer on heterogeneous FPGA-CPU platforms. IACR Trans. Cryptogr. Hardw. Embed. Syst. (2020). https://doi.org/10.13154/tches.v2020.i3.169-195

Giechaskiel, I.; Szefer, J.: Information leakage from FPGA routing and logic elements. In: Proceedings of the 39th international conference on computer-aided design, pp. 1–9 (2020)

Zamanzadeh, S.; Jahanian, A.: Scalable security path methodology: a cost-security trade-off to protect FPGA IPs against active and passive tampers. In: 2017 Asian hardware oriented security and trust symposium (AsianHOST), pp. 85–90 (2017). IEEE

Vliegen, J.; Mentens, N.; Verbauwhede, I.: Secure, remote, dynamic reconfiguration of FPGAs. ACM Trans. Reconfig. Technol. Syst. 7 (4), 1–19 (2015). https://doi.org/10.1145/2629423

Yao, Y.; Kiaei, P.; Singh, R.; Tajik, S.; Schaumont, P.: Programmable Ro (Pro): a multipurpose countermeasure against side-channel and fault injection attack. (2021) arXiv:2106.13784

Schultz, T.; Jha, R.; Casto, M.; Dupaix, B.: Vulnerabilities and reliability of ReRAM based PUFs and memory logic. IEEE Trans. Reliabil. 69 (2), 690–698 (2019)

Nannipieri, P.; Di Matteo, S.; Baldanzi, L.; Crocetti, L.; Belli, J.; Fanucci, L.; Saponara, S.: True random number generator based on Fibonacci-Galois ring oscillators for FPGA. Appl. Sci. 11 (8), 3330 (2021)

Zhang, F.; Wang, Z.; Shen, H.; Yang, B.; Wu, Q.; Ren, K.: DARPT: defense against remote physical attack based on TDC in multi-tenant scenario. In: Proceedings of the 59th ACM/IEEE design automation conference, pp. 559–564 (2022)

Staub, D.; Jha, R.; Kapp, D.: A CRISPR-Cas-inspired mechanism for detecting hardware Trojans in FPGA devices. (2020) https://doi.org/10.48550/arXiv.2005.07332

Proulx, A.; Chouinard, J.-Y.; Fortier, P.; Miled, A.: A survey on FPGA cybersecurity design strategies. ACM Trans. Reconfig. Technol. Syst. (2022). https://doi.org/10.1145/3561515

Agrawal, R.; Castro, L.; Yang, G.; Juvekar, C.; Yazicigil, R.; Chandrakasan, A.; Vaikuntanathan, V.; Joshi, A.: FAB: An FPGA-based accelerator for bootstrappable fully homomorphic encryption. In: 2023 IEEE International symposium on high-performance computer architecture (HPCA), pp. 882–895 (2023). IEEE

Di Matteo, S.; Gerfo, M.L.; Saponara, S.: VLSI design and FPGA implementation of an NTT hardware accelerator for homomorphic seal-embedded library. IEEE Access 11 , 72498–72508 (2023)

Ch, M.L.; Raj, A.B.; Abhikshit, L.: Design and implementation of a secure physical unclonable function in FPGA. In: 2020 Second international conference on inventive research in computing applications (ICIRCA), pp. 1083–1089 (2020). IEEE

Download references

Acknowledgements

This work was supported by the Interdisciplinary Research Center for Intelligent Secure Systems at King Fahd University of Petroleum & Minerals under Grant INSS2310. All co-authors contributed equally to this work.

Author information

Authors and affiliations.

Computer Engineering Department, King Fahd University of Petroleum & Minerals, 31261, Dhahran, Saudi Arabia

Ali Alsuwaiyan, Aliyu Abubakar Habib, Ali Bello Imoukhuede, Mohamed Osman Omar, Aiman El-Maleh, Abdulaziz Tabbakh & Muhamad Felemban

Interdisciplinary Research Center for Intelligent Secure Systems, King Fahd University of Petroleum & Minerals, 31261, Dhahran, Saudi Arabia

Ali Alsuwaiyan, Aiman El-Maleh, Abdulaziz Tabbakh & Muhamad Felemban

Software Engineering Department, Ontario Tech University, 2000 Simcoe St N, Oshawa, ON, L1G 0C5, Canada

Md Al Maruf, Mansour Alqarni & Akramul Azim

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ali Alsuwaiyan .

Bibliographic Information of the Included Papers

This section presents Table  10 , which provides bibliographic details of the papers that have been included in this systematic review. The table serves as a resource, offering the list of 51 included articles that have been carefully selected and analyzed for their significance and relevance to the research topic at hand. The primary objective of this table is to help the reader identify articles that have successfully met the criteria of the quality assessment conducted as part of the review process. The References section comprehensively encompasses all pertinent references, including those that have been utilized to substantiate this review, as in the related work section.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Alsuwaiyan, A., Habib, A.A., Imoukhuede, A.B. et al. A Systematic Literature Review on Vulnerabilities, Mitigation Techniques, and Attacks in Field-Programmable Gate Arrays. Arab J Sci Eng (2024). https://doi.org/10.1007/s13369-024-09562-w

Download citation

Received : 26 April 2024

Accepted : 28 August 2024

Published : 23 September 2024

DOI : https://doi.org/10.1007/s13369-024-09562-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Cyberattack
• Embedded systems
• Find a journal
• Publish with us
• Track your research