what is multiple case study research design

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Research Guides

Multiple Case Studies

Nadia Alqahtani and Pengtong Qu

Description

The case study approach is popular across disciplines in education, anthropology, sociology, psychology, medicine, law, and political science (Creswell, 2013). It is both a research method and a strategy (Creswell, 2013; Yin, 2017). In this type of research design, a case can be an individual, an event, or an entity, as determined by the research questions. There are two variants of the case study: the single-case study and the multiple-case study. The former design can be used to study and understand an unusual case, a critical case, a longitudinal case, or a revelatory case. On the other hand, a multiple-case study includes two or more cases or replications across the cases to investigate the same phenomena (Lewis-Beck, Bryman & Liao, 2003; Yin, 2017). …a multiple-case study includes two or more cases or replications across the cases to investigate the same phenomena

The difference between the single- and multiple-case study is the research design; however, they are within the same methodological framework (Yin, 2017). Multiple cases are selected so that “individual case studies either (a) predict similar results (a literal replication) or (b) predict contrasting results but for anticipatable reasons (a theoretical replication)” (p. 55). When the purpose of the study is to compare and replicate the findings, the multiple-case study produces more compelling evidence so that the study is considered more robust than the single-case study (Yin, 2017).

To write a multiple-case study, a summary of individual cases should be reported, and researchers need to draw cross-case conclusions and form a cross-case report (Yin, 2017). With evidence from multiple cases, researchers may have generalizable findings and develop theories (Lewis-Beck, Bryman & Liao, 2003).

Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Los Angeles, CA: Sage.

Lewis-Beck, M., Bryman, A. E., & Liao, T. F. (2003). The Sage encyclopedia of social science research methods . Los Angeles, CA: Sage.

Yin, R. K. (2017). Case study research and applications: Design and methods . Los Angeles, CA: Sage.

Key Research Books and Articles on Multiple Case Study Methodology

Yin discusses how to decide if a case study should be used in research. Novice researchers can learn about research design, data collection, and data analysis of different types of case studies, as well as writing a case study report.

Chapter 2 introduces four major types of research design in case studies: holistic single-case design, embedded single-case design, holistic multiple-case design, and embedded multiple-case design. Novice researchers will learn about the definitions and characteristics of different designs. This chapter also teaches researchers how to examine and discuss the reliability and validity of the designs.

Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches . Los Angeles, CA: Sage.

This book compares five different qualitative research designs: narrative research, phenomenology, grounded theory, ethnography, and case study. It compares the characteristics, data collection, data analysis and representation, validity, and writing-up procedures among five inquiry approaches using texts with tables. For each approach, the author introduced the definition, features, types, and procedures and contextualized these components in a study, which was conducted through the same method. Each chapter ends with a list of relevant readings of each inquiry approach.

This book invites readers to compare these five qualitative methods and see the value of each approach. Readers can consider which approach would serve for their research contexts and questions, as well as how to design their research and conduct the data analysis based on their choice of research method.

Günes, E., & Bahçivan, E. (2016). A multiple case study of preservice science teachers’ TPACK: Embedded in a comprehensive belief system. International Journal of Environmental and Science Education, 11 (15), 8040-8054.

In this article, the researchers showed the importance of using technological opportunities in improving the education process and how they enhanced the students’ learning in science education. The study examined the connection between “Technological Pedagogical Content Knowledge” (TPACK) and belief system in a science teaching context. The researchers used the multiple-case study to explore the effect of TPACK on the preservice science teachers’ (PST) beliefs on their TPACK level. The participants were three teachers with the low, medium, and high level of TPACK confidence. Content analysis was utilized to analyze the data, which were collected by individual semi-structured interviews with the participants about their lesson plans. The study first discussed each case, then compared features and relations across cases. The researchers found that there was a positive relationship between PST’s TPACK confidence and TPACK level; when PST had higher TPACK confidence, the participant had a higher competent TPACK level and vice versa.

Recent Dissertations Using Multiple Case Study Methodology

Milholland, E. S. (2015). A multiple case study of instructors utilizing Classroom Response Systems (CRS) to achieve pedagogical goals . Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3706380)

The researcher of this study critiques the use of Classroom Responses Systems by five instructors who employed this program five years ago in their classrooms. The researcher conducted the multiple-case study methodology and categorized themes. He interviewed each instructor with questions about their initial pedagogical goals, the changes in pedagogy during teaching, and the teaching techniques individuals used while practicing the CRS. The researcher used the multiple-case study with five instructors. He found that all instructors changed their goals during employing CRS; they decided to reduce the time of lecturing and to spend more time engaging students in interactive activities. This study also demonstrated that CRS was useful for the instructors to achieve multiple learning goals; all the instructors provided examples of the positive aspect of implementing CRS in their classrooms.

Li, C. L. (2010). The emergence of fairy tale literacy: A multiple case study on promoting critical literacy of children through a juxtaposed reading of classic fairy tales and their contemporary disruptive variants . Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3572104)

To explore how children’s development of critical literacy can be impacted by their reactions to fairy tales, the author conducted a multiple-case study with 4 cases, in which each child was a unit of analysis. Two Chinese immigrant children (a boy and a girl) and two American children (a boy and a girl) at the second or third grade were recruited in the study. The data were collected through interviews, discussions on fairy tales, and drawing pictures. The analysis was conducted within both individual cases and cross cases. Across four cases, the researcher found that the young children’s’ knowledge of traditional fairy tales was built upon mass-media based adaptations. The children believed that the representations on mass-media were the original stories, even though fairy tales are included in the elementary school curriculum. The author also found that introducing classic versions of fairy tales increased children’s knowledge in the genre’s origin, which would benefit their understanding of the genre. She argued that introducing fairy tales can be the first step to promote children’s development of critical literacy.

Asher, K. C. (2014). Mediating occupational socialization and occupational individuation in teacher education: A multiple case study of five elementary pre-service student teachers . Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3671989)

This study portrayed five pre-service teachers’ teaching experience in their student teaching phase and explored how pre-service teachers mediate their occupational socialization with occupational individuation. The study used the multiple-case study design and recruited five pre-service teachers from a Midwestern university as five cases. Qualitative data were collected through interviews, classroom observations, and field notes. The author implemented the case study analysis and found five strategies that the participants used to mediate occupational socialization with occupational individuation. These strategies were: 1) hindering from practicing their beliefs, 2) mimicking the styles of supervising teachers, 3) teaching in the ways in alignment with school’s existing practice, 4) enacting their own ideas, and 5) integrating and balancing occupational socialization and occupational individuation. The study also provided recommendations and implications to policymakers and educators in teacher education so that pre-service teachers can be better supported.

Multiple Case Studies Copyright © 2019 by Nadia Alqahtani and Pengtong Qu is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• Privacy Policy

Home » Case Study – Methods, Examples and Guide

Case Study – Methods, Examples and Guide

Table of Contents

A case study is a research method that involves an in-depth examination and analysis of a particular phenomenon or case, such as an individual, organization, community, event, or situation.

It is a qualitative research approach that aims to provide a detailed and comprehensive understanding of the case being studied. Case studies typically involve multiple sources of data, including interviews, observations, documents, and artifacts, which are analyzed using various techniques, such as content analysis, thematic analysis, and grounded theory. The findings of a case study are often used to develop theories, inform policy or practice, or generate new research questions.

Types of Case Study

Types and Methods of Case Study are as follows:

Single-Case Study

A single-case study is an in-depth analysis of a single case. This type of case study is useful when the researcher wants to understand a specific phenomenon in detail.

For Example , A researcher might conduct a single-case study on a particular individual to understand their experiences with a particular health condition or a specific organization to explore their management practices. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a single-case study are often used to generate new research questions, develop theories, or inform policy or practice.

Multiple-Case Study

A multiple-case study involves the analysis of several cases that are similar in nature. This type of case study is useful when the researcher wants to identify similarities and differences between the cases.

For Example, a researcher might conduct a multiple-case study on several companies to explore the factors that contribute to their success or failure. The researcher collects data from each case, compares and contrasts the findings, and uses various techniques to analyze the data, such as comparative analysis or pattern-matching. The findings of a multiple-case study can be used to develop theories, inform policy or practice, or generate new research questions.

Exploratory Case Study

An exploratory case study is used to explore a new or understudied phenomenon. This type of case study is useful when the researcher wants to generate hypotheses or theories about the phenomenon.

For Example, a researcher might conduct an exploratory case study on a new technology to understand its potential impact on society. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as grounded theory or content analysis. The findings of an exploratory case study can be used to generate new research questions, develop theories, or inform policy or practice.

Descriptive Case Study

A descriptive case study is used to describe a particular phenomenon in detail. This type of case study is useful when the researcher wants to provide a comprehensive account of the phenomenon.

For Example, a researcher might conduct a descriptive case study on a particular community to understand its social and economic characteristics. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a descriptive case study can be used to inform policy or practice or generate new research questions.

Instrumental Case Study

An instrumental case study is used to understand a particular phenomenon that is instrumental in achieving a particular goal. This type of case study is useful when the researcher wants to understand the role of the phenomenon in achieving the goal.

For Example, a researcher might conduct an instrumental case study on a particular policy to understand its impact on achieving a particular goal, such as reducing poverty. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of an instrumental case study can be used to inform policy or practice or generate new research questions.

Case Study Data Collection Methods

Here are some common data collection methods for case studies:

Interviews involve asking questions to individuals who have knowledge or experience relevant to the case study. Interviews can be structured (where the same questions are asked to all participants) or unstructured (where the interviewer follows up on the responses with further questions). Interviews can be conducted in person, over the phone, or through video conferencing.

Observations

Observations involve watching and recording the behavior and activities of individuals or groups relevant to the case study. Observations can be participant (where the researcher actively participates in the activities) or non-participant (where the researcher observes from a distance). Observations can be recorded using notes, audio or video recordings, or photographs.

Documents can be used as a source of information for case studies. Documents can include reports, memos, emails, letters, and other written materials related to the case study. Documents can be collected from the case study participants or from public sources.

Surveys involve asking a set of questions to a sample of individuals relevant to the case study. Surveys can be administered in person, over the phone, through mail or email, or online. Surveys can be used to gather information on attitudes, opinions, or behaviors related to the case study.

Artifacts are physical objects relevant to the case study. Artifacts can include tools, equipment, products, or other objects that provide insights into the case study phenomenon.

How to conduct Case Study Research

Conducting a case study research involves several steps that need to be followed to ensure the quality and rigor of the study. Here are the steps to conduct case study research:

• Define the research questions: The first step in conducting a case study research is to define the research questions. The research questions should be specific, measurable, and relevant to the case study phenomenon under investigation.
• Select the case: The next step is to select the case or cases to be studied. The case should be relevant to the research questions and should provide rich and diverse data that can be used to answer the research questions.
• Collect data: Data can be collected using various methods, such as interviews, observations, documents, surveys, and artifacts. The data collection method should be selected based on the research questions and the nature of the case study phenomenon.
• Analyze the data: The data collected from the case study should be analyzed using various techniques, such as content analysis, thematic analysis, or grounded theory. The analysis should be guided by the research questions and should aim to provide insights and conclusions relevant to the research questions.
• Draw conclusions: The conclusions drawn from the case study should be based on the data analysis and should be relevant to the research questions. The conclusions should be supported by evidence and should be clearly stated.
• Validate the findings: The findings of the case study should be validated by reviewing the data and the analysis with participants or other experts in the field. This helps to ensure the validity and reliability of the findings.
• Write the report: The final step is to write the report of the case study research. The report should provide a clear description of the case study phenomenon, the research questions, the data collection methods, the data analysis, the findings, and the conclusions. The report should be written in a clear and concise manner and should follow the guidelines for academic writing.

Examples of Case Study

Here are some examples of case study research:

• The Hawthorne Studies : Conducted between 1924 and 1932, the Hawthorne Studies were a series of case studies conducted by Elton Mayo and his colleagues to examine the impact of work environment on employee productivity. The studies were conducted at the Hawthorne Works plant of the Western Electric Company in Chicago and included interviews, observations, and experiments.
• The Stanford Prison Experiment: Conducted in 1971, the Stanford Prison Experiment was a case study conducted by Philip Zimbardo to examine the psychological effects of power and authority. The study involved simulating a prison environment and assigning participants to the role of guards or prisoners. The study was controversial due to the ethical issues it raised.
• The Challenger Disaster: The Challenger Disaster was a case study conducted to examine the causes of the Space Shuttle Challenger explosion in 1986. The study included interviews, observations, and analysis of data to identify the technical, organizational, and cultural factors that contributed to the disaster.
• The Enron Scandal: The Enron Scandal was a case study conducted to examine the causes of the Enron Corporation’s bankruptcy in 2001. The study included interviews, analysis of financial data, and review of documents to identify the accounting practices, corporate culture, and ethical issues that led to the company’s downfall.
• The Fukushima Nuclear Disaster : The Fukushima Nuclear Disaster was a case study conducted to examine the causes of the nuclear accident that occurred at the Fukushima Daiichi Nuclear Power Plant in Japan in 2011. The study included interviews, analysis of data, and review of documents to identify the technical, organizational, and cultural factors that contributed to the disaster.

Application of Case Study

Case studies have a wide range of applications across various fields and industries. Here are some examples:

Business and Management

Case studies are widely used in business and management to examine real-life situations and develop problem-solving skills. Case studies can help students and professionals to develop a deep understanding of business concepts, theories, and best practices.

Case studies are used in healthcare to examine patient care, treatment options, and outcomes. Case studies can help healthcare professionals to develop critical thinking skills, diagnose complex medical conditions, and develop effective treatment plans.

Case studies are used in education to examine teaching and learning practices. Case studies can help educators to develop effective teaching strategies, evaluate student progress, and identify areas for improvement.

Social Sciences

Case studies are widely used in social sciences to examine human behavior, social phenomena, and cultural practices. Case studies can help researchers to develop theories, test hypotheses, and gain insights into complex social issues.

Law and Ethics

Case studies are used in law and ethics to examine legal and ethical dilemmas. Case studies can help lawyers, policymakers, and ethical professionals to develop critical thinking skills, analyze complex cases, and make informed decisions.

Purpose of Case Study

The purpose of a case study is to provide a detailed analysis of a specific phenomenon, issue, or problem in its real-life context. A case study is a qualitative research method that involves the in-depth exploration and analysis of a particular case, which can be an individual, group, organization, event, or community.

The primary purpose of a case study is to generate a comprehensive and nuanced understanding of the case, including its history, context, and dynamics. Case studies can help researchers to identify and examine the underlying factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and detailed understanding of the case, which can inform future research, practice, or policy.

Case studies can also serve other purposes, including:

• Illustrating a theory or concept: Case studies can be used to illustrate and explain theoretical concepts and frameworks, providing concrete examples of how they can be applied in real-life situations.
• Developing hypotheses: Case studies can help to generate hypotheses about the causal relationships between different factors and outcomes, which can be tested through further research.
• Providing insight into complex issues: Case studies can provide insights into complex and multifaceted issues, which may be difficult to understand through other research methods.
• Informing practice or policy: Case studies can be used to inform practice or policy by identifying best practices, lessons learned, or areas for improvement.

Advantages of Case Study Research

There are several advantages of case study research, including:

• In-depth exploration: Case study research allows for a detailed exploration and analysis of a specific phenomenon, issue, or problem in its real-life context. This can provide a comprehensive understanding of the case and its dynamics, which may not be possible through other research methods.
• Rich data: Case study research can generate rich and detailed data, including qualitative data such as interviews, observations, and documents. This can provide a nuanced understanding of the case and its complexity.
• Holistic perspective: Case study research allows for a holistic perspective of the case, taking into account the various factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and comprehensive understanding of the case.
• Theory development: Case study research can help to develop and refine theories and concepts by providing empirical evidence and concrete examples of how they can be applied in real-life situations.
• Practical application: Case study research can inform practice or policy by identifying best practices, lessons learned, or areas for improvement.
• Contextualization: Case study research takes into account the specific context in which the case is situated, which can help to understand how the case is influenced by the social, cultural, and historical factors of its environment.

Limitations of Case Study Research

There are several limitations of case study research, including:

• Limited generalizability : Case studies are typically focused on a single case or a small number of cases, which limits the generalizability of the findings. The unique characteristics of the case may not be applicable to other contexts or populations, which may limit the external validity of the research.
• Biased sampling: Case studies may rely on purposive or convenience sampling, which can introduce bias into the sample selection process. This may limit the representativeness of the sample and the generalizability of the findings.
• Subjectivity: Case studies rely on the interpretation of the researcher, which can introduce subjectivity into the analysis. The researcher’s own biases, assumptions, and perspectives may influence the findings, which may limit the objectivity of the research.
• Limited control: Case studies are typically conducted in naturalistic settings, which limits the control that the researcher has over the environment and the variables being studied. This may limit the ability to establish causal relationships between variables.
• Time-consuming: Case studies can be time-consuming to conduct, as they typically involve a detailed exploration and analysis of a specific case. This may limit the feasibility of conducting multiple case studies or conducting case studies in a timely manner.
• Resource-intensive: Case studies may require significant resources, including time, funding, and expertise. This may limit the ability of researchers to conduct case studies in resource-constrained settings.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Descriptive Research Design – Types, Methods and...

Observational Research – Methods and Guide

Textual Analysis – Types, Examples and Guide

Exploratory Research – Types, Methods and...

Explanatory Research – Types, Methods, Guide

Transformative Design – Methods, Types, Guide

Breadcrumbs Section. Click here to navigate to respective pages.

The Multiple Case Study Design

DOI link for The Multiple Case Study Design

Get Citation

Most organizations today operate in volatile economic and social environments and qualitative research plays an essential role in investigating leadership and management problems. This unique volume offers novice and experienced researchers a brief, student-centric research methods text specifically devoted to the multiple case study design.

The multiple case study design is a valuable qualitative research tool in studying the links between the personal, social, behavioral, psychological, organizational, cultural, and environmental factors that guide organizational and leadership development. Case study research is essential for the in-depth study of participants' perspectives on the phenomenon within its natural context. Rigorously designed management and leadership case studies in the extant literature have a central focus on individual managers' and leaders' stories and their perceptions of the broader forces operating within and outside their organizations.

This is a comprehensive methodology book exploring the multiple case study design with step-by-step and easily accessible guidelines on the topic, making it especially valuable to researchers, academics, and students in the areas of business, management, and leadership.

TABLE OF CONTENTS

Chapter 1 | 6  pages, a refresher on the philosophical foundations of academic research, chapter 2 | 6  pages, research methodologies, chapter 3 | 3  pages, the role of theory in qualitative research, chapter 4 | 6  pages, how does the novice researcher design a multiple case study, chapter 5 | 5  pages, the advantage of the multiple case study design for management researchers, chapter 6 | 6  pages, applying data collection methods in multiple case study research, chapter 7 | 9  pages, the data analysis process for multiple case study research, chapter 8 | 3  pages, extending theory with multiple case study design, chapter 9 | 7  pages, incorporating multiple case design and methodologies into teaching and professional practice, chapter 10 | 9  pages, writing and publishing multiple case study research, chapter 11 | 2  pages, concluding thoughts.

• Privacy Policy
• Terms & Conditions
• Cookie Policy
• Taylor & Francis Online
• Taylor & Francis Group
• Students/Researchers
• Librarians/Institutions

Connect with us

Registered in England & Wales No. 3099067 5 Howick Place | London | SW1P 1WG © 2024 Informa UK Limited

• Section 2: Home
• Developing the Quantitative Research Design
• Qualitative Descriptive Design
• Design and Development Research (DDR) For Instructional Design
• Qualitative Narrative Inquiry Research
• Action Research Resource
• Case Study Design in an Applied Doctorate

Qualitative Research Designs

Case study design, using case study design in the applied doctoral experience (ade), applicability of case study design to applied problem of practice, case study design references.

• SAGE Research Methods
• Research Examples (SAGE) This link opens in a new window
• Dataset Examples (SAGE) This link opens in a new window
• IRB Resource Center This link opens in a new window

The field of qualitative research there are a number of research designs (also referred to as “traditions” or “genres”), including case study, phenomenology, narrative inquiry, action research, ethnography, grounded theory, as well as a number of critical genres including Feminist theory, indigenous research, critical race theory and cultural studies. The choice of research design is directly tied to and must be aligned with your research problem and purpose. As Bloomberg & Volpe (2019) explain:

Choice of research design is directly tied to research problem and purpose. As the researcher, you actively create the link among problem, purpose, and design through a process of reflecting on problem and purpose, focusing on researchable questions, and considering how to best address these questions. Thinking along these lines affords a research study methodological congruence (p. 38).

Case study is an in-depth exploration from multiple perspectives of a bounded social phenomenon, be this a social system such as a program, event, institution, organization, or community (Stake, 1995, 2005; Yin, 2018). Case study is employed across disciplines, including education, health care, social work, sociology, and organizational studies. The purpose is to generate understanding and deep insights to inform professional practice, policy development, and community or social action (Bloomberg 2018).

Yin (2018) and Stake (1995, 2005), two of the key proponents of case study methodology, use different terms to describe case studies. Yin categorizes case studies as exploratory or descriptive . The former is used to explore those situations in which the intervention being evaluated has no clear single set of outcomes. The latter is used to describe an intervention or phenomenon and the real-life context in which it occurred. Stake identifies case studies as intrinsic or instrumental , and he proposes that a primary distinction in designing case studies is between single and multiple (or collective) case study designs. A single case study may be an instrumental case study (research focuses on an issue or concern in one bounded case) or an intrinsic case study (the focus is on the case itself because the case presents a unique situation). A longitudinal case study design is chosen when the researcher seeks to examine the same single case at two or more different points in time or to capture trends over time. A multiple case study design is used when a researcher seeks to determine the prevalence or frequency of a particular phenomenon. This approach is useful when cases are used for purposes of a cross-case analysis in order to compare, contrast, and synthesize perspectives regarding the same issue. The focus is on the analysis of diverse cases to determine how these confirm the findings within or between cases, or call the findings into question.

Case study affords significant interaction with research participants, providing an in-depth picture of the phenomenon (Bloomberg & Volpe, 2019). Research is extensive, drawing on multiple methods of data collection, and involves multiple data sources. Triangulation is critical in attempting to obtain an in-depth understanding of the phenomenon under study and adds rigor, breadth, and depth to the study and provides corroborative evidence of the data obtained. Analysis of data can be holistic or embedded—that is, dealing with the whole or parts of the case (Yin, 2018). With multiple cases the typical analytic strategy is to provide detailed description of themes within each case (within-case analysis), followed by thematic analysis across cases (cross-case analysis), providing insights regarding how individual cases are comparable along important dimensions. Research culminates in the production of a detailed description of a setting and its participants, accompanied by an analysis of the data for themes or patterns (Stake, 1995, 2005; Yin, 2018). In addition to thick, rich description, the researcher’s interpretations, conclusions, and recommendations contribute to the reader’s overall understanding of the case study.

Analysis of findings should show that the researcher has attended to all the data, should address the most significant aspects of the case, and should demonstrate familiarity with the prevailing thinking and discourse about the topic. The goal of case study design (as with all qualitative designs) is not generalizability but rather transferability —that is, how (if at all) and in what ways understanding and knowledge can be applied in similar contexts and settings. The qualitative researcher attempts to address the issue of transferability by way of thick, rich description that will provide the basis for a case or cases to have relevance and potential application across a broader context.

Qualitative research methods ask the questions of "what" and "how" a phenomenon is understood in a real-life context (Bloomberg & Volpe, 2019). In the education field, qualitative research methods uncover educational experiences and practices because qualitative research allows the researcher to reveal new knowledge and understanding. Moreover, qualitative descriptive case studies describe, analyze and interpret events that explain the reasoning behind specific phenomena (Bloomberg, 2018). As such, case study design can be the foundation for a rigorous study within the Applied Doctoral Experience (ADE).

Case study design is an appropriate research design to consider when conceptualizing and conducting a dissertation research study that is based on an applied problem of practice with inherent real-life educational implications. Case study researchers study current, real-life cases that are in progress so that they can gather accurate information that is current. This fits well with the ADE program, as students are typically exploring a problem of practice. Because of the flexibility of the methods used, a descriptive design provides the researcher with the opportunity to choose data collection methods that are best suited to a practice-based research purpose, and can include individual interviews, focus groups, observation, surveys, and critical incident questionnaires. Methods are triangulated to contribute to the study’s trustworthiness. In selecting the set of data collection methods, it is important that the researcher carefully consider the alignment between research questions and the type of data that is needed to address these. Each data source is one piece of the “puzzle,” that contributes to the researcher’s holistic understanding of a phenomenon. The various strands of data are woven together holistically to promote a deeper understanding of the case and its application to an educationally-based problem of practice.

Research studies within the Applied Doctoral Experience (ADE) will be practical in nature and focus on problems and issues that inform educational practice.  Many of the types of studies that fall within the ADE framework are exploratory, and align with case study design. Case study design fits very well with applied problems related to educational practice, as the following set of examples illustrate:

Elementary Bilingual Education Teachers’ Self-Efficacy in Teaching English Language Learners: A Qualitative Case Study

The problem to be addressed in the proposed study is that some elementary bilingual education teachers’ beliefs about their lack of preparedness to teach the English language may negatively impact the language proficiency skills of Hispanic ELLs (Ernst-Slavit & Wenger, 2016; Fuchs et al., 2018; Hoque, 2016). The purpose of the proposed qualitative descriptive case study was to explore the perspectives and experiences of elementary bilingual education teachers regarding their perceived lack of preparedness to teach the English language and how this may impact the language proficiency of Hispanic ELLs.

Exploring Minority Teachers Experiences Pertaining to their Value in Education: A Single Case Study of Teachers in New York City

The problem is that minority K-12 teachers are underrepresented in the United States, with research indicating that school leaders and teachers in schools that are populated mainly by black students, staffed mostly by white teachers who may be unprepared to deal with biases and stereotypes that are ingrained in schools (Egalite, Kisida, & Winters, 2015; Milligan & Howley, 2015). The purpose of this qualitative exploratory single case study was to develop a clearer understanding of minority teachers’ experiences concerning the under-representation of minority K-12 teachers in urban school districts in the United States since there are so few of them.

Exploring the Impact of an Urban Teacher Residency Program on Teachers’ Cultural Intelligence: A Qualitative Case Study

The problem to be addressed by this case study is that teacher candidates often report being unprepared and ill-equipped to effectively educate culturally diverse students (Skepple, 2015; Beutel, 2018). The purpose of this study was to explore and gain an in-depth understanding of the perceived impact of an urban teacher residency program in urban Iowa on teachers’ cultural competence using the cultural intelligence (CQ) framework (Earley & Ang, 2003).

Qualitative Case Study that Explores Self-Efficacy and Mentorship on Women in Academic Administrative Leadership Roles

The problem was that female school-level administrators might be less likely to experience mentorship, thereby potentially decreasing their self-efficacy (Bing & Smith, 2019; Brown, 2020; Grant, 2021). The purpose of this case study was to determine to what extent female school-level administrators in the United States who had a mentor have a sense of self-efficacy and to examine the relationship between mentorship and self-efficacy.

Suburban Teacher and Administrator Perceptions of Culturally Responsive Teaching to Promote Connectedness in Students of Color: A Qualitative Case Study

The problem to be addressed in this study is the racial discrimination experienced by students of color in suburban schools and the resulting negative school experience (Jara & Bloomsbury, 2020; Jones, 2019; Kohli et al., 2017; Wandix-White, 2020). The purpose of this case study is to explore how culturally responsive practices can counteract systemic racism and discrimination in suburban schools thereby meeting the needs of students of color by creating positive learning experiences. 

As you can see, all of these studies were well suited to qualitative case study design. In each of these studies, the applied research problem and research purpose were clearly grounded in educational practice as well as directly aligned with qualitative case study methodology. In the Applied Doctoral Experience (ADE), you will be focused on addressing or resolving an educationally relevant research problem of practice. As such, your case study, with clear boundaries, will be one that centers on a real-life authentic problem in your field of practice that you believe is in need of resolution or improvement, and that the outcome thereof will be educationally valuable.

Bloomberg, L. D. (2018). Case study method. In B. B. Frey (Ed.), The SAGE Encyclopedia of educational research, measurement, and evaluation (pp. 237–239). SAGE. https://go.openathens.net/redirector/nu.edu?url=https%3A%2F%2Fmethods.sagepub.com%2FReference%2Fthe-sage-encyclopedia-of-educational-research-measurement-and-evaluation%2Fi4294.xml

Bloomberg, L. D. & Volpe, M. (2019). Completing your qualitative dissertation: A road map from beginning to end . (4th Ed.). SAGE.

Stake, R. E. (1995). The art of case study research. SAGE.

Stake, R. E. (2005). Qualitative case studies. In N. K. Denzin and Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., pp. 443–466). SAGE.

Yin, R. (2018). Case study research and applications: Designs and methods. SAGE.

• << Previous: Action Research Resource
• Next: SAGE Research Methods >>
• Last Updated: Jul 28, 2023 8:05 AM
• URL: https://resources.nu.edu/c.php?g=1013605

© Copyright 2024 National University. All Rights Reserved.

Privacy Policy | Consumer Information

• Case Reports

Qualitative Case Study Methodology: Study Design and Implementation for Novice Researchers

• January 2010
• The Qualitative Report 13(4)

• McMaster University

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Hasan Savaş

• Kürşat İlgün

• Celal Aramaz

• Anahtar Kelimeler İran
• Türkiye Özet

• Bob Algozzine

• B J Breitmayer
• Y.S. Lincoln
• ADV NURS SCI
• Margarete Sandelowski

• John W. Scheib
• Robert E. Stake
• Thomas J. Richards

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

1st Edition

The Multiple Case Study Design Methodology and Application for Management Education

• Taylor & Francis eBooks (Institutional Purchase) Opens in new tab or window

Description

Most organizations today operate in volatile economic and social environments and qualitative research plays an essential role in investigating leadership and management problems. This unique volume offers novice and experienced researchers a brief, student-centric research methods text specifically devoted to the multiple case study design. The multiple case study design is a valuable qualitative research tool in studying the links between the personal, social, behavioral, psychological, organizational, cultural, and environmental factors that guide organizational and leadership development. Case study research is essential for the in-depth study of participants' perspectives on the phenomenon within its natural context. Rigorously designed management and leadership case studies in the extant literature have a central focus on individual managers' and leaders' stories and their perceptions of the broader forces operating within and outside their organizations. This is a comprehensive methodology book exploring the multiple case study design with step-by-step and easily accessible guidelines on the topic, making it especially valuable to researchers, academics, and students in the areas of business, management, and leadership.

Table of Contents

Daphne Halkias is Professor and Distinguished Research Fellow at École des Ponts Business School in Paris, France. Michael Neubert is Associate Professor in Business and Management Studies and a Member of the Academic Council at UIBS in Zurich, Switzerland. Paul W. Thurman is Professor of Management and Analytics at Columbia University's Mailman School of Public Health, New York, USA. Nicholas Harkiolakis is on the Faculty of the School of Technology at Northcentral University, San Diego, California, USA.

About VitalSource eBooks

VitalSource is a leading provider of eBooks.

• Access your materials anywhere, at anytime.
• Customer preferences like text size, font type, page color and more.
• Take annotations in line as you read.

Multiple eBook Copies

This eBook is already in your shopping cart. If you would like to replace it with a different purchasing option please remove the current eBook option from your cart.

Book Preview

The country you have selected will result in the following:

• Product pricing will be adjusted to match the corresponding currency.
• The title Perception will be removed from your cart because it is not available in this region.

From David E. Gray \(2014\). Doing Research in the Real World \(3rd ed.\) London, UK: Sage.

Multiple Case Research Design

• First Online: 10 November 2021

Cite this chapter

• Stefan Hunziker 3 &
• Michael Blankenagel 3  

5652 Accesses

7 Citations

This chapter addresses the peculiarities, characteristics, and major fallacies of multiple case research designs. The major advantage of multiple case research lies in cross-case analysis. A multiple case research design shifts the focus from understanding a single case to the differences and similarities between cases. Thus, it is not just conducting more (second, third, etc.) case studies. Rather, it is the next step in developing a theory about factors driving differences and similarities. Also, researchers find relevant information on how to write a multiple case research design paper and learn about typical methodologies used for this research design. The chapter closes with referring to overlapping and adjacent research designs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Bruns, W. J., & McKinnon, S. M. (1993). Information and managers: A field study. Journal of Management Accounting Research, 5 , 84–108.

Google Scholar  

Eisenhardt, K. M., & Graebner, M. E. (2007). Theory building from cases: Opportunities and challenges. Academy of Management Journal, 50 (1), 25–32.

Article   Google Scholar  

Ferreira, L. D. & Merchant, K. A. (1992). Field research in management accounting and control: A review and evaluation . Emerald Group Publishing Limited.

Keating, P. J. (1995). A framework for classifying and evaluating the theoretical contributions of case research in management accounting. Journal of Management Accounting Research, 7 , 66–86.

Lillis, A. M., & Mundy, J. (2005). Cross-sectional field studies in management accounting research—closing the gaps between surveys and case studies. Journal of Management Accounting Research, 17 (1), 119–141.

Ragin, C. C. (2009). Reflections on casing and case-oriented research (pp. 522–534). The Sage handbook of case-based method.

Ridder, H.-G. (2017). The theory contribution of case study research designs. Business Research, 10 (2), 281–305.

Stake, R. E. (2005). Qualitative case studies. In N.K. Denzin & Y.S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., pp. 443–466).

Vaughan, D. (1992). Theory elaboration: The heuristics of case analysis. What is a case?. In C.C. Ragin & H.S. Becker (Eds.), Exploring the foundations of social inquiry (pp. 173–202). Cambridge University Press.

Walsham, G. (2006). Doing interpretive research. European Journal of Information Systems, 15 (3), 320–330.

Yin, R. K. (2014). Case study research. Design and methods (5th ed.). SAGE.

Download references

Author information

Authors and affiliations.

Wirtschaft/IFZ – Campus Zug-Rotkreuz, Hochschule Luzern, Zug-Rotkreuz, Zug , Switzerland

Stefan Hunziker & Michael Blankenagel

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Stefan Hunziker .

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Fachmedien Wiesbaden GmbH, part of Springer Nature

About this chapter

Hunziker, S., Blankenagel, M. (2021). Multiple Case Research Design. In: Research Design in Business and Management. Springer Gabler, Wiesbaden. https://doi.org/10.1007/978-3-658-34357-6_9

Download citation

DOI : https://doi.org/10.1007/978-3-658-34357-6_9

Published : 10 November 2021

Publisher Name : Springer Gabler, Wiesbaden

Print ISBN : 978-3-658-34356-9

Online ISBN : 978-3-658-34357-6

eBook Packages : Business and Economics (German Language)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Integrations

What's new?

In-Product Prompts

Participant Management

Interview Studies

Prototype Testing

Card Sorting

Tree Testing

Live Website Testing

Automated Reports

Templates Gallery

Choose from our library of pre-built mazes to copy, customize, and share with your own users

Browse all templates

Financial Services

Tech & Software

Product Designers

Product Managers

User Researchers

By use case

Concept & Idea Validation

Wireframe & Usability Test

Content & Copy Testing

Feedback & Satisfaction

Content Hub

Educational resources for product, research and design teams

Explore all resources

Question Bank

Research Maturity Model

Guides & Reports

Help Center

Future of User Research Report

The Optimal Path Podcast

User Research

Aug 15, 2024 • 13 minutes read

Mixed methods research explained: Combine data like a pro

From heatmaps to interviews, here’s how to blend qualitative and quantitative data for holistic user insights.

Ella Webber

Mixed methods research is one of the most popular and powerful UX research approaches—blending numbers with narrative to garner a holistic understanding of your product or research question.

Whether you’re in UX research and design, education, healthcare, or social sciences, mixed methods research can help you find insights and make better decisions.

Read on for a breakdown of what mixed methods are, their strengths and weaknesses, when to use them, and how to analyze the data.

UX research made easy

Explore the power of combining quantitative and qualitative research to discover new insights and test final solutions.

What is mixed methods research and when should you use it?

Mixed methods research involves collecting, analyzing, and integrating both quantitative and qualitative UX research methods within a single study. It is unique to other UX research techniques in that it combines data types, encouraging product teams to use qualitative feedback to explain the story behind quantitative numbers.

• Quantitative data can come from UX surveys , product analytics , usability testing , experiments, or statistical databases and provide broad numerical insights
• Qualitative data is gathered through user interviews , focus groups, or contextual inquiries and offers a deep, contextual understanding of the subject matter

Why use a mixed methods approach?

The power of mixed methods research is simple: it allows you to combine the best parts of both types of data—quantitative research methods, like surveys, give you broad trends, while qualitative methods, such as interviews, dig deep into personal experiences.

Anthony J. Onwuegbuzie and R. Burke Johnson, in Mixed Methods Research: A Research Paradigm Whose Time Has Come , highlight how blending these methods allows researchers to leverage the strengths of both approaches. They identify mixed methods research as one of the “three core research paradigms: qualitative, quantitative, and mixed methods.”

Like any technique, however, mixed methods research has both strengths and weaknesses to consider.

When should you use mixed methods research?

Mixed UX research methods are useful when neither qualitative nor quantitative data alone can fully answer your research question . Evaluative research further helps to assess the effectiveness of your mixed method research findings and ensure they meet user needs.

For example, use mixed methods research when:

• You need to go beyond numbers (generalizability): Quantitative methods, like surveys, provide broad trends and patterns that are relevant to a wider population. For example, a survey might show that most users enjoy a new app feature, but it won’t capture why some users might dislike it.
• The why matters (contextualization): Mixed methods allow you to put numerical findings in context, adding rich detail to your conclusions. For example, if analytics show that users are spending less time on your app (quantitative), interviews can help you understand the reasons behind this behavior, such as frustration with a recent update or a lack of engaging content (qualitative).
• Credibility is important (credibility and triangulation): When both data types converge on the same conclusion, it strengthens your findings. For example, the combined evidence is more credible if survey data indicates that most users prefer a particular software interface and focus groups echo this preference.
• You need to track changes (developmental purposes): Mixed methods are invaluable when one type of data informs the other. For example, initial qualitative research with a small group of beta testers can uncover key issues and user needs, which can then be explored quantitatively with a larger user base to see how widespread these issues are.
• Understand complex issues (complementary insights): Different data types can offer complementary insights. For example, in a study on software usability, quantitative data might show a drop in task completion rates, while qualitative data reveals specific pain points and user frustrations. This combined approach can guide more effective design improvements.

What are the types of mixed methods research design?

The type of mixed methods research design you choose depends on your research goals, the timing of data collection, and each data type. Here are some key factors to consider:

• Your research approach: Are you trying to understand existing findings (explanatory) or dig deeper into a topic (exploratory)?
• Your research questions: Do your questions need big-picture answers (like how many users are happy) and detailed explanations (like why some users are unhappy)?
• Existing data availability: Is there any existing information you can use from previous studies or a research repository (like user demographics)?
• Data you can collect yourself: What kind of in-depth information do you need to gather from users (through interviews, testing, etc.)?

Whether you're a data diver or a narrative novelist, understanding these research methods can make your studies more dynamic and insightful.

📚 A UX research repository is crucial for keeping track of research findings. You need a centralized database to store and manage all your qualitative and quantitative data. This ensures that your research is organized, accessible, and reusable for future studies.

Let’s look at the most common types of mixed methods research design:

Convergent parallel

Convergent parallel design involves collecting qualitative and quantitative data simultaneously but analyzing them separately. The primary goal is to merge the two datasets to provide a complete understanding of the research problem.

For example, let’s say you want to study user satisfaction with a new mobile app. Here’s how you might use the convergent parallel design:

• Qualitative results: Conduct in-depth user interviews with 30 participants to gather detailed insights into their experiences and perceptions of the app. Plus, analyze 200 user reviews from app stores. You might use prompts like, "What features do you find most valuable?" and "Please describe any difficulties you've experienced while using the app."
• Quantitative study: Use analytics data to measure user engagement metrics like session duration and feature usage, then distribute UX surveys to gather quantitative satisfaction scores.

Concurrent embedded design

Embedded design is a mixed methods research approach where qualitative and quantitative data are collected simultaneously, but one type of data is supplementary to the other.

The secondary data provides additional context and can help explain or clarify the primary findings. This approach is particularly beneficial when time or resources are limited, as it allows for a more comprehensive analysis without doubling the workload.

Explanatory sequential design

Explanatory sequential design is a popular mixed methods research approach introduced by John W. Creswell and Vicki L. Plano Clark. This research design involves collecting and analyzing quantitative data first, followed by qualitative data collection and analysis.

According to Creswell, this approach is particularly useful when researchers need to explain relationships found in quantitative data.

The process typically involves two phases:

• Quantitative phase: This involves collecting numerical data through methods like surveys or experiments. The goal here is to identify patterns, trends, or relationships.
• Qualitative phase: Qualitative phase: After analyzing the quantitative data, researchers collect qualitative data with qualitative approaches, like interviews or focus groups, to provide deeper insights. This phase helps explain the ‘why’ or ‘how’ behind the quantitative findings.

Creswell emphasizes that one of the strengths of this design is its simple structure, making it easy for researchers to manage and for audiences to understand the research process and findings.

Exploratory sequential design

Exploratory sequential design begins with qualitative data collection and analysis, followed by quantitative data collection. This immersive approach helps generate rich, detailed data that lays a strong foundation for the subsequent quantitative analysis.

For example, let’s say a researcher wants to understand why people don't meditate regularly. They could start with generative research techniques , like conducting workshops where participants discuss their daily routines and barriers to meditation. These qualitative insights reveal underlying themes and patterns, like time constraints and lack of motivation.

Next, the researcher analyzes these qualitative data to identify key factors impacting wellbeing habits. Based on these insights, they develop a survey to quantitatively measure how widespread these barriers are among a larger population.

So, that’s how you collect data. But how do you analyze it? Unsurprisingly, there are multiple analysis and interpretation methods commonly used in mixed methods research. Let’s look at some.

How to analyze mixed methods research data: 3 Ways to combine qualitative and quantitative data

Combining different types of research data can add credibility to your research findings. Let’s look at how to conduct mixed methods research:

Triangulation protocol

Following a thread, mixed methods matrix.

The triangulation protocol in mixed methods research is a systematic way to use multiple data sources, techniques, or perspectives to get a clear understanding of a research problem. The goal is to capitalize on the strengths of both types of data while minimizing their individual weaknesses.

Let's say you want to conduct a study aiming to evaluate the effectiveness of a new educational program on student performance, and you arrive at the following datasets:

• Quantitative finding: 80% of students improved their math scores after the program
• Qualitative finding: Students reported that interactive activities helped them understand math concepts better

When you merge these findings, the research concludes that the interactive activities (identified qualitatively) are likely a significant factor contributing to the improved scores (quantitatively).

The following a thread method allows researchers to trace a specific theme or concept across both qualitative and quantitative data sets.

Here’s how it works:

• Identify key themes: Begin by identifying key themes or variables that are central to your research questions. These themes will serve as the ‘threads’ you’ll follow through your data.
• Extracting data: Extract relevant data segments related to each theme from qualitative (e.g. interviews, focus groups) and quantitative (e.g. surveys, statistical data) sources. This involves coding qualitative data and identifying relevant quantitative measures.
• Mapping data: Create a map or matrix that links data segments from different sources according to the identified themes. This matrix helps visualize how different data points converge or diverge on the same theme.
• Comparative analysis: Compare the data segments within each theme to identify patterns, consistencies, and discrepancies. Look for how qualitative narratives support or contradict quantitative findings.
• Synthesis and interpretation: Synthesize the findings to develop an understanding of each theme. Interpret the data by integrating the qualitative insights with the quantitative results, explaining how they complement or contrast with each other.

A mixed methods matrix is a visual tool used to integrate and compare qualitative and quantitative data in mixed methods research. It helps researchers organize data according to key themes or variables, facilitating a comprehensive analysis and interpretation.

The matrix consists of several rows and columns:

• Rows represent key themes or research questions
• Columns represent different data sources or methods (e.g. interviews, surveys, observations)

By populating each cell with relevant data segments, researchers can easily identify areas of convergence, divergence, and complementarity. Let’s say you want to answer this research question: How does a new health intervention impact patient satisfaction and health outcomes?

You would populate the matrix as follows:

How to conduct mixed methods research: A mixed method research example

Let’s say you own a project management app and want to understand user satisfaction and identify areas for improvement. Here are eight steps to apply mixed methods research—using the convergent parallel technique—to discover user pain points and create a better user experience.

Step 1: Define your research objectives

In UX research , asking the right questions is crucial for identifying user needs and pain points effectively. But in order to write the right user research questions , you need to define a clear objective. What are you looking to understand?

Defining a clear UX research objective helps guide all other research decisions and acts as a lighthouse that guides your research project.

In our example , our research objective could be ‘to explore user experience and identify areas for improvement within our project management app’.

Step 2: Design your study and recruit participants

Ensure your study is designed to allow integration of both quantitative and qualitative data. There are various mixed method research designs to choose from—the right one for you depends on your research objectives and preferences.

At this stage, you should also establish a clear strategy for data integration and decide how you’ll combine the qualitative and quantitative data during the UX reporting and analysis phase. This might involve merging data sets for comparative analysis , or embedding one data set within the other to provide additional context.

The integration plan should reflect your research goals and ensure that the combined data offers a clear understanding. For our study, we’ll design a convergent parallel mixed methods study and triangulate our data during the analysis phase. This enables us to find our what and our why.

This is also when you need to recruit research participants for your study. Consider what you’re studying and identify your target test audience. You then need to create a call-out for your research study—either on socials, via email, or with In-Product Prompts .

Alternatively, you can find and filter research participants using Maze Panel , then manage your participant relationships using Maze Reach .

Step 3: Collect quantitative data

Next up, you want to start gathering your quantitative data. A good way to do this is with a survey to collect numerical data that can be statistically analyzed. For example, a user satisfaction survey that includes rating scales (1–10) for various aspects of the software.

For our research into app user satisfaction, we asked:

• Please rate your overall satisfaction with the app (1–10)
• How often do you use the app per week?
• How easy is the app to use on a scale of 1 to 10?
• How likely are you to recommend the app to a friend or colleague (1–10)?

❓ Need a quick and easy way to create and manage surveys? Maze Feedback Surveys simplify your feedback collection process so you can focus on making the changes your customers want to see. You can quickly create surveys tailored to your needs with Maze's survey templates .

Step 4: Collect qualitative data

Once you’ve got your quantitative data, it’s time to collect your qualitative data. Consider conducting user interviews or focus groups to obtain detailed, descriptive data that provides context and deep understanding.

For our study, we selected 20 users from the survey who gave varied ratings and conducted 30-minute interviews, asking:

• What do you like most about the app?
• What features do you find difficult to use?
• Can you describe a recent experience using the app?
• What improvements would you suggest?

💬 User interviews are resource-intensive and time-consuming. Speed them up with Maze’s end-to-end user interview solution: Interview Studies .

Step 5: Quantitative data analysis

Now you’ve got all your data—it’s time to dig in. For your quantitative data, this involves using statistical methodology to identify trends and patterns.

When we looked at our example data, we calculated:

• CSAT score: 75%
• Frequency of use: 70% use the app daily
• Ease of use average score: 6.8/10
• Net Promoter Score (NPS): 65

Step 6: Qualitative data analysis

Analyzing qualitative data involves coding and categorizing qualitative responses to uncover themes and patterns. Identify recurring themes in user feedback, such as ease of use, functionality, and improvement areas. If you’re using Maze Interview Studies to analyze your findings, you can automatically extract key themes and summaries to speed this process up.

When reviewing qualitative data, we found a number of interesting nuggets in our qualitative data:

• Users express dissatisfaction with the app’s usability, specifically the navigation between different functionalities
• Users wish they could access their billing details via the app, instead of solely via the web
• User find the core functionality—the project management features—to be highly valuable to their day-to-day, but also report finding the interface to be clunky and unintuitive

Step 7: Integrate data and interpret findings

Following your analysis, combine the findings from both data sets and draw conclusions. Look for correlations and insights that span both types of data.

Example integration:

• High satisfaction scores (75%) but lower ease of use (6.8/10) prove a strong product market fit but call for a more intuitive experience
• Further qualitative research agreed with this conclusion and identified specific areas for improvement, such as adding additional functionalities and improving the interface

Step 8: Report findings to stakeholders for buy-in

Present the integrated results to highlight how qualitative insights support or explain quantitative trends.

The format of your report will depend on your audience:

• Internal stakeholders (project managers, designers): Consider a concise report with clear visuals like charts, graphs, and user quotes to highlight key findings and actionable recommendations
• External stakeholders (clients, investors): Create a formal report with a clear introduction, methodology section, and comprehensive results presentation, summarizing key findings and highlighting the impact on user satisfaction and app usage

Always strive to go beyond what the data says and explain why it matters.

For example, once we’d conducted our research and drawn conclusions, we compiled this into a report that shared:

• Research methods: We used mixed methods research (surveys and interviews) to explore existing user pain points and satisfaction levels.
• Overall findings: User satisfaction is moderately high (7.5/10), indicating a generally positive reception. However, the ease of use score (6.8/10) and qualitative feedback highlight significant usability issues for new users.
• Actionable next steps based on findings: Simplify the user interface to improve the experience for new users, potentially increasing overall satisfaction and ease of use scores.

Conducting mixed methods research with Maze

Mixed methods research is one of the most effective ways to boost your UX insights, and gather a more rounded understanding of your users’ problems and perspectives. Combining research methods and types of data can uncover insights you may otherwise miss. And while there are ideal times to conduct qualitative, quantitative, or mixed methods research, ultimately it really is as simple as more research = more insights .

If you’re looking for the ideal research companion to help conduct mixed methods research, consider Maze. Maze is the user research platform that empowers all teams with the research methods they need to get game-changing insights. Whether it’s a mixed methods study or a one-off test—Maze helps you gather accurate insights, faster, for more informed decision-making.

Frequently asked questions about mixed methods research

What is the purpose of mixed methods research?

The purpose of mixed methods research is to combine quantitative and qualitative data to provide a more complete understanding of a research problem. This approach helps validate findings, explore complex issues from multiple perspectives, and produce more reliable and actionable results.

What’s the difference between qualitative and quantitative research?

• Qualitative research explores non-numerical data to understand concepts, opinions, or experiences. It uses methods like interviews, focus groups, and observations to gather in-depth insights.
• Quantitative research focuses on numerical data to quantify variables and uncover patterns. It uses methods like surveys, experiments, and statistical analysis to measure and analyze data.

What is the difference between mixed methods and multiple methods?

Mixed methods research integrates qualitative (e.g. interviews) and quantitative (e.g. surveys) data within a single study. Multiple methods research uses various research approaches, but they can be either qualitative or quantitative. For example, it might use surveys and experiments (quantitative) or interviews and focus groups (qualitative) in different parts of a study without combining the data.

Title Page Setup

A title page is required for all APA Style papers. There are both student and professional versions of the title page. Students should use the student version of the title page unless their instructor or institution has requested they use the professional version. APA provides a student title page guide (PDF, 199KB) to assist students in creating their title pages.

Student title page

The student title page includes the paper title, author names (the byline), author affiliation, course number and name for which the paper is being submitted, instructor name, assignment due date, and page number, as shown in this example.

Title page setup is covered in the seventh edition APA Style manuals in the Publication Manual Section 2.3 and the Concise Guide Section 1.6

Related handouts

• Student Title Page Guide (PDF, 263KB)
• Student Paper Setup Guide (PDF, 3MB)

Student papers do not include a running head unless requested by the instructor or institution.

Follow the guidelines described next to format each element of the student title page.

Professional title page

The professional title page includes the paper title, author names (the byline), author affiliation(s), author note, running head, and page number, as shown in the following example.

Follow the guidelines described next to format each element of the professional title page.

This paper is in the following e-collection/theme issue:

Published on 14.8.2024 in Vol 8 (2024)

A Chatbot (Juno) Prototype to Deploy a Behavioral Activation Intervention to Pregnant Women: Qualitative Evaluation Using a Multiple Case Study

Authors of this article:

Original Paper

• Elisa Mancinelli 1, 2 , BSc, MSc   ; 
• Simone Magnolini 3 , PhD   ; 
• Silvia Gabrielli 2 , PhD   ; 
• Silvia Salcuni 1 , PhD  

1 Department of Developmental and Socialization Psychology, University of Padova, Padova, Italy

2 Digital Health Lab, Centre for Digital Health and Wellbeing, Fondazione Bruno Kessler, Povo, Trento, Italy

3 Intelligent Digital Agents, Centre for Digital Health and Wellbeing, Fondazione Bruno Kessler, Povo, Trento, Italy

Corresponding Author:

Elisa Mancinelli, BSc, MSc

Department of Developmental and Socialization Psychology, University of Padova

Via Venezia 8

Padova, 35131

Phone: 39 3342799698

Email: [email protected]

Background: Despite the increasing focus on perinatal care, preventive digital interventions are still scarce. Furthermore, the literature suggests that the design and development of these interventions are mainly conducted through a top-down approach that limitedly accounts for direct end user perspectives.

Objective: Building from a previous co-design study, this study aimed to qualitatively evaluate pregnant women’s experiences with a chatbot (Juno) prototype designed to deploy a preventive behavioral activation intervention.

Methods: Using a multiple–case study design, the research aims to uncover similarities and differences in participants’ perceptions of the chatbot while also exploring women’s desires for improvement and technological advancements in chatbot-based interventions in perinatal mental health. Five pregnant women interacted weekly with the chatbot, operationalized in Telegram, following a 6-week intervention. Self-report questionnaires were administered at baseline and postintervention time points. About 10-14 days after concluding interactions with Juno, women participated in a semistructured interview focused on (1) their personal experience with Juno, (2) user experience and user engagement, and (3) their opinions on future technological advancements. Interview transcripts, comprising 15 questions, were qualitatively evaluated and compared. Finally, a text-mining analysis of transcripts was performed.

Results: Similarities and differences have emerged regarding women’s experiences with Juno, appreciating its esthetic but highlighting technical issues and desiring clearer guidance. They found the content useful and pertinent to pregnancy but differed on when they deemed it most helpful. Women expressed interest in receiving increasingly personalized responses and in future integration with existing health care systems for better support. Accordingly, they generally viewed Juno as an effective momentary support but emphasized the need for human interaction in mental health care, particularly if increasingly personalized. Further concerns included overreliance on chatbots when seeking psychological support and the importance of clearly educating users on the chatbot’s limitations.

Conclusions: Overall, the results highlighted both the positive aspects and the shortcomings of the chatbot-based intervention, providing insight into its refinement and future developments. However, women stressed the need to balance technological support with human interactions, particularly when the intervention involves beyond preventive mental health context, to favor a greater and more reliable monitoring.

Introduction

User-centered design of digital mental health interventions.

eHealth is a burgeoning field that integrates medical informatics, public health, and business. It encompasses delivering health services and information through the internet and digital technologies. In this domain, e-mental health specifically focuses on leveraging technologies, such as smartphone apps, websites, chatbots, and virtual reality, to enhance and support mental health care [ 1 - 3 ]. e-Mental health holds many advantages, including the increased scalability of mental services, in terms of screening, prevention, and treatment, leading to reduced costs for the broader health care system [ 4 - 6 ]. However, while the potential benefits of digital technology can be considerable, their actual implementation and use, especially within the field of e-mental health, often fall short. The journey from preuse considerations to initial adoption and, crucially, sustained use poses challenges that need careful navigation and understanding. In this regard, a recent review [ 7 ] exploring design methods and approaches for digital tools in mental health emphasized that human-centered design methods, thus those focusing on user experience (UX) rather than just engineering design, are not fully integrated into the field. The reported design approaches are predominantly external, lacking the perspective of the end users for whom the tool is intended. Indeed, when developing digital solutions, it is essential to consider 4 key components: the design issue and solution, the context in which the design occurs, the dynamics and organization of the design activity, and the actors contributing to the design [ 8 - 10 ]. Within the context of e-mental health intervention, the above altogether emphasizes the significance of co-design, a collaborative process strongly involving targeted end users to contribute to all stages of e-mental health intervention development. This inclusive approach encompasses needs assessment, content development, pilot-testing, and finally, dissemination [ 11 ]. The Obesity-Related Behavioral Intervention Trials (ORBIT) model [ 12 ] is instrumental to this end. The ORBIT model, which uses a user-centered design, provides a methodological framework encompassing a pliable and iterative progressive procedure, predefined clinically significant milestones for advancement, and the option to revert to a prior phase of refinement in case of suboptimal outcomes. Its primary emphasis is on pre-efficacy development and testing, yet not failing to incorporate subsequent research phases to illustrate that treatment optimization is viable even for interventions that have attained the efficacy or effectiveness stage [ 12 ].

e-Mental Health in Perinatal Care: A Focus on Prevention Interventions

The World Health Organization (WHO) [ 13 ] has consistently emphasized the significance of identifying and preventing risks, with the WHO and the United Nations Population Fund acknowledging maternal mental health as a pivotal factor in accomplishing the Millennium Development Goals [ 14 ]. The transition to motherhood involves various intrapersonal and interpersonal changes and challenges that can have negative effects on women’s mental health, increasing the risk of developing peripartum depression [ 15 - 17 ]. However, despite the negative repercussions this poses on the women, the child, and the whole family [ 18 ], as well as the broader society [ 19 - 21 ], it often goes untreated. There are various reasons for this. On the one hand, few women proactively seek professional assistance for their mental health problems, mainly due to factors such as lack of mental health literacy; stigma; and practical barriers like childcare, professional, and financial constraints [ 22 ]. By contrast, women face limited access to specialized perinatal mental health services, which is attributed to the capacity constraints of existing services and long waiting times for those in need of support [ 23 , 24 ]. Therefore, many women never receive any support or treatment. Indeed, this situation has sparked interest in the potential of e-mental health. It can circumvent some of the aforementioned barriers, ultimately facilitating a more widespread help-seeking process; this has led to the creation and dissemination of scalable and more far-reaching tools to support the well-being and mental health of perinatal women [ 25 , 26 ]. In this context, the stepped-care model is noteworthy, as its intentions are focused on promoting the dissemination of mental health programs by facilitating coordination between primary and secondary mental health services [ 27 ], and this coordination can be facilitated through e-mental health. This would ultimately align with the evidence that engaging in help-seeking behaviors increases the likelihood of perinatal women seeking further assistance for their depression symptoms [ 28 ]. In this regard, structured, evidence-based interventions such as behavioral activation (BA) might be particularly suitable. BA is a behavioral intervention designed to alleviate symptoms of depression [ 29 - 32 ] by offering individuals practical strategies to improve their adjustment and well-being and supporting participation in enjoyable and positive activities while reducing engagement in behaviors that worsen depressive symptoms [ 29 , 33 ]. As such, these interventions hold great potential as initial broad-case preventive work. However, when specifically focusing on peripartum depression, there appears to be a deficiency in digital prevention and treatment programs at large [ 34 ], and of BA interventions as well [ 35 ], in addressing depression symptoms during pregnancy compared with the postpartum period, thus underscoring the necessity to boost the development and evaluation of primary mental health services.

This Study: Within the Iterative Design Phase

This study arises from the results obtained by a previous exploratory co-design study [ 36 ] investigating the feasibility of an internet-based BA intervention for pregnant women showing subclinical symptoms of depression. As such, it constitutes the second phase of investigation within the “design phase” foreseen by the above-reported ORBIT model [ 12 ]. This prior exploratory study not only aimed to assess the initial feasibility of the intervention but also sought to gather valuable feedback directly from pregnant women. This then guided the adjustment of the intervention’s content and structure while promoting the use of a different digital solution. More specifically, the study aimed to compare a guided and unguided version of the digital intervention, with the guided group involving psychologists who engaged in weekly text message conversations with women to support them in the intervention content revision. In this respect, data suggested that the guided group showed greater adherence and were more willing overall to finish the intervention than the unguided group. Building on this and in line with the existing literature [ 37 , 38 ] highlighting the potential benefit of including chatbots within psychological interventions by fostering intervention adherence through increased engagement and involvement, a new structuring of the BA intervention as a chatbot-delivered intervention was prototyped. Chatbots are artificial intelligence–enabled engagement technologies, falling under the category of technologies that enable interaction with patients through natural language processing by engaging in limited text conversations intending to support subsequent behavior-change tasks [ 39 ]. It is crucial to emphasize that in this context, chatbots are conceptualized as tools suitable for educational purposes, facilitating the acquisition of specific evidence-based techniques or skills [ 40 ] resulting in suitability for application in preventive contexts.

Mindful of the above, this study aims to qualitatively evaluate, through a multiple case study, pregnant women’s experience and perception of a chatbot prototype to deploy a BA preventive support tool and intervention. In this regard, incorporating a dedicated prototype evaluation during co-design can streamline the process of conducting rigorous evaluations in real-world settings during the subsequent evaluative phases, which may involve activities such as pilot-testing and subsequent randomized controlled trials [ 41 ]. Furthermore, women’s desire for improvement and technological advancements of chatbot-based technology in the field of perinatal mental health was also investigated. As such, this study bounds the design and evaluation of the chatbot and prevention intervention it deploys within the ORBIT methodological framework [ 12 ], in favor of a thorough and meticulous evaluation of the intervention design phase regarding both definition and refinement. In line with this, a multiple–case study design is used as it permits the conduct of a comparative analysis of cases, aiming to identify both similarities and differences among them and, thus, in the perception of the chatbot and the content it deploys. In addition, this approach seeks to unveil patterns and themes that arise from the cross-case analysis. By evaluating the phenomenon of interest across different contexts, a multicase study might enhance the validity of findings by investigating in depth how the phenomenon may vary or remain consistent under various circumstances [ 42 ].

Ethical Considerations

Ethical considerations adhered to the guidelines outlined in the Declaration of Helsinki [ 43 ] and European data protection laws (EU GDPR 679/2016). Approval for the study was obtained from the Ethical Committee of the Psychology Department at the University of Padova (approval 5434/2023). Participants provided their informed consent to participation and data publication for scientific reasons.

Participants and Enrolment Procedure

Women aged >18 years and between the 12th and 30th week of gestation could take part in this study. Exclusion criteria were the following: clinically significant depression symptoms (Patient Health Questionnaire-9 [PHQ-9] [ 44 ] score≥15), suicidal ideation (PHQ-9 item 9), present or past history of psychiatric disorders, and experiencing an artificially induced pregnancy. To allow participation, a Google Form link containing the baseline questionnaires was shared through social media platforms (ie, Facebook and Instagram) in pregnancy-related national groups and pages. After the inclusion and exclusion criteria evaluation, women were provided with the information needed to start the interaction with the chatbot in Telegram and sent a copy of the informed consent they had agreed on that was reported within the web-based questionnaire. To uphold confidentiality, each participant was assigned a unique alphanumeric code. Women were granted the autonomy to withdraw from participation at any point without the obligation to provide reasons and without facing any adverse consequence. Furthermore, they were clearly informed that the software (ie, Telegram and the chatbot) did not constitute a medical device, as its use does not extend to the diagnosis, prevention, monitoring, prediction, prognosis, treatment, or alleviation of diseases. It was, instead, clarified that the developed support intervention and related software were exclusively intended for research purposes and used for the sole collection, storage, transmission of data and administration of questionnaires.

A total of 12 women completed the baseline questionnaire. Among them, 2 dropped out after the first interaction (week 1), 2 after completing the interaction in week 2, and 2 following the third interaction (week 3). One participant withdrew after completing the interaction in week 4. Among those who dropped out in the early weeks, 5 reported medical conditions: Crohn disease, risk of miscarriage associated with a shortened cervix and hypertonic pelvic floor, gestational diabetes, hypothyroidism, and fibroma. Ultimately, 5 participants were included in the multiple–case study evaluation, with none reporting any medical conditions. Of them, 4 (80%) participants reached the postintervention questionnaire evaluation, while 1 (20%) had to interrupt the interaction after week 4; however, she agreed to participate in the final semistructured interview. Given that this study aimed to qualitatively evaluate the perception and experience with a chatbot prototype, the decision was made to include this participant despite not finishing the study since she nonetheless was able to engage with the chatbot for more than half of the anticipated interactions.

The Intervention Content and Structuring

This study aligns with the iterative process outlined in the ORBIT model [ 12 ] for intervention design and evaluation. Specifically, it falls within the refined subphase of the initial design phase, in which practical aspects such as mode and agent of delivery, as well as the frequency and duration of contact, are evaluated to identify the most efficient ways to achieve clinical targets. Parallel to this, and in reference to the Digital Product Lifecycle, we care to emphasize that this project is at the beginning stages of the product life cycle, thus moving back and forth between the “definition phase” (in which the product or intervention concepts and related digital requirements are defined) and the “design phase” (which involves prototyping and pilot-testing the product) [ 45 , 46 ].

Accordingly, this study focuses on evaluating a revised version of an intervention based on an evidence-based BA intervention protocol (behavioral activation treatment for depression-revised) [ 47 ]. This revised intervention represents a second evaluation that builds upon exploratory testing conducted in a preceding co-design study [ 36 ]; as such, thorough information on the intervention content and rationality can be found in this prior study paper. However, commencing with the results from this latter study, in this study, the intervention was organized into 6 weekly sessions ( Figure 1 A), omitting the 3 additional ones previously included. The intervention content was streamlined by eliminating separate in-between–session homework. Instead, the essential components of the homework were incorporated within the main sessions or interactions as on-the-moment exercises strategically designed to promote the original intent of the homework. In this context, it is noteworthy that while the original protocol may not explicitly encompass a comprehensive functional analysis, several treatment components seamlessly aligned within such a framework and were further enhanced in the modified version of the intervention. This alignment is underscored by BA’s dual objectives of pinpointing factors that sustain or reinforce depressive behaviors (both positive and negative reinforcement) and identifying positive reinforcers that can support healthy behavioral patterns ( Figure 1 A). This process thus forms the basis for understanding the functional aspects of behavior, laying the groundwork for targeted strategies that can aid the person in autonomously addressing and modifying the maladaptive behavioral patterns effectively.

Moreover, there was a modification in the mode of delivering the intervention. Specifically, the content, previously structured as an e-learning course, was facilitated through a rule-based chatbot named Juno, operationalized within the Telegram platform with the sole purpose of delivering the intervention. As such, information was delivered through text messages, complemented by explanatory videos and images using the Telegram interface. The text messages were sent by Juno, which adhered to a preestablished protocol that had to be followed sequentially, enabling structured dialogues in which women engaged primarily by selecting the predefined buttons to navigate the conversation. Due to the rule-based nature of Juno, individualized feedback was not provided. In this regard, Figure 1 B depicts a simulation of the interaction between Juno and the user, showing how Juno responds and guides the user during the on-the-moment exercises (the reported example is an exercise conducted during week 2 reported in Figure 1 A “The bidirectional link between behavior and emotions”).

Moreover, by using the Telegram interface, participants can access multimedia resources such as videos and images in the multimedia section of the app. In addition, they could scroll back through the chat history to review past topics, although there was not a specific page summarizing the weekly intervention topics. This allowed participants to revisit and reinforce previous discussions as needed while maintaining the logical sequencing of the intervention. Conversations were structured to last around 10 minutes per session.

The Implementation of the Chatbot Juno in Telegram

The chatbot Juno was developed drawing inspiration from the methodology used in designing Motibot, a chatbot dedicated to providing psychosocial support to adults with diabetes mellitus [ 48 ]. Leveraging the capabilities of the Rasa open-source platform [ 49 ], which has been explicitly tailored for chatbot development and training, became viable owing to the domain-agnostic nature of Motibot’s core structure, which provided a remarkably flexible foundation. The Rasa platform seamlessly integrates advanced machine learning techniques and harnesses pretrained embeddings from language models. This integration empowers the construction of a chatbot finely tuned to a specific language. The synergy of machine learning techniques with crafted rules ensures a chatbot that is not only dynamic but also highly responsive. Within Juno, the pivotal role played by natural language understanding [ 50 ] became evident in interpreting user messages while considering the conversational history. A carefully defined set of variables facilitated a smooth transition between turns in the dialogue.

For instance, named entity recognition, a specific natural language understanding task, was used to interpret the intent “say your name” and identify the entity “user’s name.” Juno optimally used Telegram as its user interface, offering numerous advantages to users while streamlining the development process. In addition, Telegram’s built-in support for interactive tools, including buttons, links, and images, enhanced the overall UX. In this regard, enhancing UX involves using personalized interaction time frames. Juno, as part of its intervention process, prompted users at the end of the initial day to specify when they prefer follow-up contacts. This proactive approach assisted users in scheduling their intervention; Juno uses Rasa’s reminder interface to accomplish this task. However, potential server malfunctions can affect this tool. To mitigate such issues, Juno allows users to initiate the interaction (eg, by writing the message “Can we start?”) if the reminder date passes without any notification. Despite being a solution to a possible interaction problem, this approach should maintain a positive UX. Furthermore, it should be acknowledged that in this initial phase of development, the possibility that the chatbot could occasionally overlook an appointment was a possibility.

Moreover, in line with what was reported above, it is noteworthy that Juno follows an expert-written structured script to maintain focus on the intervention content and avoid deviating from the intended topics. Users can provide input by selecting predefined buttons or providing written responses, but they do not receive personalized feedback based on their input. If users attempt to engage with Juno outside the scope of the intervention, Juno informs them that it cannot respond to such queries and returns to the predefined interaction by starting from where they had left off.

With regard to data storage, no further development was required, as the native support of Rasa for storing interactions in a MongoDB database (ie, a universal time stamp) ensures both consistency and the archiving of users’ data (ie, log-in information, the time spent by each user interacting with Juno, etc).

Measurement Instruments

During the baseline assessment, women were asked the following demographic information: age, gestational week, if the pregnancy was physiological or induced through medically assisted techniques, marital status, educational level, category of occupation, living location, past and present psychiatric history, and presence of any medical condition (both pregnancy-related and not). Moreover, during the baseline assessment, participants completed questionnaires assessing psychological symptoms, levels of BA, and perceived environmental reward. The same questionnaires were administered at the end of the sixth week of interactions, facilitated by Juno in the Telegram Chat, for a postintervention evaluation; the UX and user engagement (UE) measures were included in the postintervention assessment.

Psychological Symptoms

Depression symptoms were evaluated through 2 unidimensional self-report tools: the PHQ-9 [ 44 , 51 ] and the Edinburgh Postnatal Depression Scale [ 52 , 53 ]. The PHQ-9 assesses the severity of depression symptoms over the past 2 weeks through 9 items measured on a 4-point Likert scale (0=“not at all”; 3=“almost every day”). Items align with the diagnostic criteria of the Diagnostic and Statistical Manual of Mental Disorders , Fourth Edition [ 54 ]. A score of ≤9 indicates mild or no depression symptoms, between 10 and 14 indicates moderate symptoms, and ≥15 indicates severe symptoms. Item 9 specifically assesses suicidal ideation. The Edinburgh Postnatal Depression Scale also assesses the severity of depression symptoms, yet on the previous week and more specifically in association with the perinatal period. It comprises 10 items measured on a 4-point Likert scale (0=“no, not at all”; 3=“yes, always”), with item 10 assessing suicidal ideation. Scores range between 0 to 30, and a score of ≥13 suggests probable depression. Anxiety symptoms were measured through the Generalized Anxiety Disorder-7 [ 51 , 55 ], a unidimensional self-report tool gauging the severity of these symptoms over the past 2 weeks through 7 items measured on a 4-point Likert scale (0=“never”; 3=“almost every day”). Scores range between 0 and 21; a score between 0 and 4 suggests minimal anxiety, between 5 and 9 mild anxiety symptoms, between 10 and 14 moderate anxiety symptoms, and ≥15 severe anxiety symptoms. Finally, stress symptoms were assessed through the Perceived Stress Scale-10 [ 56 , 57 ], a unidimensional self-report tool assessing stress symptoms over the past month using 10 items measured on a 4-point Likert scale (0=“never”; 3=“quite often”). Scores range between 10 and 40, with scores ranging from 0 to 13 suggesting lower stress levels, between 14 and 26 moderate stress levels, and ≥27 high perceived stress levels.

BA Measures

The BA for Depression Scale-Short Form [ 58 ] was used to measure changes in avoidance and activation during BA interventions for depression over the past week. It is a self-report featuring 9 items measured on a 7-point Likert scale (0=“not at all”; 6=“completely”), providing scores for BA, behavioral avoidance, and a total score ranging from 0 to 54. The Environmental Reward Observation Scale [ 59 ], a unidimensional self-report tool, was also used; it measures the level of environmental reward perceived in recent months through 10 items rated on a 4-point Likert scale (1=“strongly disagree”; 4=“strongly agree”). Scores range from 10 to 40.

The UX was evaluated through the Mobile Application Rating Scale [ 60 ], a self-report tool evaluating the quality of an app and its features. Comprising 23 items scored on a 5-point Likert scale (1=“poor”; 5=“excellent”), it assesses 4 dimensions of objective quality: engagement, functionality, esthetics, and information, along with a subjective quality scale. Only subscales related to “information,” “subjective app quality,” and “app-specific” (function) were considered for this study, totaling 17 items. UE was instead evaluated through the User Engagement Scale-Short Form [ 61 ], a short self-report tool assessing UE with a digital solution. With 12 items based on a 5-point Likert scale (1=“strongly disagree”; 5=“strongly agree”), it encompasses factors such as focused attention, perceived usability, esthetic attractiveness, and reward. Higher scores index a more positive evaluation.

Semistructured Interviews

Semistructured interviews, conducted by the first author between October and December 2023, featured 15 main questions tailored to the study. The semistructured interview comprehends 3 main blocks of questions and related probing questions: one focused on women’s personal experience with the intervention content (4 questions), another focused on their experience with the chatbot and the overall platforms (5 questions), and the last one inquired on opinions for future technological advancements (6 questions). Before asking the last block of questions, participants were provided with the definitions of digital intervention and technological advancement within the context of chatbot technologies, as reported in the Multimedia Appendix 1 . The interviews were conducted approximately 10 to 14 days after the participants had finished the interactions with the chatbot Juno; they were conducted either by phone call or through Google Meet, based on the participants’ preference. With the participants’ consent, the interviews were audio recorded for transcription and evaluation.

Data Analysis

All the analyses were computed with RStudio (RStudio IDE). Participants’ questionnaire scores were assessed, and score differences in psychological symptoms and activation levels, between preintervention and postintervention time points, were calculated by subtracting the postintervention scores from the baseline ones. Relying on the qualitative meaning of response points (particularly for the psychological symptoms, measured on a 4-point Likert scale), differences between time points were commented on when they differed by a minimum of +3 or –3 score points.

The semistructured interviews were evaluated in 2 different but complementary manners. First, a purely qualitative descriptive evaluation was conducted by extracting and evaluating the key points reported by each case in the related interview transcript for each question. Subsequently, a text-mining analysis was performed using the R package quanteda [ 62 ]. To this end, (1) the transcripts, written in Italian, were tokenized by using the specific quanteda function and converting uppercase letters into lowercase letters, removing numbers, punctuation, and stop words. Subsequently, (2) user responses were subdivided and grouped based on the question they referred to in separate .txt files. Finally, (3) recurrent words (ie, word stems) and their diagrams (ie, pairs of reoccurring word stems) were extracted; the former were considered recurrent if they appeared at least 3 times, while the latter if they appeared at least 2 times across interviews. The 3-occurrences criterion threshold was defined in line with past research [ 48 ]. In particular, the 3-occurrences criterion for including a stem was chosen based on the assumption that through this, an occurrence is expected to belong to the 5% most recurrent ones. This criterion resulted in the extraction of between 2.4% and 9.2% of the most recurrent stems (average 5.5%) for the different questions, reasonably complying with the assumed 5% threshold. In addition, for a given question, the average occurrence of stems was 1.3; thus, a 3-occurrences threshold was equivalent to the condition of a stem recurring with a frequency corresponding to more than twice the average occurrence.

Cases Presentation

Table 1 shows the participant’s demographic information. Figure 2 shows their scores regarding psychological symptoms, activity level, and environmental reward at baseline ( Figure 2 A) and the postintervention time point ( Figure 2 B), further plotting the difference between the 2 time points ( Figure 2 C). Of the 5 cases, participant E completed the interaction with Juno until (and including) week 4 because of a technical issue with the server provider of the chatbot (the update of the server’s public certificate resulted in a compromised connection between the Rasa server and Telegram, and despite efforts within the support time frame, communication restoration was unsuccessful). As such, her postintervention evaluation measurements are not available. It should also be noted that because of technical issues linked to the temporalization of the interactions, participant B skipped the interaction of week 2, participant A skipped the interaction of week 3 ( Figure 1 ), and participant D skipped the interaction of weeks 3 and 5. Furthermore, all participants had to autonomously prompt the interaction with Juno at least once.

Differences and Similarities Across Cases: Questionnaire Scores

Regarding the trend of change between the 2 time points, women showed comparable levels of psychological symptoms, BA, and environmental reward at baseline, which instead seem slightly different at the postintervention time point. More specifically, participant C stands out as the only one showing a reduction in all psychological symptom variables, with changes ranging from 3 to 4 score points. However, the levels of BA and environmental reward appear seemingly unchanged. By contrast, participant A seems to exhibit a peak in the reduction of stress symptoms and an increase in BA. Interestingly, participant B demonstrated a trend of increase in anxiety symptoms, alongside a trend of reduction in depression symptoms and a notable peak in increased BA.

In contrast, participant D appears to demonstrate a negative peak in BA (ie, a decrease), while the other dimensions seem unchanged. Notwithstanding, it should be stressed that at either time point, none of the participants reported clinically relevant symptoms in terms of depression, anxiety, and stress symptoms. Finally, Figure 3 plots the participants’ evaluation of UX and UE. Taken together, participant B provided, in all dimensions, the lowest UX and UE scores, while participant D had the highest scores. More specifically, all showed a quite high appreciation for the esthetic of the interactions (mean 4, SD 0) and a modest to high perceived usability of the chat (mean 3.67, SD 0.82), which although seems particularly true for participant D, while less so for participant B.

Furthermore, the latter reported a particularly low sense of absorption during the interaction, which was quite low also for participant C. This sense of absorption was instead moderate for participants A and D (mean 2.42, SD 1.17). These 2, together with participant C, also reported a moderate to quite high sense of reward from the interactions (mean 3.33, SD 0.72), instead lower for participant B. A comparable pattern emerged regarding UX-information (mean 4.08, SD 0.63); in addition, participant B, for whom the information was of modest quality, participants A, C, and D instead evaluated them as high-quality information in terms of credible sources, quantity, and clearness. An almost equal score distribution emerged for the app-specific function (ie, the app operation in terms of easy learning, logical flow, and gesture interaction design; mean 3.38, SD 0.75) and subjective quality (ie, the actual availability of using the app; mean 2.69, SD 1.01), with the latter being way lower.

Multimedia appendix 1 shows participants’ specific scores reported in Figures 2 and 3 .

Differences and Similarities Across Cases: Qualitative Evaluation of the Semistructured Interviews’ Answers and Text-Mining Results

A summary of the key concepts that emerged from the answers provided during the semistructured interviews is reported in Tables 2 - 4 , separately for each case. In this regard, it is worth noting the answers provided for q00 regarding the motivation for participation; only participant B reported a more personal motive linked to a desire to enrich her pregnancy experience. Differently, participants C, D, and E (participant E revealed during the semistructured interview that she is a perinatal psychologist and that she participated in the study because she was curious to experience firsthand the potentiality of digital tools in this context) were pushed by curiosity and a personal propensity to help with research. Finally, participant A reported that her curiosity was sparked by seeing one of the institutions that is part of this study.

a d/n: did not know what to answer.

a FAQ: frequently asked question.

b Not applicable.

Focusing on the text-mining analysis performed, the interview length ranged between 27.41 and 60.02 (mean 42.3, SD 12.63) minutes. After deleting the stop words, transcripts included a mean of 1732.8 (SD 795.78) words per participant. Overall, the aggregated results (text mining) are shown in Figures 4 - 6 . The nodes (ie, word stems) dimension illustrates the proportion of concept occurrences across transcripts for a specific question, all appearing at least 3 times. Word stems connected by arrows represent diagrams that have occurred at least twice. The word stems are translated after analysis for inclusion in the plots; therefore, the direction of the arrows reflects the Italian syntax.

Personal Experience With the Chatbot Juno

As for the text-mining aggregated results, the transcript of the interviews regarding participants’ personal experience with Juno included, after deleting the stop words, a mean of 67.56 (SD 28.39) words per question. Results are summarized for each question in Figure 4 . Instead, Table 2 summarizes, separately, the participants’ answers to each question. In this respect, all participants reported feeling negative regarding the technical problems encountered, except participant C, who felt indifferent to them. Noteworthy is that although all skipped at least 1 interaction, participant C is the sole one that had not skipped any interaction, while further reporting that she “knew that this is a research project,” thereby highlighting that she had foreseen some issues to occur. In line with this, the text-mining results highlighted the feelings of displeasure , untimely making the experience less impactful ( made it→less→impactful; q01).

Nonetheless, the experience (in terms of the content of the interactions) was liked and felt very interesting , in particular, the content of the exercises/questions (q02). However, participant A specified that she would have preferred if the latter were proposed in a more structured manner while also allowing for the possibility to continue practicing them in between the interactions to favor a sense of continuity. Furthermore, participant D felt that some of the questions (part of the exercises) were redundant. Participants B, C, and D instead stressed their appreciation for how the broader content was deployed in terms of videos and images, while participant E specifically appreciated how the messages were phrased. The overall content of the interactions, particularly the initial psychoeducation, was felt as pertinent and adequate. Coherently, the text-mining results highlighted that the interactions’ content was felt useful , allowing participants to take a moment to pay attention to themselves (q03). In this regard, they all reported that the content was pertinent to the pregnancy period ( period→pregnancy ) but could also be useful during the postpartum and the future in general , supporting them in asking for help ( ask→help ; q03 and q04) and in general favoring a self-awareness that can transversally be applied to life in favor of well-being. However, focusing on the subjective answers, while participant B felt that the content was suited for the beginning of the second trimester, participant C felt that such a period was already too late and that the support provided by the chatbot was better suited for the emotional tumult of the first trimester.

At last, all women felt the 6-week length of the intervention was adequate , although, given the length of the pregnancy period, they could have followed it even for a longer time (q05). This latter aspect was stressed by all those who had skipped at least 1 week of interaction and not by participant C, who had followed all 6.

UX With the Chatbot Juno in Telegram

The transcript of the interviews regarding participants’ UX with the chatbot Juno in Telegram included, after deleting the stop words, a mean of 69.65 (SD 29.95) words per question. Results are summarized for each question in Figure 5 , while Table 3 reports participants’ answers. With regard to women UX in interacting with Juno, as previously outlined, experiences were quite different, albeit the technical problems with the chatbot Juno have emerged as a matter to particularly account for (q08). In this regard, participant B pointed out the importance of providing clearer guidance, ideally beforehand, on how to autonomously deal with technical issues to help avoid feelings of confusion. Notwithstanding, all women showed appreciation for the esthetic of the material ( esthetic→material ) describing it as cute . Furthermore, it mostly brought the focus of the UX to the way Juno answered their inputs, highlighting the relevance of this aspect, thereby wishing for an increased personalization of the answers (q06). However, despite this, participant A perceived that because of the way messages, in general, were phrased and of the overall interaction flow, these made her at times “forget that there was not a person on the other side.” This is instead different from participant B’s perception, who considered the messages to be a bit sterile. In between these 2 polarities is instead the perception of participants D and E, the former describing them as “sufficiently spontaneous and realistic” and the latter further stressing that, although she felt properly guided by Juno, perceiving clearly that Juno was virtually created made her feel reassured. Coherently, when asked about their opinion on using a chatbot as a means to deploy psychological content (q07), participant A reported that the interactions’ limits (in terms of chatbot freedom) were both a limit and a strength. Nonetheless, overall, women felt that it could be an effective medium that can provide a kind of momentary containment ( type→containment ) and that it might work as a cue to subsequently reach for in- person support. Indeed, they felt that beyond its application in preventive contexts, a psychologist is needed ( go→psychologist→instead ; q07), and even in the context of this study, participant B felt the need for human contact at least by telephone call.

Finally, focusing on the app itself, women all agreed on the convenience ( convenient→app ) of Telegram as an interface, allowing them to avoid downloading another app and describing it as an optimal channel that they already knew and that is easy to use (q09).

Opinions on Future Technological Advancement

The transcript of the interviews’ answers regarding the participants’ desired technological advancement included, after deleting the stop words, a mean of 159.49 (SD 53.01) words per question. Results are summarized for each question in Figure 6 ; Table 4 reports the participants’ answers. Overall, when asked about opinions on future technical advancements, women’s answers were quite cohesive. In line with this, when asked about how they would image an ideal app in the context of perinatal care (q10), the greater focus was on the information content ( content→information ) related to what happens during pregnancy and in the different trimesters ( happens→trimester ) as well psychologically ( well-being→psychological ). It was also focused on the possibility of searching for this information and reading about it ( go→search ) freely.

Furthermore, they reported interest in having a chat with a chatbot within the app mainly to ask personal questions related to their personal experience ( linked to→experience→personal ). In this regard, focusing specifically on the potential technological advancements that could be foreseen for chatbots like Juno (q11), women showed a lack of interest in including vocal commands in terms of sending and receiving audios ( command→vocal→no ) and did not show a particular interest in personalizing the chatbot appearance ( personalization→appearance→chatbot ), albeit recognizing that others might. The sole exception was participant E; she reported voice commands as the first thing she would have liked to add, perceiving it as a way to optimize time. Regarding personalization , this aspect was again prominent among all women, stressing their desire to receive more personalized ( personal ) answers . However, albeit desired, such increased personalization and freedom of the chatbot also emerged as women’s main concern regarding the application of these tools in the mental health context (q12). As such, women reported the need to maintain human→monitoring . Indeed, worries were expressed regarding the kind of information the chatbot might give if unsupervised.

Furthermore, they expressed worries related to increased freedom and resemblance to human interactions, with the idea that this might lead to an overreliance on these tools. Indeed, they stressed the risk of these substituting interactions ( substitute→interaction ) with professionals and psychologists ( support→psychologist→risk→more→freedom→chatbot ), which was not desired. In line with this, participants believed that although a main advantage of these technological→tools is that they can be valuable in supporting psychological→well-being in preventive contexts ( preventive→terms ) or to satisfy specific needs without waiting to make an appointment , they cannot equate a therapeutic→intervention delivered in-person, particularly during pregnancy (q13). To deal with the concerns and risks reported, women agreed on the importance of underlining and reminding of what to expect from these tools ( meaning→tools ) and clearly stating their limits , thereby distinguishing the kind of support that can be received by a physical person versus a digital tool ( digital→person ). This would then also work as a disclaimer, thus preventing them from feeling disappointment when perceiving the limits of these tools (q14). In line with the above, women expressed a strong desire for an app that could be integrated within the health (care) →system, perceiving it as something that could create a shared space that facilitates interactions with gynecologists , thereby allowing the latter to account for women’s psychological well-being together with the medical aspects.

Principal Findings

This study aimed to use a multiple–case study design to evaluate and compare pregnant women’s experience and perception of Juno, a chatbot prototype to deploy a BA preventive support intervention; their opinions regarding desired improvements and technological advancements were also investigated. The insights gained from this study are valuable and in line with previous studies emphasizing the importance and essential nature of evaluating prototypes during the design stages of a digital tool and chatbot in particular [ 41 , 63 ]. Within this context, the adoption of a multicase study design [ 42 ] allowed us to gather valuable in-depth information on the similarities and differences in pregnant women’s perceptions, opinions, and desires while also evaluating the technical issues encountered and their impact on women’s experience [ 63 ].

Focusing on the implementation and operationalization of Juno in Telegram allowed women to benefit from the lack of installation requirements, experiencing an interface within a familiar environment; this is an advancement from the platform used in the previous study [ 36 ]. Instead, feedback regarding the materials’ esthetic and intervention content at large was again appreciated, and the content, in particular, was described as sound and useful. Women expressed specific appreciation for the exercises proposed by Juno as part of the BA intervention, assessing that they favored self-reflection. Differently from the previous study [ 36 ], most women would have liked for the intervention to be longer. This might be linked to the weeks of interaction skipped since only participant C who had completed all 6 interactions would have not lengthened the intervention. Another explanation could, instead, be linked to the change of platform and even more the new structuring of the exercises. Compared with the previous structuring [ 36 ], they have been changed so to be as short, simple, and effortless as possible, and as such, they were turned into on-the-moment reasoning exercises guided by Juno and no longer as in-between–session homework. This altogether seems to have been appreciated by women, except for 1 participant A, who instead stressed that she would have preferred to have the possibility to continue training them autonomously through practical exercises also in between the interactions. The desire for continuity became evident throughout her interview, suggesting that the digital tool was perceived as a companion to turn to when extra support was needed by providing a personal space to freely take care of herself. Participant E also highlighted this, considering it as something with added value, especially during the postpartum period, helping her take care of herself to then potentially better care for the newborn.

Mindful of the above, it is pivotal to remember that women in this study can be deemed “healthy,” as none of them present medical conditions, and all psychological symptom variables were below the clinical thresholds. Notwithstanding, it is worth noting that participant C, who besides having cleared the 6 interactions, also reported the highest symptomatic scores at baseline, showed the greatest and most consistent trend of reduction in symptom scores. However, despite some pretest-posttest changes in symptom variables scores, none of the participants ever reached clinical relevance, so they should be regarded as normal fluctuations in the state of well-being and ill-being occurring during pregnancy [ 64 ]. As such, the positive feedback on the meaning of the content and the exercises proposed is important since within a preventive context, the goal is not symptom reduction or resolution but to emphasize the awareness of psychosocial functioning and the intricate relationship between emotions and behaviors. This would ultimately allow the development of transversally applicable personal resources that can be applied across life situations, thus fostering adaptation capacities at large. Women here appeared to have perceived these benefits, acknowledging that the intervention content was relevant to the pregnancy period and could potentially be helpful during the postpartum and in the future. However, it is worth highlighting a difference in its perceived usefulness as a function of the pregnancy period during which they thought that the intervention should be deployed. Participant B considered it suitable for the early pregnancy period, around the beginning of the second trimester, while participant C suggested it was more beneficial for the emotional tumult of the first trimester. It is noteworthy that both women followed the intervention during the same gestational week. Such individual differences in the perception of need are though important in terms of motivation in following the preventive intervention and in the foreseen impact of the information received.

Notwithstanding these individual differences, consistent patterns across women’s feedback were identified, particularly when asking them how their ideal digital tool developed to provide psychological support during pregnancy should be. The first thing they stressed regarded the content; within a preventive context, all women felt the need for more holistic information in which the medical and the psychological aspects are integrated, helping them understand how the 2 influence each other to then receive guidance in understanding what is “normal” and what is not. They expressed a preference for this information to be readily available, giving them the freedom to access it as they preferred. This could indeed support their empowerment [ 26 ] and aligns with the evidence highlighting that engaging in help-seeking–related behaviors increases the likelihood of perinatal women seeking further assistance in the future if needed [ 28 ]. As for receiving support for their own subjective experience, women pointed to chatbots, seeing them as having the potential to provide a 24/7 means to answer their pregnancy-related questions. This was viewed in the context of containing worries without the need to wait or continually seek assistance for potentially smaller concerns. However, all women consistently emphasized that chatbots should not be intended to substitute in-person support or human relationships more broadly. They highlighted the importance of the perception of contact and vicinity for pregnant women. Furthermore, although recognizing the potential benefits in preventive contexts, in situations of greater need and increased psychological symptoms, women stressed that chatbots should always be accompanied by human monitoring.

Keeping this in mind, it is important to reason about the technical problems encountered with the chatbot Juno. Except for the sole woman who did not skip any interactions (participant C), all other women reported their dissatisfaction with the technical problems encountered. Their reactions varied from feeling that the intervention content became less impactful to experiencing frustration, disappointment, confusion, and a perception of loss of control. These aspects are particularly significant in a mental health context, even if preventive, since such perceptions might reduce the willingness to follow the intervention by hindering their sense of agency. Notably, participant B, who participated with the desire to enrich her pregnancy experience, reported higher and more personal expectations toward the intervention. Consistently, her scores on the UX and UE questionnaires and the results from the semistructured interview suggest that she felt the most negative about the technical problems encountered. In contraposition, participant C, who assessed that she knew that interactions with Juno were “part of a research project,” acknowledged the technical problems but remained indifferent. However, it is crucial to consider the reported feelings of confusion and loss of control. While desiring a more personalized chatbot to receive answers that are more in tune with their individual needs, participants themselves emphasized the importance of clear explanations and reminders about the chatbot’s capabilities and limitations as it becomes more sophisticated and autonomous. This is essential to prevent overreliance on the chatbot and to avoid potential disappointment and iatrogenic effects that could decrease the likelihood of seeking help in the future. When discussing the use of chatbots like Juno as a means to deliver psychological content, participants acknowledged its effectiveness in providing momentary containment and serving as an initial step before seeking further support. However, they also underscored the need for human psychologists or professionals in preventive contexts. Participant B, in particular, expressed a desire for some “human” contact, even by telephone while interacting with Juno. This resonates with a compelling argument made by Sedlakova and Trachsel [ 40 ]; they conducted an epistemic analysis of chatbots’ adoption within mental health or therapeutic contexts, prompting the need to carefully reason about how chatbots can be perceived. As such, in line with women’s desire for increased chatbot personalization but worries linked to its potential increased freedom, the authors [ 40 ] suggested balancing the number of humanlike characteristics and features of chatbots and that their application should be confined to specific functions.

Focusing on the broader real-life application of apps and chatbots in contexts such as the health care system, beyond being highly desired, they were seen as tools that could bridge between women and clinical professionals. Moreover, in line with the above, women reported that having such tools would make them feel like their psychological well-being was accounted for together with their physical well-being since the former is felt neglected. Another aspect that has emerged is that they could help favor self-monitoring and monitoring from the clinician; existing literature does indicate that tools of this nature are acceptable to perinatal women as a means of monitoring mood symptoms [ 65 ]. In this regard, the Interactive Centre of Perinatal Excellence developed by the Australian Centre of Perinatal Excellence [ 66 , 67 ] is noteworthy. It is an interactive digital screening app integrated into the health care system and designed to facilitate screening for perinatal depression and anxiety symptoms. It provides women with feedback on the screening results while generating related reports for the clinician. It can support the prevention of perinatal mental health disorders by empowering women, streamlining the screening process, and saving time and resources for both women and clinicians. In such a context, the inclusion of a tool like Juno within an app to “educate” and guide women through their pregnancy and postpartum while allowing for symptom monitoring might hold great potential; what if, within such an app, the preventive BA intervention deployed through Juno was proposed to women showing mild and/or moderate depression symptoms?

The literature highlights consistent prevalence metrics of depression symptoms throughout the whole pregnancy period (worldwide, 20.7%; Europe, 17.9% [ 68 ]; Italy, 6%-22% [ 69 - 72 ]), pointing at it as among the main predictors of postpartum depression [ 16 , 73 ], with repercussions on the quality of life of women [ 74 ] as well as on the child’s development and well-being [ 75 - 77 ]. These metrics highlight the necessity of collaborative efforts in designing and implementing tailored programs, particularly in primary prevention (to prevent symptoms before they start) and secondary prevention (targeting individuals at risk or with subclinical symptoms) [ 78 ]. This is especially crucial, given the unique characteristics of peripartum depression, referring to its direct association with the challenges and bodily changes inherent to the perinatal period [ 79 ] and stressing the need for tailored intervention programs. In this regard, taken together, our results suggest that Juno holds potential for apps in a preventive context, which is of value considering the paucity of preventive perinatal tools [ 34 ]. However, it has also emerged that within this context, a tool like Juno is not deemed as sufficient. In this regard, the comments made by participant A are emblematic; she felt that what Juno could give within the broader perinatal period was like “a drop in the ocean.” This is further exemplified by the dropout of all women with medical conditions, which suggests that in its current form, the intervention deployed through Juno would have limited application. As such, data indicate that its real-life adoption might be scarce if not inserted within a broader context that can better signify its value while allowing us to account for women’s differences in need, which influences the type and amount of use they would make of it. Furthermore, beyond ensuring a better functioning of the tool itself, a thorough action plan linked to problem resolution should be defined and provided to women (both in research and real-life contexts). However, the evaluation of these issues resonated with literature emphasizing the advantages of incorporating dedicated prototyping and implementation phases during the co-design of digital tools [ 41 ].

Study Limitations and Future Directions

Although the results’ generalizability cannot a priori be expected in this study design, women’s high educational level and residency in northern Italy still represent a limitation. Perinatal depression symptoms tend to be higher among women with lower educational levels [ 80 ], suggesting a potential bias in the sample. In addition, the mentioned sample’s characteristics may reduce the variability of analyzed cases, impacting the generalizability of the findings. A further limitation regards data collection, as it relied on self-reports and semistructured interviews, which are indeed vulnerable to social desirability biases. Moreover, in this study, women’s experience with depression symptoms and their use of e-mental health tools were not measured, thus representing a limit of the study. Nonetheless, assessing these matters could provide valuable insights into their perceptions and potential use of chatbots during pregnancy. These dimensions warrant consideration in future studies to better understand the factors influencing women’s engagement with digital interventions during this critical period.

Being at the beginning of the product life cycle [ 45 , 46 ], referring specifically to the technical problems encountered, while they represent a limitation in the study, their management by the research team was invaluable in providing insights into the software used and the potential of Rasa. This understanding contributes to a more flexible problem-solving approach for addressing current and potential future issues. Proactively addressing such problems helps users maintain a sense of control and proficiency with the tool. In addition, these issues offer important information on how problem resolution, or lack thereof, impacts the overall UX. In this regard, it is noteworthy that the users did not express dissatisfaction with the simplicity of the solution, which primarily operated on rule-based mechanisms. This emphasizes the significance of incorporating user-centered design principles in developing natural language processing solutions that effectively meet end users’ needs and expectations.

Conclusions

In line with this, good practices can be outlined to construct appropriate validity and mitigate any negative effects on the user, thus ensuring ethical standards [ 81 - 86 ]: (1) ground intervention content in evidence-based data pertinent to the perinatal literature; (2) seek input from both end users and clinical professionals to evaluate needs and gather feedback on intervention content and e-mental health tool usability; (3) conduct feasibility and pilot-testing to ensure acceptability, feasibility, efficacy, and effectiveness, along with evaluating e-mental health tool use (both frequency and duration); (4) use adequate measures and evaluate appropriate outcomes to assess intervention success; (5) ensure that end users are provided with a clear informed consent regarding intervention purpose, content, and e-mental health tool capacities, risks, and limits; (6) incorporate safety measures, including clear procedures for managing situations of need and heightened distress, such as providing crisis support services and establishing connections with reference clinicians or public health services, while also monitoring end user mental health; (7) continuously monitor intervention progress to refine effectiveness and minimize potential negative effects; and (8) ensure clinical professionals are properly guided and informed with up-to-date evidence on available e-mental health interventions, their effectiveness, suitability, and safety. Aligning with this, to ensure a consistently high-quality technical solution for end users, substantial investments in assistance and infrastructure are imperative. The insights from this study underscore the importance of prioritizing UX and technical reliability to enhance the effectiveness and adoption of preventive perinatal tools like Juno in real-world contexts. Although Juno already aligns with ethical standards 1 to 5, the results of this study indicate that the tool’s capacities, risks, and limitations need to be greatly reported (point 5). In addition, safety measures were limited to self-reported depression, anxiety, and stress levels, with no specific process for monitoring intervention progress (point 6). Therefore, future developments of Juno should incorporate comprehensive safety measures and test their feasibility and acceptability. This includes integrating technological requirements to establish a more specific procedure for monitoring intervention progress (point 7).

Conflicts of Interest

None declared.

Definitions of digital intervention and technological advancement provided to participants during the semistructured interviews and the participants’ scores.

• Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res. Dec 19, 2010;12(5):e74. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Torous J, Bucci S, Bell IH, Kessing LV, Faurholt-Jepsen M, Whelan P, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. Oct 2021;20(3):318-335. [ FREE Full text ] [ CrossRef ] [ Medline ]
• mHealth: new horizons for health through mobile technologies: second global survey on eHealth. World Health Organization. 2011. URL: https://iris.who.int/handle/10665/44607 [accessed 2024-04-29]
• Fairburn CG, Patel V. The impact of digital technology on psychological treatments and their dissemination. Behav Res Ther. Jan 2017;88:19-25. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim J, Marcusson-Clavertz D, Yoshiuchi K, Smyth JM. Potential benefits of integrating ecological momentary assessment data into mHealth care systems. Biopsychosoc Med. 2019;13:19. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wang Q, Su M, Zhang M, Li R. Integrating digital technologies and public health to fight COVID-19 pandemic: key technologies, applications, challenges and outlook of digital healthcare. Int J Environ Res Public Health. Jun 04, 2021;18(11):6053. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vial S, Boudhraâ S, Dumont M. Human-centered design approaches in digital mental health interventions: exploratory mapping review. JMIR Ment Health. Jun 07, 2022;9(6):e35591. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dorst K, Dijkhuis J. Comparing paradigms for describing design activity. Des Stud. Apr 1995;16(2):261-274. [ CrossRef ]
• Dorst K. Design beyond design. She Ji J Des Econ Innov. Feb 2019;5(2):117-127. [ CrossRef ]
• Dorst K. Design research: a revolution-waiting-to-happen. Des Stud. Jan 2008;29(1):4-11. [ CrossRef ]
• Boyd H, McKernon S, Mullin B, Old A. Improving healthcare through the use of co-design. N Z Med J. Jun 29, 2012;125(1357):76-87. [ Medline ]
• Czajkowski SM, Powell LH, Adler N, Naar-King S, Reynolds KD, Hunter CM, et al. From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. Oct 2015;34(10):971-982. [ FREE Full text ] [ CrossRef ] [ Medline ]
• WHO recommendations on maternal and newborn care for a positive postnatal experience. World Health Organization. URL: https://www.who.int/publications/i/item/9789240045989 [accessed 2024-04-29]
• The millennium development goals report 2013. WHO-UNFPA. URL: https://www.unfpa.org/publications/millennium-development-goals-report-2013 [accessed 2024-04-29]
• Sánchez-Polán M, Franco E, Silva-José C, Gil-Ares J, Pérez-Tejero J, Barakat R, et al. Exercise during pregnancy and prenatal depression: a systematic review and meta-analysis. Front Physiol. 2021;12:640024. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hutchens BF, Kearney J. Risk factors for postpartum depression: an umbrella review. J Midwifery Womens Health. Jan 22, 2020;65(1):96-108. [ CrossRef ] [ Medline ]
• Walker AL, de Rooij SR, Dimitrova MV, Witteveen AB, Verhoeven CJ, de Jonge A, et al. Psychosocial and peripartum determinants of postpartum depression: findings from a prospective population-based cohort. The ABCD study. Compr Psychiatry. Jul 2021;108:152239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Howard LM, Khalifeh H. Perinatal mental health: a review of progress and challenges. World Psychiatry. Oct 2020;19(3):313-327. [ CrossRef ] [ Medline ]
• Clark M. Maternal depression costs society billions each year, new model finds. Center For Children and Families. 2019. URL: https:/​/ccf.​georgetown.edu/​2019/​05/​31/​maternal-depression-costs-society-billions-each-year-new-model-finds/​ [accessed 2024-04-29]
• Pollack LM, Chen J, Cox S, Luo F, Robbins CL, Tevendale HD, et al. Healthcare utilization and costs associated with perinatal depression among Medicaid enrollees. Am J Prev Med. Jun 2022;62(6):e333-e341. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rokicki S, McGovern M, Von Jaglinsky A, Reichman NE. Depression in the postpartum year and life course economic trajectories. Am J Prev Med. Feb 2022;62(2):165-173. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Button S, Thornton A, Lee S, Shakespeare J, Ayers S. Seeking help for perinatal psychological distress: a meta-synthesis of women’s experiences. Br J Gen Pract. Aug 28, 2017;67(663):e692-e699. [ CrossRef ]
• Fonseca A, Ganho-Ávila A, Lambregtse-van den Berg M, Lupattelli A, Rodriguez-Muñoz MF, Ferreira P, et al. Emerging issues and questions on peripartum depression prevention, diagnosis and treatment: a consensus report from the cost action riseup-PPD. J Affect Disord. Sep 01, 2020;274:167-173. [ CrossRef ] [ Medline ]
• Fonseca A, Silva S, Canavarro MC. Depression literacy and awareness of psychopathological symptoms during the perinatal period. J Obstet Gynecol Neonatal Nurs. Mar 2017;46(2):197-208. [ CrossRef ] [ Medline ]
• Hussain-Shamsy N, Shah A, Vigod SN, Zaheer J, Seto E. Mobile health for perinatal depression and anxiety: scoping review. J Med Internet Res. Apr 13, 2020;22(4):e17011. [ FREE Full text ] [ CrossRef ] [ Medline ]
• van den Heuvel JF, Groenhof TK, Veerbeek JH, van Solinge WW, Lely AT, Franx A, et al. eHealth as the next-generation perinatal care: an overview of the literature. J Med Internet Res. Jun 05, 2018;20(6):e202. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Richards DA. Stepped care: a method to deliver increased access to psychological therapies. Can J Psychiatry. Apr 01, 2012;57(4):210-215. [ CrossRef ] [ Medline ]
• Fonseca A, Gorayeb R, Canavarro MC. Women׳s help-seeking behaviours for depressive symptoms during the perinatal period: socio-demographic and clinical correlates and perceived barriers to seeking professional help. Midwifery. Dec 2015;31(12):1177-1185. [ CrossRef ] [ Medline ]
• Dimidjian S, Barrera M, Martell C, Muñoz RF, Lewinsohn PM. The origins and current status of behavioral activation treatments for depression. Annu Rev Clin Psychol. 2011;7:1-38. [ CrossRef ] [ Medline ]
• Dimidjian S, Goodman SH, Sherwood NE, Simon GE, Ludman E, Gallop R, et al. A pragmatic randomized clinical trial of behavioral activation for depressed pregnant women. J Consult Clin Psychol. Jan 2017;85(1):26-36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jacobson NS, Martell CR, Dimidjian S. Behavioral activation treatment for depression: returning to contextual roots. Clin Psychol Sci Pract. Feb 2001;8(3):255-270. [ CrossRef ]
• Soucy Chartier I, Provencher MD. Behavioural activation for depression: efficacy, effectiveness and dissemination. J Affect Disord. Mar 05, 2013;145(3):292-299. [ CrossRef ] [ Medline ]
• Kanter JW, Manos RC, Bowe WM, Baruch DE, Busch AM, Rusch LC. What is behavioral activation? a review of the empirical literature. Clin Psychol Rev. Aug 2010;30(6):608-620. [ CrossRef ] [ Medline ]
• Mancinelli E, Bassi G, Gabrielli S, Salcuni S. The efficacy of digital cognitive-behavioral interventions in supporting the psychological adjustment and sleep quality of pregnant women with sub-clinical symptoms: a systematic review and meta-analysis. Int J Environ Res Public Health. Aug 03, 2022;19(15):9549. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mancinelli E, Dell'Arciprete G, Pattarozzi D, Gabrielli S, Salcuni S. Digital behavioral activation interventions during the perinatal period: scoping review. JMIR Pediatr Parent. Feb 28, 2023;6:e40937. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mancinelli E, Gabrielli S, Salcuni S. A digital behavioral activation intervention (JuNEX) for pregnant women with sub-clinical depression symptoms: an explorative co-design study. JMIR Hum Factors. May 16, 2024;11:e50098. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Provoost S, Lau HM, Ruwaard J, Riper H. Embodied conversational agents in clinical psychology: a scoping review. J Med Internet Res. May 09, 2017;19(5):e151. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaffney H, Mansell W, Tai S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Ment Health. Oct 18, 2019;6(10):e14166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kellogg KC, Sadeh-Sharvit S. Pragmatic AI-augmentation in mental healthcare: key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians. Front Psychiatry. Sep 6, 2022;13:990370. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sedlakova J, Trachsel M. Conversational artificial intelligence in psychotherapy: a new therapeutic tool or agent? Am J Bioeth. May 2023;23(5):4-13. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Noorbergen TJ, Adam MT, Teubner T, Collins CE. Using co-design in mobile health system development: a qualitative study with experts in co-design and mobile health system development. JMIR Mhealth Uhealth. Nov 10, 2021;9(11):e27896. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baxter P, Jack S. Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep. Jan 14, 2015;13(4):544-559. [ CrossRef ]
• World Medical Association. Dichiarazione di helsinki della world medical association principi etici per la ricerca biomedica che coinvolge gli esseri umani. Evidence. 1964;5(10):e1000059. [ FREE Full text ]
• Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ji X, Abdoli S. Challenges and opportunities in product life cycle management in the context of industry 4.0. Procedia CIRP. 2023;119:29-34. [ CrossRef ]
• Kiritsis D, Bufardi A, Xirouchakis P. Research issues on product lifecycle management and information tracking using smart embedded systems. Adv Eng Inform. Jul 2003;17(3-4):189-202. [ CrossRef ]
• Lejuez CW, Hopko DR, Acierno R, Daughters SB, Pagoto SL. Ten year revision of the brief behavioral activation treatment for depression: revised treatment manual. Behav Modif. Mar 2011;35(2):111-161. [ CrossRef ] [ Medline ]
• Bassi G, Giuliano C, Perinelli A, Forti S, Gabrielli S, Salcuni S. A virtual coach (Motibot) for supporting healthy coping strategies among adults with diabetes: proof-of-concept study. JMIR Hum Factors. Jan 21, 2022;9(1):e32211. [ FREE Full text ] [ CrossRef ] [ Medline ]
• The future of customer experience. RASA Technologies. URL: https://rasa.com/ [accessed 2023-11-30]
• Dinesh PM, Sujitha V, Salma C, Srijayapriya B. A review on natural language processing: back to basics. In: Raj JS, Iliyasu AM, Bestak R, Baig ZA, editors. Lecture Notes on Data Engineering and Communications Technologies. Singapore, Singapore. Springer; 2021:655-661.
• Shevlin M, Butter S, McBride O, Murphy J, Gibson-Miller J, Hartman TK, et al. Measurement invariance of the patient health questionnaire (phq-9) and generalized anxiety disorder scale (GAD-7) across four European countries during the COVID-19 pandemic. BMC Psychiatry. Mar 01, 2022;22(1):154. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Benvenuti P, Ferrara M, Niccolai C, Valoriani V, Cox JL. The Edinburgh postnatal depression scale: validation for an Italian sample. J Affect Disord. May 1999;53(2):137-141. [ CrossRef ] [ Medline ]
• Cox JL, Holden JM, Sagovsky R. Detection of postnatal depression. Development of the 10-item Edinburgh postnatal depression scale. Br J Psychiatry. Jun 2, 1987;150(06):782-786. [ CrossRef ] [ Medline ]
• American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 4th edition. New York, NY. American Psychiatric Press; 1994.
• Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [ CrossRef ] [ Medline ]
• Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. Dec 1983;24(4):385. [ CrossRef ]
• Mondo M, Sechi C, Cabras C. Psychometric evaluation of three versions of the Italian Perceived Stress Scale. Curr Psychol. Jan 08, 2019;40(4):1884-1892. [ CrossRef ]
• Manos RC, Kanter JW, Luo W. The behavioral activation for depression scale-short form: development and validation. Behav Ther. Dec 2011;42(4):726-739. [ CrossRef ] [ Medline ]
• Armento ME, Hopko DR. The Environmental Reward Observation Scale (EROS): development, validity, and reliability. Behav Ther. Jun 2007;38(2):107-119. [ CrossRef ] [ Medline ]
• Domnich A, Arata L, Amicizia D, Signori A, Patrick B, Stoyanov S, et al. Development and validation of the Italian version of the mobile application rating scale and its generalisability to apps targeting primary prevention. BMC Med Inform Decis Mak. Jul 07, 2016;16(1):83. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O’Brien HL, Cairns P, Hall M. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. Int J Hum Comput Interact. Apr 2018;112:28-39. [ CrossRef ]
• Benoit K, Watanabe K, Wang H, Nulty P, Obeng A, Müller S, et al. quanteda: an R package for the quantitative analysis of textual data. J Open Source Softw. 2020;3(30):774. [ FREE Full text ] [ CrossRef ]
• Yan SJ, Wadley G, Vo T, Xiao Y, Truong H. Design and evaluation of a prototype ‘hibaby’ for pregnancy assistance. In: Proceedings of the 2022 International Conference on Human Machine Interaction. 2022. Presented at: ICHMI '22; May 6-8, 2022:53-59; Beijing, China. URL: https://dl.acm.org/doi/abs/10.1145/3560470.3560477 [ CrossRef ]
• Bjelica A, Cetkovic N, Trninic-Pjevic A, Mladenovic-Segedi L. The phenomenon of pregnancy — a psychological view. Ginekol Pol. Feb 28, 2018;89(2):102-106. [ CrossRef ]
• Varma DS, Mualem M, Goodin A, Gurka KK, Wen TS, Gurka MJ, et al. Acceptability of an mHealth app for monitoring perinatal and postpartum mental health: qualitative study with women and providers. JMIR Form Res. Jun 07, 2023;7:e44500. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Blackmore R, Boyle JA, Gray KM, Willey S, Highet N, Gibson-Helm M. Introducing and integrating perinatal mental health screening: development of an equity-informed evidence-based approach. Health Expect. Oct 24, 2022;25(5):2287-2298. [ FREE Full text ] [ CrossRef ] [ Medline ]
• ICOPE: perinatal mental health digital screening. Center of Perinatal Excellence. URL: https://www.cope.org.au/health-professionals/icope-digital-screening/ [accessed 2024-04-29]
• Yin X, Sun N, Jiang N, Xu X, Gan Y, Zhang J, et al. Prevalence and associated factors of antenatal depression: systematic reviews and meta-analyses. Clin Psychol Rev. Feb 2021;83:101932. [ CrossRef ] [ Medline ]
• Agostini F, Neri E, Salvatori P, Dellabartola S, Bozicevic L, Monti F. Antenatal depressive symptoms associated with specific life events and sources of social support among Italian women. Matern Child Health J. May 11, 2015;19(5):1131-1141. [ CrossRef ] [ Medline ]
• Cena L, Mirabella F, Palumbo G, Gigantesco A, Trainini A, Stefana A. Prevalence of maternal antenatal and postnatal depression and their association with sociodemographic and socioeconomic factors: a multicentre study in Italy. J Affect Disord. Jan 15, 2021;279:217-221. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Corbani IE, Rucci P, Iapichino E, Quartieri Bollani M, Cauli G, Ceruti MR, et al. Comparing the prevalence and the risk profile for antenatal depressive symptoms across cultures. Int J Soc Psychiatry. Nov 14, 2017;63(7):622-631. [ CrossRef ] [ Medline ]
• Giardinelli L, Innocenti A, Benni L, Stefanini MC, Lino G, Lunardi C, et al. Depression and anxiety in perinatal period: prevalence and risk factors in an Italian sample. Arch Womens Ment Health. Feb 29, 2012;15(1):21-30. [ CrossRef ] [ Medline ]
• Caparros-Gonzalez RA, Romero-Gonzalez B, Strivens-Vilchez H, Gonzalez-Perez R, Martinez-Augustin O, Peralta-Ramirez MI. Hair cortisol levels, psychological stress and psychopathological symptoms as predictors of postpartum depression. PLoS One. Aug 28, 2017;12(8):e0182817. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lagadec N, Steinecker M, Kapassi A, Magnier AM, Chastang J, Robert S, et al. Factors influencing the quality of life of pregnant women: a systematic review. BMC Pregnancy Childbirth. Nov 23, 2018;18(1):455. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Betts KS, Williams GM, Najman JM, Alati R. The relationship between maternal depressive, anxious, and stress symptoms during pregnancy and adult offspring behavioral and emotional problems. Depress Anxiety. Feb 30, 2015;32(2):82-90. [ CrossRef ] [ Medline ]
• Field T, Diego M, Hernandez-Reif M, Figueiredo B, Deeds O, Ascencio A, et al. Comorbid depression and anxiety effects on pregnancy and neonatal outcome. Infant Behav Dev. Feb 2010;33(1):23-29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Talge NM, Neal C, Glover V, Early Stress‚ Translational Research and Prevention Science Network: Fetal and Neonatal Experience on Child and Adolescent Mental Health. Antenatal maternal stress and long-term effects on child neurodevelopment: how and why? J Child Psychol Psychiatry. Mar 2007;48(3-4):245-261. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baker J. Three levels of health promotion/disease prevention. In: Baker J, editor. Contemporary Health Issues. Davis, CA. LibreTexts; 2020.
• Batt MM, Duffy KA, Novick AM, Metcalf CA, Epperson CN. Is postpartum depression different from depression occurring outside of the perinatal period? a review of the evidence. Focus (Am Psychiatr Publ). Apr 2020;18(2):106-119. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Papadopoulou SK, Pavlidou E, Dakanalis A, Antasouras G, Vorvolakos T, Mentzelou M, et al. Postpartum depression is associated with maternal sociodemographic and anthropometric characteristics, perinatal outcomes, breastfeeding practices, and mediterranean diet adherence. Nutrients. Sep 04, 2023;15(17):3853. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bächle TC, Wernick A. The Futures of eHealth, Social, Ethical and Legal Challenges. Berlin, Germany. Alexander von Humboldt Institute for Internet and Society; 2019.
• Balcombe L. AI chatbots in digital mental health. Informatics. Oct 27, 2023;10(4):82. [ CrossRef ]
• Jokinen A, Stolt M, Suhonen R. Ethical issues related to eHealth: an integrative review. Nurs Ethics. Mar 2021;28(2):253-271. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maeckelberghe E, Zdunek K, Marceglia S, Farsides B, Rigby M. The ethical challenges of personalized digital health. Front Med (Lausanne). Jun 19, 2023;10:1123863. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wykes T, Lipshitz J, Schueller SM. Towards the design of ethical standards related to digital mental health and all its applications. Curr Treat Options Psych. Jul 5, 2019;6(3):232-242. [ CrossRef ]
• Wykes T, Schueller S. Why reviewing apps is not enough: transparency for trust (T4T) principles of responsible health app marketplaces. J Med Internet Res. May 02, 2019;21(5):e12390. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 21.03.24; peer-reviewed by AR Yameogo, C-M Huang; comments to author 24.04.24; revised version received 14.05.24; accepted 17.06.24; published 14.08.24.

©Elisa Mancinelli, Simone Magnolini, Silvia Gabrielli, Silvia Salcuni. Originally published in JMIR Formative Research (https://formative.jmir.org), 14.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

• Advanced search

Advanced Search

First contact physiotherapy: an evaluation of clinical effectiveness and costs

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• ORCID record for Nicola E Walsh
• ORCID record for Serena Halls
• ORCID record for Rachel Thomas
• ORCID record for Alice Berry
• ORCID record for Cathy Liddiard
• ORCID record for Margaret E Cupples
• ORCID record for Heather Gage
• ORCID record for Daniel Jackson
• ORCID record for Fiona Cramp
• ORCID record for Hannah Stott
• ORCID record for Paula Kersten
• ORCID record for Justin Jagosh
• ORCID record for Peter Williams
• Figures & Data

Background First contact physiotherapy practitioners (FCPPs) are embedded within general practice, providing expert assessment, diagnosis, and management plans for patients with musculoskeletal disorders (MSKDs), without the prior need for GP consultation.

Aim To determine the clinical effectiveness and costs of FCPP models compared with GP-led models of care.

Design and setting Multiple site case-study design of general practices in the UK.

Method General practice sites were recruited representing the following three models: 1) GP-led care; 2) FCPPs who could not prescribe or inject (FCPPs-standard [St]); and 3) FCPPs who could prescribe and/or inject (FCPPs-additional qualifications [AQ]). Patient participants from each site completed outcome data at baseline, 3 months, and 6 months. The primary outcome was the SF-36 Physical Component Summary (PCS) score. Healthcare usage was collected for 6 months.

Results In total, 426 adults were recruited from 46 practices across the UK. Non-inferiority analysis showed no significant difference in physical function (SF-36 PCS) across all three arms at 6 months ( P = 0.667). At 3 months, a significant difference in numbers improving was seen between arms: 54.7% ( n = 47) GP consultees, 72.4% ( n = 71) FCPP-St, and 66.4% ( n = 101) FCPP-AQ ( P = 0.037). No safety issues were identified. Following initial consultation, a greater proportion of patients received medication (including opioids) in the GP-led arm (44.7%, n = 42), compared with FCPP-St (18.4%, n = 21) and FCPP-AQ (24.7%, n = 40) ( P <0.001). NHS costs (initial consultation and over 6-month follow-up) were significantly higher in the GP-led model (median £105.5 per patient) versus FCPP-St (£41.0 per patient) and FCPP-AQ (£44.0 per patient) ( P <0.001).

Conclusion FCPP-led models of care provide safe, clinically effective patient management, with cost-benefits and reduced opioid use in this cohort.

• general practice
• physiotherapy
• delivery of health care
• musculoskeletal diseases
• Introduction

General practice is experiencing unprecedented demand for appointments at a time when the number of fully qualified GPs is falling, part-time working is increasing, and average patient caseload is rising. 1 The Additional Roles Reimbursement Scheme was introduced in 2019 with the intention of growing the capacity of the primary care workforce. 2 First contact physiotherapy practitioners (FCPPs) were one of five professional roles initially identified for expedited implementation, 2 in recognition of the growing demands musculoskeletal disorders (MSKDs) place on general practice, which account for up to 30% of consultations. 3 FCPPs have an extended appointment time (normally 20 minutes) to assess, diagnose, and determine the most appropriate interventions and manage onward referral for patients without the prior need for GP consultation. 4 Some FCPPs also have the capability to provide injection therapy, and following legislation change in 2013, licensed physiotherapists can independently prescribe, including, since 2015, some controlled drugs. 5 By 2024, all adults in England consulting with a suspected MSKD should be offered a consultation with a FCPP within their local practice. 6

Since its inception, local service evaluations indicate that FCPPs reduce the need for GP consultation, referral to secondary care services, and prescribed medications, while improving patient and staff satisfaction. 7 The only large-scale evaluation of FCPP was conducted as part of an NHS England national pilot of the initiative and reported against pre-determined criteria including the following: re-consultation rates with the GP; improvements in patient symptoms at 3 months; provision of self-management and/or exercise advice for the condition; and impact on ability to work. 8 Pre-determined criteria were largely successfully met, apart from limited information on presenteeism and the ability to work. While this evaluation provided important data on the potential of FCPP, there was no insight regarding longer-term clinical outcomes, use of healthcare resources, or differences in outcomes compared with traditional GP-led models of care.

The current study aimed to determine the impact of FCPP on clinical outcomes and healthcare resource use for 6 months post-consultation compared with GP-led models of care.

How this fits in

Setting and practice recruitment

General practices across the UK were invited to participate either via expressions of interest in response to a previous survey regarding FCPP provision, 9 or through advertisement via Clinical Research Networks. The aim was to recruit across all four nations, from a range of urban and rural areas, and differing levels of deprivation; deprivation index was based on practice report and confirmed by nationally available data. 10 – 13

Description of services

General practice study sites were categorised into the following three study arms, according to their existing service provision:

no FCPP service: MSKD management with GP-led consultation (‘GP’);

standard FCPP with no additional competencies for prescribing and/or injecting (‘FCPP-St’); and

FCPP with additional qualifications to prescribe and/or inject (‘FCPP-AQ’).

Participant recruitment

Patients who attended appointments for MSKDs in the study sites were given recruitment materials by the clinician or an allocated practice staff member. They were invited to contact the study team for further information, or to express their willingness to participate. Volunteers were screened for eligibility.

The inclusion criteria were as follows: 1) patients consulting with a suspected MSKD episode, defined as any acute or chronic disorder related to the spinal or peripheral musculoskeletal (MSK) system; 2) patients not consulted for the same problem in preceding 3 months; and 3) patients aged ≥18 years. The exclusion criteria were as follows: 1) receiving palliative care; and 2) non-English speaking and unwilling to provide informed consent and communicate through an interpreter.

Eligible participants provided written, informed consent. Recruitment started in December 2019, slowed in January 2020, owing to the emerging COVID-19 pandemic, and paused in March 2020. Recruitment re-started under COVID-19 restrictions in July 2020 and ended in April 2022. Final assessments were completed in October 2022.

Data collection

Information on age, gender, reason for consultation, MSK risk (using STarT MSK), education, and employment were collected by telephone at baseline (post-consultation). Participants were also asked about their consultation experience and any safety concerns (to be reported elsewhere). There were no notable differences across groups.

Questionnaires regarding Patient Reported Outcome Measures (PROMs) were posted to participants following initial consultation (baseline) and at 3 months and 6 months post-consultation. The questionnaires were self-completed and returned by post. The primary outcome measure was the change from baseline to 6 months in the SF-36 Physical Component Summary (PCS) score. 14 Secondary clinical outcomes were SF-36 Mental Component Summary score; Musculoskeletal Health Questionnaire (MSK-HQ, total and physical); perceived safety of health care, using the healthcare experience in general practice survey, short form (Patient Reported Experiences and Outcomes of Safety in Primary Care; PREOS-PC Q5), on a 10-point scale: completely unsafe (0) to completely safe (10); and Roland–Morris Disability Questionnaire (for patients with low back pain). EQ-5D-5L, a generic measure of health-related quality of life, was gathered for use in the economic evaluation. 15

Sample size

The total participants required per arm was 181 across 39 sites. This was based on a non-inferiority margin of 2 units in SF-36 PCS scale, 14 a minimal clinically important difference of 4 points 16 and standard deviation (SD) 6.5, 17 a one-sided P = 0.05 non-inferior hypothesis test, with 80% power, a design effect of 1.09 for a cluster size of 14 and an intraclass correlation coefficient (ICC) of 0.0075, 18 and 20% attrition. COVID-19 impacted recruitment, so figures were revisited. Actual attrition rates were used (5%) and number of sites were increased ( n = 46), which required a total sample size of n = 462 ( n = 154 per arm).

Data analyses

The primary outcome was the change in SF-36 PCS score from baseline to 6 months compared between arms, using a one-way analysis of variance; in case of difference, a post-hoc unpaired t -test was performed. Further comparisons were undertaken in the context of stepwise linear regression modelling, incorporating demographic and clinical data, including baseline SF-36 PCS score. Outcomes from baseline to 3 months are also reported.

Economic analysis

The base case economic analysis adopted an NHS and social care perspective. Information on service use related to the MSK condition was gathered retrospectively by telephone interview at 3 months and 6 months, using a tailored version of the Client Service Receipt Inventory (CSRI). 19 This included: NHS and private healthcare services (primary, community, accident and emergency [A&E], outpatient referrals, and inpatient stays) and social care. Unit costs 20 , 21 were applied to service use and summed (months 1–6) at the participant level, including the cost of the index consultation (see Supplementary Information S1). Group costs were inspected and compared. Owing to the skewed nature of the total costs data, stepwise logistic regression was used to model the presence or absence of additional costs over and above the cost of the initial presentation, with service model as a dummy variable and baseline demographic and clinical factors as covariates. A societal perspective was included through consideration of self-reported days off work and inability to perform usual activities, and the private perspective through out-of-pocket expenditures.

Analyses were carried out using IBM SPSS Statistics (version 27). Database access can be requested via: http://researchdata.uwe.ac.uk/703 .

A total of 426 participants were recruited from 46 general practices across the UK, with a range of deprivation indices and rural or urban locations. Of the 426 participants, there were 110 (25.8%) from GP-led care, 124 (29.1%) from FCPP-St, and 192 (45.1%) from FCPP-AQ. A total of 46 GP practices were involved: 13 GP-led care practices (with 1, 2, 2, 5, 6, 6, 7, 10, 11, 14, 14, 15, and 17 participants), 15 FCPP-St practices (with 1, 3, 3, 3, 4, 4, 5, 7, 7, 9, 9, 14, 15, 17, and 23 participants), and 18 FCPP-AQ practices (with 1, 1, 4, 4, 6, 8, 8, 9, 11, 12, 14, 15, 15, 16, 16, 16, 17, and 19 participants). The study completion rates in each arm for PROMs and CSRIs, along with attrition patterns, can be seen in Supplementary Table S1.

Mean age was 63 years (SD 13.2); 34.1% ( n = 145) were male and 97.8% ( n = 408) reported White ethnicity. There were no statistically significant differences in individual baseline demographics between arms. There was some discrepancy in practice-level deprivation across arms, with a higher representation of low deprived practices in the FCPP-St arm ( Table 1 ). Data were returned at all three time points by 377 (88.5%) participants, including 320 (75.1%) who provided completed PROM and CSRI data. Details of attrition from the study are given in Supplementary Table S1.

• View inline

Baseline demographics: summary statistics with comparison of the three service models

Clinical data revealed no statistically significant differences between arms at baseline, except for the EQ-5D-5L (visual analogue scale [VAS]; better state of health reported in FCPP-St model) and for MSK-HQ total (a more desirable MSK status was indicated in FCPP-St model). Participants reported a range of peripheral and spinal diagnoses (up to two pain sites); given the previously reported high incidence of low back pain in primary care, 18 a 24.9% ( n = 106/426) prevalence was noted ( Table 2 ).

Baseline clinical summary for each of the three service models

Outcomes analysis

The primary outcome variable was the change in SF-36 PCS score from baseline to 6 months; in an unadjusted analysis, no statistically significant difference was found between arms ( Table 3 ). This was confirmed under linear regression, with a final model ( R 2 = 0.138, n = 332) predicting change = 15.074–0.333x (SF-36 PCS score at baseline) + 2.377 (if university educated) + 2.402 (if in full-time employment). Service model along with age at baseline, gender (male: yes/no), ethnic origin (White: yes/no), whether MSKD area at baseline included back (yes/no), whether MSKD area at baseline included knee or leg or hip or foot or ankle (yes/ no), and whether the presented MSK condition had affected employment or ability to perform usual activities (yes/ no) were not significant (see Supplementary Table S2).

Primary and secondary outcome changes from baseline to 3 months and from baseline to 6 months (positive changes indicate improvement)

However, when each of these change outcomes was simplified from the change in continuous score to an improved or worsened/stayed the same scenario, a statistically significant difference between arms was seen in two instances. At 3 months, the FCPP-St and FCPP-AQ service models delivered a statistically significant greater improvement rate for the primary outcome variable SF-36 PCS score compared with the GP-led service model ( P = 0.037). At 6 months, the FCPP-St and FCPP-AQ service models delivered a statistically significant greater improvement rate for the secondary outcome MSK-HQ physical compared with the GP-led service model ( P = 0.016; Table 3 ). No other statistically significant differences in outcomes were found between arms. No safety issues were identified.

Healthcare utilisation and costs

The initial consultation was assumed to be face-to-face with a GP, FCPP-St, or FCPP-AQ. CSRI data were available for 370/426 (86.9%) of participants at 3 months, 348 (81.7%) at 6 months (see Supplementary Table S1). Health service use after the initial consultation was low in all arms, most being within general practice; few participants reported hospital use. Key health service usage (GP and physiotherapist) and prescribing outcomes are shown in Table 4 . In the 3 months following initial consultation, a greater proportion of patients received medication (including opioids) in the GP-led arm (44.7%; n = 42) compared with FCPP-St (18.4%; n = 21) and FCPP-AQ (24.7%; n = 40) (χ 2 P <0.001) . A full breakdown of NHS service use, including medication prescribing, at 3 months and 6 months, is shown in Supplementary Tables S3 and S4. There was scattered use of the private sector while use of over-the-counter medications was commonplace (see Supplementary Tables S5 and S6).

Key self-reported NHS service usages associated with the presenting musculoskeletal condition, not including initial presentation, at 3 months and 6 months

Group mean total costs (health services, excluding medications) over 6-month follow-up for the three service models are shown in Table 5 . Comparisons were performed both excluding and including inpatient (planned MSK surgery) events, and assuming the FCPP-St and FCPP-AQ were both working at salary level band 7; a sensitivity analysis was performed with the FCPP-AQ costed at the higher band 8a. In each comparison, there is a statistically significant difference in costs between the three models ( P <0.001) with the GP model the more costly (median £105.5 per patient versus £41.0 for FCPP-St and £44.0 for FCPP-AQ in the band 7 calculation), and no statistically significant difference between the FCPP-St and FCPP-AQ. In the band 8a comparison, the FCPP-AQ was significantly more costly than the FCPP-St. Regarding days lost through inability to work or perform usual activities, the FCPP-St model showed greater reductions in days lost compared with GP-led care and FCPP-AQ, but there was no statistically significant difference between GP-led care and FCPP-AQ ( Table 6 ). Only eight participants had absences covered by sick notes in the first 3 months and three during the second period (two of which were new).

Total costs (£) summary statistics, months 0–6

Changes in days lost (unable to work or perform usual activities), with comparisons of the three service models

Backwards stepwise logistic regression to model the presence or absence of additional health service costs in months 0–6 over and above the initial presentation (excluding inpatient), with re-running of the final model to include additional participants for whom data were missing only for non-significant predictors, led to the model in Supplementary Table S2 (with Nagelkerke R 2 = 0.072, n = 334). The model demonstrates a significantly (2.181 times) higher likelihood of incurring additional costs after the initial consultation with a GP-led service model compared with a FCPP-St or FCPP-AQ service model. Higher scores in baseline SF-36 PCS score are also significantly associated with a lower likelihood of incurring additional cost (adjusted odds ratio of 0.966 implies that a participant with a baseline SF-36 PCS score, which is 10 points higher than another participant, is 0.966 10 = 0.708 times less likely to incur additional cost). No other predictors were statistically significant.

The analysis demonstrated that neither FCPP model was inferior in relation to clinical outcome at 6-month post-consultation compared with the GP-led model, but both were significantly less costly; P <0.001. There were no significant differences in quality-of-life changes (based on EQ-5D-5L) between the models at 3 months or 6 months, so given the cost differentials, no formal cost-effectiveness analysis was undertaken ( Tables 3 and 5 ).

Analysis demonstrated no statistically significant difference in clinical outcomes between different service models after 6 months. However, the GP-led model of care was approximately 2.5 times costlier than the FCPP-St and FCPP-AQ models. Furthermore, at 3 months, a greater proportion of patients who consulted with FCPPs had improved, compared with those who had consulted with GPs, and time off work or unable to perform usual activities was reduced in the FCPP-St consultees.

Strengths and limitations

To the authors’ knowledge, this is the first study that has compared GP-and FCPP-led models of care for MSKDs and included data from all four UK nations. It provides a robust overview of the service innovation to support decision making, and a qualitative analysis, which was conducted concurrently, will allow further interpretation of findings.

Recruitment was severely hampered by the COVID-19 pandemic, yet this study still provides the most extensive dataset of FCPPs to date. There was uneven recruitment across study arms and sites because the drive for FCPP recruitment, resulting from the Additional Roles Reimbursement Scheme, made the identification of GP-led sites challenging; and recruitment within some individual sites was lower than anticipated. At site level, there was some variation in deprivation across arms: the FCPP-St consisted of relatively more practices with lower levels of deprivation compared with the other arms, which may explain the higher levels of quality of life (EQ-5D-5L [VAS] and MSK-HQ) reported at baseline within this arm. However, while these differences were of statistical significance, neither was of clinical significance, based on previously reported levels of minimum clinical important difference 23 , 24 and, importantly, there was no difference in the primary outcome measure at baseline across arms. All sites that expressed an interest in participation were recruited, so this variation did not result from selective recruitment. Furthermore, at the level of individual participants, no significant differences were found between groups regarding levels of education or employment.

The sample was almost exclusively White and not representative of practice cohorts despite efforts for diverse recruitment at practice and patient level. Only 12/46 (26.1%) sites returned requested data regarding numbers invited to participate in the study, so how representative the study sample is of those eligible is unable to be reported. Much of the recruitment was undertaken under COVID-19 restrictions, which disproportionately impacted people of ethnic minority heritage, which may have influenced decision to participate, although in consultation with recruitment sites, it was identified that fewer people from ethnic minority communities consult FCPP staff. There was potential recruitment bias as not all eligible participants consented to join the study.

Comparison with existing literature

To the authors’ knowledge, this is the first study to show a comparison between GP and FCPP clinical outcomes and resource use, confirming the proposed benefits of the new model of care. While at 6 months there were no differences in patient improvement across the models studied, at 3 months a significantly greater proportion of patients who consulted with FCPPs had improved compared with GP consultees, with positive impact on ability to work or perform usual activities in FCPP-St ( P = 0.005). Previous work highlighted GP propensity for pharmacological management rather than guideline-based self-management and rehabilitation strategies, which may account for these differences; 25 – 27 indeed, a greater proportion of patients under GP-led care were prescribed medication, including opioid derivatives. The authors are unable to identify any factors in the study design that would account for this finding and believe this is a result of clinical decision making. Other work has shown that FCPPs with a licence to prescribe are still reluctant to use this intervention, instead choosing to use their capability to deprescribe where possible and intervene with non-pharmacological measures. 28

From an onward resource use perspective, data showed minimal reliance on other services within each model and therefore relatively low costs. For services that were used, there was a greater number of referrals onto outpatient physiotherapy by GPs, as would be expected; other work has suggested GP overuse of magnetic resonance imaging (MRI), but this was not found. 29 These data were obtained through self-report so may have been subject to recall bias. It is noted, however, that other studies report the similarities in self-report versus medical record review, and in some cases note greater accuracy with patient recall. 30

A previous evaluation in England reported that GP workload was positively impacted by FCPPs. It found most patients did not consult their GP with the same problem within 3 months of seeing the FCPP. 8 This concurs with the present study’s findings that only 23/276 (8.3%) of patients consulted the GP for the same problem having seen the FCPP, whereas many more (30.9%) initial GP consultees re-consulted the GP for the same problem within the study period ( Table 4 ).

A predominant aim of introducing FCPPs is to make better use of resources in general practice. The present study shows clear cost benefits to implementing FCPP models compared with GP-led care given the extent of MSKD consultations in primary care. 3

Implications for research and practice

This research supports continued implementation of FCPP in general practice as a safe, clinically effective, and cost-beneficial approach to managing people with MSKDs. Given FCPPs’ low reliance on prescription medications, it may also assist in reducing opioid prescriptions in primary care. Further research is required to understand why there appears to be disproportionate consultations from people of ethnic minority heritage to ensure appropriate access for all.

• Acknowledgments

The FRONTIER team would like to thank all participants for their time and valuable contribution to the study. The authors would also like to thank Gemma Artz, Pete Young, Jude Hancock, Alison Diaper, the Study Steering Committee, and the research team at Bristol, North Somerset and South Gloucestershire Integrated Care Board for their expertise and support.

This study was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research Programme (reference: 16/116/03). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

Ethical approval

Granted on 18 June 2019 (Integrated Research Application System ID: 261530; Research Ethics Committee reference: 19/NI/0108). Health Research Authority approval was granted on 25 June 2019.

University of the West of England Database Repository Database access can be requested via: http://researchdata.uwe.ac.uk/703 .

Freely submitted; externally peer reviewed.

Competing interests

The authors have declared no competing interests.

Discuss this article:

bjgp.org/letters

• Received October 25, 2023.
• Revision requested November 28, 2023.
• Accepted January 22, 2024.
• © The Authors

This article is Open Access: CC BY 4.0 licence ( http://creativecommons.org/licences/by/4.0/ ).

• British Medical Association
• NHS England, NHS Improvement
• NHS England
• Chartered Society of Physiotherapy
• Home Office
• Jordan KP ,
• 10. ↵ Ministry of Housing, Communities & Local Government . The English Indices of Deprivation 2019 Frequently asked questions (FAQs) , https://assets.publishing.service.gov.uk/media/5dfb3d7ce5274a3432700cf3/IoD2019_FAQ_v4.pdf (accessed 29 Jul 2024).
• Scottish Government
• Northern Ireland Statistics and Research Agency
• Kosinski M ,
• Bjorner JB ,
• EuroQol Research Foundation
• Aeschlimann A ,
• Ogollah RO ,
• Salisbury C ,
• Montgomery AA ,
• Hollinghurst S ,
• Personal Social Service Research Unit (PSSRU)
• Devlin NJ ,
• Reneman MF ,
• Speijer BLGN ,
• Lancaster G ,
• Hayward R ,
• Wallis JA ,
• Ackerman IN ,
• Brusco NK ,
• Smithson J ,
• Parkunan A ,
• Wallace E ,
• Moriarty F ,
• McGarrigle C ,

Online First

Thank you for recommending British Journal of General Practice.

NOTE: We only request your email address so that the person to whom you are recommending the page knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager

• Tweet Widget
• Facebook Like
• Google Plus One

Jump to section

More in this toc section.

• The first 100 days after childbirth: cross-sectional study of maternal clinical events and health needs from primary care
• Media depictions of primary care teleconsultation safety: a thematic analysis of UK newspapers

Related Articles

Cited by....

British Journal of General Practice

• ENVD 1120: Fundamentals of Architecture
• Browse Architecture Case Studies in Databases
• Browse Architecture Books

Ask A Librarian

Make an appointment

Chat, Call, Text

Email: [email protected]

Engineering, Science and Design Librarian

Art & Architecture Librarian

Browse Architecture Case Studies

• Birkhauser Building Types Database containing 750 architectural case studies each including technical drawings and photographs of the exterior and interior. Architectural plans including vector-based plans available for download. Includes multiple buildings types such as residential, office buildings, industrial complexes, museums, schools and other building types.

• Birkhauser Building Types Video Tutorial Transcript
• DETAIL Inspiration A full-text, image and reference database for architects in English and German sourced from DETAIL print magazine.

• DETAIL Inspiration Video Tutorial Transcript
• Next: Browse Architecture Books >>
• University of Colorado Boulder Libraries
• Research Guides
• .Site: Research By Course and Topic
• Last Updated: Aug 14, 2024 2:50 PM
• URL: https://libguides.colorado.edu/envd1120
• © Regents of the University of Colorado