delphi method research paper

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements

Daniel beiderbeck.

a Center for Sports and Management, WHU, Otto Beisheim School of Management, Erkrather Str. 224a, 40233 Düsseldorf, Germany

Nicolas Frevel

Heiko a. von der gracht.

b School of International Business and Entrepreneurship, Steinbeis University, Kalkofenstr. 53, 71083 Herrenberg, Germany

Sascha L. Schmidt

Vera m. schweitzer.

c Chair of Leadership, WHU, Otto Beisheim School of Management, Erkrather Str. 224a, 40233 Düsseldorf, Germany

Delphi is a scientific method to organize and structure an expert discussion aiming to generate insights on controversial topics with limited information. The technique has seen a rise in publication frequency in various disciplines, especially over the past decades. In April 2021, the term Delphi method yielded 28,200 search hits in Google Scholar for the past five years alone. Given the increasing level of uncertainty caused by rapid technological and social change around the globe, collective expert opinions and assessments are likely to gain even more importance. Therefore, the paper at hand presents technical recommendations derived from a Delphi study that was conducted amid the outbreak of the COVID-19 pandemic in 2020.

• • The paper comprehensively demonstrates how to prepare, conduct, and analyze a Delphi study. In this regard, it combines several methodological advancements of the recent past (e.g., dissent analyses, scenario analyses) with state-of-the-art impulses from other disciplines like strategic management (e.g., fuzzy clustering), psychology (e.g., sentiment analyses), or clinical trials (e.g., consensus measurement).
• • By offering insights on the variety of possibilities to exploit Delphi-based data, we aim to support researchers across all disciplines in conducting Delphi studies and potentially expand and improve the method's field of application.

Graphical Abstract

Specifications table

Method Basics and Co-Submitted Research

Basics of the delphi study.

The Delphi technique is a scientific method to organize and manage structured group communication processes with the aim of generating insights on either current or prospective challenges; especially in situations with limited availability of information [21 , 48 , 74 , 77] . As such, it has been frequently used in various scientific disciplines ranging from health care [14 , 29 , 51 , 62] , medicine [24 , 43 , 63 , 86] , education [15 , 72 , 88] , business [19 , 95 , 98] , engineering and technology [11 , 82] , social sciences [10 , 89] , to information management [4 , 81] , and environmental studies [83] . Irrespective of the focus in time or content, the Delphi technique builds on the anonymity of participating experts who are invited to assess and comment on different statements or questions related to a specific research topic [47 , 59] . Quantitative assessments traditionally include probability, impact, and desirability of occurrence, but are not limited to these. Further dimensions could refer to innovativeness, urgency, or (technical) feasibility, for instance. Moreover, participant-related information such as confidence or expertise can be collected [25 , 32 , 87] . In addition, especially in medical and clinical research, Delphi studies make use of rank-order questions, rating scales, or open questions, while often being designed to examine levels of consensus among experts [14 , 72 , 86] . In a Delphi survey, the aggregated group opinion is fed back to participants across multiple discussion rounds of the same set of theses. During this multi-round procedure, the rounds can be performed sequentially, or – with the help of dedicated software – immediately (so-called real-time Delphi) [2 , 35 , 36] . After each round, panelists have the possibility to review the aggregated results and to reconsider their assessment based on the added quantitative and qualitative information [12 , 53] . This structured group communication process is supposed to lead to a convergence – or divergence – of opinions, hence, producing more accurate results than traditional opinion-polling techniques [59] . Moreover, the Delphi method has advantages over in-person techniques such as group discussions or brainstorming sessions, as it rules out personal sensitivities among the experts and therefore avoids potentially destructive group dynamics [99] . The results of a Delphi survey can deliver stand-alone insights but are increasingly linked to scenario analytics, to fulfill idea-generation, consolidation, or judgment functions [68] .

Co-submitted research

The technical paper at hand builds on a Delphi study including scenario analysis, which is dealing with the impact of COVID-19 on the European football ecosystem [8] . The study included 110 international experts and was conducted amid the COVID-19 outbreak between April and May 2020. In times of deep uncertainty, participants evaluated the regulatory, economic, social, and technological implications of the pandemic on the European football ecosystem [97] . In this context, the study served two main purposes: on the one hand, it facilitated an expert discussion that was valuable for all participants as they faced a similar level of unprecedented ambiguity and thus shared common challenges. On the other hand, it aimed to advance the Delphi technique from a methodological point of view by offering a comprehensive analysis and combine cross-disciplinary features. For example, the authors conducted dissent analyses from the field of risk and emergency preparedness, while introducing a sentiment analysis of the field of psychology. The latter was of particular importance in times of crisis in order to interpret the experts' assessments against the backdrop of their individual situation or constitution. All in all, the Delphi method proved to be a suitable technique to manage a systematic online dialogue among experts while at the same time assuring scientific rigor to derive accurate results.

Structure and aim of this paper

We structure this technical paper following the three major phases of a Delphi-based research project: preparing, conducting, and analyzing. Each phase consists of different steps (as depicted in Fig. 1 ), which will be thoroughly explained in this paper. In this context, we provide a comprehensive overview of potential features and recent advancements in all three phases (see Table 1 , [5] , [6] , [17] , [23] , [26] , [28] , [30] , [31] , [37] , [39] , [41] , [46] , [55] , [57] , [67] , [64] , [65] , [79] , [85] ) and therefore complement and substantially extend recent methodic publications such as Schmalz et al. [80] . Thereby, we aim to support the research community in utilizing the Delphi technique for their respective disciplines by following a replicable, but still highly customizable approach.

Three Phases of Delphi-based Research

Overview of Potential Delphi Features.

Preparing a Delphi study (Phase One)

Thorough preparation is critical to ensure the validity and accuracy of a Delphi study [45 , 80] . In general, this phase pursues four different goals: (1) Definition of research goals , (2) definition of Delphi format , (3) definition of Delphi statements , and (4) definition of additional questions (see Fig. 2 ). To achieve these goals, we started with an initial conceptualization phase followed by two creative workshops in order to define our research goals and the Delphi format . Simultaneously, we conducted desk research to understand the current body of research and to identify the major challenges in the industry. Given the topicality of events around the pandemic, the existing body of research on the impact of COVID-19 on sports industries was scarce. Therefore, we decided to involve experts early in the process to define our overarching topics and thus our Delphi statements . To refine these, we conducted 17 formulation sessions with the research team and fed back the proposed statements as well as additional questions to our experts. Eventually, we also tested our statements with previously not involved researchers and experts to ensure the comprehensibility of our statements. To allow the research community to thoroughly understand and adapt this research process, we will describe each step in more detail below.

Goals and Time Estimates for Delphi Phases

Initial conceptualization

The initial conceptualization was necessary to define the overarching research goal , which – in our case – was twofold. On the one hand, we wanted to facilitate an expert discussion in the European football industry amid the COVID-19 crisis to thus provide practical added value to all participants who faced unprecedented challenges due to the pandemic. On the other hand, we wanted to gain accurate insights on the short-, mid-, and long-term effects of COVID-19 on European football by conducting a state-of-the-art Delphi study. To achieve these two goals, we compiled a research team with expertise in terms of content (i.e., European football) as well as methodology (i.e., Delphi technique). Given the urgency, we also developed a tight timeline for preparing and conducting our research – with roughly 5 weeks from initial conceptualization in mid-March 2020 to the actual survey launch in mid-April 2020.

Technical recommendation for “2.1 Initial conceptualization”

• • We deliberately included researchers with different expertise in terms of industry specifics, methodological experience, and statistical knowhow. We made good experience with this composition and motivate researchers to include methodological and statistical expertise in the research team as this significantly accelerates the research process.
• • We encourage scholars to put dedicated effort into defining a research direction as well as reviewing and assessing the suitability of the Delphi technique and potential software solutions.

Creative workshops

The creative workshops were used to define the Delphi format . For us, the Delphi format includes three central elements: (1) scope, (2) theory/framework, and (3) sequential or real-time conduction. In terms of scope, we decided to focus on the European football ecosystem both because we wanted to include experts from different backgrounds, organizations (i.e., clubs, leagues and associations, academia, football-related adjacencies), and from all of the five core European football markets (i.e., Germany, United Kingdom, Spain, France, Italy). This helped us to cover a broad range of perspectives and allowed us to get an international perspective on the impact of the pandemic.

From a theory/framework perspective, we conducted a literature review to identify an adequate structure on which we could base our research. To cover a wide range of potential effects of COVID-19 on the European football ecosystem, we decided to build on the PEST framework (political, economic, socio-cultural, technological) [42] and extended the political dimension with a regulatory perspective, which appeared to be more suitable in the context of football, so that we introduced the REST framework (Regulatory, Economic, Social, Technological) for our context [61 , 68] . In our second workshop, we discussed this framework with five previously not involved industry experts to obtain additional and unbiased perspectives. As a result, we decided to split the economic angle of our REST framework into two separate buckets focusing on revenue-related and cost-related economic effects. This modification towards a REEST structure (Regulatory, Economic – revenue, Economic – cost, Socio-cultural, Technological) helped us to refine our Delphi format by putting more emphasis on the economic pressure that many football-related organizations felt during the first lockdown in April 2020 and therefore increased the relevance of our study. We encourage researchers to not blindly follow existing frameworks but to adjust them to their needs as appropriate.

The decision for a sequential or real-time Delphi was made in favor of the real-time format due to the ambitious timeframe of the actual survey conduction and due to the improved user experience for participants, which often results in higher participation and lower drop-out rates [2 , 36] . For a more detailed discussion on decision criteria between sequential or real-time Delphi see Gnatzy et al. [35] .

Technical recommendation for “2.2 Creative workshops”

• • Defining a framework for the initial list of Delphi statements helped to structure our thinking and to involve experts for different topic areas. In addition, it contributes to the completeness of the set. We therefore highly recommend using a framework to structure and cluster Delphi statements.
• • If a framework is not applicable for specific research contexts, it is advisable to anchor the Delphi research based on existing theories. A good example of this is Winkler et al. [100] , who used the "organizational information processing theory (OIPT)" as a foundation for their Delphi study to analyze decision making in emerging markets.
• • The real-time Delphi format allowed us to involve a larger number of experts in a shorter amount of time. We observed that much more than 50% of the experts joined the Delphi discussion after three weeks of the initial launch and then engaged vividly in the discussion. The sequential approach tends to complicate participation for late joiners so that we would recommend a real-time format in case the schedule is tight.

Desk research

Dedicated desk research was performed in between and after the two creative workshops. As Schmalz et al. [80] conclude, a thorough literature review is indispensable for a Delphi study. However, this does not necessarily need to be limited to the scientific body of research – particularly in the case of prospective, forecast studies for which existing literature might be scarce. In the special case of our co-submitted research, for example, there was almost no existing research on the consequences of COVID-19 at the beginning of the crisis. Therefore, we also focused on the popular press to identify the most urgent issues for the European football ecosystem. To do so, we screened international newspapers and pertinent sports management magazines to get a first idea for potential Delphi statements . This initial long list of statements was captured in Microsoft Excel and shared with the five above-mentioned experts who participated in our second workshop. Their input was used to further expand the statement long list which then served as a basis for our initial expert interviews.

Technical recommendation for “2.3 Desk research”

• • The desk research should be structured based on a framework or theory, too. The lack of a proper foundation jeopardizes not only the quality of desk research but also the comprehensiveness of the literature review that should be part of all scientific publications.

Initial expert interviews

Based on the modified REEST framework, we decided to conduct three initial expert interviews for each of our five framework dimensions, following a semi-structured approach [1] . The panel was meant to represent all stakeholders within the European football ecosystem, which is why we interviewed subject matter experts from all five European target countries as well as the four stakeholder groups. In total, we contacted 21 experts via email or directly via phone and achieved a final response rate of 71 percent. We recommend activating contacts from (wider) personal networks to increase the response rate and to speed up the process so that two weeks from first inquiry to final interview becomes a realistic target.

We scheduled all interviews for 60 min and spent roughly 15 min explaining our research goals as well as the characteristics of a Delphi survey. We then spent 30 min discussing the main challenges caused by COVID-19 for the expert's respective area of expertise and developed/refined potential Delphi statements and saved the last 15 min for open questions and follow-up information. The latter included an invitation to the actual Delphi survey as well as an inquiry to nominate a list of potential experts as proposed by Belton et al. [9] . After each interview, members of the research team reviewed the findings and conducted formulation sessions, which are described in the next section. The results of these sessions were then used for the next interview so that we iteratively developed our Delphi statements . At the end of the process, we shared the short list of statements in Microsoft Excel with all experts and received their proposed prioritization which helped us identify our final set of 15 statements.

Technical recommendation for “2.4 Initial expert interviews”

• • We made a very good experience involving experts outside the research team early in the process. However, the amount of 15 initial experts was surely the upper limit. For most prospective Delphi studies, 5 to 8 initial experts should be sufficient.
• • Contacting experts from the closer network, accelerated our process significantly, since we were able to speak to most experts on short notice with a very high response rate. However, research teams should bear in mind potential biases and homogeneous thinking within the close network.

Formulation sessions

The accurate wording of statements is central to the quality of Delphi studies as it can reduce biases and increase response variance [27 , 57] . Therefore, we conducted regular formulation and review sessions (17 iterations in total) with at least two participants (one permanent and four alternating research team members). This setup guaranteed that the core research team member was aware of all information while being challenged by others in terms of subjective biases [101] . The goal of our formulation sessions was not only to define the final set of Delphi statements , but also to decide on question formats, related information, and additional questions .

For the formulation of Delphi statements , we followed the guidelines by Markmann et al. [57] and iteratively shaped the wording with our experts. To balance the trade-off between the gain of insight and participation effort, we included 15 statements (three for each dimension of our REEST framework) in our study. Moreover, we discussed the question format and decided to query the expected probability (EP) of occurrence as our main variable, given the prospective nature of our Delphi. Moreover, we used desirability (D) and impact (I) of occurrence as complementary variables, and confidence (C) in assessing the respective statement as a bias control variable. For the dimensions D, I, and C we chose a traditional five-point Likert scale from very low (1) to very high (5). The EP dimension, in turn, can have different question formats, such as fixed formats (e.g., Liker-scale, or 0–100 percent scale to assess the expected probability of occurrence by a certain time) or flexible formats (e.g., assessment of time when occurrence is most likely, or assessment of expected probability of occurrence at several points in time in the future). For the co-submitted research, we decided to mix fixed and flexible statements, because we wanted to have both a focus on short-term effects of COVID-19 (which we tested with fixed statements with the end date 2022, e.g., ``in 2022, (strategic) investors got more shares in European football clubs due to COVID-19´´ ) and an indication for medium- to long-term consequences of the pandemic (which we tested with flexible statements, e.g., ``A salary cap for professional football players has been introduced´´ ). We also discussed relevant information associated with our statements and decided to present two exemplary pro arguments as well as two exemplary contra arguments for each statement as initial conditions [35] . This information provided a common basis for all experts and was supposed to motivate participants to think of both supporting and opposing arguments, which is a way to mitigate biases such as framing, anchoring, or desirability bias [13] . For the same reason, we asked participants to separately share qualitative comments in favor and against the occurrence of the respective statement. To gain further insights, we also included an open-comment option for the impact of occurrence. To keep the survey length reasonable, we decided to dispense free-text fields for the desirability and confidence dimensions.

Last, we used the formulation sessions to agree on additional questions . These included classic demographic questions such as gender, age, country of residence, type of organization, and years of work experience within the European football industry. In addition to these surface-level criteria, we also asked for deep-level characteristics, because we wanted to learn about the values and beliefs of participating experts, which might affect their opinions [56 , 87] . These consisted of the respective area(s) of expertise (e.g., strategy, sponsoring, marketing, digital, legal) as well as personality-related information [56] . The latter included COVID-19-related questions to assess experts' level of optimism and a short version of the positive affect negative affect scale (PANAS) to judge on experts' sentiments [78 , 91 , 97] . In conclusion, the formulation sessions eventually determined the Delphi format, Delphi statements , and additional questions , which is why we want to emphasize the importance of this step within a Delphi research project.

Technical recommendation for “2.5 Formulation sessions”

• • While we conducted 17 formulation iterations, we think that fewer sessions would be sufficient. Also, these formulation sessions can be very informal.
• • We recommend involving at least 3 researchers in the process of formulation in order to avoid subjective perspectives and biases.
• • The mix of fixed and flexible statements complicated subsequent analyses significantly (as described in the respective lessons learned in the analysis section). Therefore, we recommend sticking to consistent scales for each individual dimension of assessment (i.e., expected probability, desirability, and impact).
• • Three open-text questions per Delphi statement seemed to be the maximum for our amount of 15 statements. We experienced that responses in the impact-related commentary field often referred to aspects that were mentioned in the probability-related comments before. Therefore, we argue that one or two open-text questions per Delphi statement are suitable to get enough qualitative input, while not risking increased survey fatigue.
• • We understand that in other disciplines qualitative feedback might not be at the core of investigation. Therefore, also larger amounts of statements (up to 100) with few questions (e.g., importance and relevance) can be suitable [15] .
• • We made a good experience illustrating only one Delphi statement per webpage, in order to avoid the necessity to scroll online. From our experience, this style of presentation prevented experts from overlooking free-text fields and allowed participants to get accustomed to a consistent format. In order to guide participants through the survey, we added an overall progress bar and included a "half-time message", indicating that 50 percent of the survey was completed.
• • We highly encourage researchers to include additional questions, because they can help to learn more about experts' personal predispositions. For more details see section ``4.3 Sentiment Analysis´´.

Survey pre-tests

In between the last two formulation sessions, we selectively pre-tested our Delphi format, Delphi statements, and additional questions with fellow researchers and experts from the creative workshop in order to ensure clear comprehensibility and guarantee high reliability [69 , 70] . Based on these pre-tests, we slightly adjusted our final wording. In the co-submitted paper itself, we referred to our Delphi statements as Delphi projections , which is particularly common in the context of foresight. For the remainder of this technical paper, we stick to the broader expression of Delphi statements .

Technical recommendation for “2.6 Survey pre-tests”

• • While we put the main focus of our pre-tests on the content of our study, we would highly recommend researchers to also pre-test the average time to complete the survey, as survey length is known to be a critical factor with regard to survey fatigue and elevated drop-out rates [38] . Based on feedback from participants we learned that our survey length was about 45 min and therefore at the upper limit for the context of our field of study. However, reasonable durations might be shorter or longer for other settings, which should be pre-tested with representatives of the respective expert panel.

Conducting a Delphi study (Phase Two)

In terms of the actual conduction of the Delphi survey, this technical paper will focus on software selection and programming as well as the identification and interaction with experts. To the best of our knowledge, real-time Delphi software has only been applied in business and forecasting studies so far. We encourage scholars of all other disciplines to consider such applications during the survey design in future research endeavors.

Software selection

As mentioned earlier, we decided to conduct a real-time Delphi in order to account for the ambitious timeframe and to allow participating experts to review the most recent results at any point in time. In general, we advise defining the type of Delphi (i.e., sequential or real-time) early in the research process. Based on the respective research goals , one or the other type might be more suitable. While web-based software is strictly required for real-time Delphi surveys, sequential studies can still be distributed via mail or even phone; although this is rather an exception nowadays [14] . In terms of web-based software, Aengenheyster et al. [2] compared state-of-the-art providers regarding features, data output, user-friendliness, and ease of administration. Based on their assessment and our own market screening, we decided to choose Surveylet as our preferred platform. The provider, Calibrum , offers different service packages, which range from pure platform access to full-service support. For the co-submitted research, we acquired a medium package including basic service support and individualization options.

Technical recommendation for “3.1 Software selection”

• • Surveylet offers a variety of options, which might even go beyond the relevant set of functions for most Delphi studies. It also allows for individualization options and service support, which typically require more expensive contracts. Thus, we recommend checking occurring costs. Setting up an account takes roughly one week and should therefore be initiated sufficiently earlier than the actual survey programming.
• • Meanwhile, there might also be additional online Delphi platforms available beyond the scope of the review of Aengenheyster [2] . A more recent example is the BOHEMIA Delphi (Beyond the Horizon – Foresight in Support of the Preparation of the EU‘s Future Policy in Research and Innovation) [34] .

Survey programming

While software selection and preparations can be performed early in the process, we highly recommend finishing phase one (i.e., definition of Delphi format, Delphi statements , and additional questions ) before starting the actual survey programming. Subsequent changes to format and statements lead to extra effort and significantly increase the error-proneness. Therefore, we captured and refined all relevant text modules in Microsoft Excel, prior to programming the survey. These included the survey introduction, the actual statements, pro and contra arguments, as well as all additional questions and an outro.

Special attention should be paid to the survey introduction, particularly in web-based Delphi studies, as a proper understanding of the process is crucial for panelists. We recommend a short, but very concise introduction, including (1) the purpose and anticipated duration of the study, (2) contact details of the research team, and (3) information about the Delphi process . For the explanation of the Delphi process, we recommend mentioning the anonymity of participants and the iterative character of the method. In this context, we encouraged participants to also share (and review) qualitative comments. Moreover, we explicitly draw attention to potential biases, that might affect participants' evaluations. By addressing these issues, we aimed to sensitize participants to deliberately avoid these biases [13] . Eventually, we offered a link to a short online tutorial (approximately 90 seconds), that explained the overall Delphi process with visual support.

In terms of Surveylet as the software of choice for our co-submitted research, we made good experience with the following settings and programming steps: First, we recommend tracking all possible statistics, which include more than 30 variables such as mean values, standard deviations, and interquartile ranges. These should generally be displayed to the survey administrator and can selectively be displayed to the participants. While more data result in more information for the experts, they can also trigger biases, so that we decided to only share mean values with our participants [13] . We refrained from using real-time text analyses, as these were – at the time of our survey – not fully mature, causing significantly longer loading times of the website. An option that appeared quite useful to us was the randomization of statements. That is, every Delphi statement along with the related questions was presented in randomized order, which prevented the risk that experts put more effort into early statements or get collectively biased due to previous answers.

Technical recommendation for “3.2 Survey programming”

• • When it comes to survey programming, we recommend enough preparation time (at least 2 weeks). For novice users, we also recommend basic service support for the first Delphi study or more preparation time to understand the most important features.

Expert selection and invitation

The initial identification of experts can be a challenging task, depending on the subject that is supposed to be explored [25 , 32] . Based on the existing body of literature and the experience from our co-submitted research, we suggest considering five aspects when composing a Delphi expert panel: (1) Size of the panel, (2) level of expertise, (3) level of heterogeneity, (4) level of interest, and (5) access to the panel .

While the specific context of investigation will surely have an impact on the panel composition, it is always advisable to address all five aspects early in the process. In our co-submitted research, we wanted to gain an understanding of prospective developments and aimed to include different stakeholder groups to obtain a comprehensive view of an entire ecosystem. Therefore, the size of the panel needed to be rather large. In general, we recommend a larger number of participants for more holistic topics (as often found in management research) and a more condensed set of experts for specialized topics (as often found in the clinical context). For statistical purposes, it is advisable to have at least 15 to 20 experts in any given sub-group of experts, if significant differences between these sub-groups are supposed to be statistically analyzed. Moreover, we learned that the variety in additional qualitative comments typically decreases from a quantity of 30 to 40 participants. Similar to the size of the panel, the level of expertise depends on the subject. While there might be a need for specific domain knowledge in some cases, other Delphi surveys might benefit from a broader more generalist perspective of participants. In any case, it is necessary to predefine criteria for level of expertise, such as age, years of work experience, occupation, academic degree, or the number of publications in a certain field of research. These criteria then help to justify the panel selection and potentially allow to distinguish between groups based on expertise. Another important aspect of panel composition is the level of heterogeneity . Especially in more holistic – often future-related – settings, a heterogeneous sample can mitigate cognitive biases [13] . Moreover, a variety of backgrounds offers room for inter-group analyses. Possible categories for preselection include dedicated experts from academia, politics, the broader public, and obviously the specific industry that is supposed to be evaluated. Based on our past experience, we also encourage researchers to assess the level of interest that certain participants might have with regard to the survey results while bearing the risk of a potential self-selection bias in mind [40] . Time and attention of subject matter experts are scarce and therefore personal investment of participants can increase response rates and quality of comments. Similarly, access to the panel should be evaluated early in the process. While there are always experts for each and every topic, it is not always easy to reach out to them directly.

To invite experts, the software tool Surveylet offers a variety of options. Based on the size of the panel, we recommend either pre-populated links (i.e., one individual link for each participant based on the participant's e-mail address) for smaller panels with available contact details or in case of larger panels an open link, in which each expert has to insert his or her e-mail address as a unique identifier. At this point, it is important to assure participants that the e-mail address purely serves as an identifier to revise previous inputs.

Technical recommendation for “3.3 Expert selection and invitation”

• • A first and crucial step is to define the criteria on how to measure expertise for the research endeavor. This holds especially true for Delphi studies, where it is rather not about the representativeness of a population but the identification and inclusion of the highest-level of expertise in the panel. A systematic review of expert identification methods can, for example, be found in Mauksch et al. [58] .
• • Our goal was to include more than 80 participants in our survey because we wanted to differentiate between four sub-groups of experts (with at least 20 participants per sub-group). We argue that for a holistic prospective, forecasting Delphi survey with at least three sub-groups of experts, a quantity of 80 participants is sufficient. With a conservative average response rate of 10% (we had 16.2% in our co-submitted research), this would require an initial set of 800 invited experts, which might already be a prohibitively high number for some fields of research. In these cases, we recommend aiming for smaller samples and more focused statements.
• • For larger samples, we recommend examining the panel composition on a regular basis during the survey. If necessary, it can be helpful to additionally invite targeted experts to ensure balanced sub-groups of experts (e.g., from industry, politics, and academia).
• • For smaller samples, we recommend creating individualized links (if applicable), because this offers maximum convenience to participants. We used such links to invite our initial experts to participate in the survey and also created individual links if we had the respective contact information.
• • Our chosen Delphi software allowed adjusting the "landing page" by altering the URL. In this specific case, we highly recommend asking for participants’ e-mail addresses only. On the one hand, this allows users to not share their full names. On the other hand, the e-mail address serves as a unique identifier for the platform, thus allowing participants to access the survey from different devices. If the e-mail address is not requested, participants can only review their given responses, if they use the same device (and did not clear their cache).

Survey conduction

With regard to the actual survey conduction, we recommend an a priori definition of (cascaded) termination criteria. Typically, termination criteria are either time-related, participant-related, or consensus-related. Time-related criteria might include the number of rounds for sequential Delphi studies, or a certain time period for real-time Delphi studies [26] . Participant-related criteria could refer to the number of experts that participated in the study and – within the real-time format – revisited the survey at least once. If the Delphi study addresses consensus, also dedicated measures such as agreement thresholds (e.g., interquartile range, mode frequency), or stability measures (e.g., coefficient of variation, nonparametric χ² test) can serve as termination criteria [7 , 94] . Particularly with regard to the set of stability and agreement criteria, Dajani et al. [20] proposed a theoretical hierarchical model to stop or adjust the Delphi process. Von Briel [92] and Culot et al. [18] represent examples of this approach.

While there is a common notion that Delphi studies in principle follow a consensus-building purpose, we argue that similarly, disagreement among experts is a valid and very insightful outcome, especially in prospective studies. Therefore, we applied a cascaded termination logic with agreement and stability thresholds on the first level and a time-related criterion (maximum 8 weeks) on the second level. Since we did not reach consensus on all statements after 8 weeks, we terminated the survey and included all participants who re-visited at least once in our analysis. Over the course of our survey period, we sent out reminder emails twice: After 3 weeks we contacted all experts that had not yet participated and after 6 weeks we sent a reminder to all participants who answered the survey and asked to review and revise their inputs. For this purpose, our selected Delphi software offered a function to address different groups of participants (e.g., based on their progress within the survey) separately, which can be a helpful service.

Technical recommendation for “3.4 Survey conduction”

• • Given our real-time format with more than 100 participants, it was difficult for participants to grasp all qualitative inputs shared by their peers. To help participants distinguish between pro and contra arguments concerning the expected probability of statements, we included two separate text boxes. However, there is always a trade-off between the amount of requested information and required time spent by the experts. Thus, we recommend using less qualitative input fields for more practical-oriented studies.

Expert follow-up

In order to inform all participants about our initial results, we shared an overview of our descriptive statistics 6 weeks after the termination of the survey. In doing so, we aimed to enrich the practical discussion without having to wait for the scientific publication, which typically consumes several months including revisions. While this step is particularly important for urgent topics, we generally recommend some kind of expert follow-up in order to appreciate the time and effort that participants put into the study.

Technical recommendation for “3.5 Expert follow-up”

• • There is a risk of revealing results before analyzing them thoroughly. Therefore, we decided to share descriptive statistics after we completed the analyzing phase, but well before the actual research paper was published. To reward participants for their time and effort, we would highly recommend sharing basic results as early as possible. This is particularly important in the context of urgent and up-to-date topics.

Analyzing a Delphi study (Phase Three)

The possibilities of analyzing Delphi-based datasets are manifold. In our co-submitted research, we split our analyses into four different categories: (1) Descriptive statistics, (2) Dissent analyses, (3) Sentiment analysis, and (4) Scenario analysis . To analyze our dataset, we used the open-source software R . We made a very good experience with this software because it allows conducting almost any relevant analysis with publicly available software packages.

Descriptive statistics

Descriptive statistics of Delphi-based datasets typically include qualitative and quantitative analyses. We also motivate researchers to include a post-hoc Mann-Whitney U test at the beginning of the descriptive statistics to check for non-response bias [84] .

Qualitative analyses

Qualitative analyses particularly focus on experts' comments and can reveal insights about the participants' level of engagement as well as potential interrelations between different Delphi statements . For data type transparency, we highly recommend conducting a syntax and content analysis as suggested by Förster and von der Gracht [32] . In terms of syntax, we labeled all comments as either whole sentences, phrases, or catchwords. A high percentage of whole sentences generally indicates a solid level of engagement in the discussion and should therefore serve as a quality measure [73] . To analyze content, we had two researchers coding the comments as beliefs, differentiations, cause-effect relationships, examples, historical analogies, experiences, trends, figures, no information, or misunderstandings. To assure concordance, we calculated the level of agreement between the two coders. With an agreement rate of more than 80%, we inferred acceptable interrater reliability [54] .

To gain further insights from participants' comments, we recommend performing a cross-impact analysis (for additional illustration, see e.g., [6 , 71] ) in order to understand potential interaction effects between statements. Therefore, we assessed the active and reactive effects among our statements by considering the results of our content analysis. We then plotted the results and categorized statements as buffering (limited active or reactive effect), active, reactive, and critical (strong active and reactive effect) statements. These insights helped us to interpret our results in the scenario analysis and validated our effort to formulate largely independent Delphi statements .

Quantitative analyses

For our basic quantitative analyses, we calculated arithmetic mean values and standard deviations for our three statement-related dimensions expected probability, impact, and desirability. To assess consensus, we used interquartile ranges due to their robustness as a statistical measure. While there are multiple interpretations in literature, we argue that a threshold of a maximum of 25% of the respective scale (e.g., 25 on a scale from 0–100, or 1.25 on a scale from 1–5) can serve as an indicator for consensus. For our flexible projections, in turn, we utilized mode frequency and a visual inspection of histograms to infer information about consensus, or potential dissent schemes, as explained in the subsequent section.

Technical recommendation for “4.1 Descriptive statistics”

• • Although time-consuming, we made a good experience with two coders for all qualitative analyses. Insights from participants' comments are a valuable input for the analyses and discussion. Although often underreported in many Delphi-based journal articles, we argue that the qualitative part of the methodology should not be ignored.
• • Despite the fact that there are various quantitative measures for consensus and stability, we made good experience with interquartile ranges as a measure of choice. Alternative approaches (e.g., fuzzy statistics) can be suitable under special circumstances, but would not have revealed extra insights in our case [16] .

Dissent analyses

The major aim of the Delphi method is to systematically structure a group communication process [53] ). This process might lead to consensus, but as with for example Policy-type Delphi studies (see e.g., [22] ), researchers could be more interested in the dissent of the panel. Especially for prospective studies, we argue that dissent can reveal valuable insights for the practical and academic discussion. Therefore, we present how we applied a series of potential dissent analyses in our co-submitted research, which were initially introduced by Warth et al. [96] .

Desirability bias analysis

In many forecast surveys, participants tend to assess desirable developments as more likely than undesirable ones [101] . Therefore, we tested for a potential desirability bias, following the approach presented by Ecken et al. [27] . It includes a post hoc adjustment of expected probability values based on the desirability assessments of experts. As the calculations for this method require a restructured dataset in the long format (i.e., one row per participant per statement), it takes quite a lot of effort. Based on our experience, we would recommend using a less time-consuming technique to account for a potential desirability bias (e.g., by partializing out the influence of desirability on expected probability, or by conducting simple correlation analyses).

Outlier analysis

Outliers can have a significant effect on statistic variables, such as the interquartile range [3] . Therefore, we identified and eliminated outliers to test if these had an impact on the group's consensus. In our co-submitted research, we found no significant effect, however, we would recommend running this analysis and interpreting the results. Especially if the respective outliers shared out-of-the-norm qualitative comments, these might deliver valuable insights. Alternatively, they could also point towards (systematic) misunderstandings, which would be a potential reason to either delete the specific participant from the dataset, or to double-check the comprehensibility of the specific statement.

Bipolarity analysis

The bipolarity analysis accounts for the fact that there might be opposing groups of experts with respective intra-group consensus [77] . To test for this effect, we checked for bimodal distributions and visually inspected histograms of expected probability assessments for all statements. While we had little indication for strong bipolarity in our co-submitted research, this simple analysis should always be conducted as part of the result evaluation. Bipolarity – if present – almost prohibits consensus. Therefore, it is even more important to study the two extremes to understand if these are close together or rather far apart from each other. Either constellation could reveal valuable insights.

Stakeholder-group analysis

A classical dissent analysis that can be found in multiple disciplines is the stakeholder-group analysis. For the co-submitted research, we distinguished four stakeholder groups based on their occupation. To identify opposing views, we conducted Mann-Whitney U tests between the four groups for all 15 statements and reported (marginally) significant differences between groups. Although this analysis requires substantial time effort, it is fairly easy from a methodological point of view and we highly recommend differentiating stakeholder groups, as it provides valuable insights with practical relevance.

Technical recommendation for “4.2 Dissent analyses”

• • We see consensus and dissent analyses as two sides of the same medal. Although both directions might ask for different analysis steps, measures, and thresholds, we recommend applying both perspectives to the Delphi dataset. Especially for dissent, which is often neglected, we see high value and additional insights in analyzing the potential reasons for diverging opinions.
• • When planning the expert panel, it is advisable to think about potential stakeholder-group analyses early in the process. To obtain reliable results and bearing statistical requirements in mind, each subset should consist of at least 15 to 20 participants. This should be considered in the panel composition.
• • In addition to analyzing pre-defined stakeholder groups, an explorative group analysis based on participants’ assessment patterns inherent in the data could be beneficial. There might be strong dissent across identified groups such as technology optimists, sustainability pessimists, or transformation skeptics.

Sentiment analyses

While the importance of considering participants' sentiments in prospective studies was pointed out in the 1980s, especially the personality dimension is rarely found in Delphi-based studies in the past decades [56 , 87] . However, detailed information about the personality of participants can shed a different light on results and should therefore be considered in all Delphi studies, irrespective of the individual discipline. While there is a myriad of possibilities to cover personality and expert-related information, we covered four dimensions for sentiment analysis: (1) Expertise and experience, (2) Level of confidence, (3) Level of optimism, as well as (4) Positive and negative affect .

Expertise and experience

To assess expertise and experience, we asked for our experts' years of professional experience within the industry we examined. Based on this information, we calculated correlations between years of experience and expected probability assessments and reported significant effects. Moreover, participants were able to indicate their knowledge in specific topic areas such as strategy, marketing, sales, and digital. This served as the foundation for subset comparisons, similar to the stakeholder-group analysis.

Level of confidence

In our co-submitted research, we collected information on our experts' subjective knowledge on each statement, by asking for confidence in assessing the respective topic. We used a five-point Likert scale from 1 (very low) to 5 (very high) and then calculated correlations between confidence and expected probability for statements with linear intervals and chi-square tests for statements with non-linear intervals. While we only reported our results verbally, there are also insightful ways to illustrate these analyses, as exemplarily depicted in Fig. 3 .

Visualization of Relationship between Confidence and Expected Probability. Note. This mosaic plot is based on the data of Beiderbeck et al. [8] . It shows the relationship between confidence (measured with a five-point Likert scale from 1 = very low to 5 = very high) and expected probability (0 = statement will never occur; 1 = statement will occur long-term; 2 = statement will occur short-term). Size of the respective mosaic represents number of participants with respective confidence and expected probability assessment.

Level of optimism

While experience, expertise, and confidence find more frequent application in Delphi-based manuscripts, we rarely find other indicators for deep-level expert characteristics in use [87] . Therefore, we included the level of optimism as an indicator for a personality trait that is relevant for future predictions [56] . We posed two dedicated questions with respect to the overall future developments within the industry of investigation. Based on the responses we conducted a median split and created two subsets of rather optimistic and rather pessimistic experts. We then conducted Mann-Whitney U tests and reported significant differences between these two groups.

Positive and negative affect

Given the circumstances of the COVID-19 outbreak, we also wanted to account for the subjective wellbeing of our experts, which might have affected their respective assessments. Therefore, we used a shortened version of the PANAS (Positive and Negative Affect Schedule) [91] and asked experts to evaluate a four-item construct for two points in time: prior to the crisis and during the crisis. This helped us to calculate differences to see which expert was more or less affected by the pandemic in terms of his or her subjective wellbeing. Again, we used this information to calculate correlations and present significant effects.

Technical recommendation for “4.3 Sentiment analyses”

• • Instead of "level of optimism" it could also make sense to test for other deep-level characteristics of participants. These can be adapted based on the respective field of research. In the context of technology forecasting, for example, it could be worthwhile to test related constructs such as the trust in technology [60] , affinity for technology interaction [33] , or attitudes towards using technology [90] . In this context, the inclusion of underlying theories like the technology acceptance model [52] might also be an interesting avenue for future research.
• • To account for the individual wellbeing of participants there are numerous alternatives to PANAS, such as the "profile of mood states" or "circumplex model of affect" [75] . While these constructs allow for a more nuanced differentiation of affect, we still decided to use PANAS, because it offered a four-item short version and therefore mastered the trade-off between quality of insights and amount of effort for the participants.
• • Another dimension of the sentiment analysis could be "locus of control" – which also offers a short version – to determine participants' perceptions on heteronomy and self-determination [50] .

Scenario analysis

Particularly for prospective studies, Delphi-based insights can serve as a basis for scenario analyses [68] . While there are multiple ways of building and illustrating scenarios, we decided to apply the fuzzy c-means algorithm for our co-submitted research. With a significantly high number of experts and assessments, this method yields feasible results. Moreover, it is relatively easy to execute and visualize with R . In the case of smaller samples, hierarchical clustering might be more appropriate [49] . With dedicated software, the latent class analysis offers a further possibility to generate related groups of statements [96] .

With the help of the c-means algorithm, we created three groups and plotted the clusters on a 3D coordinate system with the axes expected probability, desirability, and impact to gain a visual impression of our results. In general, we recommend 3D visualizations, because they help the reader grasp the interrelation between three (or more) outcome variables.

Technical recommendation for “4.4 Scenario analysis”

• • To cluster statements, we needed a comparable output for each individual dimension. In this regard, the different output formats for expected probability required us to introduce a new logic in order to transform and unify the two scales. In our case, this aggregation led to a loss of informational value, because the unified scale consisted of three categories only. Hence, we recommend to not change the format of a scale within one particular dimension of the Delphi survey (e.g. expected probability) or to consider a possible transformation logic upfront.

Conclusion and future research avenues

In this technical paper, we illustrate a comprehensive Delphi preparation, conduction, and analysis process. We offer room for flexibility in adapting the research process for individual needs while sticking to a consistent framework that allows for replicability.

We encourage researchers from all disciplines to use the Delphi technique in order to organize structured expert discussions around both current and prospective challenges in the respective field of study. From a methodological point of view, we want to support the research community by offering technical recommendations from our Delphi study on the impact of COVID-19 on the European football ecosystem. At the same time, we advocate for further innovative developments of the technique, specifically with regard to the role of experts' personality traits, thinking patterns, and situational concomitant. As with any research, scholars should conclude their research articles with a critical limitations section. During our study of literature, we came across the report of Sackman [76] , obviously one of the early critical reflections of the Delphi method. Each Delphi study should include a careful elaboration on its validity and reliability (see e.g., [93] , section 3.7 for a review on quality criteria in Delphi surveys), while following evolving quality frameworks, such as described in Jünger et al. [44] , Belton et al. [9] , and Murphy et al. [66] .

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Delphi technique on nursing competence studies: a scoping review.

1. Introduction

1.1. the delphi technique, 1.1.1. selection and composition of the expert’s panel, 1.1.2. rounds, 1.1.3. data analysis and consensus, 1.1.4. reliability and validity, 1.1.5. advantages and disadvantages of the delphi technique, 1.2. rationale, context and aim of the scoping review, 2. materials and methods, 2.1. eligibility criteria, 2.2. search strategy, 2.3. study selection, 2.4. data extraction and presentation, 3.1. preparatory procedures, 3.2. access and expert selection procedures, 3.3. acquisition of experts’ inputs, 3.3.1. instrumentation, 3.3.2. first round, 3.3.3. subsequent rounds, 3.3.4. stability of the expert panel, 3.4. data analysis and consensus, 3.5. ethical–legal procedures and guarantees, 4. discussion, limitations, 5. conclusions, supplementary materials, author contributions, institutional review board statement, informed consent statement, conflicts of interest.

• Le Boterf, G. Ingénierie et Évaluation des Compétences , 5th ed.; Groupe Eyrolles: Paris, France, 2006. [ Google Scholar ]
• Benner, P. De Iniciado a Perito-Excelência e Poder na Prática Clínica de Enfermagem ; Edição Comemorativa; Quarteto: Coimbra, Portugal, 2005. [ Google Scholar ]
• Meretoja, R.; Leino-Kilpi, H.; Kaira, M. Comparison of nurse competence in different hospital work environments. J. Nurs. Manag. 2004 , 12 , 329–336. [ Google Scholar ] [ CrossRef ]
• Dunn, S.; Lawson, D.; Robertson, S.; Underwood, M.; Clark, R.; Valentine, T.; Walker, N.; Wilson-Row, C.; Crowder, K.; Herewane, D. The development of competency standards for specialist critical care nurses. J. Adv. Nurs. 2000 , 31 , 339–346. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Beeckman, D.; Vanderwee, K.; Demarre, L.; Paquay, L.; Van Hecke, A.; Defloor, T. Pressure ulcer prevention: Development and psychometric validation of a knowledge assessment instrument. Int. J. Nurs. Stud. 2010 , 47 , 399–410. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tang, Q.; Zhang, D.; Chen, J.; Liu, M.; Xiang, Y.; Luo, T.; Zhu, L. Tests on a scale for measuring the core competencies of paediatric specialist nurses: An exploratory quantitative study. Nurs. Open 2023 , 10 , 5098–5107. [ Google Scholar ] [ CrossRef ]
• Tay, C.; Yuh, A.; Lan, E.; Ong, C.; Aloweni, F.; Lopez, V. Development and validation of the incontinence associated dermatitis knowledge, attitude and practice questionnaire. J. Tissue Viability 2020 , 29 , 244–251. [ Google Scholar ] [ CrossRef ]
• Wheeler, K.; Phillips, K. The Development of Trauma and Resilience Competencies for Nursing Education. J. Am. Psychiatr. Nurses Assoc. 2021 , 27 , 322–333. [ Google Scholar ] [ CrossRef ]
• Keeney, S.; Hasson, F.; McKenna, H. Consulting the oracle: Ten lessons from using the Delphi technique in nursing research. J. Adv. Nurs. 2006 , 53 , 205–212. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Barrios, M.; Guilera, G.; Nuño, L.; Gómez-Benito, J. Consensus in the delphi method: What makes a decision change? Technol. Forecast. Soc. Chang. 2021 , 163 , 120484. [ Google Scholar ] [ CrossRef ]
• Foth, T.; Efstathiou, N.; Vanderspank-Wright, B.; Ufholz, L.; Dütthorn, N.; Zimansky, M.; Humphrey-Murto, S. The use of Delphi and Nominal Group Technique in nursing education: A review. Int. J. Nurs. Stud. 2016 , 60 , 112–120. [ Google Scholar ] [ CrossRef ]
• Avella, J. Delphi Panels: Research Design, Procedures, Advantages, and Challenges. Int. J. Dr. Stud. 2016 , 11 , 305–321. [ Google Scholar ] [ CrossRef ]
• James, D.; Warren-Forward, H. Research methods for formal consensus development. Nurse Res. 2015 , 22 , 35–40. [ Google Scholar ] [ CrossRef ]
• Hasson, F.; Keeney, S.; McKenna, H. Research guidelines for the Delphi survey technique. J. Adv. Nurs. 2000 , 32 , 1008–1015. [ Google Scholar ] [ CrossRef ]
• Fish, L.; Busby, D. The delphi technique. In Research Methods in Family Therapy , 2nd ed.; Sprenkle, D., Piercy, F., Eds.; Guilford: New York, NY, USA, 2005. [ Google Scholar ]
• Linstone, H.; Turoff, M. The Delphi Method: Techniques and Applications ; Addison-Wesley Publishing Company, Advanced Book Program: New York, NY, USA, 2002. [ Google Scholar ]
• Beiderbeck, D.; Frevel, N.; von der Gracht, H.A.; Schmidt, S.L.; Schweitzer, V.M. Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX 2021 , 8 , 101401. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Fink-Hafner, D.; Dagen, T.; Doušak, M.; Novak, M.; Hafner-Fink, M. Delphi Method: Strengths and Weaknesses. Adv. Methodol. Stat. 2019 , 16 , 1–19. [ Google Scholar ] [ CrossRef ]
• Grisham, T. The Delphi technique: A method for testing complex and multifaceted topics. Int. J. Manag. Proj. Bus. 2009 , 2 , 112–130. [ Google Scholar ] [ CrossRef ]
• Dalkey, N.; Helmer, O. An Experimental Application of the Delphi Method to the Use of Experts. Manag. Sci. 1963 , 9 , 458–467. [ Google Scholar ] [ CrossRef ]
• Hsu, C.; Sandford, B. The Delphi Tehcnique: Making sense of consensus. Pract. Assess. Res. Eval. 2007 , 12 , 1–8. [ Google Scholar ]
• Dalkey, N. An experimental study of group opinion: The Delphi method. Futures 1969 , 1 , 408–426. [ Google Scholar ] [ CrossRef ]
• Dalkey, N. Delphi ; RAND Corporation: Santa Monica, CA, USA, 1967. [ Google Scholar ]
• Adams, S. Projecting the next decade in safety management: A Delphi technique study. Prof. Saf. 2001 , 46 , 26–29. [ Google Scholar ]
• Sossa, J.; William, H.; Hernandez-Zarta, R. Delphi method: Analysis of rounds, stakeholder and statistical indicators. Foresight 2019 , 21 , 525–544. [ Google Scholar ] [ CrossRef ]
• Donohoe, H.; Stellefson, M.; Tennant, B. Advantages and Limitations of the e-Delphi Technique. Am. J. Health Educ. 2012 , 43 , 38–46. [ Google Scholar ] [ CrossRef ]
• Keeney, S.; Hasson, F.; McKenna, H. The Delphi Technique in Nursing and Health Research ; John Wiley & Sons Ltd.: London, UK, 2011. [ Google Scholar ]
• Meijering, J.; Tobi, H. The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial. Int. J. Forecast. 2018 , 34 , 216–224. [ Google Scholar ] [ CrossRef ]
• Keeney, S.; Hasson, F.; McKenna, H. A critical review of the Delphi technique as a research methodology for nursing. Int. J. Nurs. Stud. 2001 , 38 , 195–200. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• McKenna, H. The Delphi technique: A worthwhile research approach for nursing? J. Adv. Nurs. 1994 , 19 , 1221–1225. [ Google Scholar ] [ CrossRef ]
• Thangaratinam, S.; Redman, C. The Delphi technique. Obstet. Gynaecol. 2005 , 7 , 120–125. [ Google Scholar ] [ CrossRef ]
• Meyrick, J. The Delphi method and health research. Health Educ. 2003 , 103 , 7–16. [ Google Scholar ] [ CrossRef ]
• Hasson, F.; Keeney, S. Enhancing rigour in the Delphi technique research. Technol. Forecast. Soc. Chang. 2011 , 78 , 1695–1704. [ Google Scholar ] [ CrossRef ]
• Custer, R.; Scarcella, J.; Stewart, B. The Modified Delphi Technique-A Rotational Modification. J. Edu. Voc. Stud. 1999 , 15 . [ Google Scholar ] [ CrossRef ]
• Mauksch, S.; von der Gracht, H.; Gordon, T. Who is an expert for foresight? A review of identification methods. Technol. Forecast. Soc. Chang. 2020 , 154 , 119982. [ Google Scholar ] [ CrossRef ]
• Humphrey-Murto, S.; Varpio, L.; Wood, T.; Gonsalves, C.; Ufholz, L.; Mascioli, K.; Wang, C.; Foth, T. The Use of the Delphi and Other Consensus Group Methods in Medical Education Research: A Review. Acad. Med. 2017 , 92 , 1491–1498. [ Google Scholar ] [ CrossRef ]
• Förster, B.; von der Gracht, H. Assessing Delphi panel composition for strategic foresight—A comparison of panels based on company-internal and external participants. Technol. Forecast. Soc. Chang. 2014 , 84 , 215–229. [ Google Scholar ] [ CrossRef ]
• Boulkedid, R.; Abdoul, H.; Loustau, M.; Sibony, O.; Alberti, C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS ONE 2011 , 6 , e20476. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Giannarou, L.; Zervas, E. Using Delphi technique to build consensus in practice. Int. J. Appl. Manag. Sci. 2014 , 9 , 66–82. [ Google Scholar ]
• Lau, P.; Ryan, S.; Abbott, P.; Tannous, K.; Trankle, S.; Peters, K.; Page, A.; Cochrane, N.; Usherwood, T.; Reath, J. Protocol for a Delphi consensus study to select indicators of high-quality general practice to achieve Quality Equity and Systems Transformation in Primary Health Care (QUEST-PHC) in Australia. PLoS ONE 2022 , 17 , e0268096. [ Google Scholar ] [ CrossRef ]
• Naisola-Ruiter, V. The Delphi technique: A tutorial. Hosp. Res. J. 2022 , 12 , 91–97. [ Google Scholar ] [ CrossRef ]
• Massaroli, A.; Martini, J.; Lino, M.; Spenassato, D.; Massaroli, R. Método Delphi como Referencial Metodológico para a Pesquisa em Enfermagem. Texto Contexto Enferm. 2017 , 26 , e1110017. [ Google Scholar ] [ CrossRef ]
• Winkler, J.; Moser, R. Biases in future-oriented Delphi studies: A cognitive perspective. Technol. Forecast. Soc. Chang. 2016 , 105 , 63–76. [ Google Scholar ] [ CrossRef ]
• Marques, J.; Freitas, D. Método DELPHI: Caracterização e potencialidades na pesquisa em Educação. Pro-Posições 2018 , 29 , 389–415. [ Google Scholar ] [ CrossRef ]
• Birko, S.; Dove, E.; Özdemir, V. Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation. PLoS ONE 2015 , 10 , e0135162. [ Google Scholar ] [ CrossRef ]
• Meijering, J.; Kampen, J.; Tobi, H. Quantifying the development of agreement among experts in Delphi studies. Technol. Forecast. Soc. Chang. 2013 , 80 , 1607–1614. [ Google Scholar ] [ CrossRef ]
• von der Gracht, H. Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technol. Forecast. Soc. Chang. 2012 , 79 , 1525–1536. [ Google Scholar ] [ CrossRef ]
• Diamond, I.; Grant, R.; Feldman, B.; Pencharz, P.; Ling, S.; Moore, A.; Wales, P. Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. J. Clin. Epidemiol. 2014 , 67 , 401–409. [ Google Scholar ] [ CrossRef ]
• Collins, D. Pretesting survey instruments: An overview of cognitive methods. Qual. Life Res. 2003 , 12 , 229–238. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Aromataris, E.; Munn, Z. (Eds.) JBI Manual for Evidence Synthesis ; JBI: Adelaide, Australia, 2020; Available online: www.synthesismanual.jbi.global (accessed on 12 March 2023).
• Munn, Z.; Peters, M.; Stern, C.; Tufanaru, C.; McArthur, A.; Aromataris, E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med. Res. Methodol. 2018 , 18 , 143. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Peters, M.; Godfrey, C.; McInerney, P.; Munn, Z.; Tricco, C.; Khalil, H. Chapter 11: Scoping Reviews (2020 version). In JBI Manual for Evidence Synthesis ; Aromataris, E., Munn, Z., Eds.; JBI: Adelaide, Australia, 2020; Available online: www.synthesismanual.jbi.global (accessed on 12 March 2023).
• Page, M.; McKenzie, J.; Bossuyt, P.; Boutron, I.; Hoffmann, T.; Mulrow, C.; Shamseer, L.; Tetzlaff, J.; Akl, E.; Brennan, S.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ]
• Tricco, A.; Lillie, E.; Zarin, W.; O’Brien, K.; Colquhoun, H.; Levac, D.; Moher, D.; Peters, M.; Horsley, T.; Weeks, L.; et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann. Intern. Med. 2018 , 169 , 467–473. [ Google Scholar ] [ CrossRef ]
• Furtado, L. Advancing the Delphi Technique: A Critical Review of Literature on Nursing Competence Studies. Available online: https://archive.org/details/osf-registrations-kp2vw-v1 (accessed on 9 December 2023).
• Peters, M.; Marnie, C.; Colquhoun, H.; Garritty, C.; Hempel, S.; Horsley, T.; Langlois, E.; Lillie, E.; O’Brien, K.; Tunçalp, Ӧ.; et al. Scoping reviews: Reinforcing and advancing the methodology and application. Syst. Rev. 2021 , 10 , 263. [ Google Scholar ] [ CrossRef ]
• Tracy, M.; O’Grady, E. Hamric and Hanson’s Advanced Practice Nursing: An Integrative Approach ; Elsevier: Amsterdam, The Netherlands, 2019. [ Google Scholar ]
• Levac, D.; Colquhoun, H.; O’Brien, K. Scoping studies: Advancing the methodology. Implement. Sci. 2010 , 5 , 69. [ Google Scholar ] [ CrossRef ]
• Beauvais, A.; Phillips, K. Incorporating Future of Nursing Competencies Into a Clinical and Simulation Assessment Tool: Validating the Clinical Simulation Competency Assessment Tool. Nurs. Educ. Perspect. 2020 , 41 , 280–284. [ Google Scholar ] [ CrossRef ]
• He, H.; Zhou, T.; Zeng, D.; Ma, Y. Development of the competency assessment scale for clinical nursing teachers: Results of a Delphi study and validation. Nurse Educ. Today 2021 , 101 , 104876. [ Google Scholar ] [ CrossRef ]
• Janssens, I.; Van Hauwe, M.; Ceulemans, M.; Allegaert, K. Development and Pilot Use of a Questionnaire to Assess the Knowledge of Midwives and Pediatric Nurses on Maternal Use of Analgesics during Lactation. Int. J. Environ. Res. Public Health 2021 , 18 , 11555. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zhang, J.; Zhou, X.; Wang, H.; Luo, Y.; Li, W. Development and Validation of the Humanistic Practice Ability of Nursing Scale. Asian Nurs. Res. (Korean Soc. Nurs. Sci.) 2021 , 15 , 105–112. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Penataro-Pintado, E.; Rodriguez-Higueras, E.; Llaurado-Serra, M.; Gomez-Delgado, N.; Llorens-Ortega, R.; Diaz-Agea, J. Development and Validation of a Questionnaire of the Perioperative Nursing Competencies in Patient Safety. Int. J. Environ. Res. Public Health 2022 , 19 , 2584. [ Google Scholar ] [ CrossRef ]
• Wang, S.; Tong, J.; Wang, Y.; Zhang, D. A Study on Nurse Manager Competency Model of Tertiary General Hospitals in China. Int. J. Environ. Res. Public Health 2022 , 19 , 8513. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Bing-Jonsson, P.; Bjork, I.; Hofoss, D.; Kirkevold, M.; Foss, C. Competence in advanced older people nursing: Development of ‘Nursing older people-Competence evaluation tool’. Int. J. Older People Nurs. 2015 , 10 , 59–72. [ Google Scholar ] [ CrossRef ]
• Fan, L.; Gui, L.; Xi, S.; Qiao, A. Core competence evaluation standards for emergency nurse specialist: Developing and testing psychometric properties. Int. J. Nurs. Sci. 2016 , 3 , 274–280. [ Google Scholar ] [ CrossRef ]
• Zheng, Y.; Shi, X.; Jiang, S.; Li, Z.; Zhang, X. Evaluation of core competencies of nurses by novel holistic assessment system. Biomed. Res. J. 2017 , 28 , 3259–3265. [ Google Scholar ]
• Chen, H.; Pu, L.; Chen, Q.; Xu, X.; Bai, C.; Hu, X. Instrument Development for Evaluation of Gerontological Nurse Specialists Core Competencies in China. Clin. Nurse Spec. 2019 , 33 , 217–227. [ Google Scholar ] [ CrossRef ]
• Holanda, F.; Marra, C.; Cunha, I. Professional competence of nurses in emergency services: Evidence of content validity. Rev. Bras. Enferm. 2019 , 72 , 66–73. [ Google Scholar ] [ CrossRef ]
• Licen, S.; Plazar, N. Developing a Universal Nursing Competencies Framework for Registered Nurses: A Mixed-Methods Approach. J. Nurs. Scholarsh. 2019 , 51 , 459–469. [ Google Scholar ] [ CrossRef ]
• Mei, N.; Chang, L.; Zhu, Z.; Dong, M.; Zhang, M.; Zeng, L. Core competency scale for operating room nurses in China: Scale development, reliability and validity evaluation. Nurs. Open 2022 , 9 , 2814–2825. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Lakanmaa, R.; Suominen, T.; Perttilä, J.; Puukka, P.; Leino-Kilpi, H. Competence requirements in intensive and critical care nursing–Still in need of definition? A Delphi study. Intensive Crit. Care Nurs. 2012 , 28 , 329–336. [ Google Scholar ] [ CrossRef ]
• Liu, L.; Curtis, J.; Crookes, P. Identifying essential infection control competencies for newly graduated nurses: A three-phase study in Australia and Taiwan. J. Hosp. Infect. 2014 , 86 , 100–109. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Van Hecke, A.; Goeman, C.; Beeckman, D.; Heinen, M.; Defloor, T. Development and psychometric evaluation of an instrument to assess venous leg ulcer lifestyle knowledge among nurses. J. Adv. Nurs. 2011 , 67 , 2574–2585. [ Google Scholar ] [ CrossRef ]
• Hoyt, K.; Coyne, E.; Ramirez, E.; Peard, A.; Gisness, C.; Gacki-Smith, J. Nurse Practitioner Delphi Study: Competencies for practice in emergency care. J. Emerg. Nurs. 2010 , 36 , 439–449. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Chang, A.; Gardner, G.; Duffield, C.; Ramis, M. A Delphi study to validate an Advanced Practice Nursing tool. J. Adv. Nurs. 2010 , 66 , 2320–2330. [ Google Scholar ] [ CrossRef ]
• Jirwe, M.; Gerrish, K.; Keeney, S.; Emami, A. Identifying the core components of cultural competence: Findings from a Delphi study. J. Clin. Nurs. 2009 , 18 , 2622–2634. [ Google Scholar ] [ CrossRef ]
• Irvine, F. Exploring district nursing competencies in health promotion: The use of the Delphi technique. J. Clin. Nurs. 2005 , 14 , 965–975. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Hardy, D.; O’Brien, A.; Gaskin, C.; O’Brien, A.; Morrison-Ngatai, E.; Skews, G.; Ryan, T.; McNulty, N. Practical application of the Delphi technique in a bicultural mental health nursing study in New Zealand. J. Adv. Nurs. 2004 , 46 , 95–109. [ Google Scholar ] [ CrossRef ]
• European Parliament and of the Council. General Data Protection Regulation-Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016. Available online: https://gdpr-info.eu (accessed on 20 February 2024).
• Cornock, M. General Data Protection Regulation (GDPR) and implications for research. Maturitas 2018 , 111 , A1–A2. [ Google Scholar ] [ CrossRef ]
• Centers for Disease Control and Prevention. Health Insurance Portability and Accountability Act of 1996. Available online: https://www.cdc.gov/phlp/php/resources/health-insurance-portability-and-accountability-act-of-1996-hipaa.html (accessed on 20 February 2024).
• Niederberger, M.; Spranger, J. Delphi Technique in Health Sciences: A Map. Front. Public Health 2020 , 8 , 457. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Fitch, K.; Bernstein, S.; Aguilar, M.; Burnand, B.; LaCalle, J.; Lazaro, P.; van het Loo, M.; McDonnell, J.; Vader, J.; Kahan, J. The RAND/UCLA Appropriateness Method User’s Manual ; RAND Corporation: Santa Monica, CA, USA, 2001. [ Google Scholar ]
• Greatorex, J.; Dexter, T. An accessible analytical approach for investigating what happens between the rounds of a Delphi study. J. Adv. Nurs. 2000 , 32 , 1016–1024. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Casali, P.; Vyas, M. Data protection and research in the European Union: A major step forward, with a step back. Ann. Oncol. 2021 , 32 , 15–19. [ Google Scholar ] [ CrossRef ]
• Farah, M.; Helou, S.; Tufenkji, P.; El Helou, E. Data Protection in Healthcare Research: Medical Students’ Knowledge and Behavior. Stud. Health Technol. Inform. 2022 , 295 , 104–107. [ Google Scholar ] [ PubMed ]
• Gattrell, W.; Logullo, P.; van Zuuren, E.; Price, A.; Hughes, E.; Blazey, P.; Winchester, C.; Tovey, D.; Goldman, K.; Hungin, A.; et al. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med. 2024 , 21 , e1004326. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Furtado, L.; Coelho, F.; Pina, S.; Ganito, C.; Araújo, B.; Ferrito, C. Delphi Technique on Nursing Competence Studies: A Scoping Review. Healthcare 2024 , 12 , 1757. https://doi.org/10.3390/healthcare12171757

Furtado L, Coelho F, Pina S, Ganito C, Araújo B, Ferrito C. Delphi Technique on Nursing Competence Studies: A Scoping Review. Healthcare . 2024; 12(17):1757. https://doi.org/10.3390/healthcare12171757

Furtado, Luís, Fábio Coelho, Sara Pina, Cátia Ganito, Beatriz Araújo, and Cândida Ferrito. 2024. "Delphi Technique on Nursing Competence Studies: A Scoping Review" Healthcare 12, no. 17: 1757. https://doi.org/10.3390/healthcare12171757

Article Metrics

Article access statistics, supplementary material.

ZIP-Document (ZIP, 315 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Search Menu
• Sign in through your institution
• Advance articles
• Special Issues
• Author Guidelines
• Submission Site
• Open Access
• Self Archiving Policy
• Why publish with this journal?
• About Translational Behavioral Medicine
• About Society of Behavioral Medicine
• Journal Staff
• Advertising and Corporate Services
• Journals Career Network
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Lay summary, introduction, funding sources, author contributions, transparency statement.

• < Previous

Application of the Delphi method to the development of common data elements for social drivers of health: A systematic scoping review

• Article contents
• Figures & tables
• Supplementary Data

Yulia A Levites Strekalova, July D Nelson, Haley M Weber, Xiangren Wang, Sara M Midence, Application of the Delphi method to the development of common data elements for social drivers of health: A systematic scoping review, Translational Behavioral Medicine , Volume 14, Issue 7, July 2024, Pages 426–433, https://doi.org/10.1093/tbm/ibae020

• Permissions Icon Permissions

Collaborative data science requires standardized, harmonized, interoperable, and ethically sourced data. Developing an agreed-upon set of elements requires capturing different perspectives on the importance and feasibility of the data elements through a consensus development approach. This study reports on the systematic scoping review of literature that examined the inclusion of diverse stakeholder groups and sources of social drivers of health variables in consensus-based common data element (CDE) sets. This systematic scoping review included sources from PubMed, Embase, CINAHL, WoS MEDLINE, and PsycINFO databases. Extracted data included the stakeholder groups engaged in the Delphi process, sources of CDE sets, and inclusion of social drivers data across 11 individual and 6 social domains. Of the 384 studies matching the search string, 22 were included in the final review. All studies involved experts with healthcare expertise directly relevant to the developed CDE set, and only six (27%) studies engaged health consumers. Literature reviews and expert input were the most frequent sources of CDE sets. Seven studies (32%) did not report the inclusion of any demographic variables in the CDE sets, and each demographic SDoH domain was included in at least one study with age and sex assigned at birth included in all studies, and social driver domains included only in four studies (18%). The Delphi technique engages diverse expert groups around the development of SDoH data elements. Future studies can benefit by involving health consumers as experts.

Collecting and capturing social factors that affect individuals’ health is imperative. Social drivers of health data allow researchers to understand health disparities to make healthcare available, accessible, and affordable. However, collecting common health data elements has challenged researchers due to limited resources to facilitate change. Incorporating various stakeholders, such as individuals and patient advocacy groups, can effectively contribute to the research process as community advisors. This article reviews the studies that used the Delphi method and brings together experts to agree on guidelines for collecting common data elements. The article’s findings reveal that experts are healthcare professionals and researchers, leaving out the crucial input from patients and caregivers. This article emphasized that developing a standard set of data elements can improve the standardization of social drivers of health. Common data elements provide the opportunity to improve patients’ and social circumstances and their efforts toward health outcomes.

Practice: Establishing common data elements involving patients and caregivers is crucial to creating a clinical patient-centered environment.

Policy: Policymakers who want to decrease health disparities should explore the standardization of common data elements to capture social determinants of health and improve health data quality and consistency.

Research: Future research must prioritize identifying diverse stakeholders to develop common data elements to enhance data harmonization, exchange, and inter-organizational collaboration.

Common data elements for data sharing

Advancements in technology and healthcare have equipped healthcare delivery and public health organizations with the capacity to gather vast amounts of medical and health-related data [ 1 ]. The capacity to capture data paved the way for inter-organizational collaboration and data team science that stand to generate the highest level of evidence for clinical practice and population health. Yet, collaborative data science requires that captured data are standardized, harmonized, interoperable, and ethically sourced. The ability to easily share and combine data from multiple studies has the potential to increase the scientific impact of individual studies. A key strategy for promoting inter-organizational data science is creating and implementing common data element (CDE) sets [ 2 ]. A data element is a standardized, precisely defined question (or variable) coupled with a predetermined set of responses.

The interest in developing health-related CDE sets is a growing practice driven by the need to harmonize data and promote inter-organizational research and quality improvement efforts. In the United States, CDE set development has been stimulated by the 21st Century Cures Act, which mandates health data exchanges and interoperability [ 3 , 4 ]. Multiple stakeholder groups are working on developing CDE sets for specific patient cohorts, diagnoses, and health outcomes. Many therapeutic organizations and research consortia have worked to develop CDE sets that represent minimally required or essential variables specific to a particular patient population. For example, the Spinal Cord International Consortium has worked to develop data elements for core data capture [ 5 , 6 ] and subspecialty collections [ 7–9 ].

In clinical and translational research, CDE sets allow data to be collected and reported uniformly across multiple research studies and sites [ 10 ]. When systematically used across different sites, studies, or clinical trials to ensure consistent data collection, CDE sets enhance data harmonization and exchange and support policy-mandated health data sharing [ 3 ].

Consensus methods for the elicitation of diverse perspectives

Developing an agreed-upon set of elements requires that diverse perspectives on the importance and feasibility of the data elements are systematically elicited and effectively integrated across different groups of translational science stakeholders. Applying structured consensus development methods can aid in eliciting and enhancing diverse perspectives. Consensus development generally involves soliciting expert opinions, systematically capturing and integrating diverse perspectives, identifying agreement through voting, and discussing disagreements to inform final decisions. Rigorous in nature, the consensus became a valid and accepted approach for generating reliable evidence in a timely manner [ 11 ] to determine priorities and develop hypotheses [ 12 ].

The Delphi technique is the most well-known and established approach for reaching consensus for developing shared guidelines, recommendations, and CDE sets. The RAND Corporation initially developed the Delphi technique in the 1950s to forecast the effects of atomic warfare. Since its introduction in research, the method has been used in different academic fields such as health, science, technology, business, communication, policy analysis, and education [ 13 ]. The Delphi technique allows for restructuring a group communication process “so that such process is effective in allowing a group of individuals to deal with a complex problem” [ 14 ]. The overarching goal of the Delphi technique is to seek the systematic emergence of a concurrent opinion [ 15 ]. The Delphi technique promotes equitable participation and knowledge transfer among experts, who frequently bring diverse perspectives based on their scientific expertise, engagement in healthcare practice, community development, policy implementation, participation in professional societies, or lived experiences as patients and caregivers. In practice, it is an iterative process that involves completing a series of questionnaires over several rounds [ 14 , 15 ]. Such explicit focus on eliciting input and providing structure for exchanging ideas makes the Delphi method particularly suitable for meaningful stakeholder engagement in clinical and translational research.

Clinical data sharing and social drivers of health

Clinical data captured through electronic health records (EHR) constitutes a significant aspect of health indicators. However, there is evidence that capturing other data elements, including patient-reported outcomes and social drivers, can be more effective in shaping health outcomes [ 16 ]. A mere 20% of the population’s health factors pertain to medical service delivery. The residual 80%—typically referred to as individual and social domains of health (SDoH)—encompass patients’ socioeconomic status, health-promoting and limiting behaviors, physician and environmental factors, and accessibility, availability, and affordability of health care [ 17 ]. While there is widespread recognition of the importance of SDoH, overall scientific progress in addressing SDoH has been hindered by the lack of a resource to facilitate the collection of CDE sets for SDoH. Broad adoption of CDE sets on SDoH across behavioral, clinical, and translational research will facilitate cross-study analysis, domestically and internationally, accelerate translational research, and lead to a greater understanding of the causes of health disparities and the design and implementation of effective interventions to reduce health disparities.

In 2018, the National Institute on Minority Health and Health Disparities (NIMHD) led an effort to develop a CDE set for SDoH as part of the existing PhenX collections [ 10 ]. The PhenX SDoH toolkit is an expert-selected collection of CDE sets used to improve the quality and consistency of data acquisition and facilitate collaboration. The PhenX SDoH collection makes it easier for investigators to compare results, combine data from different studies, and promote the adoption of comparable data on SDoH across studies. The Core collection consists of 16 measurement protocols, including demographics (e.g. ethnicity and race, age, gender identity, annual family income, employment status) and social driver variables (e.g. English proficiency, occupational prestige, and access to health services). These protocols were designed to create SDoH CDE sets for cross-study analyses that compare or combine data from different studies, and the NIH encourages the use of the core SDoH variables for all primary data collection to connect data from various studies, advance minority health and health disparities science, promote a culture of scientific collaboration, and improve human health [ 18 ].

Study objective

As the utilization of the data and the application of consensus-building methods continue to expand, there is a critical need to evaluate (i) the extent to which the diverse perspectives around the health CDE sets are elicited and (ii) the rate of the inclusion of the social drivers of health as routinely captured data elements. Consequently, this study aims to report on the systematic scoping review of literature that examines the application of the Delphi method to achieve consensus regarding CDE sets, the inclusion of diverse stakeholder groups, and the integration of SDoH as data elements.

RQ1: What stakeholder groups are represented in Delphi-based clinical CDE sets?

RQ2: What are the sources of SDoH variables in Delphi-based clinical CDE sets?

RQ3: What types of SDoH variables are included in Delphi-based clinical CDE sets?

Study design

We conducted a systematic scoping review and followed the guidelines by the Joanna Briggs Institute in performing a literature review [ 19 ] and guidelines [ 20 ] in creating a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA-ScR) flow chart (see also Supplementary Appendix 3 for the checklist) without protocol registration.

Information sources and search strategy

A comprehensive search string was developed with guidance from a health library and information specialist (see Supplementary Appendix 1 for an example of the PubMed string). The Systematic Review Accelerator [ 21 , 22 ] was then used to translate the PubMed search string into the language used by the Embase, CINAHL, WoS MEDLINE, and PsycINFO databases.

Inclusion and exclusion criteria

Studies were included in the review if they were published in the English language in academic, peer-reviewed articles; reported on the use of Delphi or modified Delphi technique, alone or in combination with other methods; focused on developing consensus for data elements, data items, data sets, indicators, or data domains; and were related to human health. Exclusion criteria included non-English language, review, protocol, or commentary articles that referenced but did not apply the Delphi technique and studies that did not present health-related data elements or indicators. Specifically, we excluded studies that reported using the Delphi technique to develop clinical training outcomes, health policy recommendations, and clinical guidelines. Furthermore, based on the feedback from experts in SDoH research, studies were screened for geographic application. Studies developed specifically for low- and middle-income county contexts were excluded as the social drivers of health for populations in these countries differ significantly from those in high-income countries.

Screening and data charting process

The first author searched databases and uploaded the search results into the Covidence online platform for deduplication and review management. Two authors screened the title and abstract of each article. Screening decisions were discussed during weekly meetings and noted in study settings. Emerging conflicts were resolved by the senior author after discussion. Data from the studies included in the final review were extracted by three reviewers using Excel. During the extraction, verbatim text describing the engagement of stakeholder groups in the Delphi process and the inclusion of SDoH elements was captured. Finally, unstructured extraction data were systematically reduced. Descriptions of participating stakeholder groups were retained without changes, as reported in the reviewed articles. SDoH CDE sets were classified using the PhenX Core toolkit. Inclusion of demographic variables was classified using 11 protocols (Annual Family Income, Birthplace, Current Address, Current Age, Current Employment Status, Educational Attainment—Individual, Ethnicity and Race, Gender Identity, Health Insurance Coverage, Sex Assigned at Birth, Sexual Orientation), and inclusion of social drivers was classified using five protocols (Access to Health Services, English Proficiency, Food Insecurity, Health Literacy, Occupational Prestige). See Supplementary Appendix 2 for the data extraction table.

The methods and results sections of the included articles were reviewed to extract information about the groups of stakeholders involved in the development of health CDE sets. Data included Review article ID, authors, year, DOI, title, abstract, study goal, therapeutic area, geographic application, total number of involved experts, sources of CDE sets, and final consensus-based SDoH-related CDE sets (see Supplementary Appendix 2 ). All eligible studies were included in the scoping review. The critical appraisal of evidence was not conducted due to the scoping nature of the systematic review reported in this article.

A total of 384 studies matching the search string were identified, 167 duplicates were removed, and 217 were screened. Reasons for exclusion during the full-text review are shown in Fig. 1 . After the screening, 22 articles were deemed eligible for inclusion.

PRISMA flow diagram

The earliest included article was published in 2013 [ 23 ], with more articles published in 2021, 2022, and 2023. The complete list of included studies is available in Supplementary Appendix 2 . No publication date limitations were applied to the search. Geographically, CDE development projects addressed USA ( n = 8), international ( n = 8), Canada-specific ( n = 5), and Switzerland-specific ( n = 1) health contexts. The health focus of the reviewed articles covered a variety of therapeutic areas (e.g. cerebral palsy [ 24 ], epilepsy [ 25 ], spinal cord injury [ 8 ]), medical professions (e.g. nursing [ 23 ], surgery [ 26 , 27 ]), and population groups (e.g. pediatric patients [ 28–31 ]) frail adults [ 32 ].

Stakeholder groups represented in Delphi-based clinical CDE sets

The data in Delphi studies come from expert input. The studies included in this review involved between 5 and 272 experts ( M = 53.6, SD = 69.0). The studies with the smallest number of experts involved were part of more extensive data harmonization efforts for spinal cord injury [ 8 ] and pediatric sepsis [ 30 ] CDE sets. Another 14 studies involved less than 50 experts [ 23–25 , 29 , 31 , 33–41 ]. Finally, 6 studies involved 59–272 experts [ 26–28 , 32 , 42 , 43 ].

All studies involved experts with health or medical expertise directly relevant to the developed CDE set. Most studies did not distinguish between clinician scholars and researchers. However, several studies specifically differentiated and listed research (“researcher scientist”) and clinician (“PICU physician and nurse,” allied health practitioners) participants [ 24 , 28 , 32 , 37 , 40 , 42 , 43 ]. Other stakeholder groups involved in the CDE sets development included experts in health services and population health management [ 25 , 28 , 33 , 34 , 39 ] standards and accreditation [ 23 , 33 , 37 , 40 , 43 ], information technology and informatics [ 34 , 36 , 40 ] industry [ 42 , 43 ], and project management [ 28 , 32 , 38 ]. Finally, six studies included the participation of health consumers [ 24 , 28 , 32 , 43 ], caregivers [ 32 ], and patient advocacy groups [ 26 , 42 , 43 ]. These six projects represented the Delphi CDE set efforts involving a more significant number of experts. The exception to this group of studies is the study by Hirji et al. [ 27 ], who engaged 60 subject-matter experts and 137 participants from an Annual Multidisciplinary Cardiovascular and Thoracic Critical Care Conference, none of whom were reported to represent health consumers.

Sources of SDoH variables in Delphi-based clinical CDE sets

Most studies ( n = 14, 63%) used literature reviews [ 23–26 , 28–30 , 32 , 33 , 35–37 , 42 , 43 ], but only six reported that their reviews were systematic [ 24–26 , 29 , 37 , 42 ]. The second largest reported source of data elements was expert input ( n = 9, 41%) [ 28 , 32–34 , 36 , 38 , 39 , 41 , 42 ]. One study further differentiated between expert and provider input [ 28 ]. Regulatory requirements guided several studies ( n = 4, 18%) [ 36 , 40 ], existing protocols, 522 postmarket surveillance [ 43 ], and data standards [ 33 ]. Two studies (9%) used administrative [ 23 ] and clinical [ 43 ] real-world data. Finally, six studies were reported involving health consumers in the development of CDE sets [ 24 , 26 , 28 , 32 , 42 , 43 ]. Only two of them (9%) integrated patient perspectives by consulting patient-reported outcomes instruments [ 42 ] and direct input from a patient advisory group [ 26 ].

Types of SDoH variables included in Delphi-based clinical CDE sets

PhenX SDoH Core collection domains were used to extract the data about demographic and social driver domains included in the consensus-based CDE sets. Demographic SDoH domains covered by PhenX include Annual Family Income, Birthplace, Current Address, Current Age, Current Employment Status, Educational Attainment—Individual, Ethnicity and Race, Gender Identity, Health Insurance Coverage, Sex Assigned at Birth, Sexual Orientation and social driver domains include Access to Health Services, English Proficiency, Food Insecurity, Health Literacy, Occupational Prestige. Overall, nine studies (41%) reported including only demographic SDoH CDE sets [ 24–26 , 28 , 30 , 33 , 34 , 36 , 43 ], six studies (27%) also included both demographic and social driver domains [ 32 , 35 , 37 , 38 , 41 , 42 ], and seven studies (32%) did not report the inclusion of any demographic variables in the CDE sets [ 8 , 23 , 27 , 29 , 31 , 39 , 40 ]. On average, studies included 4.5 demographic domains (Min = 2, Max = 8) and 1.2 social driver domains (Min = 1, Max = 2).

Each demographic SDoH domain was included in at least one study. Age and Sex Assigned at Birth were two domains included in each of the 15 studies that considered SDoH CDE sets. In addition to Sex Assigned at Birth, five studies (23%) also included the domain of Gender Identity [ 28 , 32 , 33 , 35 , 43 ], and one study included the domain of Sexual Orientation [ 32 ]. Race and Ethnicity were included in 10 studies (45%) [ 28 , 32–36 , 38 , 41–43 ]. Employment was included in seven (32%) studies [ 32 , 35 , 37 , 38 , 41–43 ]. Current Address was included in four (18%) studies [ 28 , 33 , 34 , 41 ], and Educational Attainment was included in four (18%) studies [ 32 , 33 , 37 , 42 ]. Income [ 38 , 41 ], Health Insurance [ 32 , 34 ], and Birthplace [ 24 , 41 ] were covered by two studies (9%) each.

For the social driver domains, four studies (18%) included the Occupational Prestige domain [ 35 , 38 , 41 , 42 ], two studies (9%) reported including Access to Health Services [ 32 , 37 ], and one study included Food Insecurity [ 32 ]. The inclusion of English/Language Proficiency and Health Literacy was not reported by any study.

This study reported a systematic scoping review of the application of the Delphi technique as a consensus development method for constructing CDE sets. This review shows that the application of consensus methods provides transparency and opportunities for systematically comparing the methodologies used in CDE selection. This review contributes to the growing body of literature on consensus-based CDE set development and stands to inform future design considerations related to stakeholder, source, and type choices specific to the SDoH variables.

For stakeholder considerations, the current study showed that the studies under review identified and involved diverse stakeholder groups in developing CDE sets. Predominantly, healthcare professionals and translational scientists constituted the most frequently engaged groups. Most CDE set efforts did not involve patients or patient advocacy groups, which risks missing an opportunity to support patient-centered research and clinical practice [ 44 , 45 ]. The CDE set development efforts that involved more experts were poised to have representation from patients, caregivers, and industry groups. However, given the number of experts involved in these efforts, the extent to which patients’ voices were fully heard should be carefully considered and evaluated. The very definition of expert warrants a rigorous conceptual explication and operational definition. Clinical and research staff are only some of the best sources of expertise that can inform the development process of CDE sets. Instead, patients, caregivers, and other primary support groups are best suited for this endeavor. Individuals and patient advocacy groups have lived and health system interaction experiences [ 46 ] that can effectively contribute to the research process as community advisors or citizen scientists. The Delphi technique is a rigorous and recognized data collection method among experts, and future research is needed to identify the domains of expertise and characteristics of experts who should be involved in patient-, practitioner-, and community-centered CDE efforts. Future CDE set development studies should consider including patients or patient advocates who can bring the lived experience and contribute their expertise to identify SDoH variables that are core to their health conditions.

For SDoH source decisions, this study suggests that the inclusion of SDoH variables needs to be counterbalanced with the feasibility of collecting and extracting those data from electronic medical records. The missingness of SDoH data in medical records remains high [ 47 , 48 ]. Furthermore, low-resourced institutions and institutions that serve minoritized and low-income patients will likely face personnel shortages and technical challenges in SDoH data capture.

For the types of SDoH variables, this study showed that individual demographic domains were included in most reviewed studies, and data elements related to social factors, such as health behavior and access to care, were less common. Neighborhood-level data and social vulnerability indices can serve as a relevant proxy for social drivers of health that may affect human health. However, this review also showed that the current address is included in a small subset of data elements. Since current address data are regularly captured at the patient intake and are systematically recorded within the electronic medical records, the systematic inclusion of current address data can help alleviate this gap. Including zip codes can allow future research projects to link individual-level EHR data with existing neighborhood- and national-level statistics on the social conditions of populations within the United States.

In discussing the implications of this study, it is essential to note its focus and limitations. This systematic scoping review focused on and was limited to applying the Delphi method to develop health and clinical CDE sets. Other methods for expert consensus include structured Nominal Group Techniques and unstructured expert panels, but both are used less frequently to develop CDE sets. Furthermore, the review is limited to the studies that developed CDE sets for high-income counties. Similar CDE development efforts occur worldwide, and future reviews inform this practice by expanding the focus to include middle- and low-income countries. Despite these limitations, this scoping review has implications for clinical research and health policy development [ 49 ].

This study has implications for future research and policy development. Several US national guidelines call for the development of an infrastructure for health information exchange, data interoperability, and patient access to data. This review revealed that, despite their importance, patients and caregivers often need to be more represented in developing data elements and their inclusion as CDE sets. Therefore, conducting specific studies on implementing policies for patient and caregiver involvement could create better opportunities to promote patient-centered practices in health information exchange and data accessibility. This review also suggests that while SDoH variables are included in most efforts to develop CDE sets, their potential still needs to be fully realized. Thus, this research presents an opportunity for national task forces to develop evidence-based recommendations and SDoH standards. Future CDE set development efforts and national task forces can evaluate the feasibility of including the 16 domains articulated in the PhenX core dataset and provide recommendations for point-of-care data capture. Researchers should also continue to refine the vision for standardized data in research and clinical practice to maximize efficiency, hasten the initial stages of the study by reusing standardized metadata and tools, and reduce the load on data storage for quality and validation. These efforts would ensure more comprehensive inclusion of social drivers of health and broader engagement of stakeholder groups in policy and standards development.

The evolving landscape of technology and healthcare services has amplified the capacity to capture diverse health-related data for clinical and translational research. The translational science perspective that guided this review offers considerations for incorporating SDoH into future CDE sets and engaging stakeholder groups with diverse perspectives. The standardization and inclusion of SDoH as common data elements promise to improve the understanding of patients’ individual and social circumstances and their efforts to improve health outcomes. To harness the full potential of these data, standardization through the establishment of CDE sets is crucial. The Delphi technique effectively engages diverse expert groups around the health and SDoH CDE sets. Future studies can benefit from including health consumers as lived experience experts.

Research reported in this publication was partly supported by the NIH National Center for Advancing Translational Sciences to the University of Florida’s Clinical and Translational Science Institute through grant number UL1TR001427. The content is solely the authors’ responsibility and does not necessarily represent the official views of the National Institutes of Health.

Conflict of interest statement . The authors declare that they have no conflicts of interest.

Y.A.L.S., J.D.N., H.M.W., and X.W. contributed to the edits and literature knowledge of the sections. Y.A.L.S. contributed to the methods, results, and discussion sections. J.D.N., H.M.W., and X.W. contributed to the discussion section. S.M.M. contributed to the revisions and editing of the final manuscript. All authors read and approved the final manuscript.

Study Registration: This study was not formally registered. Analytical Plan Preregistration: Not applicable for this study. Data Availability: De-identified data not applicable. Analytical Code Availability: There is not analytical code associated with this study. Materials Availability: Materials for this study is not applicable.

Pastorino R , De Vito C , Migliara G et al. . Benefits and challenges of Big Data in healthcare: an overview of the European initiatives . Eur J Public Health 2019 ; 29 : 23 – 7 . https://doi.org/10.1093/eurpub/ckz168

Google Scholar

Sheehan J , Hirschfeld S , Foster E et al. . Improving the value of clinical research through the use of Common Data Elements . Clin Trials 2016 ; 13 : 671 – 6 . https://doi.org/10.1177/1740774516653238

Mandl KD , Kohane IS. Data citizenship under the 21st Century Cures Act . N Engl J Med 2020 ; 382 : 1781 – 3 . https://doi.org/10.1056/NEJMp1917640

Everson J , Healy D , Patel V. Experiences with information blocking in the United States: a national survey of hospitals . J Am Med Inform Assoc: JAMIA 2023 ; 30 : 1150 – 7 . https://doi.org/10.1093/jamia/ocad060

DeVivo M , Biering-Sørensen F , Charlifue S et al. . ; Executive Committee for the International SCI Data Sets Committees . International spinal cord injury core data set . Spinal Cord 2006 ; 44 : 535 – 40 . https://doi.org/10.1038/sj.sc.3101958

Biering-Sørensen F , DeVivo MJ , Charlifue S et al. . International Spinal Cord Injury Core Data Set (version 2.0)—including standardization of reporting . Spinal Cord 2017 ; 55 : 759 – 64 . https://doi.org/10.1038/sc.2017.59

Biering-Sørensen F , Burns AS , Curt A et al. . International spinal cord injury musculoskeletal basic data set . Spinal Cord 2012 ; 50 : 797 – 802 . https://doi.org/10.1038/sc.2012.102

Carroll A , Vogel LC , Zebracki K et al. . Relevance of the international spinal cord injury basic data sets to youth: an Inter-Professional review with recommendations . Spinal Cord 2017 ; 55 : 875 – 81 . https://doi.org/10.1038/sc.2017.14

New PW , Marshall R. International spinal cord injury data sets for non-traumatic spinal cord injury . Spinal Cord 2014 ; 52 : 123 – 32 . https://doi.org/10.1038/sc.2012.160

PhenX . Social Determinants of Health: Core Collection . 2018 . https://www.phenxtoolkit.org/collections/view/6 ( 1 December 2023 , date last accessed).

Jandhyala R. Delphi, non-RAND modified Delphi, RAND/UCLA appropriateness method and a novel group awareness and consensus methodology for consensus measurement: a systematic literature review . Curr Med Res Opin 2020 ; 36 : 1873 – 87 . https://doi.org/10.1080/03007995.2020.1816946

Anonymous . Group techniques for program planning: a guide to nominal group and Delphi processes . J Appl Behav Sci 1976 ; 12 : 581 . https://journals.sagepub.com/doi/abs/10.1177/002188637601200414 ( 31 July 2023 , date last accessed).

Arakawa N , Bader LR. Consensus development methods: considerations for national and global frameworks and policy development . Res Soc Admin Pharm: RSAP 2022 ; 18 : 2222 – 9 . https://doi.org/10.1016/j.sapharm.2021.06.024

Turoff M , Linstone HA. The Delphi method: techniques and applications . J Market Res 1975 ; 18 . https://doi.org/10.2307/3150755

Hohmann E , Cote MP , Brand JC. Research Pearls: expert consensus based evidence using the Delphi method . Arthroscopy 2018 ; 34 : 3278 – 82 . https://doi.org/10.1016/j.arthro.2018.10.004

NEJM Catalyst . Social Determinants of Health (SDOH) . Catalyst Carryover 2017 ; 3 : https://doi.org/10.1056/CAT.17.0312

Improvement, I. for C. S . Going Beyond Clinical Walls: Solving Complex Problems . Institute for Clinical Systems Improvement and the Robert Wood Johnson Foundation , 2014 . https://www.icsi.org/wp-content/uploads/2019/08/4.FacilitatingAConversation_BeyondClinicalWalls.pdf ( 2 May 2024, date last accessed ).

Google Preview

National Institutes of Health . NOT-MD-21-003: Notice Announcing Availability of Data Harmonization Tools for Social Determinants of Health (SDOH) via the PhenX Toolkit . National Institutes of Health , 2020 . https://grants.nih.gov/grants/guide/notice-files/NOT-MD-21-003.html ( 1 December 2023 , date last accessed).

Peters MDJ , Godfrey CM , Khalil H et al. . Guidance for conducting systematic scoping reviews . Int J Evid-Based Healthc 2015 ; 13 : 141 – 6 . https://doi.org/10.1097/XEB.0000000000000050

Tricco AC , Lillie E , Zarin W et al. . PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation . Ann Intern Med 2018 ; 169 : 467 – 73 . https://doi.org/10.7326/M18-0850

Clark J , Glasziou P , Del Mar C et al. . A full systematic review was completed in 2 weeks using automation tools: a case study . J Clin Epidemiol 2020 ; 121 : 81 – 90 . https://doi.org/10.1016/j.jclinepi.2020.01.008

SR-Accelerator . n.d . https://sr-accelerator.com/#/polyglot ( 31 July 2023 , date last accessed).

Morton P , Petersen C , Chard R et al. . Validation of the data elements for the health system domain of the PNDS . AORN J 2013 ; 98 : 39 – 48 . https://doi.org/10.1016/j.aorn.2013.05.008

Wilson YA , Smithers-Sheedy H , Ostojic K et al. . ; ICPGC Phenotype Working Group . Common data elements to standardize genomics studies in cerebral palsy . Dev Med Child Neurol 2022 ; 64 : 1470 – 6 . https://doi.org/10.1111/dmcn.15245

Sauro KM , Wiebe S , Macrodimitris S et al. . ; EMU Quality Improvement Team . Quality indicators for the adult epilepsy monitoring unit . Epilepsia 2016 ; 57 : 1771 – 8 . https://doi.org/10.1111/epi.13563

Coulman KD , Chalmers K , Blazeby J et al. . Development of a bariatric surgery core data set for an International Registry . Obes Surg 2023 ; 33 : 1463 – 75 . https://doi.org/10.1007/s11695-023-06545-y

Hirji SA , Salenger R , Boyle EM et al. . Expert consensus of data elements for collection for enhanced recovery after cardiac surgery . World J Surg 2021 ; 45 : 917 – 25 . https://doi.org/10.1007/s00268-021-05964-1

Roumeliotis N , Ramil J , Garros D et al. . Designing a national pediatric critical care database: a Delphi consensus study . Can J Anaesth 2023 ; 70 : 1216 – 25 . https://doi.org/10.1007/s12630-023-02480-9

Lamsal R , Johnson JK , Mulla M et al. . Improving communication by standardizing pediatric rapid response team documentation . J Healthc Qual 2022 ; 44 : e1 – 6 . https://doi.org/10.1097/JHQ.0000000000000334

Mawji A , Li E , Chandna A et al. . Common data elements for predictors of pediatric sepsis: a framework to standardize data collection . PLoS One 2021 ; 16 : e0253051 . https://doi.org/10.1371/journal.pone.0253051

Wismer AC , Rakic M , Kuehni CE et al. . Consensus minimal dataset for pediatric emergency medicine in Switzerland . Pediatr Emerg Care 2022 ; 38 : 511 – 6 . https://doi.org/10.1097/PEC.0000000000002841

Prorok JC , Williamson PR , Shea B et al. . An international Delphi consensus process to determine a common data element and core outcome set for frailty: FOCUS (The Frailty Outcomes Consensus Project) . BMC Geriatr 2022 ; 22 : 284 . https://doi.org/10.1186/s12877-022-02993-w

Lougheed MD , Taite A , ten Hove J et al. . Pan-Canadian asthma and COPD standards for electronic health records: a Canadian Thoracic Society Expert Working Group Report . Can J Respir Critic Care Sleep Med 2018 ; 2 : 244 – 50 . https://doi.org/10.1080/24745332.2018.1517623

Young DL , Fritz JM , Kean J et al. . Key data elements for longitudinal tracking of physical function: a modified Delphi consensus study . Phys Ther 2022 ; 102 : pzab279 . https://doi.org/10.1093/ptj/pzab279

Bulathsinhala L , Eleangovan N , Heaney LG et al. . Development of the International Severe Asthma Registry (ISAR): a modified Delphi study . J Allergy Clin Immunol Pract 2019 ; 7 : 578 – 88.e2 . https://doi.org/10.1016/j.jaip.2018.08.016

Baird CE , Marinac-Dabic D , Darbouze F et al. . Development of a coordinated registry network for women’s health technologies . Value Health 2018 ; 21 : S179 – 80 . https://doi.org/10.1016/j.jval.2018.04.1194 .

Chowdhury MI , Turk-Adawi K , Babu AS et al. . Development of the International Cardiac Rehabilitation Registry including variable selection and definition process . Glob Heart 2022 ; 17 : 1 . https://doi.org/10.5334/gh.1091

Vaccarino AL , Beaton D , Black SE et al. . Common data elements to facilitate sharing and re-use of participant-level data: assessment of psychiatric comorbidity across brain disorders . Front Psychiatry 2022 ; 13 : 816465 . https://doi.org/10.3389/fpsyt.2022.816465 .

Boesch RP , de Alarcon A , Piccione J et al. . ; Aerodigestive Research Collaborative . Consensus on triple endoscopy data elements preparatory to development of an Aerodigestive Registry . Laryngoscope 2022 ; 132 : 2251 – 8 . https://doi.org/10.1002/lary.30038

Heinemann A , Raad J , Akuthota V et al. . Scoping review to develop common data elements for lumbar spinal stenosis . Spine J 2017 ; 17 : 1045 – 57 . https://doi.org/10.1016/j.spinee.2017.04.005

Daneshvari Berry S , Kroth PJ , Edgar HJH et al. . Developing the minimum dataset for the new Mexico decedent image database . Appl Clin Inform 2021 ; 12 : 518 – 27 . https://doi.org/10.1055/s-0041-1730999

Wall D , Meah N , York K et al. . A global eDelphi exercise to identify core domains and domain items for the development of a Global Registry of Alopecia Areata Disease Severity and Treatment Safety (GRASS) . JAMA Dermatol 2021 ; 157 : 1 – 11 . https://doi.org/10.1001/jamadermatol.2020.5839

Gressler LE , Cowley T , Velezis M et al. . Building the foundation for a modern patient-partnered infrastructure to study temporomandibular disorders . Front Digit Health 2023 ; 5 : 1132446 . https://doi.org/10.3389/fdgth.2023.1132446

Chung AE , Sandler RS , Long MD et al. . Harnessing person-generated health data to accelerate patient-centered outcomes research: the Crohn’s and Colitis Foundation of America PCORnet Patient Powered Research Network (CCFA Partners) . J Am Med Inform Assoc: JAMIA 2016 ; 23 : 485 – 90 . https://doi.org/10.1093/jamia/ocv191

Bergeson SC , Dean JD. A systems approach to patient-centered care . JAMA 2006 ; 296 : 2848 – 51 . https://doi.org/10.1001/jama.296.23.2848

Rudisill AC , Eicken MGA , Gupta D et al. . Patient and care team perspectives on social determinants of health screening in primary care: a qualitative study . JAMA Network Open 2023 ; 6 : e2345444 . https://doi.org/10.1001/jamanetworkopen.2023.45444

Venzon A , Le TB , Kim K. Capturing social health data in electronic systems: a systematic review . Comput Inform Nurs 2019 ; 37 : 90 – 8 . https://doi.org/10.1097/CIN.0000000000000481

Cook LA , Sachs J , Weiskopf NG. The quality of social determinants data in the electronic health record: a systematic review . J Am Med Inform Assoc 2022 ; 29 : 187 – 96 . https://doi.org/10.1093/jamia/ocab199

Luke DA , Sarli CC , Suiter AM et al. . The translational science benefits model: a new framework for assessing the health and societal benefits of clinical and translational sciences . Clin Transl Sci 2018 ; 11 : 77 – 84 . https://doi.org/10.1111/cts.12495

Supplementary data

Email alerts

Citing articles via.

• Recommend to Your Librarian
• Advertising & Corporate Services

Affiliations

• Online ISSN 1613-9860
• Print ISSN 1869-6716
• Copyright © 2024 Society of Behavioural Medicine
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Research guidelines for the Delphi survey technique

Affiliation.

• 1 Centre for Nursing Research, University of Ulster, Newtownabbey, Northern Ireland. [email protected]
• PMID: 11095242

Consensus methods such as the Delphi survey technique are being employed to help enhance effective decision-making in health and social care. The Delphi survey is a group facilitation technique, which is an iterative multistage process, designed to transform opinion into group consensus. It is a flexible approach, that is used commonly within the health and social sciences, yet little guidance exists to help researchers undertake this method of data collection. This paper aims to provide an understanding of the preparation, action steps and difficulties that are inherent within the Delphi. Used systematically and rigorously, the Delphi can contribute significantly to broadening knowledge within the nursing profession. However, careful thought must be given before using the method; there are key issues surrounding problem identification, researcher skills and data presentation that must be addressed. The paper does not claim to be definitive; it purports to act as a guide for those researchers who wish to exploit the Delphi methodology.

PubMed Disclaimer

Similar articles

• An accessible analytical approach for investigating what happens between the rounds of a Delphi study. Greatorex J, Dexter T. Greatorex J, et al. J Adv Nurs. 2000 Oct;32(4):1016-24. J Adv Nurs. 2000. PMID: 11095243 Review.
• Consulting the oracle: ten lessons from using the Delphi technique in nursing research. Keeney S, Hasson F, McKenna H. Keeney S, et al. J Adv Nurs. 2006 Jan;53(2):205-12. doi: 10.1111/j.1365-2648.2006.03716.x. J Adv Nurs. 2006. PMID: 16422719 Review.
• A critical review of the Delphi technique as a research methodology for nursing. Keeney S, Hasson F, McKenna HP. Keeney S, et al. Int J Nurs Stud. 2001 Apr;38(2):195-200. doi: 10.1016/s0020-7489(00)00044-4. Int J Nurs Stud. 2001. PMID: 11223060 Review.
• The Delphi process in dental research. Cramer CK, Klasser GD, Epstein JB, Sheps SB. Cramer CK, et al. J Evid Based Dent Pract. 2008 Dec;8(4):211-20. doi: 10.1016/j.jebdp.2008.09.002. J Evid Based Dent Pract. 2008. PMID: 19027655
• Do we agree? Using a Delphi technique to develop consensus on skills of hand expression. Becker GE, Roberts T. Becker GE, et al. J Hum Lact. 2009 May;25(2):220-5. doi: 10.1177/0890334409333679. J Hum Lact. 2009. PMID: 19414824
• Using mixed methods and partnership to develop a program evaluation toolkit for organizations that provide physical activity programs for persons with disabilities. Lawrason SVC, DaSilva P, Michalovic E, Latimer-Cheung A, Tomasone JR, Sweet S, Forneris T, Leo J, Greenwood M, Giles J, Arkell J, Patatas J, Boyle N, Adams N, Martin Ginis KA. Lawrason SVC, et al. Res Involv Engagem. 2024 Sep 2;10(1):91. doi: 10.1186/s40900-024-00618-7. Res Involv Engagem. 2024. PMID: 39223687 Free PMC article.
• Development of a standardized consensus lexicon for terms related to micronutrient programs. Hlaing LM, Bourassa MW, Brown KH, Atkin R, Osendarp SJM, Gupta A, Hess SY. Hlaing LM, et al. PLoS One. 2024 Aug 30;19(8):e0308230. doi: 10.1371/journal.pone.0308230. eCollection 2024. PLoS One. 2024. PMID: 39213298 Free PMC article.
• Construction and Analysis of the Physical Fitness Evaluation Index System for Elite Male Singles Badminton Players: Based on Delphi and AHP Methods. Ye B, Zhu H, Yang Z, He Z, Liu G, Pan H, Guo H. Ye B, et al. Life (Basel). 2024 Jul 28;14(8):944. doi: 10.3390/life14080944. Life (Basel). 2024. PMID: 39202686 Free PMC article.
• Textbook Outcome in Colorectal Surgery for Cancer: An Italian Version. Sofia S, Degiuli M, Anania G, Baiocchi GL, Baldari L, Baldazzi G, Bianco F, Borghi F, Cavaliere D, Coco C, Coppola R, D'Ugo D, Delrio P, Fumagalli Romario U, Guerrieri M, Milone M, Morino M, Muratore A, Navarra G, Pedrazzani C, Persiani R, Petz W, Rosati R, Roviello F, Scabini S, Sica G, Solaini L, Spinelli A, Spolverato G, Urso E, Reddavid R. Sofia S, et al. J Clin Med. 2024 Aug 9;13(16):4687. doi: 10.3390/jcm13164687. J Clin Med. 2024. PMID: 39200831 Free PMC article.
• Delphi studies in social and health sciences-Recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study. Niederberger M, Schifano J, Deckert S, Hirt J, Homberg A, Köberich S, Kuhn R, Rommel A, Sonnberger M; DEWISS network. Niederberger M, et al. PLoS One. 2024 Aug 26;19(8):e0304651. doi: 10.1371/journal.pone.0304651. eCollection 2024. PLoS One. 2024. PMID: 39186713 Free PMC article.

Publication types

• Search in MeSH

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

Other Literature Sources

• The Lens - Patent Citations

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Delphi Methodology

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Last »
• Delphi survey Follow Following
• Delphi Follow Following
• Analytical Hierarchy Process (AHP) Follow Following
• Decision Making Analysis and Modeling Follow Following
• Decision Analysis Follow Following
• Free Software Follow Following
• Construction Industry Follow Following
• SPSS Software Follow Following
• Open Source and Free Software Studies Follow Following
• Open Access Publishing Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Journals
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024