Delphi methodology in healthcare research: How to decide its appropriateness

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members. The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine. The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited, ethically/logistically difficult or evidence is conflicting. However, the attempts to assess the quality standard of Delphi studies have reported significant variance, and details of the process followed are usually unclear. We recommend systematic quality tools for evaluation of Delphi methodology; identification of problem area of research, selection of panel, anonymity of panelists, controlled feedback, iterative Delphi rounds, consensus criteria, analysis of consensus, closing criteria, and stability of the results. Based on these nine qualitative evaluation points, we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019. There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members, defining consensus, closing criteria for rounds, and presenting the results. We propose our evaluation points for researchers, medical journal editorial boards, and reviewers to evaluate the quality of the Delphi methods in healthcare research.     

An approach to an index of hospital performance.

Two indexes are described, based on measures of administrative effectiveness and patient care effectiveness. The measures used were selected and ranked by a Delphi panel from a list of 30 measures drawn from the literature. Weights were assigned by the panel to 19 selected measures. The resulting indexes did well in a test on data collected from 32 Texas hospitals.     

Formative Evaluation of a Video and Training Manual on Feeding Children with Special Needs

Special Foods for Special Kids (video and manual) was developed to educate child care providers who participate in the Child and Adult Care Food Program on feeding and nutrition issues (nutrient content, food preparation, social impact, and environmental considerations) for children with special needs.To evaluate the appropriateness of the training materials, a preliminary expert panel, modified Delphi panel, and focus groups were used. Composite scores given by the Delphi panel for content, readability, and format were 4.59 ± 0.61, 4.29 ± 1.0, and 4.41 ± 0.64, respectively, on a scale from 1 (strongly disagree) to 5 (strongly agree). Suggestions from the latter two evaluation strategies to improve the materials included adding a glossary of terms, a section on autism and a section on choking, statements on safety and sanitation issues, and an explanation of the format of answers (which were upside down). Multiple, formative evaluation methods were used among developers, experts, and users to ensure that these training materials were appropriate and applicable for the end user.     

Methods for public health economic evaluation: A Delphi survey of decision makers in English and Welsh local government

Standard reference case methods recommended for health technology appraisals do not translate well to a public health setting. This paper reports on a Delphi survey designed to elicit views of public health decision makers in England and Wales, about different methodological elements of economic evaluation. This is important as methods should align with the objective function of decision makers. The Delphi survey comprised two rounds, with round 1 allowing open‐ended recommendations in addition to 5‐point Likert scale responses. The final survey comprised 36 questions, and levels and strength of agreement were assessed using median values and mean absolute deviation of the median. The Delphi panel (n = 66) achieved high levels of agreement for costs, health, well‐being, and productivity impact to be important elements within an economic evaluation. The panel agreed that evaluations should be relevant to the local context and include costs and consequences over a lifetime horizon. There was a call for the transparent reporting of costs and effects for different population subgroups, and for different sectors. Overall, the panel revealed a preference for a flexible approach, understanding that economic evidence fits within a dynamic process of decision making. These results provide empirical evidence to inform guidelines for public health economic evaluation.     

Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria

OBJECTIVES: The aim of the Consensus on Health Economic Criteria (CHEC) project is to develop a criteria list for assessment of the methodological quality of economic evaluations in systematic reviews. The criteria list resulting from this CHEC project should be regarded as a minimum standard. METHODS: The criteria list has been developed using a Delphi method. Three Delphi rounds were needed to reach consensus. Twenty-three international experts participated in the Delphi panel. RESULTS: The Delphi panel achieved consensus over a generic core set of items for the quality assessment of economic evaluations. Each item of the CHEC-list was formulated as a question that can be answered by yes or no. To standardize the interpretation of the list and facilitate its use, the project team also provided an operationalization of the criteria list items. CONCLUSIONS: There was consensus among a group of international experts regarding a core set of items that can be used to assess the quality of economic evaluations in systematic reviews. Using this checklist will make future systematic reviews of economic evaluations more transparent, informative, and comparable. Consequently, researchers and policy-makers might use these systematic reviews more easily. The CHEC-list can be downloaded freely from http://www.beoz.unimaas.nl/chec/.     

Delphi法的研究进展及其在医学中的应用

Delphi法是一种通过向专家进行几轮咨询，获得专家一致性意见的预测方法，由于其具有的匿名性、信息反馈性和对结果进行统计分析三大特点，在社会科学领域中广泛应用，近年来在医学研究中的应用也逐渐扩展。本综述了Delphi法在方法学方面的研究进展及其在医学研究中的应用，同时也指出了其不足之处。     

Delphi as a method to establish consensus for diagnostic criteria

BACKGROUND/OBJECTIVES: To achieve a consensus, among a panel of experts, on the best clinical criteria for the clinical diagnosis of carpal tunnel syndrome (CTS). METHODS: Experts rated the diagnostic importance of items from the clinical history and physical examination for CTS. The ratings were expressed on a 10-cm visual analog scale. The average and standard deviation of the scores for each item were returned to the panelists. The panel members evaluated the items a second time with knowledge of the group responses from the first round. The scores were standardized to minimize scaling variations and, after the second round, the items were ranked in order of importance assigned by the group. Cronbach's alpha was used as a measure of homogeneity for the rankings. Increasing homogeneity was considered to be an indication of consensus among the panelists. RESULTS: Cronbach's alpha increased from 0.86 after the first round to 0.91 after the second iteration. Panelists who were relative outliers on the first round demonstrated a much higher correlation with the entire group after the second round. CONCLUSIONS: Delphi is an effective method of establishing consensus for certain clinical questions. Cronbach's alpha was a useful statistic for measuring the extent of consensus among the panel members. Delphi was chosen from the possible methods of group process because of its inherent feasibility. The absence of a need by the panelists to meet in person removed any constraint on the geographic location of the panel members. In addition, the anonymous nature of Delphi was thought to be a key factor in avoiding a result that might be skewed by one or more persuasive panelists. Both of these characteristics were felt to be particularly important to the topic on which consensus was sought, the clinical diagnostic criteria for CTS. This movement in the opinions of some of the panelists appeared to result from the feedback of information describing the group opinion.     

Recommendations for Welcoming Back Nursing Home Visitors During the COVID-19 Pandemic: Results of a Delphi Panel

ObjectivesNursing homes became epicenters of COVID-19 in the spring of 2020. Due to the substantial case fatality rates within congregate settings, federal agencies recommended restrictions to family visits. Six months into the COVID-19 pandemic, these largely remain in place. The objective of this study was to generate consensus guidance statements focusing on essential family caregivers and visitors.DesignA modified 2-step Delphi process was used to generate consensus statements.Setting and ParticipantsThe Delphi panel consisted of 21 US and Canadian post-acute and long-term care experts in clinical medicine, administration, and patient care advocacy.MethodsState and federal reopening statements were collected in June 2020 and the panel voted on these using a 3-point Likert scale with consensus defined as ≥80% of panel members voting “Agree.” The consensus statements then informed development of the visitor guidance statements.ResultsThe Delphi process yielded 77 consensus statements. Regarding visitor guidance, the panel made 5 strong recommendations: (1) maintain strong infection prevention and control precautions, (2) facilitate indoor and outdoor visits, (3) allow limited physical contact with appropriate precautions, (4) assess individual residents' care preferences and level of risk tolerance, and (5) dedicate an essential caregiver and extend the definition of compassionate care visits to include care that promotes psychosocial well-being of residents.Conclusions and ImplicationsThe COVID-19 pandemic has seen substantial regulatory changes without strong consideration of the impact on residents. In the absence of timely and rigorous research, the involvement of clinicians and patient care advocates is important to help create the balance between individual resident preferences and the health of the collective. The results of this evidence-based Delphi process will help guide policy decisions as well as inform future research.     

Evaluation of explicit criteria for total hip joint replacement

We tested an appropriateness of indications tool for total hip joint replacement in patients with osteoarthritis. Criteria were developed using a modified Delphi panel judgment process. Ratings were analyzed regarding level of agreement among panelists. Another panel rated the same indications; results were compared with the main panel. Test-retest of the main panel was performed. Regression models were used to assess the contribution of each algorithm variable. Appropriateness indication judgment was applied to 84 patients and compared to health-related quality-of-life improvement before and 3 months following intervention. Main panel ratings compared to those of a second panel resulted in a kappa statistic of 0.77. Test-retest kappa for the main panel was 0.81. Patients considered appropriate candidates for surgery, based on their composite indication scores, showed more improvement in health-related quality of life after 3 months than those considered inappropriately by composite indication scores. The previous parameters tested showed acceptable results for an evaluation tool. These results support the use of this indications algorithm as a screening tool for assessing the appropriateness of hip replacement surgery in osteoarthritis.     

德尔菲法及其在公共卫生领域的应用和展望

德尔菲法(Delphi method)是一个重复进行的专家咨询过程,以整合一组专家个人意见成群组共识为目的的主观评价和预测的方法.该方法具有重复性、匿名性、可控的信息反馈性和对结果统计分析的四大特点,起初广泛应用于军事、社会和经济等领域,近年来在公共卫生领域的应用也日渐广泛.本文将介绍德尔菲法的概念、过程和特点等基本情况,阐述该方法在公共卫生领域内的应用现状,对该法未来发展作出总结和展望.     

Outcome measures for clinical genetics services: a comparison of genetics healthcare professionals and patients' views

OBJECTIVES: To explore genetics professionals' and patients' views about which outcome domains are most appropriate to measure the patient benefits of using a clinical genetics service. METHODS: A postal Delphi survey was sent to: 115 consultant geneticists; 162 genetic counsellors; 156 support group representatives; 106 patients. The survey contained 19 outcome domains and respondents assessed the usefulness of each for clinical genetics services. RESULTS: The final professional panel comprised 115 genetics healthcare professionals and the patient panel comprised 72 patients. The outcome domains that achieved consensus (at least 75% of panel rated 'useful') for the patient and professional panels were: decision-making; knowledge of the genetic condition; perceived personal control; risk perception; satisfaction; meeting expectations; ability to cope; diagnosis accuracy; quality of life. Comparison of the ratings between the professional panel and the patient panel showed there was no statistical difference (chi(2), p<0.01) between the ratings ('useful' compared to 'not useful') for 14 of the 19 outcome domains but found differences for the perceived usefulness of: level of depression; health status; spiritual well-being; test accuracy; rate of termination. CONCLUSIONS: This Delphi survey identified nine outcome domains which are good starting points to develop a core set of outcome measures for evaluating clinical genetics services.     

Applying the Delphi technique in a study of GPs' information requirements

This article describes some dilemmas and problems encountered in a Delphi study of general practitioners' (GPs) information requirements. The research involved a three-round Delphi administered to an expert panel of 50 GPs in one Welsh health authority area. The Delphi is generally perceived as a qualitative method. However, we argue that the requirements of the technique make it difficult to sustain the kind of inductive analysis – faithfully reflecting respondents' perspectives – that is axiomatic to many of the theoretical approaches that underpin qualitative inquiry. We describe how our attempts to incorporate respondents' views in near-verbatim form in the first round were undermined by the need to classify and reduce statements for later rounds, and to impose judgments about what should count as consensus. The iterative 'consensus-building' process, that is so central to the Delphi, was difficult to take forward without active intervention by the research team, but this involved a re-ordering and reduction of the data, which moved the statements included in later rounds further and further from the verbatim responses on which they were based. Whilst the findings of the study were useful on one level, the final consensus statements took a general, virtually context-free form, which contained few references to background preoccupations (largely concerned with the recent NHS reforms) that were exercising GPs at this time. The method also coped badly with the polarized opinions that existed regarding the development of information systems for commissioning. We conclude that the Delphi is best used for large-scale research in areas where opinions are well established, where the problems and choices confronting the study group are well known, and where a major programme of organizational reform is not already underway.     

Health care administration in the year 2000: practitioners' views of future issues and job requirements

This research identifies the most important domains in health care administration (HCA) from now to the year 2000 and differentiates job skill, knowledge, and ability requirements necessary for successful management. Fellows of the American College of Healthcare Executives from about half of the United States responded to two iterations of a Delphi mail inquiry. Fellows identified 102 issues that were content-analyzed into nine domains by an HCA expert panel. Domains, in order of ranked importance, were cost/finance, leadership, professional staff interactions, health care delivery concepts, accessibility, ethics, quality/risk management, technology, and marketing. In the second Delphi iteration, Fellows reviewed domain results and rated job requirements on required job importance. Results indicated that while a business orientation is needed for organizational survival, an equal emphasis on person-oriented skills, knowledge, and abilities is required.     

A Delphi-based consensus study into planning for biological incidents

OBJECTIVE: Biological incidents present a significant threat to health services in the UK. The objective of this study was to achieve consensus in all phases of biological incident planning and response. METHODS: A three-round Delphi study was conducted using a panel of 23 experts from specialities involved in the management of biological incidents. The consensus and non-consensus outcomes from the Delphi study were subsequently presented for discussion in four syndicate groups at a one-day consensus conference funded by the Health Protection Agency. RESULTS: The results of each syndicate group discussion were presented at a subsequent plenary session at the end of the conference. Further iteration of both the consensus and the non-consensus outcomes of the Delphi study resulted in the endorsement, modification, integration or rejection of individual statements. 125 consensus statements were produced. CONCLUSIONS: The 125 synopsis consensus statements that all phases of biological incident planning and response. These can be used to inform policy decisions and translated into practical guidance for emergency planners and first responders at local, regional and national levels.     

Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability.

OBJECTIVES: To develop valid, reliable indicators of the appropriateness of long-term prescribing in general practice medical records in the United Kingdom. DESIGN: A nominal group was used to identify potential indicators of appropriateness of prescribing. Their face and content validity were subsequently assessed in a two round Delphi exercise. Feasibility and reliability between raters were evaluated for the indicators for which consensus was reached and were suitable for application. PARTICIPANTS: The nominal group comprised a disciplinary mix of nine opinion leaders and prominent academics in the field of prescribing. The Delphi panel was composed of 100 general practitioners and 100 community pharmacists. RESULTS: The nominal group resulted in 20 items which were refined to produce 34 statements for the Delphi exercise. Consensus was reached on 30, from which 13 indicators suitable for application were produced. These were applied by two independent raters to the records of 49 purposively sampled patients in one general practice. Nine indicators showed acceptable reliability between raters. CONCLUSIONS: 9 indicators of prescribing appropriateness were produced suitable for application to the medical record of any patient on long term medication in United Kingdom general practice. Although the use of the medical record has limitations, this is currently the only available method to assess a patient's drug regimen in its entirety.     

Development of explicit criteria for total knee replacement

OBJECTIVES: To develop and test an appropriateness of indications tool for total knee replacement (TKR) in patients with osteoarthritis. METHODS: Criteria were developed using a modified Delphi panel judgment. Another panel rated the same indications, and the results were compared with the main panel. Test-retest of the main panel was performed. Regression models were used to assess the contribution of each algorithm variable. A classification tree was developed. RESULTS: The procedure was considered appropriate in 167 (26.8%) scenarios, and there was agreement on 112 (67.1%) of them. When the rates of the main panel were compared with those of a second panel, the result was a kappa statistic of 0.75. The test-retest kappa for the main panel was 0.78. Neither in the first case nor in the second was there an instance in which a scenario classified as appropriate shifted to inappropriate or vice versa. The regression models showed that symptomatology and radiology were the variables that explained most of the variability of appropriateness as determined by panelists. In the classification tree performed, the probability of misclassification was 3.8% with 150 scenarios, of the 156 analyzed and classified correctly. CONCLUSIONS: The previous parameters tested showed acceptable results for an evaluation tool. These results support the use of this algorithm as an aid in formulating clinical practice guidelines and to promote the appropriateness of TKR.     

Pertussis: recurrent threat

Disturbing increase of pertussis diseases in Poland in the last twelve years has been observed also in Bia?ystok district and in Bia?ystok City. Incidence rates exceed an average of morbidity in Poland. Most of illnesses are registered at children between 5 to 9 years old, completely immunized against this disease. There were more females than males. The most of patients needed hospitalization.     

Infusing Youth Development Principles in Child Welfare Practice: Use of a Delphi Survey to Inform Training

To explore ways to integrate a youth development approach into child welfare work, a Delphi survey of experts was conducted. In particular, the Delphi survey technique was used to generate suggestions regarding: 1) the training needs of child welfare workers dealing with youth transitioning out of care, and 2) potential policy approaches to change in the child welfare system for these youth. A first round survey elicited suggestions of panel respondents and a second round survey requested the rating of these suggestions. Findings illustrate the importance of a holistic approach in working with these youth, as well as the importance of connection and relationship. Training approaches also were perceived as a feasible policy response to helping youth transition from care to independent living.     

Identifying indicators of laboratory management performance: a multiple constituency approach

The challenges for laboratory management posed by cost control, managed care, organizational restructuring, information networking and health system integration call for new measures to evaluate how effectively laboratories manage emerging performance expectations. This study identifies the Delphi panel method in achieving consensus on measures of effectiveness and considers a specific application, the identification of indicators of laboratory performance, from the perspective of key constituencies.