Assessment of Interrater Reliability of the Emergency Severity Index After Implementation in Emergency Departments in Jamaica Using a Learning Collaborative Approach

IntroductionIn 2016, the Ministry of Health in Jamaica selected the Emergency Severity Index as the triage tool to be used nationally. This study evaluated the effectiveness of this approach by assessing the interrater reliability among new users trained with minimal resources by 2 experienced trainers, 1 local and 1 international.MethodsA retrospective case series review was conducted within an online learning collaborative framework. After completion of the training, the participants from each of the 19 clinical sites were asked to submit 2 triage cases per month for blinded review by the expert trainers. The triage categories assigned by each reviewer were compared with those assigned by the newly trained Emergency Severity Index providers. A weighted kappa value was calculated to assess the degree of agreement between the sites and the expert trainers.ResultsA total of 166 cases were received over the study period. Participation in the learning collaborative was consistently below 50%. The interrater reliability between the expert trainers (κ = 0.48) as well as between each scorer and each accident and emergency department site (κ_SF = 0.33, κ_PT = 0.26) was low, although there was improvement over the study period. Incomplete triage documentation limited raters' ability to assign triage categories and assess interrater reliability.DiscussionDespite a rigorous implementation process, the interrater reliability of the Emergency Severity Index skills of Jamaican emergency nurses and doctors when compared with that of the 2 experts was poor. Several areas were identified for strengthening. Considerations for the implementation of the Emergency Severity Index in countries outside of the US were also discussed.     

Reliability and Validity of Scores on the Emergency Severity Index Version 3

OBJECTIVES: No widely used triage instrument accurately assesses patient acuity. The Emergency Severity Index (ESI) promises to facilitate reliable acuity assessment and possibly predict patient disposition. However, reliability and validity of ESI scores have not been established in emergency departments (EDs) outside the original research sites, and version 3 (v.3) of the ESI has not been evaluated. The study hypothesis was that scores on the ESI v.3 show good interrater reliability and predict hospital admission, admission site, and death. METHODS: The authors conducted an ED-based cross-sectional retrospective study of 403 systematically selected ED records of patients who presented to an academic medical center. Twenty-seven variables were abstracted, including triage level assigned, admission status, site, and death. Using a standard process, the researchers determined the true triage level. Weighted kappa and Pearson correlation were used to calculate interrater reliability between true triage level and triage score assigned by the registered nurse (RN). The relationships between the true ESI level and admission, admission site, and death were assessed. RESULTS: Interrater reliability between RN ESI level and the true ESI level was kappa = 0.89; Pearson r = 0.83 (p < 0.001). Hospital admission by ESI level was as follows: 1 (80%), 2 (73%), 3 (51%), 4 (6%), and 5 (5%). A higher percentage of ESI level-1 and level-2 patients (40%, 12%) were admitted to the intensive care unit than ESI levels 3-5 (2%, 0%, 0%). Admission to telemetry for ESI levels 1-5 was 20%, 19%, 7%, 1%, and 0%, respectively. Three of four patients who died were ESI level 1 or 2. CONCLUSIONS: Scores on the ESI assigned by nurses have excellent interrater reliability and predict hospital admission and location of admission.     

Evaluation of the Emergency Severity Index (version 3) Triage Algorithm in Pediatric Patients

Objectives: As demand for emergency services outpaces available allocated resources, emergency department (ED) triage systems face increasing scrutiny. Longer waits for care make the use of reliable, valid triage systems imperative to patient safety. Little is known about the reliability and validity of triage systems in children. The purpose of this study was to evaluate the reliability and validity of the Emergency Severity Index version 3 (ESIv.3) triage algorithm in a pediatric population. Methods: This two‐phase investigation used both retrospective chart review and prospective, observational designs. Interrater reliability was evaluated using ED triage scenarios, a prospective cohort of ED patients presenting to triage, and retrospective triage assignments using the original triage note. ED triage nurses, nurse investigators, and physician investigators performed retrospective blinded triages using only the original triage note to assess reproducibility. In the second phase, validity was assessed using a retrospective analysis of observed resource use, ED length of stay, and hospitalization compared with resource utilization estimated at triage by the ESI. Results: In the reliability phase, weighted κ for ED nurse triage of standard scenarios ranged from 0.84 to 1.00, representing excellent agreement. Twenty ED pediatric patients were triaged simultaneously by an ED triage nurse and the nurse investigator. Weighted κ was 0.82 (95% confidence interval = 0.66 to 0.98), also representing strong agreement between raters. When used for retrospective chart review, the weighted κ statistics ranged from 0.42 to 0.84, representing poor to good agreement among the different categories of reviewers. During the validity phase, 510 patients were included in the final data analysis. Hospitalization, ED length of stay, and resource utilization were strongly associated with ESIv.3 category. Conclusions: The ESI triage algorithm demonstrated reliability and validity between triage assignment and resource use in this group of ED pediatric patients.     

Improving ED Emergency Severity Index Acuity Assignment Using Machine Learning and Clinical Natural Language Processing

IntroductionTriage is critical to mitigating the effect of increased volume by determining patient acuity, need for resources, and establishing acuity-based patient prioritization. The purpose of this retrospective study was to determine whether historical EHR data can be used with clinical natural language processing and machine learning algorithms (KATE) to produce accurate ESI predictive models.MethodsThe KATE triage model was developed using 166,175 patient encounters from two participating hospitals. The model was tested against a random sample of encounters that were correctly assigned an acuity by study clinicians using the Emergency Severity Index (ESI) standard as a guide.ResultsAt the study sites, KATE predicted accurate ESI acuity assignments 75.7% of the time compared with nurses (59.8%) and the average of individual study clinicians (75.3%). KATE’s accuracy was 26.9% higher than the average nurse accuracy (P <.001). On the boundary between ESI 2 and ESI 3 acuity assignments, which relates to the risk of decompensation, KATE’s accuracy was 93.2% higher, with 80% accuracy compared with triage nurses 41.4% accuracy (P <.001).DiscussionKATE provides a triage acuity assignment more accurate than the triage nurses in this study sample. KATE operates independently of contextual factors, unaffected by the external pressures that can cause under triage and may mitigate biases that can negatively affect triage accuracy. Future research should focus on the impact of KATE providing feedback to triage nurses in real time, on mortality and morbidity, ED throughput, resource optimization, and nursing outcomes.     

Implementation and Refinement of the Emergency Severity Index

OBJECTIVES: To implement a new five-level emergency department (ED) triage algorithm, the Emergency Severity Index (ESI), into nursing practice, and validate the instrument with a population-based cohort using hospitalization and ED length of stay as outcome measures. METHODS: The five-level ESI algorithm was introduced to triage nurses at two university hospital EDs, and implemented into practice with reinforcement and change management strategies. Interrater reliability was assessed by a posttest and by a series of independent paired patient triage assignments, and a staff survey was performed. A cohort validation study of all adult patients registered during a one-month period immediately following implementation was performed. RESULTS: Eight thousand two hundred fifty-one ED patients were studied. Weighted kappa for reproducibility of triage assignments was 0.80 for the posttest (n = 62 nurses), and 0.73 for patient triages (n = 219). Hospitalization was 28% overall and was strongly associated with triage level, decreasing from 58/63 (92%) of patients in triage category 1, to 12/739 (2%) in triage category 5. Median lengths of stay were two hours shorter at either triage extreme (high and low acuity) than in intermediate categories. Outcomes followed a-priori predictions. Staff nurses rated the new program easier to use, and more useful as a triage instrument than previous three-level triage. They provided feedback, which resulted in significant revisions to the algorithm and educational materials. CONCLUSIONS: Triage nurses at these two hospitals successfully implemented the ESI algorithm and provided useful feedback for further refinement of the instrument. Emergency Severity Index triage reproducibly stratifies patients into five groups with distinct clinical outcomes.     

Refining Emergency Severity Index Triage Criteria

Objectives: The Emergency Severity Index (ESI) version 3 is a five‐level triage acuity scale with demonstrated reliability and validity. Patients are rated from ESI level 1 (highest acuity) to ESI level 5 (lowest acuity). Clinical experience has demonstrated two levels of ESI level 2 patients: those who require immediate intervention and those who are stable to wait for at least ten minutes. Studies have found that few patients are rated ESI level 1, and it has been suggested that revisions to the ESI might result in appropriate reclassification of some sickest level 2 patients as level 1. The purpose of this study was to identify level 2 patients who might be reclassified as level 1 patients. Methods: This was a multisite, prospective study. The authors identified ESI level 2 patients who required immediate, lifesaving intervention and calculated chi‐square statistics and odds ratios for variables that predicted which ESI level 2 patients actually received immediate intervention. Results: Immediate lifesaving interventions were provided for 117 (20.2%) of the 589 patients included in the study. Seventeen predictors of the need for immediate intervention were identified. The strongest predictor was the triage nurse's judgment of the need for immediate intervention, especially airway and medications. Conclusions: Specific clinical findings at triage for a subset of ESI level 2 patients were associated with immediate delivery of lifesaving interventions. Revisions to the ESI level 1 criteria may be beneficial.     

Nursing Research Priorities in the Pediatric Emergency Care Applied Research Network (PECARN): Reaching Consensus Through the Delphi Method

IntroductionPediatric emergency nurses who are directly involved in clinical care are in key positions to identify the needs and concerns of patients and their families. The 2010 Institute of Medicine report on the future of nursing supports the active participation of nurses in the design and implementation of solutions to improve health outcomes. Although prior efforts have assessed the need for research education within the Pediatric Emergency Care Applied Research Network (PECARN), no systematic efforts have assessed nursing priorities for research in the pediatric ED setting.MethodsThe Delphi technique was used to reach consensus among emergency nurses in the PECARN network regarding research priorities for pediatric emergency care. The Delphi technique uses an iterative process by offering multiple rounds of data collection. Participants had the opportunity to provide feedback during each round of data collection with the goal of reaching consensus about clinical and workforce priorities.ResultsA total of 131 nurses participated in all 3 rounds of the survey. The participants represented the majority of the PECARN sites and all 4 regions of the United States. Through consensus 10 clinical and 8 workforce priorities were identified.DiscussionThe PECARN network provided an infrastructure to gain expert consensus from nurses on the most current priories that researchers should focus their efforts and resources. The results of the study will help inform further nursing research studies (for PECARN and otherwise) that address patient care and nursing practice issues for pediatric ED patients.     

Different triage categorization using Emergency Severity Index (ESI) method in emergency department

Objective

Triage is basically a categorization process to prioritize various treatments for patients based on the types of disease, severity, prognosis and resource availability. However, the term triage is more appropriate to be used in the context of natural disaster or mass casualties. Within the context of emergency situation in emergency department, the term triage refers to a method used to assess the severity of patients’ condition, determine the level of priority, and mobilize the patients to the suitable care unit. ESI is a new concept of triage using five scales in classifying the patients in emergency department. The real implementation of this concept demands nurses have to immediately make assessment about patients’ condition right away, besides they must give their final decision, whether to move the patients to the ward or to let them leave the hospital.

The Emergency Severity Index Triage Algorithm Version 2 Is Reliable and Valid

OBJECTIVES: Initial studies have shown improved reliability and validity of a new triage tool, the Emergency Severity Index (ESI), over conventional three-level scales at two university medical centers. After pilot implementation and validation, the ESI was revised to include pediatric and updated vital signs criteria. The goal of this study was to assess ESI version (v.) 2 reliability and validity at seven emergency departments (EDs) in three states. METHODS: In part 1, interrater reliability was assessed using weighted kappa analysis of written training cases and postimplementation by a random sampling of actual patient triages. In part 2, validity was analyzed using a prospective cohort with stratified random sampling at each site. The ESI was compared with outcomes including resource consumption, inpatient admission, ED length of stay, and 60-day all-cause mortality. RESULTS: Weighted kappa analysis of interrater reliability ranged from 0.70 to 0.80 for the written scenarios (n = 3289) and 0.69 to 0.87 for patient triages (n = 386). Outcomes for the validity cohort (n = 1042) included hospitalization rates by ESI triage level: level 1, 83%; 2, 67%; 3, 42%; 4, 8%; level 5, 4%. Sixty-day all-cause mortality by triage level was as follows: level 1, 25%; 2, 4%; 3, 2%; 4, 1%; and 5, 0%. CONCLUSIONS: ESI v. 2 triage produced reliable, valid stratification of patients across seven sites. ESI triage should be evaluated as an ED casemix identification system for uniform data collection in the United States and compared with other major ED triage methods.     

Prognostic performance of Emergency Severity Index (ESI) combined with qSOFA score

Objective

We conducted this study to investigate whether ESI combined with qSOFA score (ESI + qSOFA) predicts hospital outcome better than ESI alone in the emergency department (ED).

Emergency Severity Index Triage Category Is Associated with Six-month Survival

OBJECTIVE: The Emergency Severity Index (ESI) is a new five-level triage instrument. The objective of this study was to determine whether there is an association between ESI triage status and short-term survival. METHODS: This was a survival analysis for a population-based, stratified random sample of patients over the age of 14 years who visited an urban, university-affiliated hospital emergency department (ED). Measures included ESI triage category (1 through 5), vital status obtained from the Social Security Administration, date of death (if applicable), and survival time in days. Data were analyzed with Kaplan-Meier survival analysis. RESULTS: Eighty-seven percent (202/232) of a random sample of patients appeared in the Social Security vital status registry. During the 252-day follow-up period, 19 patients (9%) died, 161 (80%) lived, and 22 (11%) had an unknown vital status. The ESI triage level was strongly associated with vital status at six months (Kaplan-Meier chi-square 25.9, p<0.0001). No patient in triage categories 4 and 5 died (lower limits of the 95% confidence interval for survival, 92% and 93%, respectively); whereas survival in triage category 1 was 68%, and in categories 2 and 3 it was 86% and 83%, respectively. Most of the deaths occurred within 60 days after the index ED visit. Sensitivity analyses biased against the instrument continued to demonstrate the association between triage status and survival. CONCLUSIONS: The ESI triage status is associated with six-month survival. Patients with the highest three triage groups experienced decreased survival during the follow-up period, whereas all patients in the two lowest triage strata survived at least six months.     

Comparison between Emergency Severity Index and Heart Failure Triage Scale in heart failure patients: A randomized clinical trial

BACKGROUND: It is not clear whether Emergency Severity Index (ESI) is valid to triage heart failure (HF) patients and if HF patients benefit more from a customized triage scale or not. The aim of study is to compare the effect of Heart Failure Triage Scale (HFTS) and ESI on mistriage among patients with HF who present to the emergency department (ED).METHODS: A randomized clinical trial was conducted from April to June 2017. HF patients with dyspnea were randomly assigned to HFTS or ESI groups. Triage level, used resources and time to electrocardiogram (ECG) were compared between both groups among HF patients who were admitted to coronary care unit (CCU), cardiac unit (CU) and discharged patients from the ED. Content validity was examined using Kappa designating agreement on relevance (K*). Reliability of both scale was evaluated using inter-observer agreement (Kappa).RESULTS: Seventy-three and 74 HF patients were assigned to HFTS and ESI groups respectively. Time to ECG in HFTS group was significantly shorter than that of ESI group (2.05 vs. 16.82 minutes). Triage level between HFTS and ESI groups was significantly different among patients admitted to CCU (1.0 vs. 2.8), cardiac unit (2.26 vs. 3.06) and discharged patients from the ED (3.53 vs. 2.86). Used resources in HFTS group were significantly different among triage levels (H=25.89; df=3; P<0.001).CONCLUSION: HFTS is associated with less mistriage than ESI for triaging HF patients. It is recommended to make use of HFTS to triage HF patients in the ED.     

3L急危重病人指引模型在急诊室的应用和效果评价

目的探讨急危重病人在急诊室的应急处理流程。方法美国北卡罗大学附属医院急诊科使用的3L急危重病人指引模型(3LevelEmergencySeveritylndexModel)做为评估工具。结果应用3L指引有效评估了342例病人,符合率88.9%。需要在1L、2L、3L规范处理的病人例数分别为20例、130例和192例,3L各层面的敏感率和特征率平均分别为84.2%和75.7%。结论3L应用简便可靠,可作为急诊室护士对危重病人快速识别的护理指引。     

Accuracy of the Emergency Severity Index Triage Instrument for Identifying Elder Emergency Department Patients Receiving an Immediate Life‐saving Intervention

Objectives: The study objective was to determine the sensitivity and specificity of the Emergency Severity Index (ESI) triage instrument for the identification of elder patients receiving an immediate life‐saving intervention in the emergency department (ED). Methods: The authors reviewed medical records for consecutive patients 65 years or older who presented to a single academic ED serving a large community of elders during a 1‐month period. ESI triage scores were compared to actual ED course with attention to the occurrence of an immediate life‐saving intervention. The sensitivity and specificity of an ESI triage level of 1 for the identification of patients receiving an immediate intervention was calculated. For 50 cases, the triage nurse ESI designation was compared to the triage level determined by an expert triage nurse based on retrospective record review. Results: Of 782 consecutive patients 65 years or older who presented to the ED, 18 (2%) had an ESI level of 1, 176 (23%) had an ESI level of 2, 461 (60%) had an ESI level of 3, 100 (13%) had an ESI level of 4, and 18 (2%) had an ESI level of 5. Twenty‐six patients received an immediate life‐saving intervention. ESI triage scores for these 26 individuals were as follows: ESI 1, 11 patients; ESI 2, nine patients; and ESI 3, six patients. The sensitivity of ESI to identify patients receiving an immediate intervention was 42.3% (95% confidence interval [CI] = 23.3% to 61.3%); the specificity was 99.2% (95% CI = 98.0% to 99.7%). For 17 of 50 cases in which actual triage nurse and expert nurse ESI levels disagreed, undertriage by the triage nurses was more common than overtriage (13 vs. 4 patients). Conclusions: The ESI triage instrument identified fewer than half of elder patients receiving an immediate life‐saving intervention. Failure to follow established ESI guidelines in the triage of elder patients may contribute to apparent undertriage. ACADEMIC EMERGENCY MEDICINE 2010; 17:238–243 © 2010 by the Society for Academic Emergency Medicine     

校正改良式早期预警系统在心胸专科急诊分诊中的应用

目的将改良式早期预警(modified early warning score,MEWS)与胸科专科预检标准相结合,构建适用于心胸专科急诊分诊的校正MEWS系统,探讨其对于心胸专科急诊预检分诊工作的影响。方法便利抽样法选取上海交通大学附属胸科医院2015年9-12月急诊就诊患者8994例为对照组,2016年1-3月急诊就诊患者9138例为观察组。对照组患者按常规的急诊分诊流程处理,观察组患者实施校正MEWS系统的评分结果进行预检及分区分级处置,比较两组患者急诊分诊时间及分诊正确率、高危胸痛患者识别率、应急处理率,医生、护士及患者满意率。结果两组患者的分诊时间、分诊正确率、有效识别高危胸痛患者、应急处理率、患者满意率经比较,观察组患者均优于对照组,差异均有统计学意义(均P0.05)。结论校正MEWS评分系统便于急诊护士快速准确地分诊,同时可有效识别胸痛高危患者,提高心胸专科急诊预检分诊工作的可操作性及准确率,有助于急诊患者在最短时间内得到规范、科学、适当、合理、及时的救治。     

Reliability and Validity of the Emergency Severity Index for Pediatric Triage

Objectives: The Emergency Severity Index (ESI) triage algorithm is a five‐level triage acuity tool used by emergency department (ED) triage nurses to rate patients from Level 1 (most acute) to Level 5 (least acute). ESI has established reliability and validity in an all‐age population, but has not been well studied for pediatric triage. This study assessed the reliability and validity of the ESI for pediatric triage at five sites. Methods: Interrater reliability was measured with weighted kappa for 40 written pediatric case scenarios and 100 actual patient triages at each of five research sites (independently rated by both a triage nurse and a research nurse). Validity was evaluated with a sample of 200 patients per site. The ESI ratings were compared with outcomes, including hospital admission, resource consumption, and ED length of stay. Results: Interrater reliability was 0.77 (95% confidence interval [CI] = 0.76 to 0.78) for the scenarios (n = 155 nurses) and 0.57 (95% CI = 0.52 to 0.62) for actual patients (n = 498 patients). Inconsistencies in triage were noted for the most acute and least acute patients, as well as those less than 1 year of age and those with medical (rather than trauma) chief complaints. For the validity cohort (n = 1,173 patients), outcomes differed by ESI level, including hospital admission, which went from 83% for Level 1 patients to 0% for Level 5 (chi‐square, p < 0.0001). Nurses from dedicated pediatric EDs were 31% less likely to undertriage patients than nurses in general EDs (odds ratio [OR] = 0.31, 95% CI = 0.14 to 0.67). Conclusions: Reliability of the ESI for pediatric triage is moderate. The ESI provides a valid stratification of pediatric patients into five distinct groups. We found several areas in which nurses have difficulty triaging pediatric patients consistently. The study results are being used to develop pediatric‐specific ESI educational materials to strengthen reliability and validity for pediatric triage.