The Sydney Triage to Admission Risk Tool (START) to predict Emergency Department Disposition: A derivation and internal validation study using retrospective state-wide data from New South Wales,Australia

Background

Critical illness is a time-sensitive process which requires practitioners to process vast quantities of data and make decisions rapidly. We have developed a tool, the Checklist for Early Recognition and Treatment of Acute Illness (CERTAIN), aimed at enhancing care delivery in such situations. To determine the efficacy of CERTAIN and similar cognitive aids, we developed rubric for evaluating provider performance in a simulated medical resuscitation environments.

Methods

We recruited 18 clinicians with current valid ACLS certification for evaluation in three simulated medical scenarios designed to mimic typical medical decompensation events routinely experienced in clinical care. Subjects were stratified as experienced or novice based on prior critical care training. A checklist of critical actions was designed using face validity for each scenario to evaluate task completion and performance. Simulation sessions were video recorded and scored by two independent raters. Construct validity was assessed under the assumption that experienced clinicians should perform better than novice clinicians on each task. Reliability was assessed as percentage agreement, kappa statistics and Bland-Altman plots as appropriate.

Results

Eleven experts and seven novices completed evaluation. The overall agreement on common checklist item completion was 84.8 %. The overall model achieved face validity and was consistent with our construct, with experienced clinicians trending towards better performance compared to novices for accuracy and speed of task completion.

Conclusions

A standardized video assessment tool has potential to provide a valid and reliable method to assess 12 performances of clinicians facing simulated medical emergencies.

     

Low-Fidelity,In-Situ Pediatric Resuscitation Simulation Improves RN Competence and Self-Efficacy

ProblemED staff lack adequate exposure to critical pediatric patients to develop competence and confidence in resuscitation scenarios. Simulations of various designs have shown success at increasing health care staff performance and self-efficacy.MethodsWe developed a nurse-led, low-fidelity in situ simulation of a pediatric sepsis scenario. The primary goal was to improve staff adherence to resuscitation guidelines, as measured by the Clinical Performance Tool, a set of checklists designed to measure adherence to Pediatric Advanced Life Support algorithms by multidisciplinary teams during simulations. The secondary goal was to improve staff confidence, measured by the Confidence Scale, a 5-item Likert-type scale that can measure any psychomotor skill.ResultsA total of 43 RNs participated in 12 simulations over a period of 3 months. Mean Clinical Performance score improved by 74%, from 5.3 to 9.2 (P < 0.001). Mean confidence score for RNs improved by 56%, from 2.48 (standard deviation [SD] 0.83) to 3.88 (SD 0.66) (P < 0.001). Several systems issues were identified and addressed by multidisciplinary teams, such as increasing respiratory therapist response to the emergency department and updating of the Broselow cart.DiscussionIn situ low-fidelity simulations led by RNs contributed to significant improvement in adherence to resuscitation guidelines and in staff confidence. The simulation design had minimal impact on staffing and budget and enabled identification and correction of systems issues.     

Reliability and validity of a scoring instrument for clinical performance during Pediatric Advanced Life Support simulation scenarios

Aim

To assess the reliability and validity of scoring instruments designed to measure clinical performance during simulated resuscitations requiring the use of Pediatric Advanced Life Support (PALS) algorithms.

Methods

Pediatric residents were invited to participate in an educational trial involving simulated resuscitations that employ PALS algorithms. Each subject participated in a session comprised of four scenarios (asystole, dysrhythmia, respiratory arrest, shock). Video-recorded sessions were independently reviewed and scored by four raters using instruments designed to measure performance in terms of timing, sequence, and quality. Validity was assessed by two-factor analysis of variance with postgraduate year (PGY-1 versus PGY-2) as an independent variable. Reliability was assessed by calculation of overall interrater reliability (IRR) as well as a generalizability study to estimate variance components of individual measurement facets (scenarios, raters) and associated interactions.

Results

20 subjects were scored by four raters. Based on a two-factor ANOVA, PGY-2s outperformed PGY-1s (p < 0.05); significant differences in difficulty existed between the four scenarios, with dysrhythmia scores being the lowest. Overall IRR was high (0.81) and most variance could be attributed to subject (17%), scenario (13%), and the interaction between subject and scenario (52%); variance attributable to rater was minimal (1.4%).

Conclusions

The instruments assessed in this study measure clinical performance during PALS scenarios in a reliable and valid manner. Measurement error could be minimized further through the use of additional scenarios but additional raters, for a given scenario, would not improve reliability. Further studies should assess validity of measurement with respect to actual clinical performance during resuscitations.     

Evaluation of the team performance observation tool with targeted behavioral markers in simulation-based interprofessional education

Abstract

The primary aim of this study was to decrease the subjectivity of the Team Performance Observation Tool (TPOT) and determine the psychometric properties of this tool when using scenario-specific targeted behavioral markers (TBMs). We used a convenience sample of 47 physical therapy and 25 nursing students at an academic medical center who were organized in interprofessional teams of three to care for a simulated patient. The TPOT demonstrated satisfactory validity and reliability with the use of TBMs. We demonstrated significant correlations between the TPOT overall rating and two scenario-specific outcomes: (1) a negative correlation between the TPOT overall rating and the number of medical errors committed by the 24 teams (r?=??0.531, p?=?0.008) and (2) a positive correlation between the TPOT overall rating and a time-based functional outcome (r?=?0.803, p?<?0.001). We demonstrated substantial test–retest reliability (kappa?=?0.707, p?<?0.001), inter-rater reliability (kappa?=?0.730), and good internal consistency reliability (Cronbach’s α?=?0.921). The results of our study support the use of scenario-specific TBMs to enhance the reliability and validity of the TPOT for use in the evaluation of team-based simulation scenarios. Further evaluation of the TPOT with TBMs from other simulation and training contexts is warranted.     

Reliability and Validity of Survey Instruments to Measure Work-Related Fatigue in the Emergency Medical Services Setting: A Systematic Review

Abstract

Background: This study sought to systematically search the literature to identify reliable and valid survey instruments for fatigue measurement in the Emergency Medical Services (EMS) occupational setting. Methods: A systematic review study design was used and searched six databases, including one website. The research question guiding the search was developed a priori and registered with the PROSPERO database of systematic reviews: “Are there reliable and valid instruments for measuring fatigue among EMS personnel?” (2016:CRD42016040097). The primary outcome of interest was criterion-related validity. Important outcomes of interest included reliability (e.g., internal consistency), and indicators of sensitivity and specificity. Members of the research team independently screened records from the databases. Full-text articles were evaluated by adapting the Bolster and Rourke system for categorizing findings of systematic reviews, and the rated data abstracted from the body of literature as favorable, unfavorable, mixed/inconclusive, or no impact. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) methodology was used to evaluate the quality of evidence. Results: The search strategy yielded 1,257 unique records. Thirty-four unique experimental and non-experimental studies were determined relevant following full-text review. Nineteen studies reported on the reliability and/or validity of ten different fatigue survey instruments. Eighteen different studies evaluated the reliability and/or validity of four different sleepiness survey instruments. None of the retained studies reported sensitivity or specificity. Evidence quality was rated as very low across all outcomes. Conclusions: In this systematic review, limited evidence of the reliability and validity of 14 different survey instruments to assess the fatigue and/or sleepiness status of EMS personnel and related shift worker groups was identified.     

Preparedness to self-employed careers: development,validity, and reliability of an inventory for nursing students

AimsTo develop and psychometrically validate an inventory based on two scales: awareness regarding factors for determining the prices of freelance activities and knowledge regarding freelance-related norms among nursing undergraduate students in Italy.BackgroundAlthough self-employed nursing careers are strategic to enhance employability, no valid scales for assessing nursing students’ preparedness to undertake self-employed careers are available.DesignA multi-method and multi-phase design were employed. Phase one encompassed developmental tasks for generating items and phase two included a cross-sectional data collection for determining the psychometric proprieties of the developed scales and their reliability.ResultsThe final inventory encompassed two scales showing adequate validity and reliability after testing it on 882 Italian undergraduate nursing students. The first scale, factors for determining the prices of activities, is based on care complexity and logistic characteristic (two-factor structure). The second scale, knowledge regarding freelance-related characteristics, encompassed two domains: knowledge regarding administrative rules and pensions and retirement issues.ConclusionsThe developed inventory showed adequate evidence of initial validity and reliability, useful for filling the gap given by the unavailability of valid assessments for educators who pursue measuring the nursing students’ preparedness to undertake self-employed careers.     

Bringing simulation to the classroom using an unfolding video patient scenario: A quasi-experimental study to examine student satisfaction,self-confidence,and perceptions of simulation design

BackgroundTeaching strategies to promote experiential learning in the classroom are important to facilitate students' clinical reasoning abilities. Using video to simulate unfolding patient scenarios in the classroom has the potential to engage students through active learning and to enhance their ability to connect theory to practice.ObjectivesStudy objectives were to: (1) Evaluate nursing students' satisfaction and self-confidence in learning and perceptions of the design of a simulated patient scenario delivered via video in the classroom and (2) Compare findings to students' satisfaction, self-confidence, and perceptions of the design of prior high-fidelity simulation experiences in the laboratory.DesignA quasi-experimental approach was used.SettingThe study setting was a nursing classroom at a University in the southeast United States.ParticipantsConvenience sampling was utilized and 54 students in their final semester of a pre-licensure baccalaureate nursing program participated.MethodsThe intervention was an unfolding video patient scenario delivered in the classroom. Prior to the intervention, participants completed two measurement tools to collect information about their satisfaction, self-confidence, and perceptions of the design of prior high-fidelity simulations. After the intervention, these tools were administered again to collect data about the video simulation experience in the classroom.ResultsParticipants had a higher level of satisfaction (p = 0.002) and self-confidence (p < 0.001) following the classroom intervention than for prior high-fidelity simulations in the laboratory. Student perceptions of the simulation design were more favorable for the classroom intervention than for prior laboratory simulations (p < 0.001).ConclusionsUse of unfolding video patient scenarios in the classroom can promote student engagement in learning. Nurse educators should consider this teaching strategy that provides students with opportunities to connect and apply classroom content to patient care.     

The Leuven Questionnaire on Patient Knowledge of Chemotherapy (L-PaKC): Instrument development and psychometric evaluation

PurposeEvidence suggests that patients who gain knowledge perform better self-care, cope better and are more satisfied. Today, up-to-date and valid instruments for evaluating patient knowledge on chemotherapy are unavailable. Hence, our aim was to develop a valid instrument to assess patients' knowledge on chemotherapy.MethodWe performed a literature review to develop a topic list for the construct. Using a triple Delphi procedure, an expert panel of nine oncologists and oncology nurses evaluated the face and content validity of the topic list and the generated items. A preliminary psychometric evaluation of 144 patients allowed to identify and remediate items having limited applicability and item validity. A convenience sample of 440 patients was used to evaluate item statistics (item difficulty), reliability (Cronbach alpha) and construct validity (exploratory factor analysis) of the final instrument.ResultsWe developed a 20-item instrument reflecting 14 relevant themes of patient knowledge on chemotherapy in 13 questions. Twelve items cover general chemotherapy aspects, eight items cover treatment-specific knowledge. Three questions are facultative and address oral chemotherapy. Content validity was excellent (CVI = 0.78–1.00). Item difficulty ranged from 0.25 to 0.95. Internal consistency was acceptable (Cronbach's alpha = 0.67). Exploratory factor analysis defined four underlying factors: general aspects of chemotherapy treatment, negative treatment-related events, information resources and intake of oral chemotherapy.ConclusionsThe Leuven Questionnaire on Patient Knowledge of Chemotherapy (L-PaKC) demonstrated good content validity and psychometric properties, permitting application in both research and practice for evaluating patient knowledge on chemotherapy.     

Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills

Background. Emergency medical services (EMS) providers infrequently encounter seriously ill and injured pediatric patients. Clinical simulations are useful for assessing skill level, especially for low-frequency, high-risk problems. Objective. To identify the most common performance deficiencies in paramedics' management of three simulated pediatric emergencies. Methods. Paramedics from five EMS agencies in Michigan were eligible subjects for this prospective, observational study. Three clinical assessment modules (CAMs) were designed and validated using pediatric simulators with varying technologic complexity. Scenarios included an infant cardiopulmonary arrest, sepsis/seizure, and child asthma/respiratory arrest. Each scenario required paramedics to perform an assessment and provide appropriate pediatric patient care within a 12-minute time limit. Trained instructors conducted the simulations by following strict guidelines for sequences of events and responses. Videos of CAMs were reviewed by an independent evaluator to verify scoring accuracy. Percentage of steps completed for each of the three scenarios and specific performance deficiencies were recorded. Results. Two hundred twelve paramedics completed the CAMs. The average percentages of steps completed were as follows: arrest CAM, 45.3%; asthma CAM, 51.6%; and sepsis CAM, 47.1%. Performance deficiencies included lack of airway support or protection; lack of support of ventilations or cardiac function; inappropriate use of length-based treatment tapes; and inaccurate calculation and administration of medications and fluids. Conclusion. Multiple deficiencies in paramedics' performance of pediatric resuscitation skills were objectively identified using three manikin-based simulations. EMS educators and EMS medical directors should target these specific skill deficiencies when developing continuing education in prehospital pediatric patient care.     

Assessment of Simulated Emergency Scenarios: Are Trained Observers Necessary?

Objectives: Simulation-based medical training is associated with superior educational outcomes and improved cost efficiency. Self- and peer-assessment may be a cost-effective and flexible alternative to expert-led assessment. We compared accuracy of self- and peer-assessment of untrained raters using basic evaluation tools to expert assessment using advanced validation tools including validated questionnaires and post hoc video-based analysis. Methods: Twenty-eight simulated emergency airway management scenarios were observed and video-recorded for further assessment. Participants consisted of 28 emergency physicians who were involved in four different airway management scenarios with different roles: One scenario as a team leader, one as an assisting team member, and two as an observer. Non-technical skills (NTS) and technical skills (TS) were analyzed by three independent groups: 1) the performing team (PT) consisted of the two emergency physicians acting either in the role of team leader or team member (self-assessment); 2) the observing team (OT), consisted of two of the participating emergency physicians not involved in the current clinical scenario (peer-assessment) and assessment occurred during (OT) or directly after (PT) the simulation without prior specific interpretational training but using standardized questionnaires; and 3) the expert team (ET) consisted of two specifically trained external observers (one psychologist and one emergency physician) using video-assisted objective assessment combined with standardized questionnaires. Results: Intragroup reliability demonstrated by intra-class correlation (ICC) was moderate to good for TS (ICC 0.42^*) and NTS (ICC 0.55^*) in PT and moderate to good for TS (ICC 0.41^*) or poor for NTS (ICC 0.27) in OT. ET showed an excellent intragroup reliability for both TS (ICC 0.78^*) and NTS (ICC 0.81^*). Interrater reliability was significantly different between ET and PT and between ET and OT for both TS and NTS. There was no difference between OT and PT for neither TS nor NTS; ^*p < 0.05. Conclusions: Expert assessment of simulation-based medical training scenarios using validated checklists and performance of post hoc video-based analysis was superior to self- or peer-assessment of untrained observers for both TS and NTS.     

Reliability and validity of the Lower Limb Function Questionnaire when completed by young adult orthotic and prosthetic device users

Purpose The Lower Limb Function Questionnaire (LLFQ) was developed as a self-report assessment of lower-limb functional ability for orthotic and prosthetic (O&;P) device users to be suitable for a wide range of conditions, cultures, and ages. The measure aims to address an existing gap in tools for the assessment of functional ability in this population. The purpose of this study is to evaluate LLFQ reliability and validity in a sample of young adult O&;P users. Methods Adolescents from a secondary school in Kenya completed the LLFQ twice, 6 d apart, and test–retest reliability was assessed using intra-class correlation coefficients. Validity evaluations involved Timed Up-and-Go, 6-min walk, 6-min obstacle course, and/or spatiotemporal gait assessments. Oxygen consumption was measured during walk tests. Associations between the LLFQ and each measure were evaluated using Pearson correlation coefficients for construct validity.

Results LLFQ reliability was acceptable (ICC?=?0.79, 95% CIs 0.64–0.89). Construct validity was demonstrated via moderate correlation (r > 0.60) with obstacle course distance, gait velocity, stride length, and stance/single support/double support percent of gait cycle. Conclusions Both LLFQ reliability and validity were acceptable in the sample of youth in Kenya. Further testing is required to determine applicability in other cultural contexts.

• Implications for Rehabilitation

• The LLFQ may be clinically useful across a variety of cultures and conditions to provide feedback on the effectiveness of rehabilitative treatment or assistive devices for youth with lower limb impairments.

• The LLFQ may enable specific strengths and challenges to lower limb function to be identified to enable planning of well-targeted rehabilitation.

     

Impediments to and impact of checklists on performance of emergency interventions in primary care: an in situ simulation-based randomized controlled trial

ObjectiveMedical crises occur rather seldom in the primary care setting, but when they do, initial management impacts on morbidity and mortality. Factors that impede the performance of emergency interventions in primary care have not been studied through in-situ simulation. Checklists reportedly improve crisis management.DesignThis randomized controlled trial evaluated emergency intervention performance during two scenarios (hypoglycemia-coma and anaphylaxis-cardiac arrest) simulated at primary care centers, and whether checklist access improved performance.SettingTwenty-two primary care centers in Southern Sweden participated in the study.SubjectsA total of 347 personnel performed 100 simulations, 45 with and 55 without checklist access.Main outcome measuresTime and impediments to performance of five emergency interventions in each scenario.ResultsOn 28 of the 37 occasions when the adrenalin auto-injector was employed, the administration technique was incorrect. In 9 of 49 scenarios, teams had trouble locating the 30% glucose solution. Median time to supplemental oxygen administration during the first scenario was 186 s compared with 96 s during the second scenario (p < 0.001). Checklist access had no significant impact on time to performance of emergency interventions, aside from shorter time to adequate glucose or glucagon administration (median times 632 s with, 756 s without checklist access; p = 0.03).ConclusionUnfamiliarity with local emergency equipment impedes the performance of emergency interventions during crises simulated in the primary care setting. Simply providing checklist access does not improve the performance of emergency interventions.

KEY POINTS

• Little is known about the factors that affect the performance of emergency interventions in the primary care setting.
• Unfamiliarity with local emergency equipment impedes the performance of emergency interventions during crises simulated in the primary care setting.
• Simply providing crisis checklist access does not improve the performance of emergency interventions in the primary care setting.

     

Comparison of Errors Using Two Length-Based Tape Systems for Prehospital Care in Children

Background: The use of a length/weight-based tape (LBT) for equipment size and drug dosing for pediatric patients is recommended in a joint statement by multiple national organizations. A new system, known as Handtevy?, allows for rapid determination of critical drug doses without performing calculations. Objective: To compare two LBT systems for dosing errors and time to medication administration in simulated prehospital scenarios. Methods: This was a prospective randomized trial comparing the Broselow Pediatric Emergency Tape? (Broselow) and Handtevy LBT? (Handtevy). Paramedics performed 2 pediatric simulations: cardiac arrest with epinephrine administration and hypoglycemia mandating dextrose. Each scenario was repeated utilizing both systems with a 1-year-old and 5-year-old size manikin. Facilitators recorded identified errors and time points of critical actions including time to medication. Results: We enrolled 80 paramedics, performing 320 simulations. For Dextrose, there were significantly more errors with Broselow (63.8%) compared to Handtevy (13.8%) and time to administration was longer with the Broselow system (220 seconds vs. 173 seconds). For epinephrine, the LBTs were similar in overall error rate (Broselow 21.3% vs. Handtevy 16.3%) and time to administration (89 vs. 91 seconds). Cognitive errors were more frequent when using the Broselow compared to Handtevy, particularly with dextrose administration. The frequency of procedural errors was similar between the two LBT systems. Conclusion: In simulated prehospital scenarios, use of the Handtevy LBT system resulted in fewer errors for dextrose administration compared to the Broselow LBT, with similar time to administration and accuracy of epinephrine administration.     

Professional Assessment Tool for Team Improvement: An assessment tool for paediatric intensive care unit nurses' technical and nontechnical skills

BackgroundCardiorespiratory arrests are rare in paediatric intensive care units, yet intensive care nurses must be able to initiate resuscitation before medical assistance is available. For resuscitation to be successful, instant decision-making, team communication, and the coordinating role of the first responsible nurse are crucial. In-house resuscitation training for nurses includes technical and nontechnical skills.ObjectivesThe aim of this study was to develop a valid, reliable, and feasible assessment instrument, called the Professional Assessment Tool for Team Improvement, for the first responsible nurse's technical and nontechnical skills.MethodsInstrument development followed the COnsensus-based Standards for the selection of health Measurement Instruments guidelines and professionals' expertise. To establish content validity, experts reached consensus via group discussions about the content and the operationalisation of this team role. The instrument was tested using two resuscitation assessment scenarios. Inter-rater reliability was established by assessing 71 nurses in live scenario sessions and videotaped sessions, using intraclass correlation coefficients and Cohen's kappa. Internal consistency for the total instrument was established using Cronbach's alpha. Construct validity was assessed by examining the associations between raters' assessments and nurses' self-assessment scores.ResultsThe final instrument included 12 items, divided into four categories: Team role, Teamwork and communication, Technical skills, and Reporting. Intraclass correlation coefficients were good in both live and videotaped sessions (0.78–0.87). Cronbach's alpha was stable around 0.84. Feasibility was approved (assessment time reduced by >30%).ConclusionsThe Professional Assessment Tool for Team Improvement appears to be a promising valid and reliable instrument to assess both technical and nontechnical skills of the first responsible paediatric intensive care unit nurse. The ability of the instrument to detect change over time (i.e., improvement of skills after training) needs to be established.     

A smartphone oximeter with a fingertip probe for use during exercise training: usability,validity and reliability in individuals with chronic lung disease and healthy controls

Background and aimTelehealth is a strategy to expand the reach of pulmonary rehabilitation (PR). Smartphones can monitor and transmit oxygen saturation (SpO₂) and heart rate (HR) data to ensure patient safety during home-based or other exercise. The purpose of this study was to evaluate the usability, validity and reliability of a Kenek O₂ pulse oximeter and custom prototype smartphone application (smartphone oximeter) during rest and exercise in healthy participants and those with chronic lung disease.MethodsFifteen individuals with chronic lung disease and 15 healthy controls were recruited. SpO₂ and HR were evaluated at rest and during cycling and walking. SpO2 was valid if the mean bias was within +± 2%, the level of agreement (LoA) was within ± 4%; HR was valid if the mean bias was within ± 5 beats per min (bpm), LoA was within ± 10 bpm. Usability was assessed with a questionnaire and direct observation.ResultsThe smartphone oximeter was deemed easy to use. At rest, SpO₂ measures were valid in both groups (bias <2%, lower bound LoA ?2 to 3%). During exercise, SpO₂ measurement did not meet validity and reliability thresholds in the patients with chronic lung disease, but was accurate for the healthy controls. HR recording during exercise or rest was not valid (LoA > 10 bpm) in either group.ConclusionsThe smartphone oximeter did not record HR or SpO₂ accurately in patients with chronic lung disease during exercise, although SpO₂ was valid at rest. During exercise, patients with chronic lung disease should pause to ensure greatest accuracy of SpO₂ and HR measurement.     

What can be improved in learning to care for people with autism? A qualitative study based on clinical nursing simulation

Aim/objectiveThe aim of this study was to identify the main patterns of errors that 4th year nursing students made in simulated clinical practice with scenarios of care for people with Autism Spectrum Disorder (ASD).BackgroundClinical simulation currently plays a major role in the training of nursing students and provides the participants with the opportunity to practice and develop their clinical sills with a pediatric patient diagnosed with ASD.DesignA retrospective longitudinal qualitative study was performed.MethodsContent analysis of the existing debriefing records from a period of 7 academic years (2016–2022) was carried out. The scenario was simulated by a standardized patient diagnosed with ASD, with 23 groups of nursing students.ResultsThe results showed different patterns of errors. These patterns were grouped into 1 main category (weaknesses) and 5 major subcategories: clinical, communication, knowledge about ASD, emotions, and behavior towards parents. The most repeated errors were excessive use of verbal communication, abundant stimuli, low demand for information from primary caregivers, low demand for information about the child's emotions and interests, and a lack of knowledge of the profile of the child with ASD.ConclusionFrom the findings of this study, we can highlight the lack of training that students received on the practical approach for providing care to these individuals. It has also been inferred in this study that clinical simulation is a tool that favors reflection and experiential learning for students when they are faced with caring for people with ASD.