False-Positive Rates of Reliable Change Indices for Concussion Test Batteries: A Monte Carlo Simulation

Background Neurocognitive testing is widely performed for the assessment of concussion. Athletic trainers can use preseason baselines with reliable change indices (RCIs) to ascertain whether concussed athletes'' cognitive abilities are below preinjury levels. Although the percentage of healthy individuals who show decline on any individual test is determined by its RCI''s confidence level (eg, 10% false-positive rate using an RCI with an 80% confidence interval), the expected rate of 1 or more significant RCIs across multiple indices is unclear.Objective To use a Monte Carlo simulation procedure to estimate the normal rate (ie, base rate) of significant decline on 1 or more RCIs in multitest batteries.

Results & Conclusion

 For batteries producing 7 or more uncorrelated RCIs (80% confidence intervals), the majority of normal individuals would show significant declines on at least 1 RCI. Expected rates are lower for tests with fewer indices, higher inter-RCI correlations, and more stringent impairment criteria. These reference points can help testers interpret RCI output for multitest batteries.Key Words: neurocognitive assessment, base rates, impairment

Key Points

• Clinicians evaluating concussed athletes often rely on the results of multiple tests or subtests to determine whether the athletes remain impaired.
• Base rates (ie, false-positive rates) of impairment are higher across multiple tests than for single tests, yet joint base rates of impairment are often not published.
• This simulation study illustrates how the properties of a test battery affect the expected base rates of impairment on 1 or more indices within the test battery.
• These data can be referenced when making decisions about how to set cut scores for determining impairment in the context of postconcussive evaluations.

Formal assessment of concussed athletes is commonplace in sports medicine, and a number of assessment tools are available to quantify symptoms, cognitive impairments, and other injury sequelae.^1–4 Computerized neurocognitive tests (CNTs) are especially popular for assessing neuropsychological abilities,^5,6 with nearly 40% of athletic trainers reporting use of a CNT in their concussion-management protocols in 2009–2010.⁵ The vast majority (85.9% to 94.7%) of athletic trainers who use CNTs perform preseason assessments so that concussed athletes'' postinjury scores can be compared with their individual premorbid estimates of ability.^5,7 The CNT programs facilitate the comparison of postinjury scores with baseline scores using output about the significance of reliable change indices (RCIs), which estimate the extent to which changes in athletes'' performance are statistically unusual after taking into account measurement error inherent to a test. In addition to CNTs, RCI cutoffs have been published for other concussion tests, including the Sport Concussion Assessment Tool 3 (SCAT3)^8,9 and paper-and-pencil tests of psychomotor speed.¹⁰Yet there is little published guidance about how to interpret RCI output for batteries with multiple indices. For any RCI, the expected false-positive rate is determined by the confidence interval (CI) applied to that RCI. For example, an RCI with a 90% confidence level should classify 5% of normal, healthy individuals as significantly declined from baseline (and, likewise, 5% as significantly improved). Similarly, 80% and 95% CIs should classify 10% and 2.5%, respectively, of normal individuals as significantly declined on average. Clinicians should select thresholds for significance according to their preferences for balancing sensitivity and specificity, with more lenient criteria expected to identify more impairment in concussed athletes (ie, increasing sensitivity) while inevitably also falsely identifying more healthy individuals as impaired (ie, diminishing specificity).Although the specificity for 1 RCI is predictable, the base rates (ie, rates of abnormal scores in the normal population) of significant decline across sets of RCIs have not been well documented. This is a problem because clinical decisions usually involve interpreting the results of multiple indices simultaneously, and the base rates of impairment for 1 or more RCIs considered together should be higher than the rate for individual RCIs. Knowing the joint probability of producing various numbers of significant indices in a set is critical to making informed clinical decisions for any test battery, as the neuropsychology literature has documented. For example, among community participants who completed a comprehensive neuropsychological assessment (comprising an average of 24 tests producing 43 scores), 71.9% of individuals showed at least 1 impaired score using a 1-standard deviation cutoff for impairment.¹¹ For a shorter neuropsychological battery (Wechsler Adult Intelligence Scale–III) with only 4 composite (index) scores, 24% of healthy individuals produced at least 1 abnormal index score (below the 10th percentile).¹² These rates depend on a number of factors, including the number of indices in the battery, their intercorrelations, and the impairment threshold for each measure. Supporting this principle in the context of concussion testing, 2 published studies on the Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) battery demonstrated that 22.2% to 46% of healthy individuals produced at least 1 significantly declined RCI out of 5.^13,14The aim of our study was to estimate the base rates of significant decline in 1 or more of a set of RCIs simulating a range of conditions that match most concussion-assessment batteries. This was achieved using a Monte Carlo simulation method that has been found to accurately estimate overall impairment base rates in other neuropsychological batteries.^11,12,15 By varying the number of indices per battery, the correlations among indices, and the criteria for significance, the relationship between these factors and test specificity was illustrated. I discuss these data in the context of the advantages and costs of applying different impairment criteria for concussion-management decisions.