Assessing Effort: Differentiating Performance and Symptom Validity

The current study aimed to clarify the relationship among the constructs involved in neuropsychological assessment, including cognitive performance, symptom self-report, performance validity, and symptom validity. Participants consisted of 120 consecutively evaluated individuals from a veteran’s hospital with mixed referral sources. Measures included the Wechsler Adult Intelligence Scale-Fourth Edition Full Scale IQ (WAIS-IV FSIQ), California Verbal Learning Test-Second Edition (CVLT-II), Trail Making Test Part B (TMT-B), Test of Memory Malingering (TOMM), Medical Symptom Validity Test (MSVT), WAIS-IV Reliable Digit Span (RDS), Post-traumatic Check List-Military Version (PCL-M), MMPI-2 F scale, MMPI-2 Symptom Validity Scale (FBS), MMPI-2 Response Bias Scale (RBS), and the Postconcussive Symptom Questionnaire (PCSQ). Six different models were tested using confirmatory factor analysis (CFA) to determine the factor model describing the relationships between cognitive performance, symptom self-report, performance validity, and symptom validity. The strongest and most parsimonious model was a three-factor model in which cognitive performance, performance validity, and self-reported symptoms (including both standard and symptom validity measures) were separate factors. The findings suggest failure in one validity domain does not necessarily invalidate the other domain. Thus, performance validity and symptom validity should be evaluated separately.     

Comparisons of Five Performance Validity Indices in Bona Fide and Simulated Traumatic Brain Injury

A number of performance validity tests (PVTs) are used to assess memory complaints associated with traumatic brain injury (TBI); however, few studies examine the concordance and predictive accuracy of multiple PVTs, specifically in the context of combined models in known-group designs. The present study compared five widely used PVTs: the Test of Memory Malingering (TOMM), Medical Symptom Validity Test (MSVT), Reliable Digit Span (RDS), Word Choice Test (WCT), and California Verbal Learning Test – Forced Choice (CVLT-FC). Participants were 51 adults with bona fide moderate to severe TBI and 58 demographically comparable healthy adults coached to simulate memory impairment. Classification accuracy of individual PVTs was evaluated using logistic regression and receiver operating characteristic (ROC) curves, examining both the dichotomous cutting scores as recommended by the test publishers and continuous scores for the measures. Results demonstrated nearly equivalent discrimination ability of the TOMM, MSVT, and CVLT-FC as individual predictors, all of which markedly outperformed the WCT and RDS. Models of combined PVTs were examined using Bayesian information criterion statistics, with results demonstrating that diagnostic accuracy showed only small to modest growth when the number of tests was increased beyond two. Considering the clinical and pragmatic issues in deriving a parsimonious assessment battery, these findings suggest that using the TOMM and CVLT in conjunction or the MSVT and CVLT in conjunction maximized predictive accuracy as compared to a single index or an assortment of these widely used measures.     

Analysis of the dementia profile on The Medical Symptom Validity Test

The Medical Symptom Validity Test (MSVT) was administered as part of a neuropsychological battery to a mixed clinical sample of 286 consecutively referred individuals. Of the 47% of the sample who failed in the easy subtests, 48% were considered to have the “dementia profile.” The remaining 52% of individuals failing the easy subtests were considered by the task to have “poor effort.” Comparing the neuropsychological test performance among these three groups (Pass, Dementia Profile, Poor Effort) found that on most tasks those individuals passing the easy subtests of the MSVT perform significantly better than the other two groups, which did not differ from each other. Individuals meeting criteria for the Dementia Profile performed worse on tasks of motor functioning and list learning in comparison to the Poor Effort group. The results suggest that the algorithm creating a Dementia Profile does not effectively differentiate groups of individuals who fail the easy subtests of the MSVT. Consideration of a more liberal cutoff score for the easy subtests is offered.     

The base rate of suboptimal effort in a pediatric mild TBI sample: Performance on the Medical Symptom Validity Test

Performance on the Medical Symptom Validity Test (MSVT) was examined in 193 consecutively referred patients aged 8 through 17 years who had sustained a mild traumatic brain injury. A total of 33 participants failed to meet actuarial criteria for valid effort on the MSVT. After accounting for possible false positives and false negatives, the base rate of suboptimal effort in this clinical sample was 17%. Only one MSVT failure was thought to be influenced by litigation. The present results suggest that a sizable minority of children is capable of putting forth suboptimal effort during neuropsychological exam, even when external incentives are not readily apparent. The MSVT appears to have good potential value as an objective measure for detecting symptom invalidity in school-age youth.     

PTSD and Cognitive Functioning: Importance of Including Performance Validity Testing

Many studies have observed an association between post-traumatic stress disorder (PTSD) and cognitive deficits across several domains including memory, attention, and executive functioning. The inclusion of response bias measures in these studies, however, remains largely unaddressed. The purpose of this study was to identify possible cognitive impairments correlated with PTSD in returning OEF/OIF/OND veterans after excluding individuals failing a well-validated performance validity test. Participants included 126 men and 8 women with a history of mild traumatic brain injury (TBI) referred for a comprehensive neuropsychological evaluation as part of a consortium of five Veterans Affairs hospitals. The PTSD CheckList (PCL) and Word Memory Test (WMT) were used to establish symptoms of PTSD and invalid performance, respectively. Groups were categorized as follows: Control (PCL < 50, pass WMT), PTSD-pass (PCL ≥ 50, pass WMT), and PTSD-fail (PCL ≥ 50, fail WMT). As hypothesized, failure on the WMT was associated with significantly poorer performance on almost all cognitive tests administered; however, no significant differences were detected between individuals with and without PTSD symptoms after separating out veterans failing the WMT. These findings highlight the importance of assessing respondent validity in future research examining cognitive functioning in psychiatric illness and warrant further consideration of prior studies reporting PTSD-associated cognitive deficits.     

Differentiating poor validity from probable impairment on the medical symptom validity test: a cross-validation study

Aims: In neuropsychological evaluations, it is often difficult to ascertain whether poor performance on measures of validity is due to poor effort or malingering, or whether there is genuine cognitive impairment. Dunham and Denney created an algorithm to assess this question using the Medical Symptom Validity Test (MSVT). We assessed the ability of their algorithm to detect poor validity versus probable impairment, and concordance of failure on the MSVT with other freestanding tests of performance validity.

Methods: Two previously published datasets (n?=?153 and n?=?641, respectively) from outpatient neuropsychological evaluations were used to test Dunham and Denney’s algorithm, and to assess concordance of failure rates with the Test of Memory Malingering and the forced choice measure of the California Verbal Learning Test, two commonly used performance validity tests.

Results: In both datasets, none of the four cutoff scores for failure on the MSVT (70%, 75%, 80%, or 85%) identified a poor validity group with proportionally aligned failure rates on other freestanding measures of performance validity. Additionally, the protocols with probable impairment did not differ from those with poor validity on cognitive measures.

Conclusions: Despite what appeared to be a promising approach to evaluating failure on the easy MSVT subtests when clinical data are unavailable (as recommended in the advanced interpretation program, or advanced interpretation [AI], of the MSVT), the current findings indicate the AI remains the gold standard for doing so. Future research should build on this effort to address shortcomings in measures of effort in neuropsychological evaluations.     

Evaluating the Clinical Utility of the Medical Symptom Validity Test (MSVT): A Clinical Series

Performance validity tests (PVTs) are not widely used beyond medico-legal contexts in the UK. A UK survey suggests clinicians have reservations about their accuracy in clinical settings. This study sought to explore the validity of PVTs in an acute adult neuropsychology setting and to establish a potential “false positive” (FP) base rate. Failures on the Medical Symptom Validity Test (MSVT) in a consecutive clinical series of 405 patients were evaluated systematically and allocated to groups depending on clinical context. All failures were checked against the test’s “dementia profile”. Of the 405 participants, 329 passed the MSVT (81.2%), while 76 participants (18.8%) failed based on standard criteria. A 5.2% rate of potentially ‘unexplained’ failures was found. Other reasons for failure were classified as: presumed malingered neurocognitive dysfunction (4.6%), dementia/significant cognitive impairment (3.7%), technical/visual problems (1.8%), and “unexplained failure” with contributory factors (2.4%). These results suggest test specificity between 0.95 and 0.90. Most of the clinically significantly impaired patients matched the dementia profile (86.7%). Our results support the sensitivity, but not the specificity, of the dementia profile. However, approximately 1 in 20 patients failed the MSVT despite an otherwise unremarkable neuropsychological presentation; moreover, mood and pain may affect MSVT performance. Clinical implications for interpreting test scores are discussed.     

The Effects of Coaching on the Verbal and Nonverbal Medical Symptom Validity Tests

Evaluation of resistance to coaching is an important step in the validation of symptom validity tests (SVTs) for clinical use in neuropsychological evaluations. In the present study coaching effects were evaluated for two recently developed SVTs, the Medical Symptom Validity Test (MSVT) and Nonverbal Medical Symptom Validity Test (NVMSVT) as compared with a well-validated existing SVT, the Test of Memory Malingering (TOMM). This study used a simulation design that included 103 healthy younger study volunteers who were randomly assigned into one of four conditions: Symptom Coaching, Test Coaching, Combined Coaching, or Best Effort Control. Specificity for all SVTs was excellent (96–100%). Test Coaching, either alone or combined with Symptom Coaching, was more effective than Symptom Coaching alone in producing raw scores suggestive of “better” effort for all SVTs. However, there were only modest declines in the obtained sensitivity, which remained above 80% for all SVTs. These results provide empirical support for the classification accuracy of the MSVT and NVMSVT, even when challenged with combined coaching interventions. However, further validation using known-groups designs and clinical samples is needed.     

Young Child With Severe Brain Volume Loss Easily Passes the Word Memory Test and Medical Symptom Validity Test: Implications for Mild TBI

The Word Memory Test (WMT) and Medical Symptom Validity Test (MSVT) are two commonly used free-standing measures of test-taking effort. The use of any test as a measure of effort is enhanced when evidence shows that it can be easily passed by patients with severe neurological conditions. The opportunity arose to administer the WMT and MSVT to a 9-year-old girl (referred to as CJ) with severe congenital bilateral brain tissue loss (shown via a compelling brain MRI image), chronic epilepsy, an extremely low Full Scale IQ, extremely low adaptive functioning, developmental delays, numerous severe cognitive impairments, and treatment with multiple high-dose benzodiazepines. She received extensive early intervention services and numerous academic accommodations. Despite this set of problems, CJ passed the WMT and MSVT at perfect to near perfect levels. Implications for failure on these tests among patients with known or alleged mild traumatic brain injury are discussed.     

Identification and Treatment of TBI and Co-occurring Psychiatric Symptoms Among OEF/OIF/OND Veterans Seeking Mental Health Services Within the State of Colorado: Establishing Consensus for Best Practices

This paper highlights the results of a consensus meeting regarding best practices for the assessment and treatment of co-occurring traumatic brain injury (TBI) and mental health (MH) problems among Operation Enduring Freedom/Operation Iraqi Freedom/Operation New Dawn Veterans seeking care in non-Veterans Affairs Colorado community MH settings. Twenty individuals with expertise in TBI screening, assessment, and intervention, as well as the state MH system, convened to establish and review questions and assumptions regarding care for this Veteran population. Unanimous consensus regarding best practices was achieved. Recommendations for improving care for Veterans seeking care in community MH settings are provided.     

Neuropsychological Effects of Self-Reported Deployment-Related Mild TBI and Current PTSD in OIF/OEF Veterans

Current combat veterans are exposed to many incidents that may result in mild traumatic brain injury (mTBI) and/or posttraumatic stress disorder (PTSD). While there is literature on the neuropsychological consequences of PTSD only (PTSD-o) and mTBI alone (mTBI-o), less has been done to explore their combined (mTBI+PTSD) effect. The goal of this study was to determine whether Operation Iraqi Freedom (OIF) and Operation Enduring Freedom (OEF) veterans with mTBI+PTSD have poorer cognitive and psychological outcomes than veterans with PTSD-o, mTBI-o, or combat exposure-only. The final sample included 20 OIF/OEF veterans with histories of self-reported deployment mTBI (mTBI-o), 19 with current PTSD (PTSD-o), 21 with PTSD and self-reported mTBI (mTBI+PTSD), and 21 combat controls (CC) (no PTSD and no reported mTBI). Groups were formed using structured interviews for mTBI and PTSD. All participants underwent comprehensive neuropsychological testing, including neurocognitive and psychiatric feigning tests. Results of cognitive tests revealed significant differences in performance in the mTBI+PTSD and PTSD-o groups relative to mTBI-o and CC. Consistent with previous PTSD literature, significant differences were found on executive (switching) tasks, verbal fluency, and verbal memory. Effect sizes tended to be large in both groups with PTSD. Thus, PTSD seems to be an important variable affecting neuropsychological profiles in the post-deployment time period. Consistent with literature on civilian mTBI, the current study did not find evidence that combat-related mTBI in and of itself contributes to objective cognitive impairment in the late stage of injury.     

Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury

Objectives: (1) To examine the rate of poor performance validity in a large, multicenter, prospectively accrued cohort of community dwelling persons with medically documented traumatic brain injury (TBI), (2) to identify factors associated with Word Memory Test (WMT) performance in persons with TBI. Method: This was a prospective cohort, observational study of 491 persons with medically documented TBI. Participants were administered a battery of cognitive tests, questionnaires on emotional distress and post-concussive symptoms, and a performance validity test (WMT). Additional data were collected by interview and review of medical records. Results: One hundred and seventeen participants showed poor performance validity using the standard cutoff. Variable cluster analysis was conducted as a data reduction strategy. Findings revealed that the 10 cognitive tests and questionnaires could be summarized as 4 indices of emotional distress, speed of cognitive processing, verbal memory, and verbal fluency. Regression models revealed that verbal memory, emotional distress, age, and injury severity (time to follow commands) made unique contribution to prediction of poor performance validity. Conclusions: Poor performance validity was common in a research sample of persons with medically documented TBI who were not evaluated in conjunction with litigation, compensation claims, or current report of symptoms. Poor performance validity was associated with poor performance on cognitive tests, greater emotional distress, lower injury severity, and greater age. Many participants expected to have residual deficits based on initial injury severity showed poor performance validity.     

The Relationship Between Performances on Neuropsychological Symptom Validity Testing and the MCMI-III in Patients With Fibromyalgia

Fibromyalgia is a disorder that frequently presents with both cognitive complaints and psychiatric symptoms. This study investigated the association between Symptom Validity Test (SVT) performance and psychiatric symptoms as measured by the Millon Clinical Multiaxial Inventory-III (MCMI-III), a common measure of psychopathology. A total of 72 fibromyalgia patients at a tertiary care clinic completed the MCMI-III, an embedded cognitive symptom validity test (Reliable Digit Span), and a stand-alone cognitive symptom validity test (the Word Memory Test or Test of Memory Malingering). Of these patients, 21% failed a stand-alone SVT, whereas an additional 15% failed both a stand-alone and embedded SVT. Individuals who failed both stand-alone and embedded cognitive SVTs had higher scores on a number of MCMI-III personality subscales and had elevated scores on MCMI-III modifying indices compared to individuals who passed cognitive SVTs. Moreover, SVT performance was significantly correlated with multiple MCMI-III scores, including modifying indices, as well as the somatoform, depression, and anxiety subscales. In sum, cognitive and psychological symptom validity scores were significantly related. Given the new emphasis on cognitive complaints as part of the fibromyalgia diagnostic criteria, neuropsychological evaluation of both cognitive and psychological symptom validity should be a part of a comprehensive diagnostic assessment.     

Longitudinal study of objective and subjective cognitive performance and psychological distress in OEF/OIF Veterans with and without traumatic brain injury

Objective: To describe changes in post-deployment objective and subjective cognitive performance in combat Veterans over 18 months, relative to traumatic brain injury (TBI) status and psychological distress. Method: This prospective cohort study examined 500 Veterans from Upstate New York at four time points, six months apart. TBI status was determined by a structured clinical interview. Neuropsychological instruments focused on attention, memory, and executive functions. Subjective cognitive complaints were assessed with the Neurobehavioral Symptom Inventory (NSI). A psychological distress composite included measures of post-traumatic stress disorder (PTSD), depression, and generalized anxiety. Results: Forty-four percent of the sample was found to have sustained military-related TBI, 97% of which were classified as mild (mTBI), with a mean time since injury of 41 months. Veterans with TBI endorsed moderate cognitive symptoms on the NSI. In contrast to these subjective complaints, mean cognitive test performance was within normal limits at each time point in all domains, regardless of TBI status. Multilevel models examined effects of TBI status, time, and psychological distress. Psychological distress was a strong predictor of all cognitive domains, especially the subjective domain. Substantial proportions of both TBI+ and TBI? groups remained in the clinically significant range at the initial and final assessment for all three distress measures, but the TBI+ group had higher proportions of clinically significant cases. Conclusions: Objective cognitive performance was generally within normal limits for Veterans with mTBI across all assessments. Psychological distress was elevated and significantly related to both objective and subjective cognitive performance.     

Examining the relation between posttraumatic stress disorder and suicidal ideation in an OEF/OIF veteran sample

This study examined the relation between posttraumatic stress disorder (PTSD) and suicidal ideation among U.S. military veterans deployed during Operation Enduring Freedom and/or Operation Iraqi Freedom. Specific aims included investigation of (1) whether PTSD was associated with suicidal ideation after controlling for combat exposure and history of suicide attempt(s), (2) whether PTSD was associated with suicidal ideation absent a co-occurring depressive disorder (MDD) or alcohol use disorder (AUD), (3) whether co-occurring MDD or AUD increased risk of suicidal ideation among those with PTSD and (4) whether PTSD/MDD symptom clusters were differentially associated with suicidal ideation. Results pointed to unique effects associated with prior suicide attempt(s), PTSD and MDD. PTSD-diagnosed participants with co-occurring MDD or AUD were not significantly more likely to endorse suicidal ideation than PTSD-diagnosed participants without such comorbidity. The ‘emotional numbing’ cluster of PTSD symptoms and the ‘cognitive-affective’ cluster of MDD symptoms were uniquely associated with suicidal ideation.     

Posttraumatic stress disorder in OEF/OIF veterans with and without traumatic brain injury

Veterans of Operation Enduring Freedom (OEF) and Operation Iraqi Freedom (OIF) are presenting with high rates of co-occurring posttraumatic stress disorder (PTSD) and traumatic brain injury (TBI). The purpose of this study was to compare the clinical presentations of combat-veterans with PTSD and TBI (N = 40) to those with PTSD only (N = 56). Results suggest that the groups present two distinct clinical profiles, with the PTSD + TBI group endorsing significantly higher PTSD scores, higher overall anxiety, and more functional limitations. The higher PTSD scores found for the PTSD + TBI group appeared to be due to higher symptom intensity, but not higher frequency, across PTSD clusters and symptoms. Groups did not differ on additional psychopathology or self-report of PTSD symptoms or executive functioning. Further analysis indicated PTSD severity, and not TBI, was responsible for group differences, suggesting that treatments implicated for PTSD would likely be effective for this population.     

Clinical utility of the memory complaints inventory to detect invalid test performance

Objective: This investigation was designed to examine the classification statistics of Memory Complaints Inventory (MCI) scores relative to the Medical Symptom Validity Test (MSVT) and the Non-Verbal Medical Symptom Validity Test (NV-MSVT), as well as various validity scales on the Personality Assessment Inventory (PAI) and Minnesota Multiphasic Personality Inventory-2 Restructured Form(MMPI-2-RF). Method: The sample consisted of 339 active duty service members with a history of concussion who completed performance validity tests (PVTs), symptom validity tests (SVTs), and the MCI. Results: Those who failed the MSVT and NV-MSVT had significantly higher scores across all MCI scales. In addition, those who scored above specified cut scores on the evaluated PAI and MMPI-2-RF validity scales also had significantly higher MCI scale scores. Receiver operator characteristics analysis demonstrated acceptable area under the curve (AUC) across the evaluated SVTs for the mean of all MCI subtests with values ranging from (.77 to .86), with comparable findings for PVTs (MSVT AUC = .75; NV-MSVT AUC = .72). Conclusions: In general the MCI scales demonstrated better classification statistics relative to SVTs vs. PVTs, which is consistent with the nature of the MCI as a self-report instrument.     

Victoria Symptom Validity Test Performance in a Heterogenous Clinical Sample

We retrospectively reviewed Victoria Symptom Validity Test (VSVT) in 374 patients who underwent neuropsychological assessment in an academic hospital-based practice. Patients were classified as either non-TBI clinically referred (generally patients referred from neurology, neurosurgery, or medicine), clinically referred TBI (no known external financial incentive), and non-clinical referrals (e.g., attorney-referred, Worker's Compensation). Three patients were not classified into any group and considered separately. Intentional response distortion, defined as statistically less than chance performance on hard VST items, was present in only 1/306 (0.3%) clinically referred non-TBI patients, and no clinically referred TBI patient obtained scores significantly less than chance on this measure. One additional clinically referred patient with a non-neurologic diagnosis who was subsequently found to be pursuing a disability claim also performed worse than chance. In contrast, 5/25 patients (20%) referred by attorneys or otherwise deemed a priori to be at-risk for deficit exaggeration performed less than chance. These data suggest that intentional response distortion in patients referred for non-forensic neuropsychological evaluation is rare. Performances by specific diagnosis using different classification criteria are also presented.     

The effects of coaching on the verbal and nonverbal medical symptom validity tests

Evaluation of resistance to coaching is an important step in the validation of symptom validity tests (SVTs) for clinical use in neuropsychological evaluations. In the present study coaching effects were evaluated for two recently developed SVTs, the Medical Symptom Validity Test (MSVT) and Nonverbal Medical Symptom Validity Test (NVMSVT) as compared with a well-validated existing SVT, the Test of Memory Malingering (TOMM). This study used a simulation design that included 103 healthy younger study volunteers who were randomly assigned into one of four conditions: Symptom Coaching, Test Coaching, Combined Coaching, or Best Effort Control. Specificity for all SVTs was excellent (96-100%). Test Coaching, either alone or combined with Symptom Coaching, was more effective than Symptom Coaching alone in producing raw scores suggestive of "better" effort for all SVTs. However, there were only modest declines in the obtained sensitivity, which remained above 80% for all SVTs. These results provide empirical support for the classification accuracy of the MSVT and NVMSVT, even when challenged with combined coaching interventions. However, further validation using known-groups designs and clinical samples is needed.     

Reducing the probability of false positives in malingering detection of Social Security disability claimants

The Symptom Validity Scale (SVS) for low-functioning individuals (Chafetz, Abrahams, & Kohlmaier, 2007) employs embedded indicators within the Social Security Psychological Consultative Examination (PCE) to derive a score validated for malingering against two criterion tests: Test of Memory Malingering (TOMM) and Medical Symptom Validity Test (MSVT). When any symptom validity test is used with Social Security claimants there is a known rate of mislabeling (1-specificity), essentially calling a performance biased (invalid) when it is not, also known as a false-positive error. The great costs of mislabeling an honest claimant necessitated the present study, designed to show how multiple positive findings reduce the potential for mislabeling. This study utilized a known-groups design to address the impact of using multiple embedded indicators within the SVS on the diagnostic probability of malingering. Using four SVS components, Sequence, Ganser, and Coding errors, along with Reliable Digit Span (RDS), the positive predictive power was computed directly or by the chaining of likelihood ratios. The posterior probability of malingering increased from one to two to three failed indicators. With three failed indicators, there were essentially no false positive errors, and the total SVS score was in the range consistent with Definite Malingering, as shown in Chafetz et al. (2007). Thus, in a typical PCE when an examiner might have only a few embedded indicators, more confidence in a diagnosis of malingering might be obtained with a finding of multiple failures.