Rumination: One Construct, Many Features in Healthy Individuals, Depressed Individuals, and Individuals with Lupus

Rumination has been associated with depression and negative health effects. Yet measures of rumination appear to index multiple constructs that may be differentially related to clinical phenomena. To clarify this literature, we explored convergence and divergence among self-report measures of rumination in 349 undergraduates, 59 depressed adults, 81 healthy adults, and 15 never-depressed adults with Systemic Lupus Erythematosis (SLE). Results suggested there are separate constructs labeled rumination with different relationships to depression. Yet, aggregate measures index a central construct. Depressed individuals ruminated more, across measures, than individuals with SLE, who ruminated more than healthy individuals; this relationship was mediated by dysphoria. Thus, administering multiple rumination measures and attending to constructs assessed by rumination measures appears important in clinical studies.     

Psychological Research with Pediatric Conditions: If We Specialize, Can We Generalize?

Considered threats to validity and generalizability of psychologicalresearch with pediatric populations and proposed ways to betterunderstand and manage these problems. Variation due to characteristicsof acute and chronic conditions (e.g., duration, severity, illnesscourse), settings and referral patterns, family environmentalcharacteristics, as well as investigators' decisions concerningcriteria and recruitment limit inferences concerning the impactof pediatric conditions on psychological development and generalizabilityof findings. These problems call for an expanded use of replication,cross-center, population-based, and theory-driven research.Scientific progress will be made by increasing dialogues andcollaboration among researchers concerning generalizabilityproblems and improving research training in epidemiologicalmethods, analysis of large-scale data sets, and meta-analysis.     

Effects of Clinical Experience and Explorer Type on Judged Crown Margin Acceptability

Purpose: This study was undertaken to simultaneously compare instrumentation type and operator characteristics in judgments of clinical acceptability of crowns exhibiting a controlled range of marginal gaps. The research was conducted in a laboratory setting and generalizability analysis was used as a statistical technique to identify the sources contributing to variation in the judgment outcome. Materials and Methods: A crown was seated on an ivorine tooth in a device that permitted continuous adjustment in intervals of 25 μm to produce known marginal gaps ranging from zero to 250 μm. Forty‐nine students and six faculty members used five types of explorers each to determine, by tactile examination, the point on the controlled increasing marginal gap where they would no longer regard the gap as clinically acceptable. Results: There were no differences across type of explorer. Operators with clinical experience had a threshold that rejected crowns at a smaller gap than did those operators without clinical experience (p= 0.007). Faculty members maintained a higher individual degree of consistency in their personal judgments than did students (p= 0.02); however, the inter‐operator consistency was significantly lower for faculty members than for students (p < 0.05). Conclusions: Differences among operators in a simulation of the decision regarding gaps in crowns accounted for 63% of the variance; type of explorer used in assisting this decision accounted for about half as much variance. Faculty members making such judgments exhibited high intra‐operator consistency but significantly lower inter‐operator consistency than did students. The study suggests that the internal standards dentists use for clinical decision making deserves further study as they may be as significant as the equipment used.     

Pain,Please: An Investigation of Sampling Bias in Pain Research

Experimental pain research frequently relies on the recruitment of volunteers. However, because experimental pain research often involves unpleasant and painful sensations, it may be especially susceptible to sampling bias. That is, volunteers in experimental pain research might differ from nonvolunteers on several relevant variables that could affect the generalizability and external validity of the research. We conducted 2 studies to investigate potential sampling bias in experimental pain research. In study 1 we assessed participants' (N?=?275; age = 17–30 years) perceived likelihood of participating in pain research. Pain catastrophizing, fear of pain, illness and injury sensitivity, depression, anxiety, sensation-seeking, gender identity, body appreciation, and social desirability were also assessed as potential predictors of the likelihood to participate. In study 2, participants (N?=?87; Age = 18–31 years) could sign up for 2 nearly identical studies, with only one involving painful sensations. Thirty-six participants signed up for the pain study and 51 participants signed up for the no-pain study. Study 1 showed that lower levels of fear of pain, higher levels of sensation-seeking, and older age predicted the perceived likelihood of participating in pain research. Study 2 showed significantly higher levels of sensation-seeking in participants who signed up for the pain study compared with those who signed up for the no-pain study. The implications of these findings for future research, as well as the clinical conclusions on the basis of experimental pain research, are discussed.

Generalizability of machine learning for classification of schizophrenia based on resting‐state functional MRI data

Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within‐site and between‐site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting‐state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within‐site generalizability of the classification framework in the main data set using cross‐validation. Then, we trained a model in the main data set and investigated between‐site generalization in the validated data set using external validation. Finally, recognizing the poor between‐site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between‐site classification performance. Cross‐validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within‐site cross‐validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.     

Item Analysis to Improve Reliability for an Internal Medicine Undergraduate OSCE

Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ‘‘problem stations’’ were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.     

Using a Longitudinal Database to Assess the Validity of Preceptors' Ratings of Clerkship Performance

PURPOSE: To examine the validity of using scores from a clinical evaluation form as an assessment of clinical competence. METHOD: Investigators collected a longitudinal clinical skills assessment database that included scores reflecting performance on standardized patient interactions, case-based learning performance, scores on multiple-choice clinical examinations, and preceptors' ratings of students during their clerkships. Pearson correlation coefficients and coefficients corrected for attenuation were calculated between the mean preceptor rating CEF score and the other measures collected during years one and two. RESULTS: Estimates from an earlier across-clerkship generalizability study of the CEF demonstrated that a mean rating computed across 28 forms and four clerkships (the average measurement frequencies observed in this study), yielded an estimated G-coefficient of 0.62 (Kreiter and Ferguson, 2001). Reliabilities for the other measures ranged from 0.23 to 0.56. Nine of the twelve clinical skill measures correlated with the CEF. For those significantly correlated measures where reliabilities could be calculated, correlations corrected for attenuation ranged from 0.46 to 0.58. CONCLUSION: This study indicates that the skills measured by the CEF are related to other clinical performance measures and, conversely, that pre-clinical measures of skills that are believed to be important during clinical years are in fact predictors of preceptors' ratings of clinical performance later in medical school. In addition, the magnitude of the disattenuated coefficients suggests that ratings on the CEF are dependent on important aspects such as clinical knowledge. This study suggests that when averaged over a large number of observations, mean CEF scores demonstrate validity coefficients large enough to support their use as part of an evaluation of students' clinical performance.     

Psychometric properties of the problem solving inventory in a group of Turkish university students

The Problem Solving Inventory (PSI; Heppner & Petersen, 1982) is a widely used self-report measure of applied problem solving in the United States. This study examined the psychometric properties of the PSI in a Turkish cultural context, specifically with regard to normative, reliability, and validity information. Subjects were 244 Turkish university students (153 women, 71 men) who completed the Turkish version of the PSI, Beck Depression Inventory, and the State-Trait Anxiety Inventory. The Turkish data provide additional psychometric support for the PSI and enhance the generalizability of some of the previous findings based on U.S. samples. Moreover, the Turkish data also suggest that the psychological construct of problem-solving appraisal may have some utility in understanding applied problem solving in the Turkish culture. Finally, the study provides cross-cultural information that increases our knowledge about the utility of problem-solving constructs across diverse environments, particularly with regard to an approach/avoidance dimension across problem-solving strategies.     

External Validity, Generalizability, and Knowledge Utilization

Purpose: To examine the concepts of external validity and generalizability, and explore strategies to strengthen generalizability of research findings, because of increasing demands for knowledge utilization in an evidence-based practice environment.
Framework: The concepts of external validity and generalizability are examined, considering theoretical aspects of external validity and conflicting demands for internal validity in research designs. Methodological approaches for controlling threats to external validity and strategies to enhance external validity and generalizability of findings are discussed.
Conclusions: Generalizability of findings is not assured even if internal validity of a research study is addressed effectively through design. Strict controls to ensure internal validity can compromise generalizability. Researchers can and should use a variety of strategies to address issues of external validity and enhance generalizability of findings. Enhanced external validity and assessment of generalizability of findings can facilitate more appropriate use of research findings.