GIST 2.0: A scalable multi-trait metric for quantifying population representativeness of individual clinical studies

The design of randomized controlled clinical studies can greatly benefit from iterative assessments of population representativeness of eligibility criteria. We propose a multi-trait metric - GIST 2.0 that can compute the a priori generalizability based on the population representativeness of a clinical study by explicitly modeling the dependencies among all eligibility criteria. We evaluate this metric on twenty clinical studies of two diseases and analyze how a study’s eligibility criteria affect its generalizability (collectively and individually). We statistically analyze the effects of trial setting, trait selection and trait summarizing technique on GIST 2.0. Finally we provide theoretical as well as empirical validations for the expected properties of GIST 2.0.     

An Investigation of the Generalizability of Medical School Grades

Construct/Background: Medical school grades are currently unstandardized, and their level of reliability is unknown. This means their usefulness for reporting on student achievement is also not well documented. This study investigates grade reliability within 1 medical school. Approach: Generalizability analyses are conducted on grades awarded. Grades from didactic and clerkship-based courses were treated as 2 levels of a fixed facet within a univariate mixed model. Grades from within the 2 levels (didactic and clerkship) were also entered in a multivariate generalizability study. Results: Grades from didactic courses were shown to produce a highly reliable mean score (G = .79) when averaged over as few as 5 courses. Although the universe score correlation between didactic and clerkship courses was high (r = .80), the clerkship courses required almost twice as many grades to reach a comparable level of reliability. When grades were converted to a Pass/Fail metric, almost all information contained in the grades was lost. Conclusions: Although it has been suggested that the imprecision of medical school grades precludes their use as a reliable indicator of student achievement, these results suggest otherwise. While it is true that a Pass/Fail system of grading provides very little information about a student's level of performance, a multi-tiered grading system was shown to be a highly reliable indicator of student achievement within the medical school. Although grades awarded during the first 2 didactic years appear to be more reliable than clerkship grades, both yield useful information about student performance within the medical college.     

A Prelude to Modeling the Expert: A Generalizability Study of Expert Ratings of Performance on Computerized Clinical Simulations

Context: The University of Iowa College of Medicine has developed a series of computer-based clinical simulations and successfully integrated them into the clinical clerkship curriculum. The computerized patient simulations provide a high degree of realism in simulating a clinical encounter. In an effort to improve the validity of our clinical skills assessment, we have initiated testing research utilizing these simulations. Because of the high costs associated with employing expert raters for performance scoring, automated scoring was deemed essential. Purpose: This study is designed to address the preliminary research questions related to utilizing the simulations for performance assessment and developing a psychometrically sound automated scoring mechanism. Specifically, it addresses issues of reliability in relation to rater and simulation characteristics, and provides essential data required for designing a sound methodology to obtain ratings for modeling. Design: The judgements of 3 expert clinician/raters, grading the responses of 69 third-year medical students, to 2 computerized simulations, are analyzed in a generalizabilty study. A random effects (persons by raters) ANOVA was performed to estimate variance components for modeling. A case facet was added to the anlaysis to provide data regarding performance assessment characteristics. Estimation of the magnitude of each variance component represents the outcome of the generalizability study. Variance estimates are used in the decision study phase of the research. Results: Only moderate levels of inter-rater reliability were obtained. Four or more raters were indicated to obtain adequate reliability. A high level of task/simulation specificity was found. Three or more simulations were indicated for performance assessment. Suggestions for improving ratings were offered. This revised version was published online in June 2006 with corrections to the Cover Date.     

An Investigation of the Sources of Measurement Error in the Post-Encounter Written Scores from Standardized Patient Examinations

Purpose. Post-encounter written exercises (e.g., patient notes) have been included in clinical skills assessments that use standardized patients. The purpose of this study was to estimate the generalizability of the scores from these written exercises when they are rated by various trained health professionals, including physicians. Method. The patient notes from a 10 station clinical skills examination involving 10 first year emergency medicine residents were analytically scored by four rater groups: three physicians, three nurses, three fourth year medical students, three billing clerks. Generalizability analyses were used to partition the various sources of error variance and derive reliability-like coefficients for each group of raters. Results. The generalizability analyses indicated that case-to-case variability was a major source of error variance in the patient note scores. The variance attributable to the rater or to the rater by examinee interaction was negligible. This finding was consistent across the four rater groups. Generalizability coefficients in excess of 0.80 were achieved for each of the four sets of raters. Physicians did, however, produce the most dependable scores. Conclusion. There is little advantage, from a reliability perspective, in using more than one trained physician, or other health professional who is adequately trained to score the patient note. Measurement error is introduced primarily by case sampling variability. This suggests that, if required, increases in the generalizability of the patient note scores can be made through the addition of cases, and not the addition of raters. This revised version was published online in July 2006 with corrections to the Cover Date.     

Implications of Eligibility Criteria on the Generalizability of Alcohol and Drug Treatment Outcome Research: A Study of Real-World Treatment Seekers in Sweden and in Australia

Background: Clinical studies of alcohol and drug treatment outcomes frequently apply participant eligibility criteria (EC), which may exclude real-world treatment seekers, impairing the representativeness of studied samples. Some research exists on the impact of EC on alcohol treatment seekers. Little is known about drug treatment and country differences. Objectives: We tested and compared the degree to which commonly used EC exclude real-world treatment seekers with problem alcohol and drug use in Sweden and Australia, and compared the impact of EC on outcomes. Methods: Two large naturalistic and comparative service user samples were used. Respondents were recruited in Stockholm County (n = 1,865; data collection 2000–2002), and Victoria and Western Australia (n = 796; in 2012–2013). Follow-up interviews were conducted after 1 year. Cross-tabulations, Chi-square (χ²) tests and logistic regressions were used. Results: Percentages of the samples excluded by individual EC ranged from 5% (lack of education/literacy) to 70% (social instability) among Swedish alcohol cases and from 2% (low alcohol problem severity) to 69% (psychiatric medication) among Australian counterparts; and from 2% (age 60+ years) to 82% (social instability) among Swedish drug cases and from 1% (age 60+ years) to 67% (psychiatric medication) among Australian counterparts. Country differences and differences across substances appeared independent of country effect. Co-morbid psychiatric medication, noncompliance, poly drug use, and low education EC caused positive 1-year outcome bias; whereas female sex and old age introduced negative outcome bias. Conclusions/Importance: Commonly used EC exclude large proportions of treatment seekers. This may impair generalizability of clinical research, and the effects of many EC differ by country and drug type.