Faculty and resident evaluations of medical students on a surgery clerkship correlate poorly with standardized exam scores

Background

The clinical knowledge of medical students on a surgery clerkship is routinely assessed via subjective evaluations from faculty members and residents. Interpretation of these ratings should ideally be valid and reliable. However, prior literature has questioned the correlation between subjective and objective components when assessing students' clinical knowledge.

Methods

Retrospective cross-sectional data were collected from medical student records at The Johns Hopkins University School of Medicine from July 2009 through June 2011. Surgical faculty members and residents rated students' clinical knowledge on a 5-point, Likert-type scale. Interrater reliability was assessed using intraclass correlation coefficients for students with ≥4 attending surgeon evaluations (n = 216) and ≥4 resident evaluations (n = 207). Convergent validity was assessed by correlating average evaluation ratings with scores on the National Board of Medical Examiners (NBME) clinical subject examination for surgery. Average resident and attending surgeon ratings were also compared by NBME quartile using analysis of variance.

Results

There were high degrees of reliability for resident ratings (intraclass correlation coefficient, .81) and attending surgeon ratings (intraclass correlation coefficient, .76). Resident and attending surgeon ratings shared a moderate degree of variance (19%). However, average resident ratings and average attending surgeon ratings shared a small degree of variance with NBME surgery examination scores (ρ² ≤ .09). When ratings were compared among NBME quartile groups, the only significant difference was for residents' ratings of students with the lower 25th percentile of scores compared with the top 25th percentile of scores (P = .007).

Conclusions

Although high interrater reliability suggests that attending surgeons and residents rate students with consistency, the lack of convergent validity suggests that these ratings may not be reflective of actual clinical knowledge. Both faculty members and residents may benefit from training in knowledge assessment, which will likely increase opportunities to recognize deficiencies and make student evaluation a more valuable tool.     

Maternal parenting style and adjustment in adolescents with type I diabetes

OBJECTIVE: To investigate the cross-sectional relationship between maternal parenting style and indicators of well-being among adolescents with diabetes. METHODS: Seventy-eight adolescents (ages 11.58-17.42 years, M = 14.21) with type 1 diabetes and their mothers separately reported perceptions of maternal parenting style. Adolescents reported their own depressed mood, self-efficacy for managing diabetes, and diabetes regimen adherence. RESULTS: Adolescents' perceptions of maternal psychological control were associated with greater depressed mood regardless of age and gender. Firm control was strongly associated with greater depressed mood and poorer self-efficacy among older adolescents, less strongly among younger adolescents. Adolescents' perceptions of maternal acceptance were associated with less depressed mood, particularly for girls and with better self-efficacy for diabetes management, particularly for older adolescents and girls. Maternal reports of acceptance were associated only with adherence. CONCLUSIONS: Maternal parenting style is associated with well-being in adolescents with diabetes, but this association is complex and moderated by age and gender.     

Validity of faculty and resident global assessment of medical students' clinical knowledge during their pediatrics clerkship

ObjectiveMedical knowledge is one of six core competencies in medicine. Medical student assessments should be valid and reliable. We assessed the relationship between faculty and resident global assessment of pediatric medical student knowledge and performance on a standardized test in medical knowledge.MethodsRetrospective cross-sectional study of medical students on a pediatric clerkship in academic year 2008–2009 at one academic health center. Faculty and residents rated students’ clinical knowledge on a 5-point Likert scale. The inter-rater reliability of clinical knowledge ratings was assessed by calculating the intra-class correlation coefficient (ICC) for residents’ ratings, faculty ratings, and both rating types combined. Convergent validity between clinical knowledge ratings and scores on the National Board of Medical Examiners (NBME) clinical subject examination in pediatrics was assessed with Pearson product moment correlation correction and the coefficient of the determination.ResultsThere was moderate agreement for global clinical knowledge ratings by faculty and moderate agreement for ratings by residents. The agreement was also moderate when faculty and resident ratings were combined. Global ratings of clinical knowledge had high convergent validity with pediatric examination scores when students were rated by both residents and faculty.ConclusionsOur findings provide evidence for convergent validity of global assessment of medical students’ clinical knowledge with NBME subject examination scores in pediatrics.     

Perceived Diabetes Task Competence Mediates the Relationship of both Negative and Positive Affect with Blood Glucose in Adolescents with Type 1 Diabetes

Background  Adolescents dealing with type 1 diabetes experience disruptions in affect and diabetes management that may influence their blood glucose. Purpose  A daily diary format examined whether daily fluctuations in both negative and positive affect were associated with adolescents’ perceived diabetes task competence (DTC) and blood glucose, and whether perceived DTC mediated the relationship between daily affect and blood glucose. Methods  Sixty-two adolescents with type 1 diabetes completed a 2-week daily diary, which included daily measures of affect and perceived DTC, then recorded their blood glucose readings at the end of the day. We utilized hierarchical linear modeling to examine whether daily perceived DTC mediated the relationship between daily emotion and blood glucose. Results  Daily perceived DTC mediated the relationship of both negative and positive affect with daily blood glucose. Conclusions  This study suggests that within the ongoing process of self-regulation, daily affect may be associated with blood glucose by influencing adolescents’ perception of competence on daily diabetes tasks.     

Decoding moral judgments from neural representations of intentions

Intentional harms are typically judged to be morally worse than accidental harms. Distinguishing between intentional harms and accidents depends on the capacity for mental state reasoning (i.e., reasoning about beliefs and intentions), which is supported by a group of brain regions including the right temporo-parietal junction (RTPJ). Prior research has found that interfering with activity in RTPJ can impair mental state reasoning for moral judgment and that high-functioning individuals with autism spectrum disorders make moral judgments based less on intent information than neurotypical participants. Three experiments, using multivoxel pattern analysis, find that (i) in neurotypical adults, the RTPJ shows reliable and distinct spatial patterns of responses across voxels for intentional vs. accidental harms, and (ii) individual differences in this neural pattern predict differences in participants’ moral judgments. These effects are specific to RTPJ. By contrast, (iii) this distinction was absent in adults with autism spectrum disorders. We conclude that multivoxel pattern analysis can detect features of mental state representations (e.g., intent), and that the corresponding neural patterns are behaviorally and clinically relevant.