Expertise in performance assessment: assessors’ perspectives

The recent rise of interest among the medical education community in individual faculty making subjective judgments about medical trainee performance appears to be directly related to the introduction of notions of integrated competency-based education and assessment for learning. Although it is known that assessor expertise plays an important role in performance assessment, the roles played by different factors remain to be unraveled. We therefore conducted an exploratory study with the aim of building a preliminary model to gain a better understanding of assessor expertise. Using a grounded theory approach, we conducted seventeen semi-structured interviews with individual faculty members who differed in professional background and assessment experience. The interviews focused on participants’ perceptions of how they arrived at judgments about student performance. The analysis resulted in three categories and three recurring themes within these categories: the categories assessor characteristics, assessors’ perceptions of the assessment tasks, and the assessment context, and the themes perceived challenges, coping strategies, and personal development. Central to understanding the key processes in performance assessment appear to be the dynamic interrelatedness of the different factors and the developmental nature of the processes. The results are supported by literature from the field of expertise development and in line with findings from social cognition research. The conceptual framework has implications for faculty development and the design of programs of assessment.     

Determinants of seafarers’ fatigue: a systematic review and quality assessment

Purpose

Fatigue jeopardizes seafarer’s health and safety. Thus, knowledge on determinants of fatigue is of great importance to facilitate its prevention. However, a systematic analysis and quality assessment of all empirical evidence specifically for fatigue are still lacking. The aim of the present article was therefore to systematically detect, analyze and assess the quality of this evidence.

Methods

Systematic searches in ten databases were performed. Searches considered articles published in scholarly journals from 1980 to April 15, 2016. Nineteen out of 98 eligible studies were included in the review. The main reason for exclusion was fatigue not being the outcome variable.

Results

Most evidence was available for work time-related factors suggesting that working nights was most fatiguing, that fatigue levels were higher toward the end of watch or shift, and that the 6-h on–6-h off watch system was the most fatiguing. Specific work demands and particularly the psychosocial work environment have received little attention, but preliminary evidence suggests that stress may be an important factor. A majority of 12 studies were evaluated as potentially having a high risk of bias.

Conclusions

Realistic countermeasures ought to be established, e.g., in terms of shared or split night shifts. As internal as well as external validity of many study findings was limited, the range of factors investigated was insufficient and few studies investigated more complex interactions between different factors, knowledge derived from studies of high methodological quality investigating different factors, including psychosocial work environments, are needed to support future preventive programs.

     

Inter-rater variability as mutual disagreement: identifying raters’ divergent points of view

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting ‘idiosyncratic rater variance’ is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters’ comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from “most consistent with my impression” to “most contrary to my impression”. Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians’ divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.     

Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment

Advances in Health Sciences Education - Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance...     

Teachers’ ability and help attributions and children's math performance and task persistence

The present longitudinal study examined the reciprocal relationships between teachers’ causal attributions and children's math performance and task persistence. In total, 760 elementary school children and their teachers participated in this study. The children were tested in math twice, at the end of the second and third grades. At both time points teachers also filled out questionnaires measuring their causal attributions and assessed children's task persistence. The results indicated that the more the teachers attributed children's math success to their own help or math failure to children's ability, the poorer the children's math performance was. Likewise, the lower the children's math performance was, the more teachers thought that their math success was due to teachers’ help or math failure was due to children's ability. In contrast, teachers’ perception of success attributions to ability was based on children's task persistence as well as math performance. Persistent behaviour in turn increased children's math performance and vice versa. The findings highlight the importance of teachers’ causal attributions in children's learning.     

Freedom of choice and health services’ performance: Evidence from a national health system

Public policies fostering the freedom of choice of provider in the healthcare sector are increasingly common in many countries and regions, where policymakers wish to empower patients and improve health service performance. However, in the literature there is not clear consensus about the impact of expanded patient choice on healthcare quality yet. This study investigates whether increasing patients' freedom of choice influences health system outcomes in terms of various non-clinical aspects of care, a dimension often overlooked by researchers in this field. Our study considers a “natural experiment” that took place within the Spanish National Health System in 2009 under which citizens of the Community (region) of Madrid were allowed to freely choose among any GP and/or specialist in their region. The empirical analysis was conducted by using Spanish microdata for the period 2002–2016 and used synthetic control estimation techniques. The key findings show the reform had a strong and long-lasting impact, reducing average waiting times and increasing patients' satisfaction with the specialist attention received. We did not detect any statistically significant impact of the reform on the other responsiveness domains analysed. Our analysis shows that freedom of choice policies could improve health system performance if they are combined with appropriate economic incentives for health providers.     

Birth registration and access to health care: an assessment of Ghana’s campaign success

Problem

Birth registration remains far from complete in many developing countries. This was true of Ghana before a major registration campaign was undertaken.

Approach

This study, based on survey data, assesses the results of a registration campaign initiated in 2004–2005 in Ghana. Key strategies included: extending the legal period for free registration of infants; incorporating registration in child health promotion weeks; training community health workers to register births; using community registration volunteers; registering children during celebrations, and piloting community population registers. This paper discusses the contribution of these strategies to the increase in registration rates and shows the degree of association between birth registration and various health-care access indicators and family characteristics.

Local setting

The Ghana Births and Deaths Registry worked together with international organizations, mainly Plan International and the United Nations Children’s Fund, to implement the birth registration campaign.

Relevant changes

Unlike many other sub-Saharan African countries, Ghana saw a substantial rise in registration rates over the campaign period. Campaign strategies improved accessibility and shortened distance to registration centres. Survey data show that the registration rate for children younger than 5 years rose from 44% in 2003 to 71% in 2008.

Lessons learnt

Incorporation of birth registration into community health care, health campaigns and mobile registration activities can reduce the indirect costs of birth registration, especially in poorer communities, and yield substantial increases in registration rates. The link between the health sector and registration activities should be strengthened further and the use of community population registers expanded.     

El Salvador's Ministry of Health, 1975–1986: a provisional performance assessment

Since it peaked in 1977, the Ministry of Health's real per capita expenditure level has dropped by nearly one-third. Central government budget reallocations to fund the costly civil war and a faltering economy have been chief causes of the decreased funding levels of the MOH. Both have been manifested in shortages of medicines, drugs and general supplies in MOH facilities, which has probebly reduced the effectiveness and utilization of their facilities. The more fundamental sources of the problems, however, have been of a longer-standing institutional nature: (1) the historical mode of organization; and (2) the resource allocation and decision-making processes within the Ministry of Health.The MOH has two, largely unrelated, health care delivery constellations. One is comprised of the health centers, units and posts; the other consists of the 14 hospitals. Saddled with two distinct and largely independent systems with very different needs, and suffering consistent erosion of its available resources, the Ministry was simultaneously confronted with the ever-increasing recurrent costs generated by the coming online of a (still) rapidly expanding, donor-funded, health infrastructure. More facilities meant that more personnel were needed. The Ministry's longestablished practice of historical budget-based resource allocation resulted in the growing percentage of recurrent costs being spent on personnel at the expense of the growing shortage of all other medical care inputs. The bittersweet role of foreign assistance is analyzed, focussing on the role of the Inter-American Development Bank, the United States Agency for International Development, and the Pan American Health Organization. Policy relevant information voids are noted, and policy recommendations offered.     

Ethics and health technology assessment: handmaiden and/or critic?

OBJECTIVES: This study examines the content and role of ethical analysis in health technology assessment (HTA) and horizon scanning publications. It proposes that ethical analysis in HTA is of at least two different types: an ethics of HTA and an ethics in HTA. METHODS: I examine the critical differences between these approaches through the examples of the analysis of genetic screening for breast cancer and home blood glucose testing in diabetes. I then argue that, although both approaches subscribe to similar views concerning HTA and ethics, they use different theoretical and methodological traditions to interpret and explain them. RESULTS AND CONCLUSIONS: I conclude by suggesting that we need the interpretive insights of both these approaches, taken together, to explain why ethics has not been able yet to contribute fully to HTA and to demonstrate the scope and complexity of ethical work in this domain.     

Qualitative analysis of MMI raters’ scorings of medical school candidates: A matter of taste?

Recent years have seen leading medical educationalists repeatedly call for a paradigm shift in the way we view, value and use subjectivity in assessment. The argument is that subjective expert raters generally bring desired quality, not just noise, to performance evaluations. While several reviews document the psychometric qualities of the Multiple Mini-Interview (MMI), we currently lack qualitative studies examining what we can learn from MMI raters’ subjectivity. The present qualitative study therefore investigates rater subjectivity or taste in MMI selection interview. Taste (Bourdieu 1984) is a practical sense, which makes it possible at a pre-reflective level to apply ‘invisible’ or ‘tacit’ categories of perception for distinguishing between good and bad. The study draws on data from explorative in-depth interviews with 12 purposefully selected MMI raters. We find that MMI raters spontaneously applied subjective criteria—their taste—enabling them to assess the candidates’ interpersonal attributes and to predict the candidates’ potential. In addition, MMI raters seemed to share a taste for certain qualities in the candidates (e.g. reflectivity, resilience, empathy, contact, alikeness, ‘the good colleague’); hence, taste may be the result of an ongoing enculturation in medical education and healthcare systems. This study suggests that taste is an inevitable condition in the assessment of students’ performance. The MMI set-up should therefore make room for MMI raters’ taste and their connoisseurship, i.e. their ability to taste, to improve the quality of their assessment of medical school candidates.     

Training in assessment and appraisal: who needs it?

The aim of this project was to develop a training programme to help consultants supervising postgraduate medical trainees to fulfil their role as Educational Supervisors. A training needs analysis was conducted which revealed that consultants did not appreciate the importance of some of their new duties. This was mainly because of the lack of clear perception of their new roles. Hence training needs to be provided.     

Analysis of physicians’ perspectives versus patients’ preferences: direct assessment and discrete choice experiments in the therapy of multiple myeloma

Background

Against the background of patient involvement, understanding patients’ preferences for treatments is crucial: Do physicians have the same or a different perception of the patients’ preferences? As there is currently no cure for patients with multiple myeloma, primary objectives of treatment are to extend survival at the best possible quality of life. In this study, physicians’ beliefs about patients’ preferences regarding the treatment of multiple myeloma (MM) were explored in a direct assessment and a discrete choice experiment (DCE), and were compared to the previously explored patients’ views. How much do physicians know about their patients’ preferences?

Methods

In a preceding study with German multiple myeloma patients, relevant attributes of an ideal multiple myeloma treatment were collected by reviewing the literature and by conducting a qualitative study with focus groups. The attributes were analyzed using both a direct measurement (16 items on a five-point Likert scale) and a DCE (eight pairs with eight characteristics). For the present study, 30 German physicians reviewed the treatment attributes from the previous study for completeness. A total of 243 physicians participated in the study (including the 30 participants in the pre-test). The direct assessment and the DCE covered four major preference dimensions that both the literature review and the focus groups revealed: Aspects of medical effectiveness (including prolonged life expectancy, effectiveness and long duration of effect), side effects, quality of life (including social, physical and emotional quality of life) and flexibility (breaks in therapy and further treatment options).

Results

In the direct measurement of patients’ preferences, physicians rated physical quality of life (specified as “reduced mobility or good mobility”), rare side effects and effectiveness aspects (duration of effect, maximal prolonged life expectancy and effectiveness) as the most important attributes from the patients’ perspective, followed by emotional quality of life (specified as “Not always think of the disease”) and therapy-free intervals. Especially further treatment options and dosage were more important to patients than physicians believed. In this case, the physicians had quite obviously underestimated the importance of these attributes from the perspective of those affected. Physicians ranked prolonged life expectancy as relatively the most important and significantly more important than all other treatment attributes. Further treatment options were the second most important attribute and significant compared to the attributes breaks in therapy and physical quality of life, whereas the patients ordered these two attributes in reverse order. Similarly, the patients gave the opposite relative importance to the next two priorities: self-application of treatment and emotional quality of life.

Conclusions

Asking patients or physicians about the multiple myeloma patients’ treatment preferences, the combination of direct assessment and DCE proves to be a valid survey technique. Over a broad range of treatment attributes, the physicians’ perceptions of preferences were very close to those of multiple myeloma patients. Both the direct assessment of importance in order to rank the patient perceptions and the DCE provide important insights into the preference structure of patients with multiple myeloma. The findings can subsequently be used as a basis for tailoring health care services for multiple myeloma patients in reference to their preferences.

     

Self-reflection and academic performance: is there a relationship?

The purposes of the present study were two-fold: first, to evaluate whether reflection journal writing was effective in promoting self-reflection and learning, and whether students become better at self-reflection if they engage continuously in reflection journal writing. To that end, the reflection journals of 690 first-year applied science students at a local polytechnic were studied by means of an automated coding procedures using software. Data was collected twice, once at the beginning and again towards the end of an academic year. Outcomes of the textual content analyses revealed that students reflected on both the process and contents of their learning: critical review of past learning experiences, learning strategies and summaries of what was learned. Correlational analyses showed weak to moderate inter-relationships between the textual categories and their classroom and knowledge acquisition test grades. Taken together, the findings suggest that self-reflection on both how and what students have learned does lead to improvements in academic performance, although to a limited extent.     

Physician-leaders and hospital performance: Is there an association?

Although it has long been conjectured that having physicians in leadership positions is valuable for hospital performance, there is no published empirical work on the hypothesis. This cross-sectional study reports the first evidence. Data were collected on the top-100 U.S. hospitals in 2009, as identified by a widely-used media-generated ranking of quality, in three specialties: Cancer, Digestive Disorders, and Heart and Heart Surgery. The personal histories of the 300 chief executive officers of these hospitals were then traced by hand. The CEOs are classified into physicians and non-physician managers. The paper finds a strong positive association between the ranked quality of a hospital and whether the CEO is a physician or not (p < 0.001). This kind of cross-sectional evidence does not establish that physician-leaders outperform professional managers, but it is consistent with such claims and suggests that this area is now an important one for systematic future research.