Differential use of the CAHPS® 0–10 global rating scale by medicaid and commercial populations

The primary objective of this study was to investigate whether Medicaid managed care enrollees and commercially insured health plan participants respond differently to the CAHPS^® 2.0 health plan survey global ratings of health care, personal doctor or nurse, and health plan. A secondary objective was to examine whether and how these differences may vary by alternative approaches to collapsing the 0-10 response scale. This study is a secondary analysis of CAHPS 2.0 health plan survey data collected in 1999 and 2000. Data on 2,142 Iowa Medicaid managed care enrollees and 1,051 commercially insured State of Iowa employees were analyzed. Differences in responses between the Medicaid-enrolled and commercially insured respondents were modeled using multinomial logistic regression, adjusting for demographics, health status and CAHPS composite measures. Results of these analyses indicated that Medicaid enrollees were significantly more likely than State of Iowa employees to use the extreme ends of the CAHPS global rating scales, particularly in the approaches when the category representing the highest end of the scale was defined as a score of 10 for the analysis. Thus, the choice of cut points for collapsing the 0-10 scales influenced statistical differences on CAHPS global ratings of care, doctor and health plan between Medicaid and privately insured populations. In conclusion, a populations use of the extremes of the global rating scales should be considered when comparing or combining CAHPS data for different populations. If response contraction bias is present, a format such as the alternative approach presented here (using categories 0-4, 5-8, 9, 10) that captures that bias may be preferable to the CAHPS format, which has been shown to maximize plan differentiation.     

PRO development: rigorous qualitative research as the crucial foundation

Recently published articles have described criteria to assess qualitative research in the health field in general, but very few articles have delineated qualitative methods to be used in the development of Patient-Reported Outcomes (PROs). In fact, how PROs are developed with subject input through focus groups and interviews has been given relatively short shrift in the PRO literature when compared to the plethora of quantitative articles on the psychometric properties of PROs. If documented at all, most PRO validation articles give little for the reader to evaluate the content validity of the measures and the credibility and trustworthiness of the methods used to develop them. Increasingly, however, scientists and authorities want to be assured that PRO items and scales have meaning and relevance to subjects. This article was developed by an international, interdisciplinary group of psychologists, psychometricians, regulatory experts, a physician, and a sociologist. It presents rigorous and appropriate qualitative research methods for developing PROs with content validity. The approach described combines an overarching phenomenological theoretical framework with grounded theory data collection and analysis methods to yield PRO items and scales that have content validity.     

Interpretation of response categories in patient-reported rating scales: a controlled study among people with Parkinson's disease

Background  

Unambiguous interpretation of ordered rating scale response categories requires distinct meanings of category labels. Also, summation of item responses into total scores assumes equal intervals between categories. While studies have identified problems with rating scale response category functioning there is a paucity of empirical studies regarding how respondents interpret response categories. We investigated the interpretation of commonly used rating scale response categories and attempted to identify distinct and roughly equally spaced response categories for patient-reported rating scales in Parkinson's disease (PD) and age-matched control subjects.     

Health-Related Quality of Life and Other Patient-Reported Outcomes in the European Centralized Drug Regulatory Process: A Review of Guidance Documents and Performed Authorizations of Medicinal Products 1995 to 2003

OBJECTIVES: The objective of this study was to review and analyze the use of health-related quality of life (HRQL) and other patient-reported outcome (PRO) evaluations for the approval of new pharmaceutical products by the European Medicines Agency (EMEA). METHODS: All published EMEA guidance documents and regulatory information for products authorized at the EMEA and appearing in the European Public Assessment Report (EPAR) database between 1995 and 2003 were examined for reference to HRQL and other PROs. RESULTS: More than half of the guidance documents for clinical investigation of pharmaceutical products in specific disease areas included reference to HRQL or other PROs. Guidance notes for 10 conditions indicated PROs can serve as primary endpoints in clinical trials, among which three included HRQL outcomes. The review of EPAR documentation uncovered HRQL and other PRO data for 34% of the drugs registered during the period of the review, with cancer-related treatments most frequently including PRO data. There was a trend toward increasing HRQL and other PRO claims in regulatory documents of pharmaceutical products in recent years, with the proportion exceeding 30% from 1999 to 2003. CONCLUSIONS: There is further scope for health outcomes researchers and regulatory decision-makers to contribute to the more efficient utilization of PROs and HRQL outcomes. Health researchers need to better justify the inclusion of these outcomes in clinical trials and highlight the added value of PRO data; while the regulators should develop harmonized procedures and capacities to adequately appraise the submitted information.     

Issues in the design of Internet-based systems for collecting patient-reported outcomes

Background and objectives Although there is a growing interest in using patient-reported outcomes (PRO) to monitor disease progression and/or therapeutic response, to improve care, and to screen for physical or psychosocial problems in routine clinical practice, PRO instruments can be difficult to administer, score, and interpret in this setting. Internet-based approaches to PRO collection may help overcome these obstacles. This paper discusses the rationale for using the Internet for routine PRO collection, summarizes relevant literature and ongoing projects, and raises several key design and development issues that should guide further efforts in this area. Major findings A small number of Internet-based PRO collection applications have been or are currently being developed. The major characteristics of several of these projects are reviewed and summarized. Successful Internet-based PRO collection applications must address patient and clinician-specific needs related to workflow and to the way in which results are presented. A growing number of instruments have been adapted for and evaluated in a web-based format. Conclusions Collecting PROs via the Internet has the potential to overcome many of the challenges associated with efforts to routinely use PROs in the clinical encounter. James B. Jones, Claire F. Snyder, and Albert W. Wu for the “Website for Outpatient QOL Assessment” Research Network.     

Using patient-reported outcomes in clinical practice: challenges and opportunities

Purpose

Introduce and explore issues at an international conference about the use of patient-reported outcomes (PROs) in clinical practice.

Methods

Review of salient literature, clinical and personal experiences, conference presentations and discussions, and post-conference comments from outside experts.

Results

PROs (information from patients about a health condition and its management) have been assessed through self-reports for at least four decades. Traditional applications are in clinical and health services research. Uses in clinical practice, although increasing, are less common and more challenging. PROs can enhance the understanding of patients’ experiences and responses to therapy and inform clinical practice.

Conclusions

We pose and discuss four main questions: (1) Will clinicians accept PRO measures? (2) Will clinicians use PRO measures? (3) Will measuring PROs actually improve those outcomes? (4) Will PROs be perceived as having other, less salutary purposes? A patient-centered perspective on PRO measurement presents issues about the extent to which PROs can accurately capture patient experiences and assess psychosocial and environmental factors that influence communication with clinicians and eventual outcomes. We end with comments about the intersection of PROs and bioethics, noting contributions that PROs may make to beneficence, patient autonomy, and distributive justice.     

Measurement Qualities of a Self-Report and Therapist-Scored Functional Capacity Instrument Based on the Dictionary of Occupational Titles

Studies provide convincing arguments to support the development of functional capacity instruments based on the Dictionary of Occupational Titles (DOT). The purpose of this study is to investigate the item-level measurement qualities of a newly developed DOT-based functional capacity instrument for clients undergoing rehabilitation treatment for back pain. Client and therapist ratings were collected on 124 clients from 27 rehabilitation sites using the newly developed Occupational Rehabilitation Data Base (ORDB) functional capacity instrument. Rasch analysis was used to investigate: (1) unidimensionality, (2) hierarchical item difficulty continuum, (3) rater severity, and 4) person-item match. Overall, the functional capacity scale of the ORDB showed good measurement qualities. All items, except the Handling item fit the Rasch measurement model. Because of high fit statistics and loading on factors independent from the remainder of the items, the "handling" item was removed, from further analyses. Separate client-rated and therapist-rated instruments retained good item-level psychometrics. While client and therapist items showed similar item-difficulty hierarchical structures, clients had a tendency to be more severe in their rating and the correlation between client and therapist ratings was relatively low, 0.32. These findings suggest that Handling items should not be included as a DOT measure for clients with back pain. While the above psychometric study supports using client or therapist ratings as independent instruments, the lack of concordance between these ratings requires further investigation.     

Quality control of an OSCE using generalizability theory and many-faceted Rasch measurement

An Objective Structured Clinical Examination (OSCE) is an effective method for evaluating competencies. However, scores obtained from an OSCE are vulnerable to many potential measurement errors that cases, items, or standardized patients (SPs) can introduce. Monitoring these sources of errors is an important quality control mechanism to ensure valid interpretations of the scores. We describe how one can use generalizability theory (GT) and many-faceted Rasch measurement (MFRM) approaches in quality control monitoring of an OSCE. We examined the communication skills OSCE of 79 residents from one Midwestern university in the United States. Each resident performed six communication tasks with SPs, who rated the performance of each resident using 18 5-category rating scale items. We analyzed their ratings with generalizability and MFRM studies. The generalizability study revealed that the largest source of error variance besides the residual error variance was SPs/cases. The MFRM study identified specific SPs/cases and items that introduced measurement errors and suggested the nature of the errors. SPs/cases were significantly different in their levels of severity/difficulty. Two SPs gave inconsistent ratings, which suggested problems related to the ways they portrayed the case, their understanding of the rating scale, and/or the case content. SPs interpreted two of the items inconsistently, and the rating scales for two items did not function as 5-category scales. We concluded that generalizability and MFRM analyses provided useful complementary information for monitoring and improving the quality of an OSCE.     

Evaluating the effectiveness of rating instruments for a communication skills assessment of medical residents

The investigators used evidence based on response processes to evaluate and improve the validity of scores on the Patient-Centered Communication and Interpersonal Skills (CIS) Scale for the assessment of residents’ communication competence. The investigators retrospectively analyzed the communication skills ratings of 68 residents at the University of Illinois at Chicago (UIC). Each resident encountered six standardized patients (SPs) portraying six cases. SPs rated the performance of each resident using the CIS Scale—an 18-item rating instrument asking for level of agreement on a 5-category scale. A many-faceted Rasch measurement model was used to determine how effectively each item and scale on the rating instrument performed. The analyses revealed that items were too easy for the residents. The SPs underutilized the lowest rating category, making the scale function as a 4-category rating scale. Some SPs were inconsistent when assigning ratings in the middle categories. The investigators modified the rating instrument based on the findings, creating the Revised UIC Communication and Interpersonal Skills (RUCIS) Scale—a 13-item rating instrument that employs a 4-category behaviorally anchored rating scale for each item. The investigators implemented the RUCIS Scale in a subsequent communication skills OSCE for 85 residents. The analyses revealed that the RUCIS Scale functioned more effectively than the CIS Scale in several respects (e.g., a more uniform distribution of ratings across categories, and better fit of the items to the measurement model). However, SPs still rarely assigned ratings in the lowest rating category of each scale.     

Concepts and Instruments for Patient-Reported Outcome Assessment in Celiac Disease: Literature Review and Experts’ Perspectives

BackgroundIn diseases where there is a large subjective component, such as celiac disease (CD), patient reported-outcomes (PRO) endpoints are highly relevant. However, there is a gap in knowledge about which PRO endpoints and instruments should be used for clinical trials for treatment of celiac disease.ObjectivesTo identify patient-centered symptom, impact, and health-related quality of life (HRQoL) concepts in CD and relevant PRO instruments, and to gather expert input on concepts and instruments to inform selection of PRO endpoints for use in clinical trials of new CD treatments.MethodsA targeted literature review was conducted to identify symptom, impact, and HRQoL concepts, including those captured in PROs further reviewed against U.S. Food and Drug Administration standards for development and validation as endpoints. US and European clinicians, payers, and a patient advocate (n = 21) were interviewed to assess the identified concepts’ relative importance in measuring treatment benefit and to gauge the value of potential PROs as endpoints for market access/reimbursement.ResultsThirty-four published studies were identified: 27 elucidated patient-centered concepts and 7 detailed the development or validation of PRO instruments. The Celiac Disease Symptom Diary and Celiac Disease Patient Reported Outcome instrument were deemed most appropriate for use as endpoints; however, each had limitations related to conceptual coverage, evidence for measurement properties, and feasibility for use in clinical trials. Experts reported gastrointestinal symptoms as most important to treat, with extra-intestinal symptoms burdensome from the patient perspective as well. Payers emphasized measuring both frequency and severity of symptoms and targeting patients nonresponsive to the gluten-free diet for treatment.ConclusionsWith emerging treatment options for CD, further work is needed to operationalize PRO symptom endpoints that are meaningful to patients, valued by payers, and acceptable to regulators in demonstrating efficacy.     

Psychometric characteristics of daily diaries for the Patient-Reported Outcomes Measurement Information System (PROMIS®): a preliminary investigation

Purpose

The Patient-Reported Outcomes (PRO) Measurement Information System (PROMIS^®) has developed assessment tools for numerous PROs, most using a 7-day recall format. We examined whether modifying the recall period for use in daily diary research would affect the psychometric characteristics of several PROMIS measures.

Methods

Daily versions of short-forms for three PROMIS domains (pain interference, fatigue, depression) were administered to a general population sample (n = 100) for 28 days. Analyses used multilevel item response theory (IRT) models. We examined differential item functioning (DIF) across recall periods by comparing the IRT parameters from the daily data with the PROMIS 7-day recall IRT parameters. Additionally, we examined whether the IRT parameters for day-to-day within-person changes are invariant to those for between-person (cross-sectional) differences in PROs.

Results

Dimensionality analyses of the daily data suggested a single dimension for each PRO domain, consistent with PROMIS instruments. One-third of the daily items showed uniform DIF when compared with PROMIS 7-day recall, but the impact of DIF on the scale level was minor. IRT parameters for within-person changes differed from between-person parameters for 3 depression items, which were more sensitive for measuring change than between-person differences, but not for pain interference and fatigue items. Notably, mean scores from daily diaries were significantly lower than the PROMIS 7-day recall norms.

Conclusions

The results provide initial evidence supporting the adaptation of PROMIS measures for daily diary research. However, scores from daily diaries cannot be directly interpreted on PROMIS norms established for 7-day recall.     

Reliability and validity of the gastrointestinal symptom rating scale in patients with gastroesophageal reflux disease

The objective of this study was to evaluate the reliability and validity of the Gastrointestinal Symptom Rating Scale (GSRS) in US patients with gastroesophageal reflux disease (GERD). Five hundred and sixteen adults with predominant heartburn symptoms of GERD were recruited from gastroenterologist and family physician practices and treated with 6 weeks of 150mg ranitidine twice daily to identify poorly responsive symptomatic GERD. The GSRS, the Medical Outcomes Study Short Form-36 (SF-36) Health Survey and the Psychological General Well-being (PGWB) scale were administered at baseline and after 6 weeks of treatment. Reported ratings of GERD-related symptoms from physician and patient diaries were measured. The GSRS contains five scales: reflux syndrome, abdominal pain, constipation syndrome, diarrhoea syndrome and indigestion syndrome. The internal consistency reliabilities for the GSRS scales ranged from 0.61 to 0.83 and the intraclass correlation coefficients ranged from 0.42 to 0.60. The GSRS scale scores were correlated with the SF-36 and PGWB scales and with the number and severity of heartburn symptoms. Patients with two or three clinician-rated GERD-related symptoms reported worse GSRS scale scores compared with patients with fewer symptoms (p < 0.0001). Statistically significant differences in the mean GSRS scale scores were observed between treatment responders and non-responders (p < 0.0001) and patients showing a response to treatment had larger mean changes in their GSRS scales than patients not showing a response to treatment (p < 0.0001). The standardized response means ranged from 0.42 to 1.43 for the GSRS scale scores. It was concluded that the GSRS is a brief, fairly comprehensive assessment of common gastrointestinal symptoms. The GSRS has good reliability and construct validity and the GSRS scales discriminate by GERD symptom severity and are responsive to treatment. The GSRS is a useful patient-rated symptom scale for evaluating the outcomes of treatment for GERD.     

Conditional reliability of admissions interview ratings: extreme ratings are the most informative

CONTEXT: Admissions interviews are unreliable and have poor predictive validity, yet are the sole measures of non-cognitive skills used by most medical school admissions departments. The low reliability may be due in part to variation in conditional reliability across the rating scale. OBJECTIVES: To describe an empirically derived estimate of conditional reliability and use it to improve the predictive validity of interview ratings. METHODS: A set of medical school interview ratings was compared to a Monte Carlo simulated set to estimate conditional reliability controlling for range restriction, response scale bias and other artefacts. This estimate was used as a weighting function to improve the predictive validity of a second set of interview ratings for predicting non-cognitive measures (USMLE Step II residuals from Step I scores). RESULTS: Compared with the simulated set, both observed sets showed more reliability at low and high rating levels than at moderate levels. Raw interview scores did not predict USMLE Step II scores after controlling for Step I performance (additional r2 = 0.001, not significant). Weighting interview ratings by estimated conditional reliability improved predictive validity (additional r2 = 0.121, P < 0.01). CONCLUSIONS: Conditional reliability is important for understanding the psychometric properties of subjective rating scales. Weighting these measures during the admissions process would improve admissions decisions.     

Relationships between changes in pain severity and other patient-reported outcomes: an analysis in patients with posttraumatic peripheral neuropathic pain

Background  

The objective of this study is to use the pain numeric rating scale (NRS) to evaluate associations between change in pain severity and changes in sleep, function, and mood assessed via patient-reported outcomes (PROs) in patients with posttraumatic pain.     

Analytic global OSCE ratings are sensitive to level of training

PURPOSE: There are several reasons for using global ratings in addition to checklists for scoring objective structured clinical examination (OSCE) stations. However, there has been little evidence collected regarding the validity of these scales. This study assessed the construct validity of an analytic global rating with 4 component subscales: empathy, coherence, verbal and non-verbal expression. METHODS: A total of 19 Year 3 and 38 Year 4 clinical clerks were scored on content checklists and these global ratings during a 10-station OSCE. T-tests were used to assess differences between groups for overall checklist and global scores, and for each of the 4 subscales. RESULTS: The mean global rating was significantly higher for senior clerks (75.5% versus 71.3%, t55 = 2.12, P < 0.05) and there were significant differences by level of training for the coherence (t55 = 3.33, P < 0.01) and verbal communication (t55 = 2.33, P < 0.05) subscales. Interstation reliability was 0.70 for the global rating and ranged from 0.58 to 0.65 for the subscales. Checklist reliability was 0.54. CONCLUSION: In this study, a summated analytic global rating demonstrated construct validity, as did 2 of the 4 scales measuring specific traits. In addition, the analytic global rating showed substantially higher internal consistency than did the checklists, a finding consistent with that seen in previous studies cited in the literature. Global ratings are an important element of OSCE measurement and can have good psychometric properties. However, OSCE researchers should clearly describe the type of global ratings they use. Further research is needed to define the most effective global rating scales.     

Integrating the Use of Patient‐Reported Outcomes for Both Clinical Practice and Performance Measurement: Views of Experts from 3 Countries

Context

Patient-reported outcomes (PROs) can play an important role in patient-centered health care by focusing on the patient''s health goals guiding therapeutic decisions. When aggregated, PROs also can be used for other purposes, including comparative effectiveness research, practice improvement, assessment of the performance of clinicians and organizations, and as a metric for value-based payments. The feasibility of integrating the use of PROs for these various purposes on a wide scale has not yet been demonstrated. Our study was conducted to inform policymakers of prudent next steps for implementing PROs in clinical practice and performance measurement programs in order to maximize their impact on the quality of health care.

Methods

We conducted a qualitative study, interviewing 58 experts and leaders from 37 organizations (response rate: 88%) in the United States, England, and the Netherlands. Respondents included clinical practitioners (n = 30), measure developers (n = 11), and leaders of performance measurement programs (n = 17). We used a qualitative content analysis to assess current strategies for applying PROs in clinical practice and performance measurement and to identify barriers to and facilitators of further implementation.

Findings

The use of PROs in clinical practice and for performance measurement has developed both separately and in parallel. Experts across the stakeholder spectrum support the collection of PRO data in an integrated manner that would enable using the data for these distinct purposes. We identified 2 main concerns about the feasibility for integrated use of PRO data: the complexity of establishing routine data collection and the tension among stakeholders when using PRO data for different purposes. These contrasting stakeholder views suggested varying interests among clinicians, measure developers, and purchasers of care.

Conclusions

Data collection approaches that support the use of PROs in health care are underdeveloped, need better integration with clinical care, and must be tailored to the characteristics of the health care system. Enabling the sustainable use of PROs will require a shared vision of clinical professionals, purchasers, and patients, with a prudent selection of the steps in implementing PROs that will maximize their impact on the quality of health care.     

Trials with patient-reported outcomes registered on the Australian New Zealand Clinical Trials Registry (ANZCTR)

Aims

It is important to understand the number, types and regions of trials that include patient-reported outcomes (PROs) to appreciate how patient experiences have been considered in studies of health and interventions. Twenty-seven percent of trials registered with ClinicalTrials.gov (2007–2013) included PROs; however, a regional breakdown was not provided and no reviews have been conducted of the Australia New Zealand Clinical Trials Registry (ANZCTR). We aimed to identify trials registered with ANZCTR with PRO endpoints and describe their characteristics.

Methods

ANZCTR was systematically searched from inception (2005) to 31 March 2017 for trials with PRO endpoints. Search terms included PRO measures listed in Patient-Reported Outcomes Quality of Life Instrument Database and Grid-Enabled Measures, as well as generic PRO terms (e.g. “quality of life” (QOL)). Trial endpoints were individually coded using an established framework to identify trials with PROs for the analysis.

Results

Of 13,666 registered trials, 6168 (45.1%) included a PRO. The proportion of studies including PROs increased between 2006 and 2016 (r?=?0.74, p?=?0.009). Among the 6168 trials, there were 17,961 individual PRO endpoints, including symptoms/functional outcomes/condition-specific QOL (65.6%), generic QOL (13.2%), patient-reported experiences (9.9%), patient-reported behaviours (7.9%). Mental health was the most common category (99.8% included PROs), followed by physical medicine/rehabilitation (65.6%), musculoskeletal (63.5%), public health (63.1%), and cancer (54.2%).

Discussion

Our findings suggest growing use of PROs in the assessment of health and interventions in ANZ. Our review identifies trial categories with limited patient-reported information and provides a basis for future work on the impact of PRO findings in clinical care.

     

Preference-based measurement of health-related quality of life (HRQL) in children with chronic musculoskeletal disorders (MSKDs).

BACKGROUND: Health-related quality of life can be measured by patients' health preferences (utilities or values). No method for measuring health state preferences has been standardized for children with arthritis or other musculoskeletal disorders (MSKDs). Such a method is needed for economic evaluations of current and new pediatric treatments. OBJECTIVES: 1) To assess the feasibility of utility measurements in children with MSKDs, 2) to test the validity of the Health Utility Index (HUI) for these children, 3) to assess whether rating scale values can be mathematically converted into meaningful standard gamble (SG) utilities, and 4) to study whether parents can act as proxies for their children with respect to health state preferences. METHODS: Eighty parents of children with MSKDs were consecutively sampled. Their children, if 8 years of age or older (n = 55), were studied concurrently. Utilities of current health states were obtained by using the SG and the HUI in random order. In addition, health state preferences were assessed using categorical and analog rating scales. Traditional nonutility measures of health status (the Childhood Health Assessment Questionnaire [CHAQ] and the Activities Scale for Kids [ASK]) were also completed. Intraclass correlation coefficients (ICCs) were calculated to assess concordance between the different utility measures and also between the ratings of the parents and their children. RESULTS: Children 8 years of age or older were able to express the strength of their health state preferences using the HUI and rating scales. Children older than 10 years of age were able to use the SG method. The health state utilities of the parents were higher than those of their children. The utilities varied widely depending on the elicitation method. The expected high agreement between the SG and the HUI was not found (ICC = 0.028 for parents, ICC = 0.016 for patients). Unlike the SG, the global utilities derived from the HUI agreed better with preferences derived from rating scales (ICC = 0.23-0.25) and correlated with traditional health status measures (with CHAQ, r = -0.56; with ASK, r = 0.46) both for parents and children. It was not possible to mathematically convert rating scale preferences into SG utilities. The SG utilities were unrelated to results from the rating scales, the CHAQ, and the ASK. Especially for parents, the SG utilities were very high, even when ratings of the other measures indicated poor health. CONCLUSIONS: Although it is possible to measure health utilities for children with MSKDs, the results are highly method dependent. The properties of the HUI in this population are more like those of the traditional health status measures rather than those of the SG. Preferences derived from rating scales, although easily performed, cannot readily be converted into SG utilities. Parents' ratings for their children are impaired by risk aversion.     

Interpreting course evaluation results: insights from thinkaloud interviews with medical students

PURPOSE: To determine whether some of the fundamental assumptions that frequently underlie interpretation of course evaluation results are justified by investigating what medical students are thinking as they complete a typical basic science course evaluation. METHODS: A total of 24 students participated in thinkaloud cognitive interviews, voicing their thoughts while completing a typical evaluation instrument that included items on overall course design, educational materials and methods, and faculty teaching. Students' responses were organised to consider how they interpreted questions, formed judgements and selected response options. Major themes relevant to the meaningful interpretation of course evaluation data were identified. RESULTS: Medical students understood educational terms such as 'independent learning' in different ways from both one another and common usage. When formulating responses, students' judgements were sometimes based on unique or unexpected criteria, and they described editing their judgements by considering factors such as effort or caring on the part of teaching faculty. Students tended to avoid using the lower end of the rating scale, used the highest rating option selectively, but chose the second highest category indiscriminately. CONCLUSIONS: These results call into question fundamental assumptions that frequently underlie interpretation of course evaluation results, such as whether students understand the intended meanings of terms used in items; whether faculty members who receive the same rating are perceived similarly; whether ratings actually reflect teaching effectiveness, and whether 'positive' ratings reflect positive opinions. This study also demonstrates how thinkaloud interviews can be used in validity studies, providing information to supplement statistical and psychometric analyses.