The Validity of Reliability Assessments

This paper focuses on reliability and evaluation of health education programs in school settings. Reliability is a concept that guides researchers in selecting or developing instruments, and is used as a standard, with validity and acceptability, for judging the credibility of research findings and inferences. Reliability is defined within the context of research design, and methods for estimating the reliability of cognitive measures are reviewed. Using data gathered in a school health education curriculum evaluation as an example, possible errors in hypotheses testing that may occur when estimating internal consistency of cognitive test scores obtained in quasi-experimental designs are examined. The appropriateness of internal consistency as a measure of reliability of cognitive measures is discussed and suggestions for reliability assessment and related issues such as power analysis are presented.     

Meta-analysis of quasi-experimental research: are systematic narrative reviews indicated?

Context  Meta-analyses are commonly performed on quasi-experimental studies in medical education and other applied field settings, with little or no apparent concern for biases and confounds present in the studies synthesised. The implicit assumption is that the biases and confounds are randomly distributed across the studies and are averaged or cancelled out by the synthesis.
Objectives  We set out to consider the possibility that the results and conclusions of meta-analyses in medical education are subject to biases and confounds and to illustrate this possibility with a re-examination of the studies synthesised in an important, recently published meta-analysis of problem-based learning.
Methods  We carefully re-examined the studies in the meta-analysis. Our aims were to identify obvious biases and confounds that provided plausible alternative explanations of each study's results and to determine whether these threats to validity were considered and convincingly ruled out as plausible rival hypotheses.
Results  Ten of the 11 studies in the meta-analysis used quasi-experimental designs; all 10 were subject to constant biases and confounds that favoured the intervention condition. Threats to validity were not ruled out in the individual studies, nor in the meta-analysis itself.
Conclusions  Our re-examination of the results and conclusions of the meta-analysis illustrates our concerns about the validity of meta-analyses based primarily on quasi-experimental studies. Our tentative conclusion is that the field of medical education might be better served in most instances by systematic narrative reviews that describe and critically evaluate individual studies and their results in light of threats to their validity.     

Andragogy and medical education: are medical students internally motivated to learn?

Andragogy – the study of adult education – has been endorsed by many medical educators throughout North America. There remains, however, considerable controversy as to the validity and utility of adult education principles as espoused by the field's founder, Malcolm Knowles. Whatever the utility of andragogic doctrine in general education settings, there is reason to doubt its wholesale applicability to the training of medical professionals. Malcolm Knowles' last tenet of andragogy holds that adult learners are more motivated by internal than by external factors. The validity of this hypothesis in medical education is examined, and it is demonstrated that medical students' internal and external motivation are context-dependent, not easily distinguishable, and interrelate with one another in complex ways. Furthermore, the psychological motivation for medical student learning is determined by a variety of factors that range from internal to external, unconscious to conscious, and individual to societal. The andragogic hypothesis of increased internal motivation to learn on the part of adults in general, and medical trainees in particular, is rejected as simplistic, misleading, and counter productive to developing a greater understanding of the forces that drive medical students to learn. This revised version was published online in June 2006 with corrections to the Cover Date.     

Problems of research into medical problem solving: some remarks on theory and method

Investigators of medical problem solving, and of problem solving in general, appear to use the term problem solving to denote different concepts.
Medical problem solving sometimes refers to the doctor solving the patient's problem, whereas others use it to refer to the doctor solving his own diagnostic problem. Second, 'problem' is used by some in a subjective sense (something is only called a problem if the subject has difficulties with it), and by others in an objective sense (problem being nearly synonymous with task). Finally, there is the definitional question of whether knowledge and problem solving are regarded as independent or as intimately related.
What one means by problem solving, with one's research objectives (research may be aimed at medical education, medical practice or cognitive theory), constitute the major determinants of the choice of research design.
It is advocated that investigators, before selecting a research design, should clarify their own definitions and research objectives, and before adopting other investigators' definitions and research designs, should consider carefully their research objectives.     

Evidence-based public health: moving beyond randomized trials

Randomized controlled trials (RCTs) are essential for evaluating the efficacy of clinical interventions, where the causal chain between the agent and the outcome is relatively short and simple and where results may be safely extrapolated to other settings.

However, causal chains in public health interventions are complex, making RCT results subject to effect modification in different populations. Both the internal and external validity of RCT findings can be greatly enhanced by observational studies using adequacy or plausibility designs. For evaluating large-scale interventions, studies with plausibility designs are often the only feasible option and may provide valid evidence of impact.

There is an urgent need to develop evaluation standards and protocols for use in circumstances where RCTs are not appropriate.

     

How clinical research can grow a medical practice

Clinical research investigators can use many different tools available to them through their practice and the study sponsor to ensure that they are active enrolling clinical research sites, such as study-specific Web sites, dedicated information lines, and educational sessions. If these tools are approved by the study sponsor and used properly through outreach in conjunction with proper processes and internal controls, a clinical research study will facilitate growth within a medical practice while ensuring that the investigators become or continue to be key opinion leaders within their specialty.     

Behavior change intervention research in healthcare settings: a review of recent reports with emphasis on external validity

BACKGROUND: Information to judge both the internal and external validity of health behavior research conducted in healthcare settings is vital to translate research findings to practice. This paper reviews the extent to which this research has reported on elements of internal and external validity, with emphasis on the extent to which research has been conducted in representative settings with representative populations. METHODS: A comprehensive review was conducted of controlled interventions for dietary change, physical activity, or smoking cessation conducted in healthcare settings and published in 12 leading health behavior journals between 1996 and 2000. Using the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework, the characteristics and results of these studies were summarized to document the extent to which intervention reach, adoption, implementation, and maintenance were reported and what has been learned about each of these dimensions. RESULTS: A total of 36 studies qualified for review. Participation rates among eligible patients were reported in 69% of studies and were generally quite high; in contrast, only 30% of studies reported on participation rates among either healthcare settings or providers. Implementation data were reported in 77% of the studies and these rates were generally high, with the caveat that intervention was often delivered by paid research staff. Long-term maintenance results were reported very consistently at the individual level, but program continuation was almost never reported at the setting level. CONCLUSIONS: We conclude that a much stronger emphasis needs to be placed on the representativeness of providers and settings that are studied. Examples of how this can be done and recommendations for future research are provided.     

Family planning field research projects: balancing internal against external validity

This report discusses the experience of a two-year family planning and maternal/child health project in Nepal. Although the project was planned as an experimental field research endeavor, a series of unanticipated events repeatedly compromised the internal validity of the project and forced design changes. While unexpected events are common in the history of most field projects, they present the research evaluator with the fundamental dilemma of trying to maintain a high degree of internal validity without sacrificing external validity. Rigid research designs with tight control over the introduction and measurement of experimental variables may serve to increase internal validity but they may also create an atypical and artificial situation that fails to mirror real field conditions and thus threatens external validity.     

The Value of Interrupted Time-Series Experiments for Community Intervention Research

Greater use of interrupted time-series experiments is advocated for community intervention research. Time-series designs enable the development of knowledge about the effects of community interventions and policies in circumstances in which randomized controlled trials are too expensive, premature, or simply impractical. The multiple baseline time-series design typically involves two or more communities that are repeatedly assessed, with the intervention introduced into one community at a time. It is particularly well suited to initial evaluations of community interventions and the refinement of those interventions. This paper describes the main features of multiple baseline designs and related repeated-measures time-series experiments, discusses the threats to internal validity in multiple baseline designs, and outlines techniques for statistical analyses of time-series data. Examples are given of the use of multiple baseline designs in evaluating community interventions and policy changes.     

The role of internal pilot studies in increasing the efficiency of clinical trials

Investigators often design clinical trials without knowing precisely the values of such necessary parameters as the variances or the event rates in the control group. In order to determine reasonable values for such parameters, they may design a small pilot study external to the main trial. In this paper we propose designs, which we term internal pilot studies, that designate a portion of the main trial as a pilot phase. At the end of the internal pilot study, the investigators recompute preselected parameters and recalculate required sample size. The study then proceeds with the modifications dictated by the internal pilot. Final analyses of the results incorporate all data, disregarding the fact that part of the data came from a pilot phase. As one example of this type of design, we consider a study to compare two normally distributed means. By simulation, we show a numerical example for which the effect of the procedure on the alpha-level is negligible, but the potential gain in power considerable. We urge considering a similar approach for a number of types of endpoints.     

A novel training model to address health problems in poor and underserved populations

Health disparities are increasingly common and many U.S. practitioners have informal experience working in resource-poor settings. There are, however, few graduate medical education programs that focus on health equity. A graduate medical education program in health equity was developed at Brigham and Women's Hospital based on a review of existing literature and on a survey of junior faculty who have had informal health disparities experience. The Howard Hiatt Residency in Global Health Equity and Internal Medicine was developed as a four-year program to provide intensive training in internal medicine and health disparities. Participating residents are matched with a mentor who has clinical and research experience in the field of global health. In addition to a series of didactic teaching sessions and longitudinal seminars that focus on issues of global health equity, residents take graduate level courses in epidemiology, health policy, ethics, and medical anthropology. Residents also carry out an independent research project in a geographic area that suffers from health disparities. Two residents are selected for training per year. Participating faculty are multidisciplinary and come from diverse Harvard-affiliated institutions. Graduate medical education in the United States with a focus on health equity is lacking. It is hoped that the novel training program in health equity for internal medical residents developed at Brigham and Women's Hospital can serve as a model for other teaching hospitals based in the United States.     

The role of choice in health education intervention trials: a review and case study

Although the randomized, controlled trial (RCT) is considered the gold standard in research for determining the efficacy of health education interventions, such trials may be vulnerable to "preference effects"; that is, differential outcomes depending on whether an individual is randomized to his or her preferred treatment. In this study, we review theoretical and empirical literature regarding designs that account for such effects in medical research, and consider the appropriateness of these designs to health education research. To illustrate the application of a preference design to health education research, we present analyses using process data from a mixed RCT/preference trial comparing two formats (Group or Self-Directed) of the "Women take PRIDE" heart disease management program. Results indicate that being able to choose one's program format did not significantly affect the decision to participate in the study. However, women who chose the Group format were over 4 times as likely to attend at least one class and were twice as likely to attend a greater number of classes than those who were randomized to the Group format. Several predictors of format preference were also identified, with important implications for targeting disease-management education to this population.     

Volunteer bias in medical education research: an empirical study of over three decades of longitudinal data

CONTEXT: The issue of whether medical education research outcomes can be biased by students' refusal to allow their data to be used in outcomes research should be empirically addressed to assure the validity of research findings. Given that institutions are expected to document the outcomes of their educational programmes, evaluations of clinical performance subsequent to medical school are crucial, but are often incomplete when graduates decline to permit data collection. OBJECTIVES: This study aimed to examine the demographic and performance differences between research volunteers and others. METHODS: A total of 7415 doctors graduated from Jefferson Medical College between 1970 and 2004; 75% (n = 5575) agreed to participate in medical education research by granting written permission for the collection of data from their postgraduate training directors on their behalf (research volunteers); 20% (n = 1489) refused to grant such permission (non-volunteers), and 5% (n = 351) did not return the permission form (non-respondents). This prospective longitudinal study compared research volunteers, non-volunteers and non-respondents on gender, ethnicity, performance measures prior to, during and after medical school, scores on medical licensing examinations, and board certification status. RESULTS: Doctors who granted permission (volunteers) generally performed better during and after medical school. In addition, they scored higher on medical licensing examinations and had a higher certification rate. Women and members of ethnic minority groups were less likely to grant permission. CONCLUSIONS: The study raises questions about the validity of research findings as a result of volunteerism in medical education research. The implications for guidelines regarding the protection of human subjects in medical education research, and for educational outcomes, are discussed.     

Medicine: conflicts of interest in research

A conflict of interest is a set of conditions in which professional judgment concerning a primary interest (patient's welfare, research integrity, ethics, education) tends to be influenced by a secondary interest. Among secondary interests are intellectual conflicts of interest that include the desire for faculty advancement, competition for sponsored research funding, to win prestigious research prizes, as well as financial conflicts of interest that arise when medical investigators own equity in a company that sponsors the study they conduct or when they serve as consultants to companies or are members of the company's advisory boards. Ethical conflicts of interest predominate in undeveloped countries. Known consequences of conflicts of interests include compromising health, death, minimization of adverse reactions, lower research quality, the probability to favor the sponsor's product, and delayed publication when results obtained are not the expected. Medical consciousness should be corrected to avoid moral duality that could harm both patients and medical ethics.     

Validation of an observation instrument for measuring student engagement in health professions settings

Documenting student engagement has received increased emphasis in medical schools, as teaching strategies are changing to include more student-to-student interactions. The purpose of this study was to develop and evaluate a measure of student engagement completed by independent observers that would not interfere with student learning time. Data from 3,182 observations completed by nine observers in 32 educational classroom settings with 23 different instructors were used to evaluate the interobserver reliability and gather validity evidence for our observational instrument, named the STROBE. Results indicated that interobserver agreement was good to excellent when observations were conducted simultaneously on randomly selected students in the same classroom (84% average agreement and 0.79 average kappa coefficient) and when observations were conducted on different randomly selected students (79% average agreement). Results also provided strong evidence for validity. Overall, findings indicate that the STROBE demonstrates promise for educational research and evaluation by documenting student engagement in medical education settings.     

Using mixed methods research in medical education: basic guidelines for researchers

Context  Mixed methods research involves the collection, analysis and integration of both qualitative and quantitative data in a single study. The benefits of a mixed methods approach are particularly evident when studying new questions or complex initiatives and interactions, which is often the case in medical education research. Basic guidelines for when to use mixed methods research and how to design a mixed methods study in medical education research are not readily available.
Methods  The purpose of this paper is to remedy that situation by providing an overview of mixed methods research, research design models relevant for medical education research, examples of each research design model in medical education research, and basic guidelines for medical education researchers interested in mixed methods research.
Conclusions  Mixed methods may prove superior in increasing the integrity and applicability of findings when studying new or complex initiatives and interactions in medical education research. They deserve an increased presence and recognition in medical education research.     

Reflections on experimental research in medical education

As medical education research advances, it is important that education researchers employ rigorous methods for conducting and reporting their investigations. In this article we discuss several important yet oft neglected issues in designing experimental research in education. First, randomization controls for only a subset of possible confounders. Second, the posttest-only design is inherently stronger than the pretest–posttest design, provided the study is randomized and the sample is sufficiently large. Third, demonstrating the superiority of an educational intervention in comparison to no intervention does little to advance the art and science of education. Fourth, comparisons involving multifactorial interventions are hopelessly confounded, have limited application to new settings, and do little to advance our understanding of education. Fifth, single-group pretest–posttest studies are susceptible to numerous validity threats. Finally, educational interventions (including the comparison group) must be described in detail sufficient to allow replication.     

Placebo surgery research: a blinding imperative

In placebo surgery research, intervention group subjects receive a real surgical procedure, and control subjects receive a sham surgical procedure. The sham procedure is designed to recapitulate enough of the real procedure to blind participants to group assignment. Placebo surgery designs are used to control for placebo effects, and to eliminate the bias that might result if participants and investigators were aware of group assignment when outcomes are measured. In placebo surgery studies, in which participants are placed at increased risk to achieve the scientific validity provided by blinding, it is imperative that the blind be maintained, and maintainable, to justify putting subjects at risk to achieve it. In circumstances in which the outcome assessors are not fully blinded, or in which the intervention is not amenable to rigorous blinding, placebo surgery studies are not ethical. Investigators doing placebo surgery trials should obtain data on the integrity of the blind from both subjects and outcome assessors, to inform the validity of the results and to provide input for "blindability" decisions involving future trials.     

Reducing ultraviolet radiation exposure to prevent skin cancer methodology and measurement

Skin cancer is the most common type of cancer, and is also one of the most preventable. This paper builds on an evidence review of skin cancer prevention interventions that was conducted for the Guide to Community Preventive Services (n=85 studies), and summarizes the state of knowledge about research methodology and measurement in studies of the effectiveness of interventions to reduce ultraviolet radiation (UVR) exposure. As this field advances, researchers should strive to minimize threats to validity in their study designs, as well as to consider the balance between internal and external validity. There is a need for more longer-duration interventions, and follow-up periods that make possible conclusions about the potential of these interventions to affect intermediate markers of skin cancer or at least sustained behavior change. Also, more work is needed to minimize attrition and characterize nonresponders and study dropouts. Verbal report measures of behavior are the most widely used measures of solar protection behavior. Given their limitations, investigators should routinely collect data about reliability and validity of those measures. They should also increase efforts to complement verbal data with objective measures including observations, skin reflectance, personal dosimetry, skin swabbing, and inspection of moles. Measures of environments and policies should incorporate observations, documentation, and direct measures of ambient UVR and shade. This article places the data derived from the evidence review in the context of needs and recommendations for future research in skin cancer prevention.     

Validity: on meaningful interpretation of assessment data

CONTEXT: All assessments in medical education require evidence of validity to be interpreted meaningfully. In contemporary usage, all validity is construct validity, which requires multiple sources of evidence; construct validity is the whole of validity, but has multiple facets. Five sources--content, response process, internal structure, relationship to other variables and consequences--are noted by the Standards for Educational and Psychological Testing as fruitful areas to seek validity evidence. PURPOSE: The purpose of this article is to discuss construct validity in the context of medical education and to summarize, through example, some typical sources of validity evidence for a written and a performance examination. SUMMARY: Assessments are not valid or invalid; rather, the scores or outcomes of assessments have more or less evidence to support (or refute) a specific interpretation (such as passing or failing a course). Validity is approached as hypothesis and uses theory, logic and the scientific method to collect and assemble data to support or fail to support the proposed score interpretations, at a given point in time. Data and logic are assembled into arguments--pro and con--for some specific interpretation of assessment data. Examples of types of validity evidence, data and information from each source are discussed in the context of a high-stakes written and performance examination in medical education. CONCLUSION: All assessments require evidence of the reasonableness of the proposed interpretation, as test data in education have little or no intrinsic meaning. The constructs purported to be measured by our assessments are important to students, faculty, administrators, patients and society and require solid scientific evidence of their meaning.