Developmental evaluation: building innovations in complex environments

Developmental evaluation is an emerging approach to program evaluation that emphasizes innovation and learning. It is particularly well suited to evaluating innovative programs in their earliest stages of development and adapting existing programs to complex or changing environments. Key features of the developmental evaluation approach include a tight integration between evaluators and program staff and the use of data for continuous program improvement. This article presents developmental evaluation as a complementary approach to the traditional formative-summative evaluation cycle, especially when used for preformative evaluation. To illustrate this emerging approach, the article features a case example from the Illinois Caucus for Adolescent Health's evaluation of its school board sexuality education policy change project. The article concludes by suggesting ways that developmental evaluation can be useful in health promotion practice.     

Methodological tips for overcoming formative evaluation challenges: the case of the Arthritis Foundation Walk With Ease program

Formative evaluation is an effective first step in guiding program improvement by identifying participant preferences and yielding information pertinent to making program decisions. As program evaluators working with service providers are increasingly encouraged to adopt evidence-based health promotion programs, a discrete set of real-world recommendations may help extend the use of this methodology to respond to community-specific contexts and improve health impact. This article describes the authors' step-by-step process of conducting a formative evaluation of the Arthritis Foundation Walk With Ease (WWE) program. Data collection targets (leaders, coordinators, and participants in the original program and leaders and participants in the revised piloted program) as well as methods (written surveys, focus groups, structured telephone interviews, and expert reviews) were triangulated. The authors describe the challenges they faced and conclude with practical methodological recommendations about managing time and resources, communications with respondents, and accountability systems for organizing triangulated data.     

Challenges in Evaluating Rural Health Programs

Complex community-based prevention programs are being held to scientific evidence of their effectiveness and rural public health departments that implement such programs often are not equipped to evaluate them. Rural public health departments are fettered by small budgets, small staffs, and less access to evaluation experts and similar resources. Community-based health promotion programs can include complex designs that may work differently in rural areas and evaluation of rural programs can be hampered by lack of control groups and the instability of results from small populations. The University of Kentucky has entered into a contract with the state Department for Public Health to implement an internal, participatory model of evaluation. In this model, the university evaluation expert trains local public health department staff in technical skills for program evaluation and acts as mentor and technical consultant to local public health departments on an ongoing basis. Through training and site visits, this model is one approach to addressing the challenges of evaluating rural health promotion programs.     

Durable data. A rehabilitation center collects and distributes information for functional assessment and program evaluation

As physical rehabilitation services have expanded nationwide, both payers and providers have increasingly felt the need for adequate outcome data. In 1988 organizers at Marianjoy Rehabilitation Center, Wheaton, IL, launched the Center for Rehabilitation Outcome Analysis-a service for rehabilitation facilities that want to develop in-house functional assessment and program evaluation systems. The center has focused on meeting three major needs of rehabilitation providers: outcome data analysis and reporting, education, and research. It uses the Patient Evaluation and Conference System for a variety of needs from planning programs to establishing functional outcome expectations. The center also educates participants on the use and meaning of outcome data. In addition, it has enlisted the help of the profession's behavioral scientists, evaluators, and statisticians to scrutinize and enhance program evaluation and measurement models.     

Standardized patients: a promising tool for health education and health promotion

Standardized patients (SPs) are trained actors who are used to engage health care providers in various types of clinical encounters for the purposes of training and evaluation. In the past, SPs have most often been a tool for training clinicians in "traditional" medical skills such as the taking of medical histories or the conduct of physical exams. More recently, however, SPs have been increasingly used to assist in the development and assessment of psychosocial skills, including those related to patient-centered communication, relationship building, and motivational interviewing. Given this shift, it is argued that the time is ripe for exploring ways that health educators can adopt SP methodologies, whether for providing continuing education for individual providers or evaluating organizations or programs. This article introduces ways of using SPs and discusses strengths and challenges related to various approaches.     

Improved Outcomes Found After Implementing a Systematic Evaluation and Program Improvement Process for Tuberculosis

California''s state and local tuberculosis (TB) programs collaborated to develop the Tuberculosis Indicators Project (TIP), a program evaluation and improvement process. In TIP, local and state staff review data, identify program gaps, implement plans to improve local TB program performance, and evaluate outcomes. After 10 years of project implementation, indicator performance changes and patient outcomes were measured. Eighty-seven percent of participating programs showed a performance increase in targeted indicators after three years compared with 57% of comparison groups. Statistically significant performance change was more common in the intervention local health departments (LHDs) than in comparison groups. The most notable performance changes were in the contact investigation and case management indicators. These results indicate that this systematic evaluation and program improvement project was associated with improved LHD TB control performance and may be useful to inform improvement projects in other public health programs.In 1998, the California Department of Public Health, Tuberculosis Control Branch (TBCB) sought to enhance its program evaluation and improvement efforts. The evaluation process had two aims: (1) to measure the effectiveness of California tuberculosis (TB) control programs in implementing well-established TB control guidelines and ensuring positive patient and public health outcomes and (2) to focus interventions where gaps were identified. From 1998 to 2000, the TBCB worked closely with local health departments (LHDs) to design TB program performance indicators and a collaborative process for the state and local TB programs to use the indicators. The resulting evaluation and program improvement initiative, called the Tuberculosis Indicators Project (TIP), was launched in 2000.In this article, we describe the implementation of an evaluation framework in multiple high-morbidity local TB control programs in California. We also quantify the program performance changes associated with TIP and examine why these changes might have occurred.     

Collaborative community research consortium: a model for HIV prevention.

In 1991, the Center for AIDS Prevention Studies (CAPS) at the University of California, San Francisco, set out to develop a model of community collaborative research that would bring the skills of science to the service of HIV prevention and the knowledge of service providers into the domain of research. Essential elements of the model were training for community-based organizations (CBOs) in research protocol writing, partnership between CBOs and CAPS researchers, program research funding, support to implement studies and analyze results, and a program manager to oversee the effort and foster the relationships between CBOs and researchers. In this article, the authors describe the CAPS model of consortium-based community collaborative research. They also introduce a set of papers, written by researchers and service providers, that describes collaborative research projects conducted by research institutions and CBOs and illustrates how collaboration can change both HIV prevention research and service.     

The use of formative evaluation to assess integrated services for children. The Robert Wood Johnson Foundation Child Health Initiative

This article describes the use of formative evaluation in assessing the feasibility of implementing a new service integration effort. The Child Health Initiative, a nine-site, national demonstration project funded in 1991 by the Robert Wood Johnson Foundation, sought to implement systemic change through the creation of new mechanisms for spending service dollars more flexibly at the local site. The Child Health Initiative called for developing local child health-monitoring systems, a care coordination mechanism, and a program for decategorizing the myriad of restrictive categorical public programs serving children. Most demonstration communities experienced some degree of success in achieving the first two components, but none was able to implement decategorization during the 3- to 5-year funding period. Key lessons for evaluators include the need for (a) a flexible evaluation design that can sequentially adapt to changes in program implementation, (b) repeated longitudinal data collection measures to document changes over time, (c) avoidance of a premature focus on program outcomes, and (d) methods to establish attribution of outcomes.     

Effects of local conditions on program outcomes: analysis of contradictory findings from two programs for homeless mentally ill

A demonstration program for the homeless mentally ill that succeeded at one site and failed at another serves as the basis for analysis of the effects of local conditions on program implementation and outcomes. Chen's program implementation framework is utilized. Implications are drawn for funders, service providers, and evaluators.     

A framework for gauging the comprehensiveness of governmental HIV prevention programs.

This article presents a brief, logical framework that can be used in conducting a review of the comprehensiveness of any governmental human immunodeficiency virus (HIV) prevention program at the local, state, or national level. The framework, presented as a checklist of questions, could be used by external evaluators of an HIV prevention program, used internally for continuous quality improvement, or a combination of the two. The checklist can be used to select components of HIV prevention programs in need of particular, critical improvements.     

The Challenge of Evaluating Programs in a Large Health Sciences Centre

This article illustrates how the logic model and a course on program evaluation at a large health sciences centre were instrumental in preparing staff to evaluate their own programs. Staff and physicians need basic skills in program evaluation. The logic model is a simple yet useful tool in helping to identify key measurables. A short course to teach the theory and practice of logic models, evaluation design, choice and design of measures, and data analysis has been shown to be a practical solution in preparing staff and physicians to evaluate their own programs.     

Evaluating the Mount Sinai Leadership Enhancement Program

The principal evaluators of the Mount Sinai Leadership Exchange program used a developmental approach to evaluating the program, applying qualitative and quantitative data within the program as the program developed. Over the course of its development, evaluation, and refinement, the leadership enhancement program emerged as a vehicle for true international exchange of ideas, skills, resources, and collegiality.     

Worker-trainers as evaluators: a case study of a union-based health and safety education program

The United Automobile Workers uses worker-trainers to deliver health and safety education training to its members. The union has experimented with worker-trainers participating in program evaluation. Worker-trainers participated in the design and conduct of a telephone survey of training impact, and of an on-site survey of trainee perceptions of a large-scale program. Worker evaluators were able to analyze data and give a highly successful presentation of results to trainees at the large-scale program. The incorporation of workers as evaluators is a key step toward the goal of worker empowerment. The involvement of program participants in their program's evaluation can enhance the quality and usefulness of work-site health and safety programs in general.     

Lessons from the MAPP demonstration sites.

This article describes one component of a three-part evaluation of a community health improvement model called MAPP (Mobilizing for Action through Planning and Partnerships). This component examined the use of MAPP by nine local public health agencies that acted as demonstration sites. Using qualitative data from telephone interviews, focus group data, and document reviews, the evaluators identified four themes associated with how MAPP was implemented and the factors that influenced implementation. A level of activity was assigned to each site for each of the four themes. Combining data from multiple demonstration sites, evaluators characterized common activities for each theme. Findings reveal that those sites with health director buy-in and full-time coordination for MAPP were associated with high levels of partnership development. The MAPP Web site, in its current state, does not appear to provide an advantage over using a printed copy of MAPP. Evaluation data from training programs and Web site users (the other two components of the evaluation) support key findings from the demonstration sites and reveal crosscutting themes for all components of the MAPP evaluation. Investigating other situational and contextual factors that influence the nature and success of MAPP implementation would be useful.     

Personal privacy versus public accountability: A technological solution to an ethical dilemma

The tension between personal privacy and public accountability produces one of the major ethical dilemmas facing behavioral health program evaluators and service system researchers. This article discusses the source of this tension and introduces a research methodology that allows program evaluators to fully and equally respect both ethical principles. This methodology uses contemporary computer and statistical technology in conjunction with aggregated, de-identified information derived from existing databases to provide valid and reliable measures of the performance of treatment programs while it protects the personal privacy of individuals.     

Strategies for Ensuring Use of Needs Assessment Findings: Experiences of a Community Substance Abuse Prevention Program

This article offers specific strategies for ensuring utilization of needs assessment findings. It is designed to help prevention specialists, community planners, program evaluators, and involved citizens maximize their efforts in developing community services by tying those services closely to community needs. Program evaluators who completed a broad-based community needs assessment commissioned by their organization, found that the organization had gone ahead in developing programs and saw the needs assessment findings as unrelated to their ongoing activities. Far from being unique, this situation is common in organizations where participants feel an urgency to get programs under way to address widespread community problems.     

Intermediate Outcomes,Strategies, and Challenges of Eight Healthy Start Projects

Site visits were conducted for the evaluation of the national Healthy Start program to gain an understanding of how projects design and implement five service components (outreach, case management, health education, depression screening and interconceptional care) and four system components (consortium, coordination/collaboration, local health system action plan and sustainability) as well as program staff’s perceptions of these components’ influence on intermediate outcomes. Interviews with project directors, case managers, local evaluators, clinicians, consortium members, outreach/lay workers and other stakeholders were conducted during 3-day in-depth site visits with eight Healthy Start grantees. Grantees reported that both services and systems components were related to self-reported service achievements (e.g. earlier entry into prenatal care) and systems achievements (e.g. consumer involvement). Outreach, case management, and health education were perceived as the service components that contributed most to their achievements while consortia was perceived as the most influential systems component in reaching their goals. Furthermore, cultural competence and community voice were overarching project components that addressed racial/ethnic disparities. Finally, there was great variability across sites regarding the challenges they faced, with poor service availability and limited funding the two most frequently reported. Service provision and systems development are both critical for successful Healthy Start projects to achieve intermediate program outcomes. Unique contextual and community issues influence Healthy Start project design, implementation and reported accomplishments. All eight projects implement the required program components yet outreach, case management, and health education are cited most frequently for contributing to their perceived achievements.     

Balancing the priorities of evaluation with the priorities of the setting: A focus on positive youth development programs in school settings

This paper explores important issues that program developers and evaluators must consider when implementing and researching positive youth development programs in school settings. Issues such as ensuring utility of the program at the local level and assessing process, proximate, and distal outcomes are stressed.     

Health education evaluation: an alternate research paradigm

A major disappointment with educational evaluative research in general, and health education evaluation in particular, is all too often outcome of "no significant difference" or "no effect." Reviewing the evaluation literature one finds experienced investigators such as Donald Campbell, Marcia Guttentag, and Carol Weiss lamenting the fact that educational programs persistently fail to uncover statistical differences. This problem continues to plague evaluators whose experiential evidence indicates that the program may have produced some very desirable results. Thus, the traditional methods of evaluation are increasingly being called into question. Evaluators are presenting theories to explain the "no effects" phenomenon and alternate methodologies are being proposed in the literature. This article suggests that health educators not only acknowledge the existence of this dilemma, but develop a working knowledge of alternate or qualitative data methodologies, their possibilities, and their limitations, thus encouraging the evaluation of those health programs which have traditionally been classified as undoable.     

Qualitative process evaluation of urban community work: a preliminary view.

Community-based and participatory research have become significant activities during the past 15 years as public health practitioners and researchers have sought new ways to provide effective disease prevention and health promotion programs. It is also important that more examples of evaluation schemes be contributed for field testing. The process evaluation model offered here was based on an eclectic literature search because the evaluators did not know what directions this Centers for Disease Control and Prevention (CDC)-funded Urban Research Center would take and wanted to be thorough. Participatory action research using predominantly qualitative methods provided a research approach congruent with the capacity-building and power-sharing principles of the center. Seattle Partners for Healthy Communities is not a traditional community project, acting primarily as a broker for expertise and community needs. However, it has been successful in supporting and evaluating community health projects.