Joint Commission International accreditation: relationship to four models of evaluation.

OBJECTIVE: To describe the components of the new Joint Commission International (JCI) accreditation program for hospitals, and compare this program with the four quality evaluation models described under the ExPeRT project (visitatie, ISO, EFQM, organizational accreditation). RESULTS: All the models have in common with the JCI program the use of explicit criteria or standards, and the use of external reviewers. The JCI program is clearly an organizational accreditation approach with evaluation of all the 'systems' of a health care organization. The JCI model evaluates the ability of an organization to assess and monitor its professional staff through internal mechanisms, in contrast with the external peer assessment used by the visitatie model. The JCI program provides a comprehensive framework for quality management in an organization, expanding the boundaries of the quality leadership and management found in the EFQM model, and beyond the quality control of the ISO model. The JCI organizational accreditation program was designed to permit international comparisons, difficult under the other models due to country specific variation. CONCLUSION: We believe that the organizational accreditation model, such as the international accreditation program, provides a framework for the convergence and integration of the strengths of all the models into a common health care quality evaluation model.     

The program implications of administrative relationships between local health departments and state and local government.

A typology of organizational arrangements between state and local public health agencies was used as a framework within which the organizational environment of the local health department was studied for its effects on program development and implementation by local public health departments. Data collected in a national sample of local health officers were used in measuring the effect of four different patterns of administrative relationships on the selected characteristics of local health department programs. Important differences were observed among the four organizational types with regard to constraints on programs and program priorities, and health officers' perceptions of the primary functions of local health departments and sources of local health department funding. These findings were then used as a baseline from which to consider the possible impact of recent federal health budgetary proposals (specifically, block grants) both on existing patterns of intergovernmental relations and on the funding and operation of local health department programs. It was determined that the most likely general development arising from these proposed changes in federal budgetary policy is that the administrative control of state health agencies over those at the local level is likely to be enhanced. Other likely developments include changes in the programs and priorities of local health departments related to reductions in overall funding levels for human services and forced competition for fewer dollars by an enlarged constituency.     

Process evaluation of the Pool Cool Diffusion Trial for skin cancer prevention across 2 years

Though process evaluation of health programs has received growing attention, few interventions have reported process evaluation over multiple years. This article describes 2 years of process evaluation (2003-04) for the Pool Cool Diffusion Trial. Pool Cool is a skin cancer prevention program designed to increase sun protection habits among children and improve organizational and environmental supports for sun protection at swimming pools. Each year, 80 telephone interviews and 40 site visits at pools across the United States were completed, to examine how fully the program was implemented and the extent of use of program components between the two study conditions. Major components of the Pool Cool program, including sun safety lessons, sun safety signs and sunscreen use, had high implementation. Between the 2 years, most of the core elements were either maintained or increased in use. There were no significant differences between the basic and enhanced conditions on implementation. Reasons given for successful implementation were the provision of a toolkit, ease of implementing the program, pool staff and children enjoying the program and the field coordinators' support. These data provide information on programmatic factors that contribute to successful program diffusion.     

A Formative Evaluation in Maternal and Child Health Practice: The Partners for Life Nutrition Education Program for Pregnant Women

Objectives: This paper describes the formative evaluation of the Partners For Life Program that was developed to change dietary behavior of low income pregnant women in the Mississippi Delta region. Methods: A diverse group of nutrition/health professionals, adapted the Expanded Food and Nutrition Education Program's (EFNEP) methodology for pregnant patients in the local Women, Infants, and Children program who were receiving maternity care at the county health department. Formative evaluation activities included gathering data to determine 1) whether a new nutrition curriculum, modeled after EFNEP could be created, 2) whether peer educators could be recruited and trained to deliver the intervention, 3) whether pregnant women could be recruited to participate in the new program, 4) whether a pilot test of the new intervention would produce short-term impact in nutrition knowledge and dietary behavior, and 5) reactions of pregnant women on the usability of the new program. Data were gathered through use of the Program Implementation Index, focus groups, and a retrospective record review. Pilot test assessments included tests of nutrition knowledge and self-reported changes in dietary behavior. Results: The formative evaluation demonstrated both positive and negative outcomes. Positive data included 1) successful recruitment and training of the peer educators to deliver the intervention; 2) successful recruitment of the targeted population for the pilot study; 3) complete information on project questionnaires and measuring forms; and 4) among those who completed the program, a statistically significant improvement in nutrition knowledge and dietary behavior. Two negative aspects in this formative study were 1) the Program Implementation Index quantitatively showed that the program experienced problems retaining participants it recruited and 2) deviation of the timeframe for intervention delivery. Program length was identified as the primary reason for participant attrition. Conclusions: It is important for program developers to use results from formative evaluations to make changes in problem areas prior to implementation of a full-scale impact evaluation.     

Local health departments in Washington State use APEX to assess capacity.

THE ASSESSMENT PROTOCOL for EXCELLENCE in PUBLIC HEALTH process was carried out in the state of Washington to assess local health department capacity and to identify their self-perceived strengths and weaknesses. Staff from 24 of the 32 local health departments in Washington completed organizational capacity assessments. Fifty percent or more of the health departments identified the following eight indicators as strengths: legal authority, public policy and implementation, budget development, financial reporting and administration, audit, financial documentation, organization and structure of program management, and policy board procedures. Seven indicators were identified as weaknesses by 50% or more of the respondents: legal counsel, mission and role, data collection and analysis, planning and development, evaluation and assurance of community health assessment, community health assessment and planning, and community health policy. The results of the assessment highlight the traditional organizational and service delivery strengths of the local health departments and point out weaknesses in their ability to assess community health and to develop communitywide health policy.     

Organizational capacity's effects on the delivery and outcomes of health education programs.

Community-based public health efforts to change health behaviors and health outcomes generally involve the implementation of complex, multipronged programs, which utilize many resources, both inside and outside a single community-based organization or agency. It is becoming more apparent that the organizational capacities of these agencies influence the implementation and success of health promotion programs. However, research is limited on the specific organizational capacities (e.g., resources, training, workload, trust, communication) that influence program implementation and, thus, the ultimate impact of these programs. This article seeks to address this gap in the literature by identifying organizational capacity variables that may have influenced the delivery of a dietary change program.     

People and money matter: investment lessons from the Ontario Heart Health Program, Canada

Resource allocation is a critical issue for public health decision-makers. Yet little is known about the level and type of resources needed to build capacity to plan and implement comprehensive programs. This paper examines the relationships between investments and changes in organizational capacity and program implementation in the first phase (1998-2003) of the Ontario Heart Health Program (OHHP)--a province-wide, comprehensive public health program that involved 40 community partnerships. The study represents a subset of findings from a provincial evaluation. Investments, organizational capacity of public health units and implementation of heart health activities were measured longitudinally. Investment information was gathered annually from the provincial government, local public health units and community partners using standard reports, and was available from 1998 to 2002. Organizational capacity and program implementation were measured using a written survey, completed by all health units at five measurement times from 1994 to 2002. Combining provincial and local sources, the average total investment by year five was $1.66 per capita. Organizational capacity of public health units and implementation of heart health activities increased both before and during the first 2 years of the OHHP, and then plateaued at a modest level for capacity and a low level for implementation after that. Amount of funding was positively associated with organizational capacity, yet this association was overpowered by the negative influence of turnover of a key staff position. Regression analysis indicated that staff turnover explained 23% of local variability in organizational capacity. Findings reinforce the need for adequate investment and retention of key staff positions in complex partnership programs. Better accounting of public health investments, including monetary and in-kind investments, is needed to inform decisions about the amount and duration of public health investments that will lead to effective program implementation.     

A process evaluation of the National Cancer Institute's Data-based Intervention Research program: a study of organizational capacity building

This paper reports on a qualitative process evaluation of the Data-based Intervention Research (DBIR) program, that was funded by the National Cancer Institute (NCI) and operated in 21 states and the District of Columbia. The goal of DBIR was to build a foundation within state health agencies to ensure the translation of cancer control science into practice. NCI's objective reflected the readiness of cancer control research for public health application, the paucity of cancer control activity within public health settings and the recognition that state health agencies could play a critical role in the effective transfer of research results into public health practice. The qualitative process evaluation reported in this paper is based on one case study of four DBIR programs. The present study indicates that the four state health agencies executed the DBIR program with fidelity. Also, the four states offered a balanced assessment of NCI's role in enabling the state agency operation of DBIR, providing numerous citations illustrating how NCI successfully facilitated organizational capacity as compared to fewer mentions of ways NCI was less than successful. Thus, in funding the DBIR model, NCI was successful in raising state health agency capacity to implement cancer prevention and control programming. Implications for capacity building in state health departments are discussed.     

Process evaluation of a three-year community-based peer education intervention for HIV prevention among Yemeni young people

This community process evaluation highlights key enabling factors that facilitated the implementation of a community peer education program for youth HIV prevention in four poor and vulnerable areas of Aden, Yemen. It also explains the implementation process and provides a deeper understanding of the impact evaluation outcomes, which revealed improved HIV knowledge and risk perception and decreased stigma and risky behavior. This process evaluation was based on qualitative methodologies, where five focus group discussions and 15 in-depth interviews were conducted among 52 participants: community peer educators, community focal points, targeted young people, and local councils. The results revealed that contributing factors to implementing the peer education program had been community participation, mobilization of targeted communities, and capacity building of all those included in the intervention. Existing community-based organizations like the Social Service Centre played a key role in building trust with the targeted communities and linking HIV peer education intervention to other existing community services. This process evaluation could provide lessons learned for replicating similar youth peer education programs in conservative communities.     

Challenges in Evaluating Rural Health Programs

Complex community-based prevention programs are being held to scientific evidence of their effectiveness and rural public health departments that implement such programs often are not equipped to evaluate them. Rural public health departments are fettered by small budgets, small staffs, and less access to evaluation experts and similar resources. Community-based health promotion programs can include complex designs that may work differently in rural areas and evaluation of rural programs can be hampered by lack of control groups and the instability of results from small populations. The University of Kentucky has entered into a contract with the state Department for Public Health to implement an internal, participatory model of evaluation. In this model, the university evaluation expert trains local public health department staff in technical skills for program evaluation and acts as mentor and technical consultant to local public health departments on an ongoing basis. Through training and site visits, this model is one approach to addressing the challenges of evaluating rural health promotion programs.     

A policy analysis of the introduction and dissemination of external peer review (visitatie) as a means of professional self-regulation amongst medical specialists in The Netherlands in the period 1985-2000

By examining the introduction and dissemination of external peer review through site-visits (visitatie) amongst Dutch medical specialists, this paper sets out to deepen our insight into the dynamics of professional self-regulation and health care policy making. We explore how visitatie has been used in the political process between medical specialists and the state, serving as a strategy in protecting the autonomy of physicians. In the late eighties and early nineties, factors both internal as well as external to the medical profession all together determined the start and spread of visitatie. The conflict between state and doctors over the specialists' income, the introduction of the market oriented policies, new visions on quality assurance, the debate on the future of medical specialistic care and a new legal framework on quality assurance, challenged the medical community to find ways to reconfirm the public's trust in the self-regulating mechanism of the profession. One answer is found in carrying out 300-400 visitaties annually. During the past years, many stakeholders have perceived visitatie as a credible instrument in assuring quality patient care. The dynamics of professionalization and measurable impact of visitatie will determine whether or not it is here to stay.     

Developing a physical activity education curriculum for adults with intellectual disabilities

Adults with intellectual disabilities have high rates of physical inactivity and related chronic diseases. Researchers have called for an increase in the development and evaluation of health education programs adapted to the unique needs of this population. Formative and process evaluation strategies were applied to develop a physical activity education program. The first phase of formative evaluation included a comprehensive literature review to select educational strategies and curriculum content. The theory of planned behavior was selected as a guiding framework, and meetings with stakeholders were held to assess feasibility. The second phase of formative evaluation included an assessment of materials by an expert panel and the priority population, and pilot testing. Next, field testing was implemented, followed by process evaluation and an assessment of implementation fidelity. The final curriculum was developed as a result of the completion of the aforementioned steps and led to a successful physical activity intervention.     

A community-centered model of the academic health department and implications for assessment.

The academic health department, also referred to as the teaching health department, is receiving increased attention as an organizational structure to enhance public health workforce development. Traditionally, academic institutions have been viewed by major funders, and reported in the literature, to be the innovators of these partnerships, in part to extend the academic institution's education role in the workplace. However, the role of the local health department as innovator is emerging with implications beyond workforce development. This report illustrates how a local agency-initiated community-centered approach to an academic health department enhances the core public health function of assessment, a frequently underdeveloped function at the local level. This agency-initiated, community-centered academic health department model builds assessment capacity through a partnership that integrates academic research and public health surveillance capacity to provide comprehensive assessment, including community assessment, community-based participatory research, data analysis, and program evaluation. This organizational structure, focused on systems approaches to building community capacity rather than focusing on disease categories or high-risk populations, illustrates how a local health department can substantially enhance its assessment capacity using available resources.     

Factors Influencing Implementation of a Physical Activity Intervention in Residential Children’s Homes

The Environmental Intervention in Children’s Homes (ENRICH) study was the first published physical activity intervention undertaken in residential children’s homes (RCHs). The study revealed differences in implementation across the homes, which may be a key factor that affects program effectiveness. The purpose of this study was to examine the direct and indirect effects of organizational capacity, provider characteristics, and quality of prevention support system on level of implementation of the ENRICH intervention. This study analyzed the ENRICH process evaluation data collected from 24 RCHs. Bayesian Path analysis was used to examine the direct and indirect effects of organizational capacity, provider characteristics, and quality of prevention support system on level of implementation. Level of implementation across RCHs was variable, ranging from 38 to 97 % (M?=?68.3, SD?=?14.45). Results revealed that organizational capacity and provider characteristics had significant direct associations with level of implementation. Neither direct nor indirect associations between quality of prevention support system and level of implementation reached statistical significance. Conducting formative assessments on organizational capacity and provider characteristics and incorporating such information in implementation planning may increase the likelihood of achieving higher levels of implementation in future studies.     

Reconsidering community-based health promotion: promise,performance, and potential

Contemporary public health emphasizes a community-based approach to health promotion and disease prevention. The evidence from the past 20 years indicates, however, that many community-based programs have had only modest impact, with the notable exception of a number of HIV prevention programs. To better understand the reasons for these outcomes, we conducted a systematic literature review of 32 community-based prevention programs. Reasons for poor performance include methodological challenges to study design and evaluation, concurrent secular trends, smaller-than-expected effect sizes, limitations of the interventions, and limitations of theories used. The effectiveness of HIV programs appears to be related in part to extensive formative research and an emphasis on changing social norms.     

A collaborative approach to program evaluation of community-based teen pregnancy prevention projects

The purpose of this article is to demonstrate a model for collaboration between program providers and program evaluators. The article describes how university-based evaluators, a state health department, and local program providers collaborated to evaluate 12 projects implementing commercially developed teenage pregnancy prevention (TPP) programs in school settings. Approximately 2,200 students participate annually in the programs. Program evaluation staff and local program providers worked together to construct logic models that helped guide the intervention and evaluation design. The local providers also participated in training sessions, conducted by the evaluation team, to increase their understanding and skills related to program evaluation methods. Student-level outcomes related to knowledge, attitudes, skills, behaviors, as well as an assessment of curricula fidelity were included in the evaluation. The result of this collaborative model has been a quality program evaluation for the projects while maintaining community input regarding program improvements that reflect local population needs.     

Evaluation of implementation of a cholesterol management program in physicians' offices

This article describes an evaluation of the implementation of a cholesterol management program in family physicians' offices as part of the Physician-Based Nutrition Program to Lower Coronary Heart Disease Risk (PBNP). The evaluation, conducted through a partnership evaluation model, used multiple case study methodology and combined the use of quantitative and qualitative methods. Data sources included office staff reports and interviews, records of contacts with study personnel, patient care data, and patient telephone interviews. Data from these sources revealed gradual program implementation and considerable variation in practitioner and clinic involvement in cholesterol management. Clinic staff reported that the support provided by PBNP in the form of training, operations materials, patient education materials and ongoing assistance was very useful. This formative evaluation has implications for refinement of the PBNP and for other prevention programs in primary care settings. It demonstrates the feasibility and acceptability of a systems approach to physicians, cholesterol/nutrition educators and clinic support staff. It also suggests ways in which researchers and clinicians can implement and evaluate health care innovations.     

艾滋病/性病/安全性行为同伴教育项目评价设计

本文简述了同伴教育的涵义及特征：提出了在１９９８ 年９ － １２ 月,于北京某医科和某工科大学一年级学生中开展艾滋病／ 性病／ 安全性行为同伴教育项目评价的设计问题,内容包括形成评价、制定计划方案、同伴教育者的征募、培训评价、项目评价等,以保证该项目的合理性和有效性,并获得成功。