Methods for evaluation of small scale quality improvement projects

Evaluation is an integral component of quality improvement and there is much to be learned from the evaluation of small scale quality improvement initiatives at a local level. This type of evaluation is useful for a number of different reasons including monitoring the impact of local projects, identifying and dealing with issues as they arise within a project, comparing local projects to draw lessons, and collecting more detailed information as part of a bigger evaluation project. Focused audits and developmental studies can be used for evaluation within projects, while methods such as multiple case studies and process evaluations can be used to draw generalised lessons from local experiences and to provide examples of successful projects. Evaluations of small scale quality improvement projects help those involved in improvement initiatives to optimise their choice of interventions and use of resources. Important information to add to the knowledge base of quality improvement in health care can be derived by undertaking formal evaluation of local projects, particularly in relation to building theory around the processes of implementation and increasing understanding of the complex change processes involved.     

Can they get along without us? Sustainability of donor-supported health projects in Central America and Africa

This article presents a synthesis of five country studies of the sustainability of U.S. government-funded health projects in Central America and Africa. The studies reviewed health projects with a comparative framework to determine which project activities had continued after the donor funding ceased. This review found that health projects in Africa were less firmly sustained than those in Central America. The studies then evaluated context factors and project characteristics that were related to the sustainability of the projects. The weak economic and political context of the African cases was found to inhibit sustainability in those countries, suggesting that broader development issues be addressed before donors expect significant sustainability of health projects in Africa. Even in Central America it was found that the strength of the institution implementing the project was an important variable for sustainability, suggesting that donor attention also be shifted toward strengthening institutional development in order to assure sustainability. In addition to context factors, several project characteristics were related to sustainability in most cases and suggest sustainability guidelines for project design and implementation. The article concludes that projects should be designed and managed so as to: (1) demonstrate effectiveness in reaching clearly defined goals and objectives; (2) integrate their activities fully into established administrative structures; (3) gain significant levels of funding from national sources (budgetary and cost-recovery) during the life of the project; (4) negotiate project design with a mutually respectful process of give and take: and (5) include a strong training component.     

Intermediate Outcomes,Strategies, and Challenges of Eight Healthy Start Projects

Site visits were conducted for the evaluation of the national Healthy Start program to gain an understanding of how projects design and implement five service components (outreach, case management, health education, depression screening and interconceptional care) and four system components (consortium, coordination/collaboration, local health system action plan and sustainability) as well as program staff’s perceptions of these components’ influence on intermediate outcomes. Interviews with project directors, case managers, local evaluators, clinicians, consortium members, outreach/lay workers and other stakeholders were conducted during 3-day in-depth site visits with eight Healthy Start grantees. Grantees reported that both services and systems components were related to self-reported service achievements (e.g. earlier entry into prenatal care) and systems achievements (e.g. consumer involvement). Outreach, case management, and health education were perceived as the service components that contributed most to their achievements while consortia was perceived as the most influential systems component in reaching their goals. Furthermore, cultural competence and community voice were overarching project components that addressed racial/ethnic disparities. Finally, there was great variability across sites regarding the challenges they faced, with poor service availability and limited funding the two most frequently reported. Service provision and systems development are both critical for successful Healthy Start projects to achieve intermediate program outcomes. Unique contextual and community issues influence Healthy Start project design, implementation and reported accomplishments. All eight projects implement the required program components yet outreach, case management, and health education are cited most frequently for contributing to their perceived achievements.     

A model for evaluating the sustainability of community-directed treatment with ivermectin in the African Program for Onchocerciasis Control

Onchocerciasis is controlled by mass treatment of at‐risk populations with ivermectin. Ivermectin is delivered through community‐directed treatment (CDTI) approach. A model has been developed to evaluate the sustainability of the approach and has been tested at 35 projects in 10 countries of the African Program for Onchocerciasis Control (APOC). It incorporates quantitative and qualitative data collection and analysis, taking account of two factors identified as crucial to project sustainability. These are (i) the provision of project performance information to partners, and (ii) evidence‐based support for project implementation. The model is designed to provide critical indicators of project performance of the model to implementing, coordinating, and funding partners. The model's participatory and flexible nature makes it culturally sensitive and usable by project management. This model is able to analyze the different levels involved in project implementation and arrive at a judgment for the whole project. It has inbuilt mechanisms for ensuring data reliability and validity. The model addresses the complex issue of sustainability with a cross‐sectional design focusing on how and at which operational level of implementation to strengthen a CDTI project. The unique attributes and limitations of the model for evaluating the sustainability of projects were described. Copyright © 2012 John Wiley & Sons, Ltd.     

Successfully sustaining sex and gender issues in undergraduate medical education: a case study

Although several projects have addressed the importance of gender health issues in medical education, the sustainability of change efforts in medical education has rarely been addressed. Understanding the possible facilitators or barriers to sustainability may help to develop future interventions that are effective in maintaining gender health issues as a topic in medical curricula. The aim of this study is to provide a longitudinal evaluation of changes regarding gender health issues that occurred in the past decade and the factors that influenced this process. The coursebooks of eight theoretical courses of the Nijmegen medical curriculum were screened on the basis of criteria for an integrated gender perspective in medical education. To assess the sustainability of gender health issues, the screening results from 2014 were compared with those of a similar project in 2005. In addition, open interviews were conducted with eight coordinators to identify facilitators and barriers influencing the sustainability of gender health issues. Analysis showed that, over the past decade, the implementation of gender health issues was mainly sustained and additional changes were made, resulting in an ongoing gender perspective in the Nijmegen medical curriculum. The coordinators mentioned several factors that influenced the sustainability of implementation in medical education: coordinators’ and teachers’ gender-sensitive attitude, competing demands, the presence of sex and gender in learning objectives, examinations and evaluation, organizational support and curriculum revisions. Our findings suggest that, in implementing sex and gender in medical education, medical faculties need to focus on top-down support in incorporating sex and gender into core objectives and time spent on incorporating sex and gender into medicine, and on the continuous training of teaching staff.     

Enhancing community participation in intervention research: farmworkers and agricultural chemicals in North Carolina.

The participation of affected communities in the development of public health intervention research improves project sustainability and effectiveness by making projects more relevant and acceptable to the communities. This article presents a multimode, multidomain model approach for community participation in different project components, which ensures the benefits of participation without requiring the same level of participation in every activity or by every community sector. A case study is used to illustrate the model, describing procedures for establishing and maintaining farmworker participation in developing an intervention to reduce exposure to chemicals. Farmworkers are a poor and underserved population for which the empowering and culturally appropriate benefits of community participation are especially needed. However, this population presents challenges for participatory health projects: geographic dispersion, ethnic diversity, lack of organization, sense of powerlessness, and communication and transportation difficulties. The lessons learned in this case extend the method and theory of community participation.     

Powering the future: how Hamilton Health Sciences put cogeneration to work for healthcare

The absolute necessity of a stable and uninterrupted power supply within hospitals makes many of these facilities uniquely suited to cogeneration plants. Hamilton Health Sciences recently completed the largest hospital cogeneration project ever undertaken in the country. Spanning three acute care hospitals and generating a combined total of 22.75 megawatts of electricity, Hamilton Health Sciences' cogeneration plants address energy supply issues by offering a clean and reliable power source completely within the hospital's control, and provide the organization with the potential to generate its own revenue into the future by selling excess electricity back to the province. The following article highlights Hamilton Health Sciences' approach to the project, including some important lessons learned, and may serve as an example for other publicly funded institutions interested in implementing similar projects.     

Implementing computer information systems for hospital-based case management

Like all health care services, case management is a process that relies on information. Based on the experiences in implementing computer information systems in six hospital-based case management programs, several financial, technical, and management issues are reviewed. These issues, which are also relevant for other specialized hospital-based programs, include information priorities, user acceptance, quantifying data, data entry methods, data security, and systems integration. The lessons learned regarding these issues are discussed, and categories of software alternatives are presented.     

Implementation of MET/CBT 5 for Adolescents

Implementation of effective substance abuse treatment programs in community settings is a high priority. The selection of a proven cost-effective model is a first step; however, difficulty arises when the model is imported into a community setting. The Center on Substance Abuse Treatment selected a brief substance abuse treatment program for adolescents, the MET/CBT-5 program, determined to be the most cost-effective protocol in the Cannabis Youth Treatment trial, for implementation in two cohorts of Effective Adolescent Treatment grantees. A qualitative investigation of the protocol implementation with nine sites in the second cohort chronicled adaptations made by grantees and prospects for sustainability. The study found that agencies introduced adaptations without seeming to be aware of potential effects on validity. In most sites, sessions were lengthened or added to accommodate individual client needs, address barriers to client participation, and provide consistency with current norms of treatment. Implications for fidelity of future implementation projects are addressed.     

Lessons Learned and Hypotheses Generated from the W. K. Kellogg Agricultural Safety and Health Initiative Cluster Evaluation

ABSTRACT

The W. K. Kellogg Agricultural Safety and Health (ASH) initiative represented a major effort by a philanthropic organization to address the continuing high prevalence of injuries and illnesses within the agricultural sector. A major emphasis within the initiative was an overall evaluation of the entire group. This overall evaluation discovered eighteen general “lessons learned” which represented common patterns and characteristics seen across all projects. The projects sought to further this evaluation process by offering specific research hypotheses, based upon the “lessons learned,'' as those lessons applied to their individual projects. These specific research hypotheses illustrated opportunities available for cooperative endeavors. This paper used experiences of the University of Illinois Cooperative Extension Service project as an example.     

Another lesson from Santa Barbara

In this commentary, Don Holmquest, speaking from the perspective of a statewide regional health information organization (RHIO), responds to papers by Robert Miller and Bradley Miller and by David Brailer on lessons learned from the Santa Barbara County Care Data Exchange project and its subsequent demise. He posits that one of the critical lessons learned through the Santa Barbara experience is the need for a careful analysis of who benefits from health information exchange as part of creating a sustainable business model for health information exchange projects.     

International development through educational linkages. I: A guide to a successful outcome

Educational linkages are an important source of foreign aid to Asian countries. In this first of two articles, the authors' experiences with such a project form the background for a discussion of the factors that contributed to the outcome of the project: partner selection, strategic planning, balancing flexibility and accountability, and coping with differences in culture and values. Key concepts of ‘international development’ and ‘sustainability’ are introduced. In the second article, the focus is on curriculum development and implementing educational standards of the World Federation of Occupational Therapy (WFOT). The information and experiences shared in these articles may be useful to those embarking on similar projects.     

Health Care Reform in Massachusetts: Implementation of Coverage Expansions and a Health Insurance Mandate

Context: Much can be learned from Massachusetts's experience implementing health insurance coverage expansions and an individual health insurance mandate. While achieving political consensus on reform is difficult, implementation can be equally or even more challenging.
Methods: The data in this article are based on a case study of Massachusetts, including interviews with key stakeholders, state government, and Commonwealth Health Insurance Connector Authority officials during the first three years of the program and a detailed analysis of primary and secondary documents.
Findings: Coverage expansion and an individual mandate led Massachusetts to define affordability standards, establish a minimum level of insurance coverage, adopt insurance market reforms, and institute incentives and penalties to encourage coverage. Implementation entailed trade-offs between the comprehensiveness of benefits and premium costs, the subsidy levels and affordability, and among the level of mandate penalties, public support, and coverage gains.
Conclusions: National lessons from the Massachusetts experience come not only from the specific decisions made but also from the process of decision making, the need to keep stakeholders engaged, the relationship of decisions to existing programs and regulations, and the interactions among program components.     

Performance of predictors: evaluating sustainability in community-directed treatment projects of the African programme for onchocerciasis control

The predictors of sustainability of community-directed treatment with ivermectin (CDTI) at four implementation levels were evaluated in 41 African Programme for Onchocerciasis Control (APOC) projects, encompassing 492 communities in 10 countries. A model protocol provided information on indicators corresponding to nine aspects of a project that is likely to be sustainable at community level after the cessation of external support. Six of the nine aspects had components of community ownership as predictors of project sustainability. Quantitative and qualitative assessments were used to obtain individual community scores and an overall sustainability score for each project graded on a scale of 0-4. Of the 41 projects evaluated, 70% scored "satisfactorily" to "highly sustainable" at the community level. We found variations among countries and that health system weaknesses could hamper community efforts in sustaining a project, such as when ivermectin was delivered late. Community ownership was of primary importance to the community score, and the community-level scores correlated with overall project sustainability. The therapeutic coverage achieved in each project correlated with the ratio of volunteer ivermectin distributors per population served. Surprisingly, the performance of these distributors was not affected by the direct incentives offered, and coverage appeared to be highest when cash or in-kind compensation was not given at all. Although further research is required, anecdotal evidence pointed to diverse indirect benefits for distributors-political goodwill, personal satisfaction and altruistic fulfillment. The results demonstrate that community ownership is among the important determining factors of sustainability of community-based programmes.     

Improving immunization data quality in Peru and Mexico: Two case studies highlighting challenges and lessons learned

IntroductionThe Global Vaccine Action Plan and the Regional Immunization Action Plan of the Americas call for countries to improve immunization data quality. Immunization information systems, particularly electronic immunization registries (EIRs), can help to facilitate program management and increase coverage. However, little is known about efforts to develop and implement such systems in low- and middle-income countries. We present the experiences of Mexico and Peru in implementing EIRs.MethodsWe conducted case studies of an EIR in Mexico and of a population registry in Peru. Information was gathered from technical documents, stakeholder focus groups, site visits, and semi-structured interviews of national stakeholders. We organized findings into narratives that emphasized challenges and lessons learned.ResultsMexico built one of the world’s first EIRs, incorporating novel features such as local-level tracking of patients; however, insufficient resources and poor data registration practices led to the system’s discontinuation. Peru created an information system to improve affiliation to social programs, including the immunization program and quality of demographic data. Mexico’s experience highlights lessons in failed sustainability of an EIR and a laudable effort to reform a country’s information system. Peru’s demonstrates that attempts to improve health and other data may strengthen health systems, including immunization data. Major challenges in information system implementation and sustainability in Peru and Mexico related to funding, clear governance structures, and resistance among health workers.DiscussionThese case studies reinforce the need for countries to ensure adequate funding, plans for sustainability, and health worker capacity-building activities before implementing EIRs. They also suggest new approaches to implementation, including economic incentives for sub-national administrative levels and opportunities to link efforts to improve immunization data with other health and political priorities. More information on best practices is needed to ensure the successful adoption and sustainability of immunization registries in low- and middle-income countries.     

农村地区培养卫生人才的国际经验与启示

目的:描述国外为农村地区培养卫生人才的政策和项目,分析增加农村地区卫生人力的做法和措施,为我国订单定向免费医学生培养政策提供参考和依据。方法:采用文献综述方法,搜索国内外为农村地区培养卫生人才相关文献48篇,对上述文献进行逻辑分析归纳,系统回顾分析国内外相关策略和做法,总结经验启示。结果:为农村地区培养卫生人才的常见措施包括:招收农村背景学生、导师制医学培养、农村卫生机构实习、经济激励措施、毕业后强制农村服务。对我国的启示包括:充足的财政资金是保证项目顺利实施的先决条件;做好项目管理和评价工作可以提高项目实施效果;选择合适的做法组合实施此类项目可以提高其效果。结论:订单定向免费医学生培养政策适合当前我国国情,同时不少国外经验值得我们借鉴:加大资金投入,加强管理和评价,加强部门合作,进一步完善订单定向医学生培养。     

Programming for safe motherhood: a guide to action

The Safe Motherhood Initiative has successfully stimulated much interest in reducing maternal mortality. To accelerate programme implementation, this paper reviews lessons learned from the experience of industrial countries and from demonstration projects in developing countries, and proposes intervention strategies of policy dialogue, improved services and behavioural change. A typological approach with three hypothetical settings from resource poor to resource rich environments is used to address the variability in health behaviours and infrastructure encountered when programming for safe motherhood.     

Community participation in externally funded health projects: lessons from Cambodia

This article provides lessons learned on establishing effective community participation in two externally funded, NGO-implemented health projects working at district level in Cambodia. The first project was implemented in accordance with the Cambodian national guidelines on community participation. The second - using lessons and experiences gained as a result of the first project - worked with Buddhist pagoda volunteers. Primary research was conducted in both settings to assess the effectiveness of the two participation strategies. The article concludes that the success of community participation in externally funded health projects with relatively short implementation timeframes requires engagement with existing community-based organizations and agencies. In Cambodia, where Theravada Buddhism is the dominant religion, pagodas and associated volunteers appear to represent such an organization. Community participation structured around pagoda volunteers - who are held in high esteem within their local communities - is more effective and sustainable than newly (and externally) established community structures with formally elected representatives. Pagodas and associated volunteers in rural Cambodia offer the advantages of effective leadership, local organization, resource mobilization and management. It is recommended that programmes and agencies wishing to adopt community participation strategies in health utilize participatory research to identify the most appropriate local organization to lead such initiatives.     

Agenda de género en la formación en ciencias de la salud: experiencias internacionales para reducir tiempos en España

ObjectiveTo analyse experiences and lessons learned in the process of incorporating gender perspective into health sciences degrees in the international arena.MethodNarrative literature review with no date limit. Keywords “curricula”, medical education”, “nursing education”, “health science”, “gender education”, “gender medicine” and “gender health issues”. Other articles were snowballed from those identified in the review.ResultsContent on sex-gender appears in learning outcomes and evaluation. Learning based on discussing innovations and assistive applications on gender issues. Resistance: scepticism as a useful academic exercise; lack of time for teacher training and difficulty in finding concrete solutions, due to the diversity of contents; Curriculum overload; the idea of dealing exclusively with women's issues. Difficulties in implementing change policies in centres arising from the difficulty in identifying and presenting gender biases to academic authorities.ConclusionsThe key to gender mainstreaming in university degrees lies at the level of political, organizational and cultural change, as well as at faculty level. There are facilitating factors and individual and interpersonal barriers; the organizational level is essential with institutional support through norms that enforce the incorporation of this perspective and ensure its sustainability.