Factors affecting the utilization of systematic reviews. A study of public health decision makers.

OBJECTIVE: To determine the extent to which public health decision makers used five systematic reviews to make policy decisions, and to determine which characteristics predict their use. METHODS: This cross-sectional follow-up study of public health decision makers in Ontario collected primary data using a telephone survey and a short, self-administered organizational demographics questionnaire completed by the administrative assistant for each Medical Officer of Health. Independent variables included characteristics of the innovation, organization, environment, and individual. Data were entered into a computerized database developed specifically for this study, and multiple logistic regression analysis was conducted. RESULTS: The participation rate was very high, with 85% of public health units and 96% of available decision makers completing the survey. In addition, 63% of respondents stated they had used at least one of the systematic reviews in the previous 2 years to make a decision. The most important predictors of use were one's position, expecting to use a review in the future, and perceptions that the reviews were easy to use and that they overcame the barrier of limited critical appraisal skills. CONCLUSIONS: Utilization of the systematic reviews in Ontario was very high. The utilization rates found in this study were significantly higher than those reported in previous utilization studies. One's position was found to be the strongest predictor of use, identifying program managers and directors as the most appropriate audience for systematic reviews.     

Creating an organizational culture for evidence-informed decision making

A public health department in Ontario, Canada, set a 10-year strategic direction for evidence-informed decision making, defined as the systematic application of research evidence to program decisions. The multifaceted approach has identified eight key lessons for leadership, funding, infrastructure, staff development, partnerships, and change management. Results after 4 years include systematic and transparent application of research to >15 program decisions and, increasingly, evidence-informed decision making as a cultural norm.     

A knowledge management tool for public health: health-evidence.ca

Background

The ultimate goal of knowledge translation and exchange (KTE) activities is to facilitate incorporation of research knowledge into program and policy development decision making. Evidence-informed decision making involves translation of the best available evidence from a systematically collected, appraised, and analyzed body of knowledge. Knowledge management (KM) is emerging as a key factor contributing to the realization of evidence-informed public health decision making. The goal of health-evidence.ca is to promote evidence-informed public health decision making through facilitation of decision maker access to, retrieval, and use of the best available synthesized research evidence evaluating the effectiveness of public health interventions.

Methods

The systematic reviews that populate health evidence.ca are identified through an extensive search (1985-present) of 7 electronic databases: MEDLINE, EMBASE, CINAHL, PsycINFO, Sociological Abstracts, BIOSIS, and SportDiscus; handsearching of over 20 journals; and reference list searches of all relevant reviews. Reviews are assessed for relevance and quality by two independent reviewers. Commonly-used public health terms are used to assign key words to each review, and project staff members compose short summaries highlighting results and implications for policy and practice.

Results

As of June 2010, there are 1913 reviews in the health-evidence.ca registry in 21 public health and health promotion topic areas. Of these, 78% have been assessed as being of strong or moderate methodological quality. Health-evidence.ca receives approximately 35,000 visits per year, 20,596 of which are unique visitors, representing approximately 100 visits per day. Just under half of all visitors return to the site, with the average user spending six minutes and visiting seven pages per visit. Public health nurses, program managers, health promotion workers, researchers, and program coordinators are among the largest groups of registered users, followed by librarians, dieticians, medical officers of health, and nutritionists. The majority of users (67%) access the website from direct traffic (e.g., have the health-evidence.ca webpage bookmarked, or type it directly into their browser).

Conclusions

Consistent use of health-evidence.ca and particularly the searching for reviews that correspond with current public health priorities illustrates that health-evidence.ca may be playing an important role in achieving evidence-informed public health decision making.     

Systematic review to inform prevention and management of chronic disease for Indigenous Australians: overview and priorities

Objective: To describe the main characteristics of systematic reviews addressing questions of chronic disease and related risk factors for Indigenous Australians. Methods: We searched databases for systematic reviews meeting inclusion criteria. Two reviewers assessed quality and extracted characteristics using pre‐defined tools. Results: We identified 14 systematic reviews. Seven synthesised evidence about health intervention effectiveness; four addressed chronic disease or risk factor prevalence; and six conducted critical appraisal as per current best practice. Only three reported steps to align the review with standards for ethical research with Indigenous Australians and/or capture Indigenous‐specific knowledge. Most called for more high‐quality research. Conclusion: Systematic review is an under‐utilised method for gathering evidence to inform chronic disease prevention and management for Indigenous Australians. Relevance of future systematic reviews could be improved by: 1) aligning questions with community priorities as well as decision maker needs; 2) involvement of, and leadership by, Indigenous researchers with relevant cultural and contextual knowledge; iii) use of critical appraisal tools that include traditional risk of bias assessment criteria and criteria that reflect Indigenous standards of appropriate research. Implications: Systematic review method guidance, tools and reporting standards are required to ensure alignment with ethical obligations and promote rigor and relevance.     

Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions

U.S. policymakers are making greater use of findings from systematic reviews, the principal product of the discipline of research synthesis. This paper summarizes the methodology and availability of systematic reviews and the brief history of their introduction to policymakers in the public and private sectors and health professionals in the United States. Then, as a case study, the paper describes how officials in a consortium of states are using systematic reviews to inform decisions about coverage for pharmaceuticals. Finally, it explores the prospects for wider use of systematic reviews by policymakers.     

Global priority setting for Cochrane systematic reviews of health promotion and public health research

BACKGROUND: Systematic reviews of health promotion and public health interventions are increasingly being conducted to assist public policy decision making. Many intra-country initiatives have been established to conduct systematic reviews in their relevant public health areas. The Cochrane Collaboration, an international organisation established to conduct and publish systematic reviews of healthcare interventions, is committed to high quality reviews that are regularly updated, published electronically, and meeting the needs of the consumers. AIMS: To identify global priorities for Cochrane systematic reviews of public health topics. METHODS: Systematic reviews of public health interventions were identified and mapped against global health risks. Global health organisations were engaged and nominated policy-urgent titles, evidence based selection criteria were applied to set priorities. RESULTS: 26 priority systematic review titles were identified, addressing interventions such as community building activities, pre-natal and early infancy psychosocial outcomes, and improving the nutrition status of refugee and displaced populations. DISCUSSION: The 26 priority titles provide an opportunity for potential reviewers and indeed, the Cochrane Collaboration as a whole, to address the previously unmet needs of global health policy and research agencies.     

Judicial intervention in alcohol regulation: an empirical legal analysis

Objective: While governments draft law and policy to promote public health, it is through cases put before the judiciary that the implementation of law can be challenged and where its practical implications are typically determined. In this paper, we examine the role of court judgements on efforts in Australia to regulate the harmful use of alcohol. Methods: Australian case law (2010 to June 2015) involving the judicial review of administrative decisions relating to development applications or liquor licences for retail liquor outlets (bottle shops), hotels, pubs and clubs was identified using a case law database (WestLaw AU). Data were extracted and analysed using standard systematic review techniques. Results: A total of 44 cases were included in the analysis. Of these, 90% involved appeals brought by industry actors against local or state government stakeholders seeking to reject applications for development applications and liquor licences. The proportion of judicial decisions resulting in outcomes in favour of industry was 77%. Conclusions: Public health research evidence appeared to have little or no influence, as there is no requirement for legislation to consider public health benefit. Implications for public health: A requirement that the impact on public health is considered in legislation will help to offset its strong pro‐competition emphasis, which in turn has strongly influenced judicial decision making in this area.     

Opinion polling and decision making: a critical appraisal of quality of life assessment

The relationship between quality of life (QOL) assessments and decision making, in relation to the delivery of health services, is subjected to critical appraisal. Three levels of decision making in the health care system are taken into account in the analysis. Criticisms of opinion polling provide the basis for the appraisal. Examples of criticisms considered are: Might the use of QOL information be manipulative? Could the interviews or questionnaires used to obtain QOL data influence personal opinions? Are the methods used sometimes defective and/or superficial? Will QOL information always be used in decision making in ways that are ascertainable and justifiable? It is concluded that the time has come for the main focus of critical appraisal in QOL research to shift, from an emphasis on evaluation of the quality of methods used for assessments of QOL, toward an emphasis on the practical usefulness of QOL data.     

Evidence and ethics in public health: the experience of SARS in Canada

Making decisions on the basis of evidence is a central tenet of all health-care disciplines, including public health. However, it is not entirely clear what it means to base decisions on evidence; debates on evidence-based approaches often lack a clear understanding of the nature of evidence and obscure the normative underpinnings of evidence. Public health decision making requires an acceptance of limitations such as the availability of funding for research to provide complete evidence for any given decision, the ethical constraints on the creation of certain types of evidence and the ongoing dilemma between the need to take action and the need to gather more information. Using the example of the SARS outbreak in Canada, the inter-relationships between evidence and ethics are explored. I outline a set of critical questions for the global public health community to discuss regarding the nature of the relationship between evidence-based public health practice and ethics.     

Evaluating the effectiveness of public health interventions: the role and activities of the Cochrane Collaboration

Public health decision makers, funders, practitioners, and the public are increasingly interested in the evidence that underpins public health decision making. Decisions in public health cover a vast range of activities. With the ever increasing global volume of primary research, knowledge and changes in thinking and approaches, quality systematic reviews of all the available research that is relevant to a particular practice or policy decision are an efficient way to synthesise and utilise research efforts. The Cochrane Collaboration includes an organised entity that aims to increase the quality and quantity of public health systematic reviews, through a range of activities. This paper aims to provide a glossary of the terms and activities related to public health and the Cochrane Collaboration.     

People and money matter: investment lessons from the Ontario Heart Health Program, Canada

Resource allocation is a critical issue for public health decision-makers. Yet little is known about the level and type of resources needed to build capacity to plan and implement comprehensive programs. This paper examines the relationships between investments and changes in organizational capacity and program implementation in the first phase (1998-2003) of the Ontario Heart Health Program (OHHP)--a province-wide, comprehensive public health program that involved 40 community partnerships. The study represents a subset of findings from a provincial evaluation. Investments, organizational capacity of public health units and implementation of heart health activities were measured longitudinally. Investment information was gathered annually from the provincial government, local public health units and community partners using standard reports, and was available from 1998 to 2002. Organizational capacity and program implementation were measured using a written survey, completed by all health units at five measurement times from 1994 to 2002. Combining provincial and local sources, the average total investment by year five was $1.66 per capita. Organizational capacity of public health units and implementation of heart health activities increased both before and during the first 2 years of the OHHP, and then plateaued at a modest level for capacity and a low level for implementation after that. Amount of funding was positively associated with organizational capacity, yet this association was overpowered by the negative influence of turnover of a key staff position. Regression analysis indicated that staff turnover explained 23% of local variability in organizational capacity. Findings reinforce the need for adequate investment and retention of key staff positions in complex partnership programs. Better accounting of public health investments, including monetary and in-kind investments, is needed to inform decisions about the amount and duration of public health investments that will lead to effective program implementation.     

Making Good Decisions in Healthcare with Multi-Criteria Decision Analysis: The Use,Current Research and Future Development of MCDA

Healthcare decision making is usually characterized by a low degree of transparency. The demand for transparent decision processes can be fulfilled only when assessment, appraisal and decisions about health technologies are performed under a systematic construct of benefit assessment. The benefit of an intervention is often multidimensional and, thus, must be represented by several decision criteria. Complex decision problems require an assessment and appraisal of various criteria; therefore, a decision process that systematically identifies the best available alternative and enables an optimal and transparent decision is needed. For that reason, decision criteria must be weighted and goal achievement must be scored for all alternatives. Methods of multi-criteria decision analysis (MCDA) are available to analyse and appraise multiple clinical endpoints and structure complex decision problems in healthcare decision making. By means of MCDA, value judgments, priorities and preferences of patients, insurees and experts can be integrated systematically and transparently into the decision-making process. This article describes the MCDA framework and identifies potential areas where MCDA can be of use (e.g. approval, guidelines and reimbursement/pricing of health technologies). A literature search was performed to identify current research in healthcare. The results showed that healthcare decision making is addressing the problem of multiple decision criteria and is focusing on the future development and use of techniques to weight and score different decision criteria. This article emphasizes the use and future benefit of MCDA.     

What health care services does the public want and who should decide? Ask them!

Most of the parties involved in healthcare decisions – governments, politicians, healthcare professionals, pharmaceutical companies, special interest groups – actively work to make their desires known. In Israel the public is part of the decision committee; in Germany health care decision are made more or less without the public being involved. In a recently published IJHPR article, Giora Kaplan and Orna Baron-Epel raise the question of how well acquainted senior decision makers in the Israeli health system are with the public’s priorities regarding the services being considered for inclusion in the public funding list. This commentary speculates about the reasons for the discrepancies found in that article between the decision makers’ and the public’s view. Furthermore, it reports on survey results from Germany about who should be part of the decision making committee and briefly touches upon the situation in other OECD countries. While public opinion may not be the determining factor, all authors advocate a strengthening of the public’s contribution to the health care decision making process, including steps to make decision makers aware of public priorities on an ongoing basis.     

Criteria for the systematic review of health promotion and public health interventions

Systematic reviews of public health interventions are fraught with challenges. Complexity is inherent; this may be due to multi-component interventions, diverse study populations, multiple outcomes measured, mixed study designs utilized and the effect of context on intervention design, implementation and effectiveness. For policy makers and practitioners to use systematic reviews to implement effective public health programmes, systematic reviews must include this information, which seeks to answer the questions posed by decision makers, including recipients of programmes. This necessitates expanding the traditional evaluation of evidence to incorporate the assessment of theory, integrity of interventions, context and sustainability of the interventions and outcomes. Unfortunately however, the critical information required for judging both the quality of a public health intervention and whether or not an intervention is worthwhile or replicable is missing from most public health intervention studies. When the raw material is not available in primary studies the systematic review process becomes even more challenging. Systematic reviews, which highlight these critical gaps, may act to encourage better reporting in primary studies. This paper provides recommendations to reviewers on the issues to address within a public health systematic review and, indirectly, provides advice to researchers on the reporting requirements of primary studies for the production of high quality systematic reviews.     

A survey of the decision-making needs of Canadians faced with complex health decisions

Objective To describe the decision‐making needs of Canadians when faced with ‘complex’ health decisions characterized by balancing advantages against disadvantages. Although a national report emphasized that public confidence in the health‐care system depends on support for personal knowledge and decision‐making, there has been no systematic investigation of the Canadian population's decision‐making needs. Design Cross‐sectional telephone survey using random digit dialling. Participants National sample of 635 adults over 18 years of age, living in Canada. Results Forty‐two percentage of eligible contacts participated. Sixty‐five percent of contacts reported making ‘complex’ health decisions, commonly about medical or surgical treatments or birth control, and more commonly by women and by married/separated individuals. Most respondents took an active role in their decisions, often sharing the process with their partner or family. Being younger was associated with a more independent role. Physicians were more often involved in the decisions of respondents with less education. Fifty‐nine percent of respondents experienced decisional conflict; more conflict was seen with those who were female and feeling uninformed about options, pressured to select one particular option, and unready or unskilled in decision‐making. Less decisional conflict was seen in those who reported birth control decisions and in those who were 70 years and older. Participants used several strategies when deliberating about choices including: information gathering, clarifying their values, and seeking support and information from others. Personal counselling and printed information materials were commonly preferred methods of learning about options. ‘Essential’ criteria for judging satisfactory decision‐making included: having sufficient knowledge about the options, outcomes, and probabilities; being clear about values; selecting and implementing a choice that agrees with personal values; and expressing satisfaction with the choice. Conclusions Canadians, particularly women, face difficult decisions and need support and information from credible sources.     

Understanding the role of contextual influences on local health-care decision making: case study results from Ontario, Canada

Approaches to involving the public in local health care decision making processes (and analyses of these approaches) have tended to treat participation and publics uniformly in search of the ideal method of involving the public or providing the same opportunities for public participation regardless of differing socio-economic, cultural, insitutional or political contexts within which decisions are made. Less attention has been given to the potential for various contextual factors to influence both the methods employed and the outcomes of such community decision-making processes. The paper explores the role that context (three sets of contextual influences more specifically) plays in shaping community decision-making processes. Results from case studies of public participation in local health-care decision making in four geographic communities in Ontario are presented. During the study period, two of these communities were actively involved in health services restructuring processes while one had recently completed its process and the fourth had not yet engaged in one. Several themes emerge from the case studies regarding the identification and role of contextual influences in differentially shaping participation in local health care decision-making. These include the propensity for communities with different social and structural attributes to engage in different "styles" of participation; the importance attached to "community values" in shaping both the qualitative and quantitative aspects of participation: the role of health councils, local government and inter-organizational collaboration as participation "enablers"; and the politicization of participation that occurs around contentious issues such as hospital closures.     

Applicability and transferability of interventions in evidence-based public health

The context in which public health programmes operate can play an important role in influencing their implementation and effectiveness. An intervention that has been shown to be effective in one setting may turn out to be ineffective somewhere else, even supposing it can be implemented there. Therefore, systematic reviews of public health interventions should appraise the applicability of the intervention process and the transferability of the intervention effectiveness to other localities. However, applicability and transferability appraisal is seldom reported in systematic reviews of public health and health promotion interventions. This paper aims to introduce an innovative approach to bridging this gap. A list of attributes that may impact on applicability and transferability can be developed, based on knowledge of the proposed intervention. Then the applicability and transferability of the intervention to the local setting can be rated, and given a score, based on knowledge of the local setting. This approach provides a useful tool for evaluating public health interventions and provides a reliable basis for informed decision making in resource-poor settings, where rigorous primary studies are lacking and where very limited resources put a high demand on evidence-based approaches to health promotion.     

What factors influence case managers' resource allocation decisions? A systematic review of the literature.

OBJECTIVE: Case managers' decisions directly affect the amount and type of services individual clients receive, as well as overall home care program available resources. We know little about the resource allocation decision-making processes of case managers. The question guiding this review was, "What factors influence case managers' resource allocation decisions in home care?' METHODS: The authors did a systematic literature review to answer the above question. After assessing the articles for inclusion, they assessed the quality (internal validity) of each included study. They described the characteristics of the studies and provided a synthesis of the findings of the primary studies. RESULTS: Five qualitative and 6 quantitative articles met the inclusion criteria for this review. The findings of these studies are equivocal. Despite this, the authors were able to create a preliminary taxonomy of the factors that influence case manager resource allocation decisions. Despite evidence-based decision making receiving so much attention in contemporary health care literature, the authors found a near absence of reference to research use in the context of case manager decision making. CONCLUSIONS: Currently, there are relatively few studies in the literature on the factors that influence, and how they are used in, case manager resource allocation decisions. Studies are often lacking in terms of conceptual clarity and theoretical framing. They are often not guided by theoretical frameworks and are not situated within the larger field of decision making or even within the clinical decision-making literature. These issues are impeding progress in this area.