Systematic reviews and economic evaluations conducted for the National Institute for Health and Clinical Excellence in the United Kingdom: a game of two halves?

BACKGROUND: Decision analytic models, as used in economic evaluations, require data on several clinical parameters. The gold standard approach is to conduct a systematic review of the relevant clinical literature, although reviews of economic evaluations indicate that this is rarely done. Technology appraisals for the National Institute for Health and Clinical Excellence (NICE), which are fully funded, represent the best case scenario for the close integration of economic evaluations and systematic reviews. The objective of this study was to assess the extent to which the systematic review of the clinical literature informs the economic evaluation in NICE technology appraisals. METHODS: All NICE technology assessment reports (TARs) published between January 2003 and July 2006 were considered. Data were abstracted on the TAR topics, the primary measure of clinical effectiveness, the approach to pooling in the clinical review, the measure of economic benefit and the use, or non-use, of the systematic review in the economic evaluation. RESULTS: Forty-one TARs were published in the period studied, all of which contained a systematic review. Most of the economic evaluations (85 percent) were cost-utility analyses, reflecting NICE's guidelines for economic evaluation. In seventeen cases, the clinical data were not pooled in the review, owing to heterogeneity in the clinical data or the limited number of studies. In these cases, the economists used alternative approaches for estimating the key effectiveness parameter in the model. The results of the review (when pooled) were always used when the primary clinical effectiveness measure corresponded with the measure of economic benefit (e.g., survival). However, because preference-based quality of life measures are rarely included in clinical trials, the results of the systematic review were never directly used in the cost-utility analyses. Nevertheless, the outputs of the systematic review were used when the data were useful in estimating components of the quality-adjusted life-year (QALY) (e.g., the life-years gained, or the frequencies of health states to which QALYs could be assigned). Problems occurred mainly when the clinical data were not pooled, or when the measure of clinical benefit could not be converted into health states to which QALYs could be assigned. CONCLUSIONS: Economic evaluations can benefit from systematic reviews of the clinical literature. However, such reviews are not a panacea for conducting a good economic evaluation. Much of the relevant data for estimating QALYs are not contained in such reviews and the chosen method for summarizing the clinical data may inhibit the assessment of economic benefit. Problems would be reduced if those undertaking the technology assessments discussed the data requirements for the economic model at an early stage.     

Systematic review of economic evaluations of vaccination programs in mainland China: Are they sufficient to inform decision making?

The purpose of the study was to systematically review economic evaluations of vaccine programs conducted in mainland China. We searched for economic evaluations of vaccination in China published prior to August 3, 2015 in eight English-language and three Chinese-language databases. Each article was appraised against the 19-item Consensus on Health Economic Criteria list (CHEC-list). We found 23 papers evaluating vaccines against hepatitis B (8 articles), Streptococcus pneumoniae (5 articles), human papillomavirus (3 articles), Japanese encephalitis (2 articles), rotavirus (2 articles), hepatitis A (1 article), Enterovirus 71 (1 article) and influenza (1 article). Studies conformed to a mean of 12 (range: 6–18) items in the CHEC-list criteria. Five of six Chinese-language articles conformed to fewer than half of the 19 criteria items. The main criteria that studies failed to conform to included: inappropriate measurement (20 articles) and valuation (18 articles) of treatment and/or vaccination costs, no discussion about distributional implications (18 articles), missing major health outcomes (14 articles), no discussion about generalizability to other contexts (14 articles), and inadequate sensitivity analysis (13 articles). In addition, ten studies did not include major cost components of vaccination programs, and nine did not report outcomes in terms of life years even in cases where QALYs or DALYs were calculated. Only 13 studies adopted a societal perspective for analysis. All studies concluded that the appraised vaccination programs were cost-effective except for one evaluation of universal 7-valent pneumococcal conjugate vaccine (PCV-7) in children. However, three of the five studies on PCV-7 showed poor overall quality, and the number of studies on vaccines other than hepatitis B vaccine and PCV-7 was limited. In conclusion, major methodological flaws and reporting problems exist in current economic evaluations of vaccination programs in China. Local guidelines for good practice and reporting, institutional mechanisms and education may help to improve the overall quality of these evaluations.     

Systematic review of economic evaluations of the 23-valent pneumococcal polysaccharide vaccine (PPV23) in individuals 60 years of age or older

ObjectivesTo systematically review the economic evaluations of 23-valent pneumococcal polysaccharide vaccine (PPV23) in adults aged ≥60 years to inform the development of local studies through the discussion of parameters and assumptions that influence the results of the analyses.MethodsWe searched the MEDLINE, Excerpta Medica, Cochrane Library, Latin-American and Caribbean Health Sciences Literature (LILACS), Brazilian Regional Library of Medicine, National Health Service Economic Evaluation, and Centre for Reviews and Dissemination—as well as the Scopus citation index and the Web of Science for full economic evaluations of PPV23 published up to March 2016. Two independent reviewers screened the articles for relevance and extracted the data. Main study characteristics and methods (clinical and epidemiological data, cost and incremental cost-effectiveness ratios (ICERs) were extracted and compared. Costs were updated to 2016 international dollars.ResultsTwenty-seven studies published from 1980 to 2016 were reviewed. Most studies were conducted in Europe and the USA; three studies were conducted in Latin America (Brazil, 2; Colombia, 1). In addition to the scenario comparing the vaccination with the PPV23 to non-vaccination, three studies also compared PPV23 to pneumococcal conjugate 13-valent vaccine (PCV13). All studies used static models. Most used a lifetime (44.4%) or 5–6 year’s time horizon (33.3%). Only three studies considered herd protection from children immunization with PCV13 in the model. Most studies considered PPV23 cost-effective (less than US$50,000 per LYG or QALY) and sometimes cost-saving (results ranging from cost-saving to US$84,636/QALY). The estimates of disease burden, the efficacy/effectiveness of PPV23, and the effects of herd protection from childhood immunization had most influence on the results.ConclusionsWell-designed cost-effectiveness studies of PPV23 that represent the current epidemiological scenario and reduce uncertainty related to efficacy/effectiveness are extremely relevant to informing the decision-making process.     

Mind the gaps: What's missing from current economic evaluations of universal HPV vaccination?

Background

Since the original licensing of human papilloma virus (HPV) vaccination for women, evidence is accumulating of its effectiveness in preventing HPV-related conditions in men, and universal vaccination (vaccinating men and women) is now recommended in some countries. Several models of the cost-effectiveness of universal HPV vaccination have been published, but results have been mixed. This article assesses the extent to which economic studies have captured the range of values associated with universal HPV vaccination, and how this influences estimates of its cost-effectiveness.

Methods

Eight published economic evaluations of universal HPV vaccination were reviewed to identify which of the values associated with universal HPV vaccination were included in each analysis.

Results

Studies of the cost-effectiveness of universal HPV vaccination capture only a fraction of the values generated. Most studies focused on impacts on health and health system cost, and only captured these partially. A range of values is excluded from most studies, including impacts on productivity, patient time and costs, carers and family costs, and broader social values such as the right to access treatment. Further, those studies that attempted to capture these values only did so partially.

Discussion

Decisions to invest in universal HPV vaccination need to be based on a complete assessment of the value that it generates. This is not provided by existing economic evaluations. Further work is required to understand this value. First, research is required to understand how HPV-related health outcomes impact on society including, for instance, their impact on productivity. Second, consideration should be given to alternative approaches to capture this broader set of values in a manner useful to decisions-makers, such as multi-criteria decision analysis.     

Systematic review of economic evaluation studies: Are vaccination programs efficient in Spain?

Introduction

During the twentieth century, vaccination has been one of the measures of greatest public health impact. Vaccine administration has helped reduce the burden of disease and mortality from infectious diseases. At present, there is increasing concern about infectious diseases and the ability of health systems to control them, highlighting the need for evaluation of vaccination programs. The aim of this study was to conduct a systematic review of economic evaluation studies carried out regarding Spain on vaccines.

Methods

Systematic review. Search of articles in major bibliographic databases available online from January 1983 to June 2011. References identified were limited to full economic evaluations carried out regarding Spain that evaluated vaccination programs. For each of the selected papers, a set of predefined variables were extracted.

Results

A total of 46 studies met inclusion criteria. The topics studied were pneumococcal vaccination, influenza vaccination, Hepatitis B vaccination and varicella vaccination. Cost-minimization analysis, perspective of society, long time horizon, use of modeling techniques, and the inclusion of direct and indirect costs were the most common methodological characteristics. The results of the studies reviewed showed, in most cases, net savings or cost-effectiveness ratios below €30,000/QALY.

Conclusions

Although there has been an improvement in the methodological quality of studies, they still show shortcomings that should be addressed. From a public health perspective, it would be relevant to evaluate vaccines targeted to major health problems in Spain, including all relevant costs and benefits. In order to obtain a more efficient use of health resources, economic evaluation methods should be applied more rigorously and results should be used consistently in decision-making processes.     

Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go?

STUDY OBJECTIVE: There is little guidance on how to identify useful evidence about the health effects of social interventions. The aim of this study was to assess the value of different ways of finding this type of information. DESIGN: Retrospective analysis of the sources of studies for one systematic review. SETTING: Case study of a systematic review of the effectiveness of interventions in promoting a population shift from using cars towards walking and cycling. MAIN RESULTS: Only four of the 69 relevant studies were found in a "first-line" health database such as Medline. About half of all relevant studies were found through the specialist Transport database. Nine relevant studies were found through purposive internet searches and seven relevant studies were found by chance. The unique contribution of experts was not to identify additional studies, but to provide more information about those already found in the literature. CONCLUSIONS: Most of the evidence needed for this review was not found in studies indexed in familiar literature databases. Applying a sensitive search strategy across multiple databases and interfaces is very labour intensive. Retrospective analysis suggests that a more efficient method might have been to search a few key resources, then to ask authors and experts directly for the most robust reports of studies identified. However, internet publications and serendipitous discoveries did make a significant contribution to the total set of relevant evidence. Undertaking a comprehensive search may provide unique evidence and insights that would not be obtained using a more focused search.     

Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go?

STUDY OBJECTIVE: There is little guidance on how to select the best available evidence of health effects of social interventions. The aim of this paper was to assess the implications of setting particular inclusion criteria for evidence synthesis. DESIGN: Analysis of all relevant studies for one systematic review, followed by sensitivity analysis of the effects of selecting studies based on a two dimensional hierarchy of study design and study population. SETTING: Case study of a systematic review of the effectiveness of interventions in promoting a population shift from using cars towards walking and cycling. MAIN RESULTS: The distribution of available evidence was skewed. Population level interventions were less likely than individual level interventions to have been studied using the most rigorous study designs; nearly all of the population level evidence would have been missed if only randomised controlled trials had been included. Examining the studies that were excluded did not change the overall conclusions about effectiveness, but did identify additional categories of intervention such as health walks and parking charges that merit further research, and provided evidence to challenge assumptions about the actual effects of progressive urban transport policies. CONCLUSIONS: Unthinking adherence to a hierarchy of study design as a means of selecting studies may reduce the value of evidence synthesis and reinforce an "inverse evidence law" whereby the least is known about the effects of interventions most likely to influence whole populations. Producing generalisable estimates of effect sizes is only one possible objective of evidence synthesis. Mapping the available evidence and uncertainty about effects may also be important.     

Are the best available clinical effectiveness data used in economic evaluations of drug therapies?

OBJECTIVES: Use of evidence on clinical effectiveness that is of poor quality or is biased in favour of the therapy under study is a concern in economic evaluations and may contribute to a mistrust of pharmacoeconomic studies. This study aimed to determine whether the authors of economic evaluations use the best available evidence for clinical effectiveness. METHODS: One hundred economic evaluations of drug therapies (published in 2001-2003) were sampled randomly from the National Health Service Economic Evaluation Database, and the source of clinical evidence was identified. For each therapy, alternative, high quality sources of clinical effectiveness data were sought by searching the Database of Abstracts of Reviews of Effects and Health Technology Assessment databases. The magnitude and direction of the effect size in the different sources of evidence were compared. RESULTS: Relevant systematic reviews were found for only 32 of the 100 economic evaluations in the sample. In three cases these reviews had been identified by the authors of the economic evaluations and two of these cases were used in the evaluation. Comparisons were possible in 21 cases. The clinical effects reported in all 21 comparisons were similar in direction but differed in magnitude. Compared to the systematic reviews, the authors of economic evaluations used evidence that was more favourable in five cases, less favourable in four cases, and similar in 12 cases. Six of the economic evaluations and corresponding systematic reviews did not present measures of effectiveness in a manner that allowed comparison. CONCLUSIONS: Authors of economic evaluations have not made sufficient use of the evidence available from systematic reviews of clinical effectiveness. The central role of economic evaluations in health policy makes it essential that improvements in economic methods are accompanied by a structured search for the highest quality information on clinical effectiveness.     

Is a determination of medical futility ethical?

Medical futility is a timely topic and one fraught with ethical implications. Medical futility is a term used to describe medical interventions that are expected to result in little or no benefit to a patient. This Editorial examines some considerations and applications of medical futility to a specific patient or treatment, the ramifications of the use of the term futility, and its effects on physician-patient communication and the role of the health care administrator. Invoking medical futility is fraught with areas of vulnerability and implications in ethical decision making. Of concern is whether the treatment will cause more harm than good (nonmaleficence), whether it respects patients' goals and desires (autonomy), and whether the interests of patients, society, and the organization are served (justice).     

Correction to: How relevant are social costs in economic evaluations? The case of Alzheimer’s disease

The European Journal of Health Economics - A correction to this paper has been published: https://doi.org/10.1007/s10198-021-01323-y     

Response bias in community surveys of mental health: Systematic bias or random noise?

This paper examines the effects on a mental health survey of three forms of response bias— naysaying, perceived trait desirability and need for social approval. It is suggested that, although these three forms of response bias may be statistically associated with either the independent or the dependent variables, it is mmore meaningful to introduce them as controls on the relation between the independent and dependent variables. Data from a moderately poor, predominately black population suggest that these three forms of response bias are statistically associated with symptoms of depression and other forms of mental illness. Nonetheless, response bias generally does not act as a form of systematic bias that invalidates the pattern of relationships observed with traditional independent variables, but instead acts as random noise.     

Health research systems: promoting health equity or economic competitiveness?

International collaborative health research is justifiably expected to help reduce global health inequities. Investment in health policy and systems research in developing countries is essential to this process but, currently, funding for international research is mainly channelled towards the development of new medical interventions. This imbalance is largely due to research legislation and policies used in high-income countries. These policies have increasingly led these countries to invest in health research aimed at boosting national economic competitiveness rather than reducing health inequities. In the United States of America and the United Kingdom of Great Britain and Northern Ireland, the regulation of research has encouraged a model that: leads to products that can be commercialized; targets health needs that can be met by profitable, high-technology products; has the licensing of new products as its endpoint; and does not entail significant research capacity strengthening in other countries. Accordingly, investment in international research is directed towards pharmaceutical trials and product development public-private partnerships for neglected diseases. This diverts funding away from research that is needed to implement existing interventions and to strengthen health systems, i.e. health policy and systems research. Governments must restructure their research laws and policies to increase this essential research in developing countries.     

How relevant are social costs in economic evaluations? The case of Alzheimer’s disease

Background

The main objective of this study was to analyse how the inclusion (exclusion) of social costs can alter the results and conclusions of economic evaluations in the field of Alzheimer’s disease interventions.

Methods

We designed a systematic review that included economic evaluations in Alzheimer’s disease. The search strategy was launched in 2000 and ran until November 2018. The inclusion criteria were: being an original study published in a scientific journal, being an economic evaluation of any intervention related to Alzheimer’s disease, including social costs (informal care costs and/or productivity losses), being written in English, using QALYs as an outcome for the incremental cost–utility analysis, and separating the results according to the perspective applied.

Results

It was finally included 27 studies and 55 economic evaluations. Around 11% of economic evaluations changed their main conclusions. More precisely, three of them concluded that the new intervention became cost-effective when the societal perspective was considered, whereas when using just the health care payer perspective, the new intervention did not result in a cost–utility ratio below the threshold considered. Nevertheless, the inclusion of social cost can also influence the results, as 37% of the economic evaluations included became the dominant strategy after including social costs when they were already cost-effective in the health care perspective.

Conclusions

Social costs can substantially modify the results of the economic evaluations. Therefore, taking into account social costs in diseases such as Alzheimer’s can be a key element in making decisions about public financing and pricing of health interventions.

     

The economics of primary prevention of cardiovascular disease – a systematic review of economic evaluations

Background

Global resource needs estimation is a critical part of addressing the HIV/AIDS epidemic. To generate these estimates knowledge of costs and cost structures is required. The evidence base for costs of HIV prevention programmes is limited. Even less is known about the existence of economies scale and whether, as economic theory suggests, average costs form a 'u'-shaped curve as scale increases. Using an econometric analysis, this paper addresses this question by estimating marginal costs and economies of scale for HIV prevention programmes for vulnerable groups in Southern India with different levels of coverage.

Methods

Two hybrid translog-cost functions were estimated. First, expenditure data from 78 state-funded HIV prevention projects in Andhra Pradesh were used to explore the impact of scale, institutional history and price on costs; second, economic cost data from 16 commercial sex worker projects across Tamil Nadu and Andhra Pradesh were analysed to additionally assess the impact of the value of inputs not reported in expenditure data and location. Coefficient estimates were used to calculate marginal costs and economies of scale.

Results

The econometric model yielded a good fit (R² = 0.46, p < 0.001 and R² = 0.79, p < 0.001, for the expenditure and economic cost datasets, respectively). The economies of scale index was greater than 1 for both datasets and fell as coverage increased. Analysis of the expenditure data found economies of scale were not exhausted, with a 0.002% change in total cost for each extra person reached and an 11% difference in total cost between target group categories. Estimation using the economic cost data suggests a point of minimum efficient scale at around 1750–2000 people reached, a 0.03% change in total cost for each extra person reached, and 28% lower costs in Tamil Nadu than Andhra Pradesh.

Conclusion

Econometric analysis of these standardized datasets provides insights into how costs change with coverage, the impact of project location and nature of the project target group. The results demonstrate the importance of understanding the nature of the cost function when designing, budgeting and estimating resource requirements for scaling up coverage of HIV prevention projects.