Measuring health system performance: A new approach to accountability and quality improvement in New Zealand

In February 2014, the New Zealand Ministry of Health released a new framework for measuring the performance of the New Zealand health system. The two key aims are to strengthen accountability to taxpayers and to lift the performance of the system's component parts using a ‘whole-of-system’ approach to performance measurement. Development of this new framework – called the Integrated Performance and Incentive Framework (IPIF) – was stimulated by a need for a performance management framework which reflects the health system as a whole, which encourages primary and secondary providers to work towards the same end, and which incorporates the needs and priorities of local communities. Measures within the IPIF will be set at two levels: the system level, where measures are set nationally, and the local district level, where measures which contribute towards the system level indicators will be selected by local health alliances. In the first year, the framework applies only at the system level and only to primary health care services. It will continue to be developed over time and will gradually be extended to cover a wide range of health and disability services. The success of the IPIF in improving health sector performance depends crucially on the willingness of health sector personnel to engage closely with the measurement process.     

Conceptual frameworks for health systems performance: a quest for effectiveness, quality, and improvement.

ISSUES: Countries and international organizations have recently renewed their interest in how health systems perform. This has led to the development of performance indicators for monitoring, assessing, and managing health systems to achieve effectiveness, equity, efficiency, and quality. Although the indicators populate conceptual frameworks, it is often not very clear just what the underlying concepts might be or how effectiveness is conceptualized and measured. Furthermore, there is a gap in the knowledge of how the resultant performance data are used to stimulate improvement and to ensure health care quality. ADDRESSING THE ISSUES: This paper therefore explores, individually, the conceptual bases, effectiveness and its indicators, as well as the quality improvement dynamics of the performance frameworks of the UK, Canada, Australia, US, World Health Organization, and Organisation for Economic Co-operation and Development. RESULTS: We see that they all conceive health and health system performance in one or more supportive frameworks, but differ in concepts and operations. Effectiveness often implies, nationally, the achievement of high quality outcomes of care, or internationally, the efficient achievement of system objectives, or both. Its indicators are therefore mainly outcome and, less so, process measures. The frameworks are linked to a combination of tools and initiatives to stimulate and manage performance and quality improvement. CONCLUSIONS: These dynamics may ensure the proper environment for these conceptual frameworks where, alongside objectives such as equity and efficiency, effectiveness (therefore, quality) becomes the core of health systems performance.     

Measuring and managing progress in the establishment of basic health services: the Afghanistan health sector balanced scorecard

The Ministry of Public Health (MOPH) of Afghanistan has adopted the Balanced Scorecard (BSC) as a tool to measure and manage performance in delivery of a Basic Package of Health Services. Based on results from the 2004 baseline round, the MOPH identified eight of the 29 indicators on the BSC as priority areas for improvement. Like the 2004 round, the 2005 and 2006 BSCs involved a random selection of more than 600 health facilities, 1700 health workers and 5800 patient-provider interactions. The 2005 and 2006 BSCs demonstrated substantial improvements in all eight of the priority areas compared to 2004 baseline levels, with increases in median provincial scores for presence of active village health councils, availability of essential drugs, functional laboratories, provider knowledge, health worker training, use of clinical guidelines, monitoring of tuberculosis treatment, and provision of delivery care. For three of the priority indicators-drug availability, health worker training and provider knowledge-scores remained unchanged or decreased between 2005 and 2006. This highlights the need to ensure that early gains achieved in establishment of health services in Afghanistan are maintained over time. The use of a coherent and balanced monitoring framework to identify priority areas for improvement and measure performance over time reflects an objectives-based approach to management of health services that is proving to be effective in a difficult environment.     

Measuring health system performance

The improvement of health system performance has become a key policy issue in most developed nations. To that end, many initiatives to measure system performance are being put in place. This paper examines the performance measurement movement from a principal/agent viewpoint. It argues that the effectiveness of performance measurement systems depends on four key factors: the extent to which the chosen performance measures reflect faithfully the objectives of the system, the nature and quality of the data, the incentives for clinicians to scrutinize and act upon the data, and the culture of the organization within which the data are deployed. Although the optimal design of performance measurement systems depends heavily on local factors, they are likely to offer a highly cost-effective instrument for securing major improvements in system performance if properly deployed     

Development and implementation of a nationwide health care quality indicator system in Taiwan.

QUALITY ISSUES: Quality is an increasingly important issue to the health care sector. The Taiwanese government also recognizes the need to implement a nationwide health care quality indicator system to strengthen quality surveillance. CHOICE OF SOLUTION: In 1999, the Department of Health funded a 2-year project led by the Taiwan Healthcare Executive College to develop a comprehensive performance assessment system, subsequently named as Taiwan Healthcare Indicator Series (THIS). The series includes four categories of indicators, namely outpatient, in-patient, emergency care, and intensive care, and has 139 items in total. IMPLEMENTATION: The system was officially launched in 2001. Participation is voluntary. The Taiwan Healthcare Executive College processes the data and provides feedback to the participating hospitals. The information is for the participating hospitals' own use and is not released to the public. EVALUATION: Participating hospitals have increased from 45 in 2001 to 227 in 2006 and now constitute approximately 50% of the total hospital population in Taiwan. The reporting rate averaged 77.7% in 2004. The first five most reported indicators are the percentage of first-visit outpatients to outpatient clinics, the average length of in-patient stay, the nosocomial infection rate, the occupancy rate, and the crude mortality rate. LESSONS LEARNED: How the data are interpreted and how data interpretation can lead to quality improvement are the principal concerns of participating hospitals. In light of the success of the indicator series, the Bureau of National Health Insurance (BNHI) of Taiwan has proposed participation in the series as being one of the criteria to be reimbursed for quality.     

Promoting quality: the health-care organization from a management perspective.

BACKGROUND: Although agreement about the need for quality improvement in health care is almost universal, the means of achieving effective improvement in overall care is not well understood. Avedis Donabedian developed the structure-process-outcome framework in which to think about quality-improvement efforts. ISSUE: There is now a robust evidence-base in the quality-improvement literature on process and outcomes, but structure has received considerably less attention. The health-care field would benefit from expanding the current interpretation of structure to include broader perspectives on organizational attributes as primary determinants of process change and quality improvement. SOLUTIONS: We highlight and discuss the following key elements of organizational attributes from a management perspective: (i) executive management, including senior leadership and board responsibilities (ii) culture, (iii) organizational design, (iv) incentive structures and (v) information management and technology. We discuss the relevant contributions from the business and medical literature for each element, and provide this framework as a roadmap for future research in an effort to develop the optimal definition of 'structure' for transforming quality-improvement initiatives.     

Reporting quality of nursing home care to consumers: the Maryland experience.

OBJECTIVE: To design and implement a reporting system for quality of long-term care to empower consumers and to create incentives for quality improvement. To identify a model to approach this technically and politically difficult task. APPROACH: Establishment of a credible and transparent decision process using a public forum. Development of the system based on: (1) review of the literature and existing systems, and discussions with stakeholders about strengths and weaknesses; (2) focus on consumer preferences in the design; and (3) responsiveness to industry concerns in the implementation. LESSONS LEARNED: None of the existing systems appeared to be a suitable model. We decided to develop an entirely new system based on three key design principles that allowed us to tailor the system to consumer needs: (1) designing a decision tool rather than a database; (2) summarizing rather than simplifying information; and (3) accounting for the target audience in the creative execution. Industry concerns focused on the burden of the system, the potential for errors, and the possible communication of a negative impression of the industry. As methodological and data limitations prevented us from resolving those concerns, we addressed them by using cautionary language in the presentation and by making a commitment to incorporate improvements in the future. All stakeholders regarded the final design as an acceptable compromise. CONCLUSIONS: Despite its potentially controversial nature and many methodological challenges, the system has been well received by both the public and the industry. We attribute this success to two key factors: a collaborative decision process, in which all critical design and execution choices were laid out explicitly and debated with stakeholders in a public forum, and realism and honesty regarding the limitations of the system.     

Trends in socioeconomic disparities in health care quality in four countries.

OBJECTIVE: To provide a targeted portrait of socioeconomic disparities in health care quality in four countries and how those disparities have changed over time. DESIGN: Within each country, comparisons between the highest and lowest quintiles of socioeconomic status were made to determine if disparities exist and if any observed disparities have been decreasing over a 5-year period. SETTING: Small geographic areas in Canada, England, New Zealand and the United States. DATA SOURCES: Data were obtained by working with national health statistics agencies in each country. RESULTS: There were socioeconomic disparities in health care quality and health status for most of the indicators studied in all four countries. The analysis included nine quality indicators in four countries, for a total of thirty-six observations. Twenty-six observations had a ratio of highest to lowest socioeconomic quintile of <0.95 or >1.05. These disparities generally persisted over time. The relative difference between the highest and lowest quintile decreased over time in eight of the twenty-one observations with time-series data available. CONCLUSION: The fact that disparities in a variety of indicators exist in four very different health systems underscores the importance of factors common to the four systems or factors outside the health system. Some successful strategies for reducing disparities could potentially be learned from the few examples of success in these countries.     

Measuring the quality of junior hospital doctors in general medicine

During the course of a larger study aimed at relating staffing levels of junior doctors in general medicine to the safety of the care provided, it became clear that consultant doctors considered the quality of their junior staff as being at least as important as the quantity. This paper describes several attempts to develop a feasible and valid method of measuring the quality of senior house officers (SHOs) and registrars using routinely available data. Having rejected three methods and had difficulties with three other methods, a modified Delphi survey was used to explore the extent to which consultants agreed on the key attributes of a high quality SHO or registrar. Sixty-seven (60%) of all consultants in the South-West Thames and Trent regions responded to two rounds of questionnaires which revealed communication skills as being consistently the most significant factor. This was confirmed in a second, anonymous survey of 198 (78%) SHO and registrar posts in the North-West Thames region. It also became clear that consultants viewed the quality of their own SHOs and registrars as generally high. This being so it is argued that the requirement to adjust for quality, when comparing SHO and registrar levels between hospitals, is of minor importance. These results also suggest that the explicit teaching of communication skills should have a high priority in undergraduate and postgraduate education.     

医学装备质量控制体制建立对提高医疗质量影响的探讨

目的：建立医学装备质量控制体制,消除设备安全隐患。方法：解剖当前医院医学装备质量控制的现状,提出医学装备质量控制体制建立的初步路径。结果：通过医学装备质量控制体制建立,改变医院重使用,重购买,忽视医学装备质量控制的传统模式。结论：医学装备质量控制体制的建立,对提高医疗质量管理水平具有重要的意义。     

Measuring instructional quality in community-orientated medical education: looking into the black box

OBJECTIVES: Decentralizing medical education to community settings has raised issues of instructional quality. The need to evaluate community-based instruction accents the need to adopt a systems perspective, moving beyond factors known to comprise general clinical teaching effectiveness to include factors that focus on instruction as a process. Application of evaluation models using traditional input-output analysis can be flawed. This approach--dubbed the 'black box'--typically examines inputs and outputs, but often ignores throughputs. DESIGN: In this article we open the black box, using theory to examine the underlying processes that define community-based medical education. We first describe the components and processes of an instructional model that is framed by the philosophy of quality and grounded in experiential learning theory. Without examining the critical processes at work inside the black box - i.e. how students come to acquire clinical knowledge and how behaviours are influenced - it is difficult to assess which programme features contribute to success. Tensions created by the absence of consensus on the outcomes of instruction and the challenge of developing adequate measures are highlighted. SETTING: State University of New York at Buffalo. SUBJECTS: Clinician-teachers, learners and patients in the environment. RESULTS AND CONCLUSIONS: We conclude with describing a tool for evaluating community-based instruction that is guided by the context of our model.     

Bridging the quality chasm: integrating professional and organizational approaches to quality.

Current Western health care practices face the challenge to improve their quality on multiple dimensions simultaneously. This requires new ways to think about how to deliver health care services. A careful and 'flexible' standardization of care into 'care programs', we argue, is central. Yet such standardization is powerless without the application of four additional design principles: a thorough restructuring and delegation of tasks, the application of integrated planning, the use of indicators about the functioning of the care programs, and implementing process-supporting information technology. Vice versa, these additional principles can only function properly when integrated with care programs. We will only be able to improve the safety, effectiveness, patient-centeredness, and timeliness of health care, while reducing costs and improving equity, by integrating professional and organizational approaches to quality. This paper describes a series of interrelated design principles that together depict how future health care delivery could be organized.     

Health system performance assessment in small countries: The case study of Latvia

Managing the complexity that characterizes health systems requires sophisticated performance assessment information to support the decision‐making processes of healthcare stakeholders at various levels. Accordingly, in the past few decades, many countries have designed and implemented health system performance assessment (HSPA) programmes. Literature and practice agree on the key features that performance measurement in health should have, namely, multidimensionality, evidence‐based data collection, systematic benchmarking of results, shared design, transparent disclosure, and timeliness. Nevertheless, the specific characteristics of different countries may pose challenges in the implementation of such programmes. In the case of small countries, many of these challenges are common and related to their inherent characteristics, eg, small populations, small volumes of activity for certain treatments, and lack of benchmarks. Through the development of the case study of Latvia, this paper aims at discussing the challenges and opportunities for assessing health system performance in a small country. As a result, for each of the performance measurement features identified by the literature, the authors discuss the issues emerging when adopting them in Latvia and set out the potential solutions that have been designed during the development of the case study.     

社区卫生服务机构内部服务质量指标评价体系研究

目的:参考内部服务质量理论,建立一套适合我国社区卫生服务机构的内部服务质量评价体系。方法:根据文献检索、参与式观察、深入访谈等结果编制社区卫生服务机构内部服务质量调查问卷,并对北京市的两家社区卫生服务中心及其下设各服务站进行实证研究。结果:问卷的专家评分均值为88.3分,内部一致性信度为0.956,因子分析抽取的7个因子可以解释75.457%的内容,说明问卷结构效度较好。结论:社区卫生服务机构内部服务质量评价体系能较好的反映员工对内部服务质量的评价,为社区卫生服务机构进行自身服务质量评价、改进内部服务质量提供一种工具。     

妇幼保健机构质量管理体系整合的初步体会

目的为进一步健全妇幼保健机构的质量管理体系，形成标准化的管理模式，推动整体管理水平和服务质量的提高。方法在“上等达标”的创建活动中，以ISO9000：2000的技术方法为路径，以广东省妇幼保健机构等级评审细则（以下简称《细则》）为内容，设计运行体系，建立体系文件，形成统一的质量方针、质量目标和持续改进的质量管理机制。结果如期顺利完成创建任务，全员质量意识和质控技能明显提高，服务数量和质量大幅度改善。结论将ISO9000国际质量标准体系和其技术方法与《细则》结合起来进行“上等达标”的创建工作是可行的，它为多种管理体系的整合运行提供了可行性例证。     

Development of a set of strategy-based system-level cancer care performance indicators in Ontario, Canada.

OBJECTIVES: To develop a set of scientifically sound and managerially useful system-level cancer care performance indicators for public reporting in Ontario, Canada. IMPLEMENTATION: Using a modified Delphi panel method, comprising a systematic literature review and multiple rounds of structured feedback from 34 experts, the Cancer Quality Council of Ontario developed a set of quality indicators spanning cancer prevention through to end-of-life care. To be useful to decision-makers and providers, indicator selection criteria included a clear focus on the cancer system, relevance to a diversity of cancer providers, a strong link to the mission and strategic objectives of the cancer system, clear directionality of indicator results, presence of targets and/or benchmarks, feasibility of populating the indicator, and credibility of the measure as an indicator of quality. To ensure that the selected indicators would measure progress over time against specific and widely accepted goals, we created a strategy map based on the five strategic objectives of the Ontario cancer system: (i) to improve the measurement and reporting of cancer quality, (ii) to increase the use of evidence and innovation in decision-making, (iii) to improve access to cancer services and reduce waiting times, (iv) to increase efficiency across the system, (v) to reduce the burden of cancer. An analysis of the mean indicator ratings by experts, and the strategy mapping exercise resulted in the identification of 36 indicators deemed suitable for routine performance measurement of the Ontario cancer system. LESSONS LEARNED: The resulting instrument incorporates a credible evidence basis for performance measurement aligned to the five strategic goals for the Ontario cancer system. It represents the integrating of a management culture, focused on the implementation of a new strategic direction for the cancer system, with the underlying evidence-based culture of clinicians.