Developing quality indicators for older adults: transfer from the USA to the UK is feasible

Background: Measurement of the quality of health care is essential for quality improvement, and patients are an underused source of data about quality of care. We describe the adaptation of a set of USA quality indicators for use in patient interview surveys in England, to measure the extent to which older patients receive a broad range of effective healthcare interventions in both primary and secondary care.

Method: One hundred and nineteen quality indicators covering 16 clinical areas, based on a set of indicators for the care of vulnerable elderly patients in the USA, were reviewed by a panel of 10 clinical experts in England. A modified version of the RAND/UCLA appropriateness method was used and panel members were supplied with literature reviews summarising the evidence base for each quality indicator. The indicators were sent for comment before the panel meeting to UK charitable organisations for older people.

Results: The panel rated 102 of the 119 indicators (86%) as valid for use in England; 17 (14%) were rejected as invalid. All 58 indicators about treatment or continuity and follow up were rated as valid compared with just over half (13 of 24) of the indicators about screening.

Conclusions: These 102 indicators are suitable for use in patient interview surveys, including the English Longitudinal Study of Ageing (ELSA). The systematic measurement of quality of care at the population level and identification of gaps in quality is essential for quality improvement. There is potential for transfer of quality indicators between countries, at least for the health care of older people.

     

Quality assessment for three common conditions in primary care: validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes

Objectives: To field test the reliability, validity, and acceptability of review criteria for angina, asthma, and type 2 diabetes which had been developed by expert panels using a systematic process to combine evidence with expert opinion.

Design: Statistical analysis of data derived from a clinical audit, and postal questionnaire and semi-structured interviews with general practitioners and practice nurses in a representative sample of general practices in England.

Setting: 60 general practices in England.

Main outcome measures: Clinical audit results for angina, asthma, and type 2 diabetes. General practitioner and practice nurse validity ratings from the postal questionnaire.

Results: 54%, 59%, and 70% of relevant criteria rated valid by the expert panels for angina, asthma, and type 2 diabetes, respectively, were found to be usable, valid, reliable, and acceptable for assessing quality of care. General practitioners and practice nurses agreed with panellists that these criteria were valid but not that they should always be recorded in the medical record.

Conclusion: Quality measures derived using expert panels need field testing before they can be considered valid, reliable, and acceptable for use in quality assessment. These findings provide additional evidence that the RAND panel method develops valid and reliable review criteria for assessing clinical quality of care.

     

Quality standards for child and adolescent mental health in primary care

ABSTRACT: BACKGROUND: Child and adolescent mental health problems are common in primary healthcare settings. However, few parents of children with mental health problems express concerns about these problems during consultations. Based on parental views, we aimed to create quality of care measures for child and adolescent mental health in primary care and develop consensus about the importance of these quality standards within primary care. METHODS: Quality Standards were developed using an iterative approach involving four phases: 1) 34 parents with concerns about their child's emotional health or behaviour were recruited from a range of community settings including primary care practices to participate in focus group discussions, followed by validation groups or interviews. 2) Preliminary Quality Standards were generated that fully represented the parents' experiences and were refined following feedback from an expert parent nominal group. 3) 55 experts, including parents and representatives from voluntary organisations, across five panels participated in a modified two-stage Delphi study to develop consensus on the importance of the Quality Standards. The panels comprised general practitioners, other community-based professionals, child and adolescent psychiatrists, other child and adolescent mental health professionals and public health and policy specialists. 4) The final set of Quality Standards was piloted with 52 parents in primary care. RESULTS: In the Delphi process, all five panels agreed that 10 of 31 Quality Standards were important. Although four panels rated 25-27 statements as important, the general practitioner panel rated 12 as important. The final 10 Quality Standards reflected healthcare domains involving access, confidentiality for young people, practitioner knowledge, communication, continuity of care, and referral to other services. Parents in primary care agreed that all 10 statements were important. CONCLUSIONS: It is feasible to develop a set of Quality Standards to assess mental healthcare provision for children and adolescents seen within primary healthcare services. Primary care practitioners should be aware of parental perspectives about quality of care as these may influence helpseeking behaviours.     

Improving the quality of mental health services in Personal Medical Services pilots: a longitudinal qualitative study

Background: A series of government initatives in the UK have included strategies to improve the quality of services received by patients, including fundholding, the development of National Service Frameworks, clinical governance, and Personal Medical Services (PMS). PMS represents a new contractual arrangement between government and general practitioners (GPs) which provides new investment in return for more detailed specification of processes and outcomes of care.

Objectives: To evaluate the effects of PMS on the quality of primary mental health care between 1998 and 2001.

Design: Multiple longitudinal case studies. Semi-structured interviews with key staff within practices (GPs, nurses, practice managers) and outside (health authority and primary care group/trust managers).

Sample: Six first wave PMS sites which had specifically planned to improve their mental health care.

Results: Improvements in mental health care were found in some PMS practices and not in others. Five mechanisms associated with successful quality improvement in mental health were identified: clear goals, effective teamwork within the practice, routine use of protocols and audits, additional resources, and effective collaboration with community and secondary care. Sites where these factors were not present struggled to meet their objectives.

Conclusion: The five mechanisms which resulted in improved mental health care were facilitated by the new contractual arrangements in PMS. The new contracts were not a necessary part of these changes, but they enabled sites with an identified interest and motivation to make the changes. The contractual changes were not in themselves sufficient to improve care.

     

RCGP Quality Team Development programme: an illuminative evaluation

Background: There is increasing interest in quality initiatives that are locally owned and delivered, team based, multiprofessional, and formative. The Royal College of General Practitioners' Quality Team Development (QTD) programme is one such initiative aimed at developing primary healthcare teams and their services.

Aims: To evaluate QTD from the perspective of participants and assessors.

Setting: UK primary health care.

Design and method: Twelve of 14 practices and all four primary care organisations (PCOs) approached agreed to participate. Thirty four semi-structured interviews were conducted with key stakeholders. The interviews were taped, transcribed, and analysed using the constant comparative method.

Results: The QTD programme appears to be highly valued by participating organisations. Practice based respondents perceived it as acceptable and feasible, and reported positive changes in teamwork and patient services. They valued its formative, participative, and multiprofessional nature, especially the peer review element. PCOs saw QTD as a method of delivering on prevailing national policies on clinical quality and modernisation agendas as well as promoting interorganisational collaboration. The main concerns raised were the workload, particularly for assessors, and maintaining the quality of the assessments and the programme.

Conclusion: This qualitative study suggests positive benefits for participants in the QTD programme. However, such practices are a self-selecting innovative minority. Further research is needed on more typical practices to identify barriers to their participation in QTD or other formative, team based quality improvement programmes.

     

Can health care quality indicators be transferred between countries?

Objective: To evaluate the transferability of primary care quality indicators by comparing indicators for common clinical problems developed using the same method in the UK and the USA.

Method: Quality indicators developed in the USA for a range of common conditions using the RAND-UCLA appropriateness method were applied to 19 common primary care conditions in the UK. The US indicators for the selected conditions were used as a starting point, but the literature reviews were updated and panels of UK primary care practitioners were convened to develop quality indicators applicable to British general practice.

Results: Of 174 indicators covering 18 conditions in the US set for which a direct comparison could be made, 98 (56.3%) had indicators in the UK set which were exactly or nearly equivalent. Some of the differences may have related to differences in the process of developing the indicators, but many appeared to relate to differences in clinical practice or norms of professional behaviour in the two countries. There was a small but non-significant relationship between the strength of evidence for an indicator and the probability of it appearing in both sets of indicators.

Conclusion: There are considerable benefits in using work from other settings in developing measures of quality of care. However, indicators cannot simply be transferred directly between countries without an intermediate process to allow for variation in professional culture or clinical practice.

     

Developing a framework of, and quality indicators for, general practice management in Europe

OBJECTIVES: To develop a framework for general practice management made up of quality indicators shared by six European countries. METHODS: Two-round postal Delphi questionnaire in the setting of general practice in Belgium, France, Germany, The Netherlands, Switzerland and the United Kingdom. Six national expert panels, each consisting of 10 members, primarily primary care practitioners and experts in the field of quality in primary care participated in the study. The main outcome measures were: (a) a European framework with indicators for the organization of primary care; and (b) ratings of the face validity of the usefulness of the indicators by expert panels in six countries. RESULTS: Agreement was reached about a definition of practice management across five domains (infrastructure, staff, information, finance, and quality and safety), and a common set of indicators for the organization of general practice. The panellist response rate was 95%. Sixty-two indicators (37%) were rated face valid by all six panels. Examples include out of hours service, accessibility, the content of doctors' bags and staff involvement in quality improvement. No indicators were rated invalid by all six panels. CONCLUSIONS: It proved to be possible to develop a European set of indicators for assessing the quality of practice management, despite the differences in health care systems and cultures in the six different countries. These indicators will now be used in a quality assessment procedure of practice management in nine European countries. While organizational indicators are part of the new GMS contract in the UK, this research shows that many practice management issues within primary care are also of relevance in other European countries.     

The use of prescribing indicators to measure the quality of care in psychiatric inpatients

Objective: To explore the potential for using seven prescribing indicators, individually and in combination, to measure prescribing quality for hospitalised psychiatric patients.

Design and setting: The dataset included full details of all psychotropic medication prescribed over a 24 hour period to 4192 inpatients in 49 British mental health services in 1998.

Results: Despite the large size of the dataset, for three of the indicators 20 services had fewer than 10 eligible patients. There was great variation between services in indicator scores. Correlations between standardised indicator scores and total score (which omitted the indicator concerned) were above 0.3 for all but one of the indicators. Cronbach's alpha was 0.73 when this outlying indicator was removed.

Conclusions: There are no routinely collected prescribing data that allow for the quality of prescribing for psychiatric patients to be monitored. Six of the seven indicators measured during this census survey appear to reflect a common attribute of the services, and the analysis suggests that they might be combined to give an overall measure of service performance. There was, however, no relationship between performance on the seventh indicator and performance on the other six. This raises questions about case mix and service level factors that might influence indicator scores independent of prescriber decision making. The psychometric properties of prescribing indicators (occurrence rates, consistency over time) are unknown.

     

Assessing organisational development in primary medical care using a group based assessment: the Maturity MatrixTM

Objective: To design and develop an instrument to assess the degree of organisational development achieved in primary medical care organisations.

Design: An iterative development, feasibility and validation study of an organisational assessment instrument.

Setting: Primary medical care organisations.

Participants: Primary care teams and external facilitators.

Main outcome measures: Responses to an evaluation questionnaire, qualitative process feedback, hypothesis testing, and quantitative psychometric analysis (face and construct validity) of the results of a Maturity Matrix^TM assessment in 55 primary medical care organisations.

Results: Evaluations by 390 participants revealed high face validity with respect to its usefulness as a review and planning tool at the practice level. Feedback from facilitators suggests that it helped practices to prioritise their organisational development. With respect to construct validity, there was some support for the hypothesis that training and non-training status affected the degree and pattern of organisational development. The size of the organisation did not have a significant impact on the degree of organisational development.

Conclusion: This practice based facilitated group evaluation method was found to be both useful and enjoyable by the participating organisations. Psychometric validation revealed high face validity. Further developments are in place to ensure acceptability for summative work (benchmarking) and formative feedback processes (quality improvement).

     

How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings?

Background: While mental health is a core part of primary care, there are few validated quality measures and little relevant internationally published research. Consensus panel methods are a useful means of developing quality measures where evidence is sparse and/or opinions are diverse. However, little is known about the dynamics of consensus techniques and the factors that influence the judgements and ratings of panels and individual panellists.     

Assessing the quality of care of multiple conditions in general practice: practical and methodological problems

Objective: To investigate practical and methodological problems in assessing the quality of care of multiple conditions in general practice.

Setting: Sixteen general practices from two socioeconomically diverse regions in the UK.

Method: Quality of care was assessed in 100 randomly selected patient records in each practice using an established set of quality indicators covering 23 conditions commonly seen in primary care. Inter-rater reliability assessment was carried out for five of the conditions.

Results: Conducting simultaneous quality assessment across multiple conditions is highly resource intensive. Poor data quality and the low prevalence of some items of care defined by the indicators are significant problems. Scores for individual indicators require very large samples for reliable assessment. Quality scores are more reliable when reported at a higher unit of analysis. This is particularly true for indicators and conditions with low prevalence where data may need to be aggregated to the level of groups of conditions or organisational providers. There is no single ideal way of aggregating quality scores.

Conclusion: The study identified some of the practical and methodological difficulties in assessing quality of care across multiple conditions. For improved quality assessment, advances in information technology and improvements in data quality are required for more efficient and reliable data extraction from medical records, together with the development of methods for combining scores across indicators, conditions, and practices. However, electronic data extraction methods will still be based on the assumption that the care recorded reflects the care provided.

     

Health problems and psychosocial work environment as predictors of long term sickness absence in employees who visited the occupational physician and/or general practitioner in relation to work: a prospective study

Aims: To determine whether psychosocial work environment and indicators of health problems are prospectively related to incident long term sickness absence in employees who visited the occupational physician (OP) and/or general practitioner (GP) in relation to work.

Methods: The baseline measurement (May 1998) of the Maastricht Cohort Study, a prospective cohort study among 45 companies and organisations, was used to select employees at work who indicated having visited the OP and/or GP in relation to work. Self report questionnaires were used to measure indicators of health problems (presence of at least one long term disease, likeliness of having a mental illness, fatigue) and psychosocial work environment (job demands, decision latitude, social support, job satisfaction) as predictors of subsequent sickness absence. Sickness absence data regarding total numbers of sickness absence days were obtained from the companies and occupational health services during an 18 month period (between 1 July 1998 and 31 December 1999). Complete data were available from 1271 employees.

Results: After adjustment for demographics and the other predictors, presence of at least one long term disease (OR 2.36; 95% CI 1.29 to 4.29) and lower level of decision latitude (OR 1.69; 95% CI 1.22 to 2.38) were the strongest predictors for sickness absence of at least one month. A higher likelihood of having a mental illness, a higher level of fatigue, a lower level of social support at work, and low job satisfaction were also significant predictors for long term sickness absence, but their effect was less strong.

Conclusion: In detecting employees at work but at risk for long term sickness absence, OPs and GPs should take into account not only influence of the psychosocial work environment in general and level of decision latitude in particular, but also influence of indicators of health problems, especially in the form of long term diseases.

     

Patient perspectives on quality of care for depression and anxiety in primary health care teams: A qualitative study

BackgroundWidespread policy reforms in Canada, the United States and elsewhere over the last two decades strengthened team models of primary care by bringing together family physicians and nurse practitioners with a range of mental health and other interdisciplinary providers. Understanding how patients with depression and anxiety experience newer team‐based models of care delivery is essential to explore whether the intended impact of these reforms is achieved, identify gaps that remain and provide direction on strengthening the quality of mental health care.ObjectiveThe main study objective was to understand patients’ perspectives on the quality of care that they received for anxiety and depression in primary care teams.MethodsThis was a qualitative study, informed by constructivist grounded theory. We conducted focus groups and individual interviews with primary care patients about their experiences with mental health care. Focus groups and individual interviews were recorded and transcribed verbatim. Grounded theory guided an inductive analysis of the data.ResultsForty patients participated in the study: 31 participated in one of four focus groups, and nine completed an individual interview. Participants in our study described their experiences with mental health care across four themes: accessibility, technical care, trusting relationships and meeting diverse needs.ConclusionGreater attention by policymakers is needed to strengthen integrated collaborative practices in primary care so that patients have similar access to mental health services across different primary care practices, and smoother continuity of care across sectors. The research team is comprised of individuals with lived experience of mental health who have participated in all aspects of the research process.     

Patients' experiences and satisfaction with health care: results of a questionnaire study of specific aspects of care

Objective: To determine what aspects of healthcare provision are most likely to influence satisfaction with care and willingness to recommend hospital services to others and, secondly, to explore the extent to which satisfaction is a meaningful indicator of patient experience of healthcare services.

Design: Postal survey of a sample of patients who underwent a period of inpatient care. Patients were asked to evaluate their overall experience of this episode of care and to complete the Picker Inpatient Survey questionnaire on specific aspects of their care.

Sample: Patients aged 18 and over presenting at five hospitals within one NHS trust in Scotland.

Method: 3592 questionnaires were mailed to patients' homes within 1 month of discharge from hospital during a 12 month period. Two reminders were sent to non-responders; 2249 (65%) questionnaires were returned.

Results: Almost 90% of respondents indicated that they were satisfied with their period of inpatient care. Age and overall self-assessed health were only weakly associated with satisfaction. A multiple linear regression indicated that the major determinants of patient satisfaction were physical comfort, emotional support, and respect for patient preferences. However, many patients who reported their satisfaction with the care they received also indicated problems with their inpatient care as measured on the Picker Inpatient Survey; 55% of respondents who rated their inpatient episode as "excellent" indicated problems on 10% of the issues measured on the Picker questionnaire.

Discussion: The evidence suggests that patient satisfaction scores present a limited and optimistic picture. Detailed questions about specific aspects of patients' experiences are likely to be more useful for monitoring the performance of various hospital departments and wards and could point to ways in which delivery of health care could be improved.

     

Is readmission to hospital an indicator of poor process of care for patients with heart failure?

Background: Controversy exists about the appropriateness of using readmission as an indicator of the quality of care. A study was undertaken to measure the validity and predictive ability of readmission in this context.

Methods: An evaluation study was performed in patients discharged alive with heart failure from three Swiss academic medical centres. Process quality indicators were derived from evidence based guidelines for the management and treatment of heart failure. Readmissions were calculated from hospital administrative data. The predictive ability of readmissions was evaluated using bivariate and multivariate analyses, and validity by calculating sensitivity, specificity, positive and negative predictive value, using process indicators as the "gold standard".

Results: Of 1055 eligible patients discharged alive, 139 (13.2%) were readmitted within 30 days. The adjusted odds ratio (OR) for absence of measurement of left ventricular function was 0.70 (95% CI 0.45 to 1.08) for readmissions. In patients with left ventricular systolic dysfunction, three dose categories of angiotensin converting enzyme inhibitor were examined using ordinal logistic regression. The adjusted OR for these categories was 1.07 (95% CI 0.56 to 2.06) for readmissions. When using process indicators as the gold standard to assess the validity of readmissions, sensitivity ranged from 0.08 to 0.17 and specificity from 0.86 to 0.93.

Conclusions: Readmission did not predict and was not a valid indicator of the quality of care for patients with heart failure admitted to three Swiss university hospitals.

     

Association of age, sex and deprivation with quality indicators for diabetes: population-based cross sectional survey in primary care

Objectives To determine the quality of diabetes management in primary care after the publication of the National Service Framework and examine the impact of age, gender and deprivation on the achievement of established quality indicators.

Design Population-based cross sectional survey using electronic general practice records carried out between June-October 2003.

Setting Thirty-four practices in Wandsworth, South-West London, UK.

Participants 6035 adult patients (≥18 years) with diabetes from a total registered population of 201 572 patients.

Interventions None.

Main outcome measures Success rates for the diabetes quality indicators within the General Medical Services contract for general practitioners.

Results We identified large variations in diabetes management between general practitioner practices with poorer recording of quality care in younger patients (18-44 years). In addition, younger patients had a worse cholesterol and glycaemia profile, although hypertension was more common in older patients. Gender and deprivation did not appear to be important determinants of the quality of care received.

Conclusions There are large variations in diabetes management between general practitioner practices, with care seemingly worse for younger adults. Longitudinal studies are required to determine whether current UK quality improvement initiatives have been successful in attenuating existing variations in care and treatment outcomes.

     

Patient and nurse assessment of quality of care in postoperative pain management

Objective: To describe and compare patient and nurse assessments of the quality of care in postoperative pain management, to investigate differences between subgroups of patients, and to compare patient assessments in different departments.

Design: Patient and nurse questionnaires.

Setting: Five surgical wards in general surgery, orthopaedics, and gynaecology in a central county hospital in Sweden.

Sample: Two hundred and nine inpatients and 64 registered nurses. The response rates were 96% for the patients and 99% for the nurses; there were 196 paired patient-nurse assessments.

Method: The Strategic and Clinical Quality Indicators in Postoperative Pain Management patient questionnaire was used which comprises14 items in four subscales (communication, action, trust, and environment). The items were scored on a 5 point scale with higher values indicating a higher quality of care. Five complementary questions on levels of pain intensity and overall satisfaction with pain relief were scored on an 11 point scale. Twelve of the 14 items in the patient questionnaire and two of the complementary questions were adjusted for use in the nurse questionnaire.

Results: The patients' mean (SD) score on the total scale (scale range 14–70) was 58.6 (8.9) and the nurses' mean (SD) score (scale range 12–60) was 48.1 (6.2). The percentage of patients who scored 1 or 2 for an individual item (disagreement) ranged from 0.5% to 52.0%, while for nurses the percentage ranged from 0.0% to 34.8%. Forty two patients (24%) reported more pain than they expected; these patients assessed the quality of care lower. There were differences between patient and nurse assessments concerning the environment subscale, the question on overall satisfaction, and patients' experience of worst possible pain intensity.

Conclusion: The results provided valuable baseline data and identified important areas for quality improvement in postoperative pain management.

     

Team structure, team climate and the quality of care in primary care: an observational study

Objectives: To determine whether practice structure (for example, list size, number of staff) predicts team processes and whether practice structure and team process in turn predict team outcomes

Design: Observational study using postal questionnaires and medical note audit. Team process was assessed through a measure of "climate" which examines shared perceptions of organisational policies, practices, and procedures.

Setting: Primary care.

Subjects: Members of the primary health care team from 42 practices.

Main outcome measures: Objective measures of quality of chronic disease management, patients' evaluations of practices, teams' self-reported ratings of effectiveness, and innovation.

Results: Team climate was better in singlehanded practices than in partnerships. Practices with longer booking intervals provided superior chronic disease management. Higher team climate scores were associated with superior clinical care in diabetes, more positive patient evaluations of practice and self-reported innovation and effectiveness.

Conclusions: Although the conclusions are preliminary because of the limited sample size, the study suggests that there are important relationships between team structure, process, and outcome that may be of relevance to quality improvement initiatives in primary care. Possible causal mechanisms that might underlie these associations remain to be determined.

     

Barriers to managing mental health in Western Australia

Objective: To explore the views of community‐care and mental health workers on barriers to the management of mental health problems in rural Western Australia, and how these could be addressed. Design: Qualitative content analysis of semi‐structured interviews. Setting: Community and mental health services in Esperance. Subjects: One hundred per cent of relevant mental health workers, 86% of community health professionals and representatives from a wide range of community organisations were interviewed (n =38). Main outcome measures: The views of community‐care and mental health workers on barriers to the management of mental health, and how these could be addressed. Results: Barriers included confusion about the role of mental health services, limited after‐hours access and help for those in situational crisis, communication problems between services, differences in working practices and difficulties in dealing with the stigma of mental illness in rural communities. Suggested solutions were an expansion of counselling services and multi‐agency shared care with clinical streams for adults, those aged > 65 and children. Conclusion: This study revealed a number of barriers that are being addressed through a memorandum of understanding between services.     

Measuring the Quality of a Childhood Cancer Care Delivery System: Assessing Stakeholder Agreement

ObjectivesWe described previously the development of a set of quality indicators (QIs) of a childhood cancer system in Ontario, Canada. The purpose of this study was to determine the acceptability of the proposed set of QIs among stakeholders of the childhood cancer system.MethodsA modified Delphi method was used to assess stakeholder agreement on the value of the proposed QIs. A QI evaluation survey was mailed to a stakeholder group of 23 multidisciplinary health care providers, survivors, parents, and policymakers who rated each QI on specific criteria. Prior to an in-person consensus meeting, the distribution of scores was provided to panel members. At the meeting, QIs were reevaluated and discussed in three successive rounds. QIs with 80% or more of panel agreement were considered endorsed.ResultsOverall, 20 QIs were endorsed by the panel, measuring all seven quality dimensions of Ontario’s Cancer System Quality Index framework. Five QIs were endorsed by 100% of the panel as follows: Five-year event-free survival, chemotherapy admission delay, drug availability, sufficient multidisciplinary staff, and parent satisfaction. Although none of the QIs relating to end-of-life or Satellite care were endorsed, panel members emphasized the need to measure these components of the system.ConclusionsStandardized implementation of the 20 pediatric cancer QIs endorsed by the multidisciplinary stakeholder panel will provide ongoing monitoring of various dimensions of system quality and the development of benchmarks over time, greatly augmenting the ability to identify needed system improvements across populations and jurisdictions.