Are audits wasting resources by measuring the wrong things? A survey of methods used to select audit review criteria

Objectives: This study measured the extent to which a systematic approach was used to select criteria for audit, and identified problems in using such an approach with potential solutions.

Design: A questionnaire survey using the Audit Criteria Questionnaire (ACQ), created, piloted, and validated for the purpose. Possible ACQ scores ranged from 0 to 1, indicating how systematically the criteria had been selected and how usable they were.

Setting: A stratified random sample of 10 audit leads in each of 83 randomly selected NHS trusts and all practices in each of 11 randomly selected primary care audit group areas in England and Wales.

Participants: Audit leads of ongoing audits in each organisation in which a first data collection had started less than 12 months earlier and a second data collection was not completed.

Main outcome measures: ACQ scores, problems identified in the audit criteria selection process, and solutions found.

Results: The mean ACQ score from all 83 NHS trusts and the 11 primary care audit groups was 0.52 (range 0.0–0.98). There was no difference between mean ACQ scores for criteria used in audits on clinical (0.51) and non-clinical (0.52) topics. The mean ACQ scores from nationally organised audits (0.59, n=33) was higher than for regional (0.51, n=21), local (0.53, n=77), or individual organisation (0.52, n=335) audits. The mean ACQ score for published audit protocols (0.56) was higher than for locally developed audits (0.49). There was no difference in ACQ scores for audits reported by general practices (0.49, n=83) or NHS trusts (0.53, n=383). Problems in criteria selection included difficulties in coordination of staff to undertake the task, lack of evidence, poor access to literature, poor access to high quality data, lack of time, and lack of motivation. Potential solutions include investment in training, protected time, improved access to literature, support staff and availability of published protocols.

Conclusions: Methods of selecting review criteria were often less systematic than is desirable. Published usable audit protocols providing evidence based review criteria with information on their provenance enable appropriate review criteria to be selected, so that changes in practice based on these criteria lead to real improvement in quality rather than merely change. The availability and use of high quality audit protocols would be a valuable contribution to the evolution of clinical governance. The ACQ should be developed into a tool to help in selecting appropriate criteria to increase the effectiveness of audit.

     

Outcome-directed clinical practice in lumbar spine surgery

Outcome audits describe the current level of clinical performance and direct change in clinical practice. The outcome measures used should be not only relevant and easily understood but also available to all interested parties, e.g. patients, clinicians and commissioners of health. The results of audits can be used to set the standard from which clinical practice can be monitored and improved. An expectation of likely outcome also gives the patient the opportunity of being able to make a fully informed choice. This audit using prospective data examines and compares the outcome of surgery for degenerative lumbar spine disease over a two-year period. The results allow more accurate information to be given to patients, areas of service development to be identified and changes in clinical practice to be made.     

Assessing the work of medical audit advisory groups in promoting audit in general practice.

Objectives--To determine the role of medical audit advisory groups in audit activities in general practice. Design--Postal questionnaire survey. Subjects--All 104 advisory groups in England and Wales in 1994. Main measures--Monitoring audit: the methods used to classify audits, the methods used by the advisory group to collect data on audits from general practices, the proportion of practices undertaking audit. Directing and coordinating audits: topics and number of practices participating in multipractice audits. Results--The response rate was 86-5%. In 1993-4, 54% of the advisory groups used the Oxfordshire or Kirklees methods for classifying audits, or modifications of them. 99% of the advisory groups collected data on audit activities at least once between 1991-2 and 1993-4. Visits, questionnaires, and other methods were used to collect information from all or samples of practices in each of the advisory group's areas. Some advisory groups used different methods in different years. In 1991-2, 57% of all practices participated in some audit, in 1992-3, 78%, and in 1993-4, 86%. 428 multipractice audits were identified. The most popular topic was diabetes. Conclusions--Advisory groups have been active in monitoring audit in general practice. However, the methods used to classify and collect information about audits in general practices varied widely. The number of practices undertaking audit increased between 1991-2 and 1993 1. The large number of multipractice audits supports the view that the advisory groups have directed and coordinated audit activities. This example of a national audit programme for general practice may be helpful in other countries in which the introduction of quality assurance is being considered.     

Approaches to the organization of multi-practice audits in primary health care in the UK.

OBJECTIVE: To investigate the approaches taken by audit groups in primary care in organizing multi-practice audits and to identify the strengths and weaknesses of the methods used. DESIGN: Postal questionnaire survey. SETTING: One hundred and six primary care audit groups in England and Wales. RESULTS: Ninety multi-practice audits had been conducted since 1993, 46 of which were audits of diabetes and 44 of asthma care. A total of 48 completed questionnaires (24 each for asthma and diabetes) were returned (response rate 53%). Audit groups reported inviting 3338 practices to take part, of which 1157 completed the audit. The commonest methods used to encourage practice participation were a personal letter (75%), audit group newsletter (63%) and sending an audit protocol to the practice (63%). Groups used various methods for selecting audit review criteria, however only three (6%) used a systematic review of available literature. Each audit group advocated a number of methods for identifying patients and for data extraction. Forty-one (85.6%) groups reported that practices received feedback of results in an individualized practice feedback report. In 19 (39.6%) audits, the audit group had not undertaken any follow-up. CONCLUSIONS: The findings indicate that multi-practice audit can encourage the participation of large numbers of practices. Audit groups are co-ordinating multi-practice audits and feeding back information to practices on a comparative basis. However, there are weaknesses in the design and conduct of some audits. Groups should pay more attention to the selection of audit criteria, methods of identifying and sampling patients, data collection procedures, and methods for implementing changes in performance. For other countries that are beginning quality improvement activities, the results of this study emphasize the need to give attention to basic methodological principles.     

Cancer screening by primary care physicians: a comparison of rates obtained from physician self-report, patient survey, and chart audit.

OBJECTIVES. This study measured the cancer screening rates of family physicians and compared the measures obtained through physician self-reports, chart audits, and patient surveys. METHODS. A cancer screening survey was sent to 50% of the members of the Washington Academy of Family Physicians, with 326 family physicians (74% response rate) completing the survey. Sixty physicians were recruited for the patient survey and chart audit phase, with a 90% participation rate. Patient surveys were conducted with about 350 patients per physician, and chart audits were conducted on a subset of about 50 patients per physician. Each physician's rate of providing each service was computed from the self-report, the patient survey, and the chart audit. RESULTS. Physicians provided many of these services at rates different from those commonly recommended. Large discrepancies were found between the rates measured by different methods. Correlations between rates derived from chart audits and patient surveys were high; however, correlations between rates from physician self-report and either patient survey or chart audit were much lower for all services. CONCLUSIONS. Studies of physicians' provision of cancer prevention services should not rely on physician self-report, but should obtain the rates through patient surveys or chart audits.     

Why is evaluation of the cost effectiveness of audit so difficult? The example of thrombolysis for suspected acute myocardial infarction.

BACKGROUND: Cost effectiveness analysis is an established technique for evaluation of delivery of health care, but its use to evaluate clinical audit is rarely reported. Thrombolysis for suspected acute myocardial infarction is a commonly used therapy of established effectiveness and an appropriate subject for audit in many healthcare settings. OBJECTIVE: To measure the cost effectiveness of audit of thrombolysis in some district general hospitals. MAIN OUTCOME MEASURE: Cost of audit per extra patient treated with thrombolysis (incremental cost effectiveness ratio). DESIGN: Prospective agreement with physicians to undertake repeated audits of a specific aspect of the management of patients with acute myocardial infarction. Baseline measurement of the proportion of these patients given thrombolysis in each hospital were made, as were three subsequent retrospective audits, giving time series of measurements. Costs were estimated from records of staff time and other resources used in each hospital; effectiveness was estimated by fitting the results to a model which assumed a uniform rate of increase over time in the proportion of eligible patients given thrombolysis which might be accelerated by regular audit. Upper and lower limits for main outcome measure were derived from sensitivity analysis of costs and logistic regression of time series data. SETTING: Five district general hospitals in North West Thames Regional Health Authority including one control hospital were used, starting in April 1991 when widespread medical audit was first introduced. RESULTS: Between the first and last audits, the proportion of patients with suspected acute myocardial infarctions receiving thrombolysis rose in three of the hospitals undertaking audit by 20% to 37% and fell by 6% in the fourth (although this hospital started with a rate in excess of 90%). The corresponding change in the control hospital was an increase of 15%. The differences between each of the auditing hospitals and control hospital were not significant, except in one case, where 51 extra treatments per year were attributable to audit (95% confidence intervals (95% CIs) 0.5 to 61 cases per year). Estimated overall costs in each hospital ranged from 3700 Pounds to 5200 Pounds for data collection, a series of four audit meetings, and subsequent actions. The central estimate of cost effectiveness in the three responsive hospitals ranged from 101 Pounds to 392 Pounds per extra case given thrombolysis, with very wide 95% CIs. In the fourth hospital audit had zero effectiveness as defined in this study. CONCLUSIONS: Methodological difficulties were encountered which need to be considered in future economic evaluations of clinical audit and related activities. These were: (a) adequate control for other factors influencing clinical behaviour; (b) uncertainties about the sustainability of changes in behaviour associated with audit; and (c) the relative infrequency in a single hospital of specific clinical events leading to small numbers for analysis. These difficulties constitute major challenges for the economic evaluation of clinical audit. They are most likely to be overcome in a large study which compares clinical audit with other interventions aiming for the same quality improvement, such as patient specific reminders or educational programmes.     

Can Virtual Streetscape Audits Reliably Replace Physical Streetscape Audits?

There is increasing recognition that the neighborhood-built environment influences health outcomes, such as physical activity behaviors, and technological advancements now provide opportunities to examine the neighborhood streetscape remotely. Accordingly, the aims of this methodological study are to: (1) compare the efficiencies of physically and virtually conducting a streetscape audit within the neighborhood context, and (2) assess the level of agreement between the physical (criterion) and virtual (test) audits. Built environment attributes associated with walking and cycling were audited using the New Zealand Systematic Pedestrian and Cycling Environment Scan (NZ-SPACES) in 48 street segments drawn from four neighborhoods in Auckland, New Zealand. Audits were conducted physically (on-site) and remotely (using Google Street View) in January and February 2010. Time taken to complete the audits, travel mileage, and Internet bandwidth used were also measured. It was quicker to conduct the virtual audits when compared with the physical audits (χ = 115.3 min (virtual), χ = 148.5 min (physical)). In the majority of cases, the physical and virtual audits were within the acceptable levels of agreement (ICC ≥ 0.70) for the variables being assessed. The methodological implication of this study is that Google Street View is a potentially valuable data source for measuring the contextual features of neighborhood streets that likely impact on health outcomes. Overall, Google Street View provided a resource-efficient and reliable alternative to physically auditing the attributes of neighborhood streetscapes associated with walking and cycling. Supplementary data derived from other sources (e.g., Geographical Information Systems) could be used to assess the less reliable streetscape variables.     

A natural survey of audit activity across the primary-secondary care interface.

OBJECTIVE: To document the nature of audit activity at the primary-secondary care interface; to explore participants' experiences of undertaking such interface audit; to identify factors associated with these experiences; and to gather views on future interface audit activities. DESIGN: A three phase national survey by postal questionnaire with a cascade sampling approach. SETTING: England and Wales. RESULTS: Response rates were: 65% to the first questionnaire; 34% to the second questionnaire; and 45% to the third questionnaire. 56% of the audits covered some element of management of patients or disease; only 33% of the audits were within a single topic area. Most audits had more than one trigger: for 61% the trigger was a perceived problem; for 58% it was of mutual interest. Only 18% of audits were initiated collaboratively; doctors were the most frequent initiators (72%), and most audits (63%) involved collaborative groups convened specifically for the audit. 58% of groups had between three and eight members, 23% had 12 or more. Doctors were the most frequent group members. There was differential involvement of group members in various group tasks; the setting of guidelines was highly dominated by doctors. Of reportedly complete audits, only two fifths had implemented change and only a quarter had evaluated this change. There was widespread feeling of successful group work, with evidence of benefit in terms of the two sectors of care being able to consider issues of mutual concern. Levels of understanding of the group task and of participation were positively related to the duration of meetings. Joint initiation of audits facilitated greater understanding of the group task. Larger group sizes allowed primary and secondary carers to discuss issues of common concern; however, larger groups were more likely to experience disagreements. Having previously worked with group members increased trust and good working relations. The main lessons learnt from the experience included the importance of setting clear objectives and good communications between primary and secondary carers. Factors identified as important for future audit activity at the primary-secondary care interface included commitment, enthusiasm, time, and money. CONCLUSIONS: Audit at the primary-secondary care interface is taking place on a wide scale and has been an enjoyable experience for most of the respondents in this study. IMPLICATIONS: Despite being a positive experience most audits stopped short of implementing change. Care must be taken to complete the audit cycle if audit at the primary-secondary care interface is to move beyond the roles of education and professional development and to fulfil its potential in improving the quality of care.     

Use of a chart audit: teaching well child care to paediatric house officers

Summary. A prospective study employing a randomly assigned control group was conducted to assess the usefulness of a chart audit in teaching paediatric residents the components of well child care. The charts of children less than 5 years of age were reviewed and compared with audit criteria. Per cent compliance scores were calculated for five categories: present history; behaviour-development; family history; past medical history; and physical assessment. Five separate audits (10 charts per resident per audit) were conducted — two prior to giving the residents feedback, one after informing them that a study was being conducted, one a month after giving feedback, and one a year later. There was no significant difference between the baseline scores of the two groups. In addition, there were no significant changes in the experimental group's scores during the first three audits or the control group's scores over the whole 3-year course of the study. However, one month after receiving feedback, the scores of the experimental group improved significantly in present history, behaviour-development, and past history. One year later, the experimental group's scores were lower in every category than in the preceding audit. However, their scores were higher than the control group and the difference reached statistical significance in present history. We conclude that regular chart audits with feedback are a valuable addition to the primary care curriculum in a paediatric residency programme.     

Audit in infection control

Audit is a key function of infection control teams. Infection control audit programmes should include audits of infection control policies in wards and departments, and microbiological safety audits of the healthcare environment. This paper reviews the literature on healthcare audit with particular emphasis on published audits in infection control. Evidence of the efficacy of audit and feedback in improving infection control outcomes is presented, together with the nature of interventions necessary to bring about change.     

Development and validation of automated microscale walkability audit method

Measuring microscale factors of walkability has been labor-intensive and expensive. To reduce the cost, various efforts have been made including virtual audits (i.e., manual audits using street view images) and the introduction of computer vision techniques. Although studies have shown that virtual audits (i.e., manual audits using street view images) can reliably replicate in-person audits, they are still prohibitively expensive to be applied to a large geographic area. Past studies used computer vision techniques to help automate the audit process, but off-the-shelf models cannot detect some of the important microscale walkability characteristics, falling short of fully capturing the multi-facetted concept of walkability. This study is one of the earliest attempts to use the combination of custom-trained computer vision models, geographic information systems, and street view images to automatically audit a complete set of items of a validated microscale walkability audit tool. This study validates the reliability of the automated audit with virtual audit results. The automated audit results show high reliability, indicating automated audit can be a highly scalable and reliable replacement of virtual audit.     

Quality of care of patients with diabetes: collation of data from multi-practice audits of diabetes in primary care.

BACKGROUND: GPs are now playing a greater role in the care of patients with diabetes. The challenges described in the Saint Vincent Joint Task Force Report include achievement of a reduction in long-term complications by collecting key clinical information and systematically organizing care of patients with diabetes. The number of practices conducting audit and the number of primary care audit groups conducting multi-practice audits of diabetes have increased since the introduction of audit in 1991. OBJECTIVES: We aimed to determine the feasibility of collating data from multi-practice audits of diabetes in primary care and to describe the pattern of care for diabetes patients in primary care. METHODS: A confidential postal questionnaire was sent to all medical audit advisory groups that had completed a multi-practice audit of diabetic care. The main outcome measures studied were prevalence and treatment of known diabetes and annual compliance with key process measures. RESULTS: Data could be collated for 17 of the 25 audit groups that supplied data representing information from 495 practices with 38 288 diabetic patients. Seven audit groups supplied data from a population denominator comprising 1475512 patients giving a prevalence of 1.46% (range 1.1-1.7%), 50.7% (range 32.5-69.0%) were managed by general practice only, 19.1% (7.6-39.7%) by hospital care only and 30.2% (11.0-49.5%) by shared care. Annual mean compliance for process measures showed wide variations: glycated haemoglobin or fructosamine checked for 72.5% (range 25.3-89.3%), fundi checked for 67.5% (57.8-86.6%), urine checked for 65.8% (27.5-80.0%), blood pressure checked for 87.6% (76.9-96.5%), smoking checked for 71.45 (21.9-86.0%), feet checked for 67.7% (40.0-90.8%) and BMI checked for 52.5% (26.4-68.2%). CONCLUSION: This study shows the feasibility of collating audit data and the potential of this approach for describing patterns of care and highlighting general and local deficiencies. Information about levels of performance in large numbers of patients can be used to set standards or norms against which individual practitioners can compare their own activity. Comparison of the health needs of local populations with national data could be used to inform commissioning services. However, audits should employ uniform evidence-based criteria so as to facilitate collation and allow comparison.     

Home health audit tool--at last!

Dissatisfied with the use of an audit tool not applicable to home health nursing, administrators of the Missoula Home Health Agency formed a special audit committee to review audits used by agencies around the country. After examining 50 audit tools, the committee agreed that the Phaneuf public health nursing audit did not adequately emphasize nursing care and charting specific to home health agencies but did fit most of their criteria. After redesigning the Phaneuf audit and testing it through two quarterly audits, the new Missoula Home Health Audit tool has been found very satisfactory. The Missoula Audit emphasizes: 1. home health's goal of independent living and/or self-care; 2. service provider-family-patient relationship; 3. nurse-physician relationship; 4. charting and recording; 5. coordination of careproviders. Unlike many existing audit tools, the Missoula Audit is specific to home health nursing, covers the spectrum of home health services, has sensitive scoring which reflects true differences in quality of care, and initially, at least, appears to be consistent from audit to audit.     

Audit of near patient cholesterol testing in occupational health clinics

An audit of near patient cholesterol testing was carried outin occupational health clinics. The aims were to examine thestatistical agreement between Reflotron and laboratory measurementsof blood cholesterol and to formulate a policy for the use ofReflotrons in cholesterol testing. Three hundred and fifty-twostaff members attending occupational health clinics over a periodof 10 months had blood taken for both Reflotron and laboratorymeasurements. The correlation between the two methods was 0.95.The Reflotron had a negative bias compared to the laboratory,with the mean difference between the two methods of measurementbeing –0.21 mmol/l (95 per cent confidence interval –0.18to –0.25mmol/l). Despite the high correlation coefficientand small mean difference, the scatter of Reflotron-laboratorydifferences was broad, with 95 per cent of the differences lyingin the range of 0.95 mmol/l below to 0.52 mmol/l above the laboratoryresult. For Reflotron results of 5.50 mmol/l and greater, thesensitivity and specificity of the Reflotron in detecting subjectswith laboratory cholesterol levels greater than 6.5 mmol/l were100 per cent and 70 per cent respectively. The laboratory participatedin two external quality assessment schemes for cholesterol testingduring the course of the audit and all the results of thesefell within the acceptable limits. The audit demonstrated thatthe Reflotron was too imprecise to be used to give accuratemeasurements of blood cholesterol. However, providing a suitableReflotron result above which patients were sent for confirmatorylaboratory testing was selected, it was an acceptable screeningdevice in the detection of hypercholesterolaemia. Other Reflotronusers should consider carrying out similar audits.     

Hospital pharmacists'' participation in audit in the United Kingdom.

OBJECTIVE--To investigate systematically participation in audit of NHS hospital pharmacists in the United Kingdom. DESIGN--Questionnaire census survey. SETTING--All NHS hospital pharmacies in the UK providing clinical pharmacy services. SUBJECTS--462 hospital pharmacies. MAIN MEASURES--Extent and nature of participation in medical, clinical, and pharmacy audits according to hospital management and teaching status, educational level and specialisation of pharmacists, and perceived availability of resources. RESULTS--416 questionnaires were returned (response rate 90%). Pharmacists contributed to medical audit in 50% (204/410) of hospitals, pharmacy audit in 27% (108/404), and clinical audit in only 7% (29/404). Many pharmacies (59% (235/399)) were involved in one or more types of audit but few (4%, (15/399)) in all three. Participation increased in medical and pharmacy audits with trust status (medical audit: 57% (65/115) trust hospital v 47% (132/281) non-trust hospital; pharmacy audit: 34% (39/114) v 24% (65/276)) and teaching status (medical audit: 58% (60/104) teaching hospital v 47% (130/279) non-teaching hospital; pharmacy audit 30% (31/104) v 25% (68/273)) and similarly for highly qualified pharmacists (MPhil or PhD, MSc, diplomas) (medical audit: 54% (163/302) with these qualifications v 38% (39/103) without; pharmacy audit: 32% (95/298) v 13% (13/102)) and specialists pharmacists (medical audit: 61% (112/184) specialist v 41% (90/221) non-specialist; pharmacy audit: 37% (67/182) v 19% (41/218)). Pharmacies contributing to medical audit commonly provided financial information on drug use (86% 169/197). Pharmacy audits often concentrated on audit of clinical pharmacy services. CONCLUSION--Pharmacists are beginning to participate in the critical evaluation of health care, mainly in medical audit.     

Obstetric audit in resource-poor settings: lessons from a multi-country project auditing 'near miss' obstetrical emergencies

This paper outlines the practical steps involved in setting up and running multi-professional, in-depth case reviews of 'near miss' obstetrical complications. It draws on lessons learned in 12 referral hospitals in Benin, C?te d'Ivoire, Ghana and Morocco. A range of feasibility indicators are presented which measured the implementation and frequency of audit activities, the quality of participation, adherence to the planned protocol for the near-miss audits, the quality of audit discussions and the sustainability of the project. Although the principles of the audit approach were well accepted and implemented everywhere, near-miss audits appeared most successful in first referral level hospitals. Contextual factors that determine the successful implementation of near-miss audit include staff finding adequate time for audit activities, financial incentives to groups rather than individuals, involvement of senior staff and hospital managers, the ease of communication in smaller units, the employment of social workers for the incorporation of women's views at audits, and the strength of external support provided by the research team. The poor quality of information recorded in case notes was recognized everywhere as a deficiency, but did not present a major obstacle to effective case reviews. Ownership and leadership within the hospital, more easily achieved in the first-level referral hospitals, were probably the most important determinants of successful implementation. Sustainability requires a commitment to audit from policy makers and managers at higher levels of the health system and some devolution of resources for implementing recommendations.     

Completing the clinical governance cycle in dietetic gastroenterology outpatients

Background:  To overcome high rates of non-attendance, inappropriate referrals and long waiting times, an audit was undertaken of a dietetic outpatients clinic for gastroenterology patients in 2003 and then repeated in 2007. The aim of the first audit was to identify referral source, types of patient, attendance rates and if dietary advice had a positive outcome. This study aimed evaluated changes since 2003.
Method:  The clinic diary was used to identify patients booked to attend in 2003 and 2007. Information was retrieved from the patient's record on source of referral, types of referral, waiting times, attendance rates and dietary goals achieved. The results were compared to determine if changes had improved service delivery. The data were analyzed in Microsoft Excel (2003) and the attendance rates were compared using chi-squared statistics.
Results:  There were 92 new patients in 2003 and 69 patients in 2007; data were obtained for n  = 77 in 2003 and n  = 55 in 2007. There was an increase in referrals from gastroenterologists ( n  = 31, 55%) in 2007 compared with 2003 ( n  = 23, 34%). Waiting times improved between the two audits with 44% being seen within 3 months in 2003 compared to 51% in 2007. Attendance rates improved by 30% when the two audits were compared and in both years 2003 and 2007 (p = 0.009), a positive outcome was recorded for the majority of patients who had completed their treatment episodes 78% and 63% respectively.
 

     

20.

Building the community voice into planning: 25 years of methods development in social audit     

Andersson N 《BMC health services research》2011,11(Z2):S1

Health planners and managers make decisions based on their appreciation of causality. Social audits question the assumptions behind this and try to improve quality of available evidence. The method has its origin in the follow-up of Bhopal survivors in the 1980s, where "cluster cohorts" tracked health events over time. In social audit, a representative panel of sentinel sites are the framework to follow the impact of health programmes or reforms. The epidemiological backbone of social audit tackles causality in a calculated way, balancing computational aspects with appreciation of the limits of the science.Social audits share findings with planners at policy level, health services providers, and users in the household, where final decisions about use of public services rest. Sharing survey results with sample communities and service workers generates a second order of results through structured discussions. Aggregation of these evidence-based community-led solutions across a representative sample provides a rich substrate for decisions. This socialising of evidence for participatory action (SEPA) involves a different skill set but quality control and rigour are still important.Early social audits addressed settings without accepted sample frames, the fundamentals of reproducible questionnaires, and the logistics of data turnaround. Feedback of results to stakeholders was at CIET insistence--and at CIET expense. Later social audits included strong SEPA components. Recent and current social audits are institutionalising high level research methods in planning, incorporating randomisation and experimental designs in a rigorous approach to causality.The 25 years have provided a number of lessons. Social audit reduces the arbitrariness of planning decisions, and reduces the wastage of simply allocating resources the way they were in past years. But too much evidence easily exceeds the uptake capacity of decision takers. Political will of governments often did not match those of donors with interest conditioned by political cycles. Some reforms have a longer turnaround than the political cycle; short turnaround interventions can develop momentum. Experience and specialisation made social audit seem more simple than it is. The core of social audit, its mystique, is not easily taught or transferred. Yet teams in Mexico, Nicaragua, Canada, southern Africa, and Pakistan all have more than a decade of experience in social audit, their in-service training supported by a customised Masters programme.     

Building the community voice into planning: 25 years of methods development in social audit

Health planners and managers make decisions based on their appreciation of causality. Social audits question the assumptions behind this and try to improve quality of available evidence. The method has its origin in the follow-up of Bhopal survivors in the 1980s, where "cluster cohorts" tracked health events over time. In social audit, a representative panel of sentinel sites are the framework to follow the impact of health programmes or reforms. The epidemiological backbone of social audit tackles causality in a calculated way, balancing computational aspects with appreciation of the limits of the science.Social audits share findings with planners at policy level, health services providers, and users in the household, where final decisions about use of public services rest. Sharing survey results with sample communities and service workers generates a second order of results through structured discussions. Aggregation of these evidence-based community-led solutions across a representative sample provides a rich substrate for decisions. This socialising of evidence for participatory action (SEPA) involves a different skill set but quality control and rigour are still important.Early social audits addressed settings without accepted sample frames, the fundamentals of reproducible questionnaires, and the logistics of data turnaround. Feedback of results to stakeholders was at CIET insistence--and at CIET expense. Later social audits included strong SEPA components. Recent and current social audits are institutionalising high level research methods in planning, incorporating randomisation and experimental designs in a rigorous approach to causality.The 25 years have provided a number of lessons. Social audit reduces the arbitrariness of planning decisions, and reduces the wastage of simply allocating resources the way they were in past years. But too much evidence easily exceeds the uptake capacity of decision takers. Political will of governments often did not match those of donors with interest conditioned by political cycles. Some reforms have a longer turnaround than the political cycle; short turnaround interventions can develop momentum. Experience and specialisation made social audit seem more simple than it is. The core of social audit, its mystique, is not easily taught or transferred. Yet teams in Mexico, Nicaragua, Canada, southern Africa, and Pakistan all have more than a decade of experience in social audit, their in-service training supported by a customised Masters programme.