Do proficiency testing participants learn from their mistakes? Experience from the EXCEL throat culture module

CONTEXT: Participation in proficiency testing is required under the Clinical Laboratory Improvement Amendments of 1988. Although the primary purpose of this testing is evaluation of current laboratory performance, a major secondary benefit of such testing is postulated to be progressive improvement in laboratory performance over time as laboratories learn from their previous experiences and feedback. OBJECTIVE: To test the hypothesis that a secondary result of proficiency testing is improvement over time of laboratory performance. DESIGN: The performance of participants in a large proficiency testing program (EXCEL), designed for clinic and office laboratories, on a specific problematic competence, the ability to differentiate group A streptococcus from group C streptococci, was monitored during a 6-year period (1996-2001) for changes in participant performance. INTERVENTIONS: With each testing cycle, feedback on performance relative to peers and an educational discussion analyzing performance and suggesting best practices was submitted to participants. RESULTS: Despite consistent feedback, there was no significant change in participant performance throughout the period studied. CONCLUSIONS: In a large, stable proficiency testing program, a significant throat culture competence, which demonstrated less than optimal performance, did not improve over time, suggesting that current utilization of proficiency testing results in laboratory improvement programs is suboptimal.     

Development and implementation of an international proficiency testing program for a neutralizing antibody assay for HIV-1 in TZM-bl cells

Recent advances in assay technology have led to major improvements in how HIV-1 neutralizing antibodies are measured. A luciferase reporter gene assay performed in TZM-bl (JC53bl-13) cells has been optimized and validated. Because this assay has been adopted by multiple laboratories worldwide, an external proficiency testing program was developed to ensure data equivalency across laboratories performing this neutralizing antibody assay for HIV/AIDS vaccine clinical trials. The program was optimized by conducting three independent rounds of testing, with an increased level of stringency from the first to third round. Results from the participating domestic and international laboratories improved each round as factors that contributed to inter-assay variability were identified and minimized. Key contributors to increased agreement were experience among laboratories and standardization of reagents. A statistical qualification rule was developed using a simulation procedure based on the three optimization rounds of testing, where a laboratory qualifies if at least 25 of the 30 ID50 values lie within the acceptance ranges. This ensures no more than a 20% risk that a participating laboratory fails to qualify when it should, as defined by the simulation procedure. Five experienced reference laboratories were identified and tested a series of standardized reagents to derive the acceptance ranges for pass-fail criteria. This Standardized Proficiency Testing Program is the first available for the evaluation and documentation of assay equivalency for laboratories performing HIV-1 neutralizing antibody assays and may provide guidance for the development of future proficiency testing programs for other assay platforms.     

输注器具铅镉含量能力验证

目的通过输注器具铅镉含量能力验证计划评价医疗器械检验机构,为提高实验室检测能力提供指导。方法依据CNAS-CL03:2010《能力验证提供者认可准则》策划并实施本次能力验证。采用模拟输注器具溶出液作为样品,根据随机原则发放给参加者,要求参加者按照作业指导书规定的步骤采用原子吸收分光光度法测定样品中铅镉含量。通过计算专家实验室公仪值获得指定值,计算上一轮次结果的散度作为能力评定标准差。结果全国共有30个省的58家实验室参加,其中39家结果满意,18家不满意,1家退出。通过结果分析参加者应在3个方面提高检验能力,一是设备核心部分加强维护保养,二是按照SOP进行试验操作,三是加强环境监控避免干扰。结论客观准确地评价了全国医疗器械检验机构铅镉含量项目的检验能力,发现并分析了问题,为实验室提高检验能力提供指导。     

Laboratory accreditation and proficiency testing

ISO/TC 212 covering clinical laboratory testing and in vitro diagnostic test systems will issue the international standard for medical laboratory quality and competence requirements, ISO 15189. This standard is based on the ISO/IEC 17025, general requirements for competence of testing and calibration laboratories and ISO 9001, quality management systems-requirements. Clinical laboratory services are essential to patient care and therefore should be available to meet the needs of all patients and clinical personnel responsible for human health care. If a laboratory seeks accreditation, it should select an accreditation body that operates according to this international standard and in a manner which takes into account the particular requirements of clinical laboratories. Proficiency testing should be available to evaluate the calibration laboratories and reference measurement laboratories in clinical medicine. Reference measurement procedures should be of precise and the analytical principle of measurement applied should ensure reliability. We should be prepared to establish a quality management system and proficiency testing in clinical laboratories.     

The application of theoretical goals based on biological variation data in proficiency testing

Proficiency testing schemes must define acceptable standards of performance. A number of strategies have been used to define acceptable standards, or analytical goals, for imprecision, but it does appear that the current widely held view is that imprecision (as SD or coefficient of variation) should be equal to or less than one half of the average intraindividual biological variation. Since the goal for inaccuracy is that results should be unbiased, the biological variation-based goals should be used for total laboratory error. Many data on the biological variation of a wide range of analytes now exist, which facilitates goal-setting. Current proficiency testing schemes generally analyze laboratory data using either a parametric statistical approach or empirical fixed limits of acceptability. The latter strategy appears to have advantages provided that the criteria of acceptability are the objective biological variation-based goals.     

On the Origins of Clinical Psychology Faculty: Who Is Training the Trainers?

There exists little published research regarding the comparative performance of graduate programs in clinical psychology on relevant measures of program achievement. The present report thus aims to provide information about one such measure—program proficiency in training graduates to assume clinical psychology faculty positions. Degree-granting institution and year of degree completion were obtained for 1,529 individuals listed as core faculty at 150 university-based clinical psychology Ph.D. programs accredited by the American Psychological Association. On this basis, leading programs (i.e., those having trained numerous clinical faculty members) are identified. Proficiency in placing graduates in clinical faculty positions was moderately positively correlated with program reputational strength; it was not significantly associated with program size. A set of recommendations for the systematic investigation of factors germane to such proficiency, as well as to program achievement in other important (and heretofore unstudied) domains, is proffered. It is argued that no single measure is adequate as an overall gauge of program excellence.     

The Medicare and Clinical Laboratories Improvement Act of 1967 proficiency testing requirements and its relationship to the private sector

The Health Care Financing Administration of the US Department of Health and Human Services is the federal agency that has the responsibility for the administration of the Medicare and Clinical Laboratories Improvement Act of 1967 programs, including the promulgation of standards for clinical laboratories. The Health Care Financing Administration has been working with the Centers for Disease Control of the US Public Health Service to develop uniform standards for proficiency testing programs for the two regulatory programs. The current standards were developed on the premise that a combination of factors, including personnel standards, record-keeping, management requirements, quality control standards, and external quality assurance measures such as proficiency testing, could be used to make decisions on reimbursing facilities for tests or to allow a facility to test in interstate commerce. These programs were regulatory in nature and provided punitive actions for failure to comply with the standards established for participation or licensure. The provision of consultation and training was not the primary focus of the program, even if it was a desirable outcome. The private sector organizations and the regulated industry, on the other hand, viewed their programs as designed for educational and self-improvement purposes rather than for any regulatory or punitive functions. Therefore, the approaches for the regulatory agencies and the professional community have differed.     

Achieving quality reproducible results and maintaining compliance in molecular diagnostic testing of human papillomavirus

Laboratories contemplating either the addition of new molecular tests or modifying methods approved by the Food and Drug Administration for human papillomavirus testing should be aware of a variety of procedural, performance, and regulatory issues surrounding such activity. Diagnostic medical laboratory testing in the United States is regulated by the Centers for Medicare and Medicaid Services, an agency formerly known as the Health Care Finance Administration. The regulatory vehicle of the Centers for Medicare and Medicaid Services is manifested in the Clinical Laboratory Improvement Amendments (CLIA). The CLIA program has put into place specific regulations for laboratory quality control, which includes specific recommendations for method validation. Regulations that must be followed regarding personnel, quality control, quality assurance, method validation, and proficiency testing depend on the complexity category of the individual test. All molecular diagnostic tests, including those for human papillomavirus, are considered high complexity. The Centers for Medicare and Medicaid Services retains the authority to allow private, national accreditation organizations to "deem" that a laboratory is compliant with CLIA '88 requirements. Accreditation organizations, such as the Joint Commission for Accreditation of Hospitals, the Commission on Office Laboratory Accreditation, and the College of American Pathologists (CAP), as well as several state medical laboratory-accrediting agencies, possess the authority to deem laboratories as "CLIA-approved." The CAP, through its Laboratory Accreditation Program, has promoted standards for laboratory performance and method validation. In general, guidelines set forth in the CAP Laboratory Accreditation Program checklists specify that all clinical laboratory testing must essentially meet those requirements defined for high-complexity testing under CLIA '88, including test validation standards, reportable/reference ranges, performance criteria, and proficiency testing.     

Use of proficiency test performance to determine clinical laboratory director qualifications

Many activities and policies influence laboratory test quality. Proficiency test results are one measure of laboratory quality, and during the past 25 years, five studies have examined the relationship of laboratory director educational requirements to proficiency test results. Data from three studies support the association between director qualifications and quality as measured by proficiency test performance, whereas no relationship was found in the other two studies. Possible reasons for conflicting results include differences in database size and demographics; in addition, proficiency test results may be inappropriate, although widely used, as the sole measure of laboratory director performance.     

Requirements for accreditation by the College of American Pathologists Laboratory Accreditation Program.

The College of American Pathologists Laboratory Accreditation Program expects a participant laboratory or laboratory section to be able to demonstrate that it is in compliance with the Standards for Laboratory Accreditation. The program expects laboratories to demonstrate that they are continually taking steps to identify and correct deficient areas and improve performance, in compliance with the Clinical Laboratory Improvement Amendments of 1988 regulatory requirements, particularly those pertaining to proficiency testing performance, and participating as inspectors in the accreditation process.     

Reliability of white blood cell counting

The Laboratory Proficiency Testing Program has successfully operated mandatory proficiency testing for 13 years in Ontario, Canada, using committees of volunteer peers. This is an alternative to gubernatorial proficiency testing. Using white blood cell (WBC) count determinations as a focus, we present our results. Ninety-six percent of participants attained WBC count results within +/- 10% of an all-methods mean and 77% attained results within +/- 5%. Ninety-six percent displayed satisfactory precision. Details of the program, ie, testing material, computer analysis, reference results, and scope of committee action, are discussed. In 1975, clinical expectations for WBC count determinations greatly exceeded most laboratories' capabilities. By 1987, participants were able to produce reliable, clinically useful WBC count results. The paramount reason for this improvement is the transition from manual to automated counting methods resulting from the stimulus of proficiency testing and the concomitant education.     

Use of Proficiency Testing as a Tool to Improve Quality in Microbiology Laboratories

Proficiency testing (PT) is a valuable tool for assessing laboratory performance and verifying the accuracy and reliability of test results. Participation is required by the Clinical Laboratory Improvement Amendments (CLIA) of 1988 for each of the microbiology subspecialties (bacteriology, mycobacteriology, mycology, parasitology, and virology), and the regulations include specific PT requirements for each subspecialty. To determine the use and perceived value of PT beyond meeting CLIA requirements, the Centers for Disease Control and Prevention funded a cooperative agreement with the Association of Public Health Laboratories to convene a series of focus groups to query laboratory professionals responsible for PT. The seven focus groups were comprised of 60 laboratory professionals representing large and small clinical laboratories, microbiology subspecialties, and public health. While participants acknowledged the need to perform PT to meet regulatory requirements, many also cited benefits and challenges beyond regulatory compliance.     

Effects of restructuring on the performance of microbiology laboratories in Alberta

OBJECTIVE: To evaluate the error rates of organism identification and antibiotic susceptibility proficiency testing challenges before, during, and after microbiology laboratory restructuring in Alberta. METHODS: Alberta Health substantially reduced and redistributed laboratory funds to the regional health authorities in 1995, forcing a dramatic restructure of services. Many rural hospitals expanded their microbiology test menus, and urban centers consolidated microbiology testing into a centralized high-volume laboratory. The Laboratory Proficiency Testing Program of the College of Physicians and Surgeons of Alberta mailed regular test profile surveys to microbiology laboratories during the restructure period to determine the type and extent of changes in services. Based on the types of tests and the extent of analysis being done, most rural B-level and some C-level laboratories were reclassified to the A level. The Laboratory Proficiency Testing Program reviewed the error rates of proficiency challenges based on the performance of different levels of laboratories before and after the period of restructure. RESULTS: Overall performance has improved according to the number of errors documented on identification and susceptibility challenges for laboratories that remained at the same classification (ie, A or C). The number of major identification errors for laboratories that were reclassified increased, but the rate of major susceptibility errors decreased. More reclassified laboratories do not have dedicated registered technologist(s) who perform microbiology testing and are not supervised by an on-site pathologist and/or medical microbiologist compared with laboratories that remained at the same classification. CONCLUSIONS: Microbiology laboratory restructuring will have adverse effects on the quality of complex testing if experienced technologists are not retained and services are not medically supervised.     

Proficiency testing of platelet counting in Ontario.

Proficiency testing results provide data for interlaboratory comparisons. The Laboratory Proficiency Testing Program of Ontario, Canada, has tested platelet counting for a decade and other hematology parameters for 15 years. Despite the fact that all testing programs are compromised by the type of testing material used, for platelet count testing, conclusions are possible. The data indicate that most laboratories in Ontario have greater difficulty performing reproducible platelet counting as compared with other components of hematologic cytometry. A major contributor to this shortfall is the method used by the laboratory. It was not a surprise that manual counting had a high coefficient of variation. Unexpectedly, the semiautomated methods used in the mid-1980s also displayed a high coefficient of variation.     

Evaluation of athletic training students' clinical proficiencies

CONTEXT: Appropriate methods for evaluating clinical proficiencies are essential in ensuring entry-level competence. OBJECTIVE: To investigate the common methods athletic training education programs use to evaluate student performance of clinical proficiencies. DESIGN: Cross-sectional design. SETTING: Public and private institutions nationwide. PATIENTS OR OTHER PARTICIPANTS: All program directors of athletic training education programs accredited by the Commission on Accreditation of Allied Health Education Programs as of January 2006 (n = 337); 201 (59.6%) program directors responded. DATA COLLECTION AND ANALYSIS: The institutional survey consisted of 11 items regarding institutional and program demographics. The 14-item Methods of Clinical Proficiency Evaluation in Athletic Training survey consisted of respondents' demographic characteristics and Likert-scale items regarding clinical proficiency evaluation methods and barriers, educational content areas, and clinical experience settings. We used analyses of variance and independent t tests to assess differences among athletic training education program characteristics and the barriers, methods, content areas, and settings regarding clinical proficiency evaluation. RESULTS: Of the 3 methods investigated, simulations (n = 191, 95.0%) were the most prevalent method of clinical proficiency evaluation. An independent-samples t test revealed that more opportunities existed for real-time evaluations in the college or high school athletic training room (t(189) = 2.866, P = .037) than in other settings. Orthopaedic clinical examination and diagnosis (4.37 +/- 0.826) and therapeutic modalities (4.36 +/- 0.738) content areas were scored the highest in sufficient opportunities for real-time clinical proficiency evaluations. An inadequate volume of injuries or conditions (3.99 +/- 1.033) and injury/condition occurrence not coinciding with the clinical proficiency assessment timetable (4.06 +/- 0.995) were barriers to real-time evaluation. One-way analyses of variance revealed no difference between athletic training education program characteristics and the opportunities for and barriers to real-time evaluations among the various clinical experience settings. CONCLUSIONS: No one primary barrier hindered real-time clinical proficiency evaluation. To determine athletic training students' clinical proficiency for entry-level employment, athletic training education programs must incorporate standardized patients or take a disciplined approach to using simulation for instruction and evaluation.     

Proposal for improving EQA programs in clinical microbiology by the Japanese Association of Medical Technologists

External quality assessment (EQA) programs have been conducted by the Japanese Association of Medical Technologists (JAMT) since 1989. The nationwide EQA programs have provided feedback for improving clinical tests quality in individual laboratories. The participating laboratories have been expected to process the survey samples according to their usual practice for patient specimens. However, many problems relating to the EQA programs in clinical microbiology have been raised. Dishonesty in the responses, survey samples being handled in a manner that improves assessment results, surveys depending on volunteers because of time and cost limitations were some of the initial problems. In addition, the criteria used to evaluate the results were poorly understood, so that neither examiners not participants were clear as to how the evaluation worked. And finally, the nationwide EQA programs can detect only gross errors and use invalid methods for evaluating routine performances. They have been measuring only a few steps in specimens processing. To assess overall laboratory competence, other methods are needed. It is time to reform the JAMT nationwide EQA program to initiate real proficiency testing and to this end it is necessary to increase collaboration between JAMT and the regional associations of medical technologists, so that the improved testing program can be properly administered.     

Evolving approaches to management of quality in clinical microbiology.

Quality management in clinical microbiology began in the 1960s. Both government and professional societies introduced programs for proficiency testing and laboratory inspection and accreditation. Many laboratory scientists and pathologists were independently active and creative in expanding efforts to monitor and improve practices. The initial emphasis was placed on intralaboratory process. Later, attention was shifted to physician ordering, specimen collection, reporting, and use of information. Quality management in the laboratory depends in large part on the monitoring of indicators that provide some evidence of how laboratory resources are being used and how the information benefits patient care. Continuous quality improvement should be introduced. This consists of a more thorough assessment of doing the right things versus the wrong things in terms of customer demand and satisfaction and studying the cumulative effect of error when responsibility is passed from one person to another. Prevention of error is accomplished more through effective training and continuing education than through surveillance. Also, this system will force more conscious attention to meeting the expectations of the many customers that must be satisfied by laboratory services, including patients, physicians, third-party payers, and managed-care organizations.     

Medical relevance of laboratory tests. A clinical perspective

To address the role of proficiency testing in the medical usefulness of laboratory tests, nine steps involved in the generation and application of a laboratory test result are identified and discussed: test ordering, patient preparation, specimen, sample, analysis, result, reporting, recognition, and action. Clinical uses of test results are enumerated. Good clinical skills are necessary for optimal test efficiency. Clinicians should improve their selection of tests, pay attention to proper patient preparation, and refine the process of interpretation of test results using disease-based reference ranges and more formal analysis of predictive value. By better definition of the clinical uses of laboratory tests, appropriate attention can be directed to steps in the laboratory domain such as medically relevant goals for accuracy and precision. With clearer understanding of the testing process, proficiency testing for monitoring laboratory performance can be more fully utilized. Audit models should be developed that include assessment of the outcome of the laboratory testing process.     

Achieving the Health Care Financing Administration limits by quality improvement and quality control. A real-world example.

With the enactment of the Clinical Laboratory Improvement Amendments of 1988 (CLIA 88), the federal government is now using proficiency testing as the primary indicator of laboratory quality. Laboratories with proficiency test failures are now at risk of a variety of harsh penalties including large monetary fines and suspension of operations. To minimize the risk of failed proficiency testing, we initiated a continuous quality improvement program in our general chemistry laboratory in conjunction with the use of a new survey-validated quality control product. This article describes the quality improvement program and our success in reducing the long-term random error in general chemistry. Despite our improvement program, significant analytical errors (greater than 30% of the CLIA limits) still exist in analytes measured by our chemistry analyzer. These errors are present in nearly the same analytes measured by other common chemistry analyzers indicating the need for improvement in their design and manufacture.