System Accuracy Evaluation of Four Systems for Self-Monitoring of Blood Glucose Following ISO 15197 Using a Glucose Oxidase and a Hexokinase-Based Comparison Method

Background:

The standard ISO (International Organization for Standardization) 15197 is widely accepted for the accuracy evaluation of systems for self-monitoring of blood glucose (SMBG). Accuracy evaluation was performed for 4 SMBG systems (Accu-Chek® Aviva, Contour®XT, GlucoCheck XL, GlucoMen® LX PLUS) with 3 test strip lots each. To investigate a possible impact of the comparison method on system accuracy data, 2 different established methods were used.

Methods:

The evaluation was performed in a standardized manner following test procedures described in ISO 15197:2003 (section 7.3). System accuracy was assessed by applying ISO 15197:2003 and in addition ISO 15197:2013 criteria (section 6.3.3). For each system, comparison measurements were performed with a glucose oxidase (YSI 2300 STAT Plus™ glucose analyzer) and a hexokinase (cobas® c111) method.

Results:

All 4 systems fulfilled the accuracy requirements of ISO 15197:2003 with the tested lots. More stringent accuracy criteria of ISO 15197:2013 were fulfilled by 3 systems (Accu-Chek Aviva, ContourXT, GlucoMen LX PLUS) when compared to the manufacturer’s comparison method and by 2 systems (Accu-Chek Aviva, ContourXT) when compared to the alternative comparison method. All systems showed lot-to-lot variability to a certain degree; 2 systems (Accu-Chek Aviva, ContourXT), however, showed only minimal differences in relative bias between the 3 evaluated lots.

Conclusions:

In this study, all 4 systems complied with the evaluated test strip lots with accuracy criteria of ISO 15197:2003. Applying ISO 15197:2013 accuracy limits, differences in the accuracy of the tested systems were observed, also demonstrating that the applied comparison method/system and the lot-to-lot variability can have a decisive influence on accuracy data obtained for a SMBG system.     

Analytical Performance Requirements for Systems for Self-Monitoring of Blood Glucose With Focus on System Accuracy: Relevant Differences Among ISO 15197:2003, ISO 15197:2013, and Current FDA Recommendations

In the European Union (EU), the ISO (International Organization for Standardization) 15197 standard is applicable for the evaluation of systems for self-monitoring of blood glucose (SMBG) before the market approval. In 2013, a revised version of this standard was published. Relevant revisions in the analytical performance requirements are the inclusion of the evaluation of influence quantities, for example, hematocrit, and some changes in the testing procedures for measurement precision and system accuracy evaluation, for example, number of test strip lots. Regarding system accuracy evaluation, the most important change is the inclusion of more stringent accuracy criteria. In 2014, the Food and Drug Administration (FDA) in the United States published their own guidance document for the premarket evaluation of SMBG systems with even more stringent system accuracy criteria than stipulated by ISO 15197:2013. The establishment of strict accuracy criteria applicable for the premarket evaluation is a possible approach to further improve the measurement quality of SMBG systems. However, the system accuracy testing procedure is quite complex, and some critical aspects, for example, systematic measurement difference between the reference measurement procedure and a higher-order procedure, may potentially limit the apparent accuracy of a given system. Therefore, the implementation of a harmonized reference measurement procedure for which traceability to standards of higher order is verified through an unbroken, documented chain of calibrations is desirable. In addition, the establishment of regular and standardized post-marketing evaluations of distributed test strip lots should be considered as an approach toward an improved measurement quality of available SMBG systems.     

Mean Absolute Relative Difference of Blood Glucose Monitoring Systems and Relationship to ISO 15197

Background:The analytical quality of a blood glucose monitoring system (BGMS) is often assessed according to the requirements described in the international standard ISO 15197. However, the mean absolute relative difference (MARD) is sometimes used as well. This analysis aims at providing empirical data from BGMS evaluation studies conducted according to ISO 15197 and at providing an estimation of how MARD and percentage of measurement results within ISO accuracy limits are related.Methods:Results of 77 system accuracy evaluations conducted according to ISO 15197 were used to calculate MARD between BGMS and a laboratory comparison method’s results (glucose oxidase or hexokinase method). Additionally, bias and 95%-limits of agreement (LoA) using the Bland and Altman method were calculated.Results:MARD results ranged from 2.3% to 20.5%. The lowest MARD of a test strip lot that showed <95% of results within ISO limits was 6.1%. The distribution of MARD results shows that only 3.6% of test strip lots with a MARD equal to or below 7% showed <95% of results within ISO limits (2.2% of all test strip lots). Bias of test strip lots that showed ≥95% of results within the limits ranged from −10.3% to +7.4%. The half-width of the 95%-LoA of test strip lots that showed ≥95% of results within the limits ranged from 4.8% to 24.0%.Conclusion:There is a threshold MARD that may allow an estimate whether ISO 15197 requirements are fulfilled, but this statement cannot be made with certainty.     

Assessing System Accuracy of Blood Glucose Monitoring Systems Using Rectangle Target Plots

Background:

Results from accuracy assessments of systems for self-monitoring of blood glucose (SMBG) are often visualized in difference or regression plots. These approaches become more difficult to read as the number of data points displayed increases, thus limiting their use. In the recently presented rectangle target plot (RTP) approach, data from each reagent system lot or product are displayed graphically as a single rectangle, thus allowing the plot to remain comprehensible even when displaying system accuracy data from multiple reagent system lots or products.

Methods:

The RTP illustrates the accuracy of SMBG systems. Each rectangle shows the mean bias and the variability of a system. By use of statistical tolerance intervals, each rectangle most closely approximates the total error for lower (<100 mg/dL) and upper (≥100 mg/dL) glucose concentrations. RTPs were created for data from 8 different manufacturers of systems for SMBG. In total, the accuracy data of 87 different reagent system lots of 50 different SMBG systems were displayed in RTPs.

Results:

The RTP approach was suitable for 81 of the 87 reagent system lots analyzed. In the remaining cases, outliers caused excessive skewness of the distribution of measurements. The reagent system lots analyzed were grouped according to manufacturer in RTPs. Data from 3 to 15 different reagent system lots were displayed in each RTP.

Conclusion:

Applying the RTP approach to a large number of reagent system lots showed that it was suitable in more than 93% of cases analyzed. The display of system accuracy data in RTPs enables lot-to-lot variability within specific products and product reliability of specific manufacturers to be visualized in a comprehensible manner.     

The Rectangle Target Plot: A New Approach to the Graphical Presentation of Accuracy of Systems for Self-Monitoring of Blood Glucose

Background:

The measurement accuracy of systems for self-monitoring of blood glucose (SMBG) is usually analyzed by a method comparison in which the analysis results are displayed using difference plots or similar graphs. However, such plots become difficult to comprehend as the number of data points displayed increases. This article introduces a new approach, the rectangle target plot (RTP), which aims to provide a simplified and comprehensible visualization of accuracy data.

Methods:

The RTP is based on ISO 15197 accuracy evaluations of SMBG systems. Two-sided tolerance intervals for normally distributed data are calculated for absolute and relative differences at glucose concentrations <100 mg/dL and ≥100 mg/dL. These tolerance intervals provide an estimator of where a 90% proportion of results is found with a confidence level of 95%.

Results:

Plotting these tolerance intervals generates a rectangle whose center indicates the systematic measurement difference of the investigated system relative to the comparison method. The size of the rectangle depends on the measurement variability.

Conclusions:

The RTP provides a means of displaying measurement accuracy data in a simple and comprehensible manner. The visualization is simplified by reducing the displayed information from typically 200 data points to just 1 rectangle. Furthermore, this allows data for several systems or several lots from 1 system to be displayed clearly and concisely in a single graph.     

Impact of Two Different Reference Measurement Procedures on Apparent System Accuracy of 18 CE-Marked Current-Generation Blood Glucose Monitoring Systems

Background:Measurement accuracy has been assessed for many different blood glucose monitoring systems (BGMS) over the years by different study groups. However, the choice of the comparison measurement procedure may impact the apparent level of accuracy found in such studies.Materials and Methods:Measurement accuracy of 18 different BGMS was assessed in a setting based on ISO 15197 using two different comparison methods in parallel: a glucose oxidase (GOD)-based and a hexokinase (HK)-based method. Accuracy limits of ISO 15197 were applied, and additional analyses were performed, including bias, linear regression, and mean absolute relative difference (MARD) to assess the impact of possible differences between comparison methods on the apparent level of accuracy.Results:While ≈80% of BGMS met the accuracy criteria of ISO 15197 when compared with the respective manufacturers’ reference measurement procedure, only two-thirds did so against both comparison methods. The mean relative bias ranged from −6.6% to +5.7% for the analysis against the GOD-based method and from −11.1% to +1.3% for the analysis against the HK-based method, whereas MARD results ranged from 3.7% to 9.8% and from 2.3% to 10.5%, respectively. Results of regression analysis showed slopes between 0.85 and 1.08 (GOD-based method) and between 0.81 and 1.01 (HK-based method).Conclusions:The results of this study indicate that there are systematic differences between the reference measurement procedures used for BGMS calibration as well as for system accuracy assessment. Because of the potential impact on therapy of patients with diabetes resulting from these differences, further steps toward harmonization of the measurement procedures’ results are important.     

Accuracy and User Performance of a New Blood Glucose Monitoring System

Introduction:Self-monitoring of blood glucose (BG) is important in diabetes management, allowing people with diabetes (PWD) to assess responses to diabetes therapy and to inform if they are attaining their glycemic targets. This study assessed the accuracy and user performance (UP) of a new blood glucose monitoring system (BGMS), CONTOUR®PLUS ELITE, according to International Organization for Standardization (ISO) 15197:2013 criteria and also more stringent criteria.Methods:In laboratory Study 1, capillary fingertip blood samples from 100 PWD were evaluated using the new BGMS. In clinical Study 2, 130 PWD had Yellow Springs Instrument (YSI) analyzer reference measurements against subject-obtained fingertip and palm blood, and trial staff-obtained venous blood. The new BGMS was tested with test strips from three different lots. A UP questionnaire assessed ease of use.Results:Study 1: 100% of combined accuracy results fulfilled ISO criteria (±15 mg/dL at BG <100 mg/dL; ±15% at BG ≥100 mg/dL); 99.8% fulfilled more stringent criteria (±10 mg/dL at BG <100 mg/dL; ±10% at BG ≥100 mg/dL). Error grid analysis showed that 100% of results were within zone A. Study 2: >98% of subject- and 100% of trial staff-obtained performance results met ISO criteria. Most subjects (>96%) found the BGMS easy to use.Conclusion:The new BGMS exceeded minimum ISO 15197:2013-specified standards for both accuracy and UP criteria, along with the more stringent accuracy criteria. These data show that this new BGMS can be a useful tool in managing glycemic control for PWD.     

Accuracy Evaluation of Five Blood Glucose Monitoring Systems: The North American Comparator Trial

Background

This study evaluated differences in accuracy between the CONTOUR^® NEXT EZ (EZ) blood glucose monitoring system (BGMS) and four other BGMSs [ACCU-CHEK^® Aviva (ACAP), FreeStyle Freedom Lite^® (FFL), ONE TOUCH^® Ultra^®2 (OTU2), and TRUEtrack^® (TT)].

Methods

Up to three capillary blood samples (N = 393) were collected from 146 subjects with and without diabetes. One sample per subject was tested with fresh (natural) blood; the other samples were glycolyzed to lower blood glucose to <70 mg/dl. Meter results were compared with results from plasma from the same sample tested on a Yellow Springs Instruments (YSI) 2300 STAT Plus™ glucose analyzer. Blood glucose monitoring system accuracy was compared using mean absolute relative difference (MARD; from laboratory reference method results) and other analyses. Separate analyses on fresh (natural) samples only were conducted to determine potential effects of glycolysis on MARD values of systems utilizing glucose-oxidase-based test strip chemistry.

Results

Across the tested glucose range, the EZ had the lowest MARD of 4.7%; the ACAP, FFL, OTU2, and TT had MARD values of 6.3%, 18.3%, 23.4%, and 26.2%, respectively. For samples with glucose concentrations <70 mg/dl, the EZ had the lowest MARD (0.65%), compared with the ACAP (2.5%), FFL (18.3%), OTU2 (22.4%), and TT (33.2%) systems.

Conclusions

The EZ had the lowest MARD across the tested glucose ranges when compared with four other BGMSs when all samples were analyzed as well as when natural samples only were analyzed.