Use and misuse of the receiver operating characteristic curve in risk prediction

The c statistic, or area under the receiver operating characteristic (ROC) curve, achieved popularity in diagnostic testing, in which the test characteristics of sensitivity and specificity are relevant to discriminating diseased versus nondiseased patients. The c statistic, however, may not be optimal in assessing models that predict future risk or stratify individuals into risk categories. In this setting, calibration is as important to the accurate assessment of risk. For example, a biomarker with an odds ratio of 3 may have little effect on the c statistic, yet an increased level could shift estimated 10-year cardiovascular risk for an individual patient from 8% to 24%, which would lead to different treatment recommendations under current Adult Treatment Panel III guidelines. Accepted risk factors such as lipids, hypertension, and smoking have only marginal impact on the c statistic individually yet lead to more accurate reclassification of large proportions of patients into higher-risk or lower-risk categories. Perfectly calibrated models for complex disease can, in fact, only achieve values for the c statistic well below the theoretical maximum of 1. Use of the c statistic for model selection could thus naively eliminate established risk factors from cardiovascular risk prediction scores. As novel risk factors are discovered, sole reliance on the c statistic to evaluate their utility as risk predictors thus seems ill-advised.     

Logistic回归联合受试者工作特征曲线评价血小板参数对急性冠脉综合征的诊断价值

目的：探讨平均血小板体积（MPV）和大型血小板比例（P-LCR）在胸痛患者急性冠脉综合征（ACS）中的诊断价值。方法测定83例2013年1月至6月因胸痛就诊于空军总医院的ACS患者和56例非心源性胸痛患者的血小板参数,通过组间均值比较和logistic回归分析筛选出有效指标,并利用受试者工作特征（ROC）曲线评价其在ACS判断中的诊断价值。结果（1）与非心源性胸痛患者相比,ACS患者血小板数量减少[（191.28±67.07）x10^9/L vs（236.75±64.09）x10^9/L], MPV[（11.88±1.24） vs （10.73±1.08）fL]、血小板分布宽度（PDW）[（15.54±1.87） vs（13.40±2.35）fL]和P-LCR[（47.49±9.55）% vs （35.11±10.00）%]等明显升高（P＜0.05）;（2）不同临床类型的ACS患者血小板参数无显著变化（P＞0.05）;（3）logistic分析表明P-LCR和MPV是ACS的辅助诊断指标;（4）ROC曲线显示肌钙蛋白Ⅰ、P-LCR、MPV、肌酸激酶同工酶对应的诊断界值分别是0.15μg/L、38.5%、11.05fL和19.0U/L,曲线下面积分别为0.987,0.817,0.754和0.598,灵敏度为97.3%,92.8%,71.1%和45.8%,特异度为90.3%,64.3%,62.5%和73.2%,诊断符合率100%,80%,72.6%和67.2%。结论 MPV和P-LCR分别有助于ACS的发现,可作为胸痛患者ACS的辅助诊断指标,由于MPV和P-LCR升高对ACS的判断准确性较高,因此可以为ACS的早期判断和鉴别诊断提供参考。     

Use of the receiver operating characteristic curve to evaluate sensitivity, specificity, and accuracy of methods for detection of peaks in hormone time series

We utilize the "Receiver Operating Characteristic" to describe the relationship between sensitivity and specificity as the threshold for peak detection is varied systematically, to provide objective comparison of the performance of methods for detection of episodic hormonal secretion. A computer program was used to generate synthetic data with peaks with variable durations, with constant or variable height, shape and/or interpulse interval. This approach was used to compare the CLUSTER and DETECT programs. For both programs, the observed false positive rates estimated using signal-free data were in good agreement with the nominal rates, but in the presence of signal the observed false positive rates were systematically lower. Sensitivity increases with increasing signal/noise ratio, as expected. Program DETECT, using its standard options, provided excellent sensitivity (90-100%) with very low false positive rate under all conditions tested. Its performance could be further improved by the use of a more stringent definition of a peak requiring the presence of "UP" followed by a "DOWN". The CLUSTER program was found to have very poor sensitivity when using the "local variance" option. Use of the true fixed standard deviation or percent coefficient of variation resulted in a modest improvement. Optimal performance of program CLUSTER was obtained by the use of the best of 3 variance models, testing 12 different cluster sizes (from 1x1 to 4x4) and selecting the best among these: under these conditions it can achieve high sensitivity (90-100%) for very low observed false positive rate, such that its performance was comparable to that of DETECT. The methods developed and illustrated here should permit the definitive characterization and validation of the performance of any one method, the objective comparison of the relative performance of two or more methods for analysis of pulsatile hormone levels for episodic hormone secretion, and lead to the improvement of algorithms for peak detection.     

Magnetic resonance imaging of congenital heart disease: sensitivity and specificity using receiver operating characteristic curve analysis

MRI has shown promise for the evaluation of various congenital heart lesions. The current study was designed to assess the sensitivity and specificity of the technique for the evaluation of all major anatomic elements of the heart affected in simple and complex congenital heart disease. MR images of 51 patients (31 males and 20 females, ages 3 to 69 years) with a total of 110 congenital heart lesions were reviewed by a panel of two cardiac radiologists and one pediatric cardiologist, who assessed the cardiac pathoanatomy without knowledge of clinical details and assigned a confidence level to each diagnosis. The true diagnosis was established independently by the findings of angiocardiography and catheterization as well as by surgery, where applicable. ROC curves were generated from the responses, and the sensitivity at a specificity level of 90% was determined. MRI was shown to have a high sensitivity in evaluating great vessel relationships (100%), thoracic aorta abnormalities (94%), ASDs (91%) and VSDs (100%), visceroatrial situs (100%), and loop (100%). Three of five anomalies of the pulmonary and systemic veins were diagnosed by MRI. Right ventricular outflow obstructions (95%) were detected with a much higher sensitivity than lesions of the other valves (aortic valve 52%, mitral valve 62%, tricuspid valve 76%). Spin-echo MRI is a sensitive and specific method for the noninvasive assessment of congenital heart disease, with limitations in the evaluation of some valvular anomalies.     

CT-colonography in the follow-up of acute diverticulitis: patient acceptance and diagnostic accuracy

Objective. The current guidelines recommend maintenance of combined therapy for hepatitis C virus (HCV) genotype-1 chronic hepatitis when HCV-RNA is undetectable or ≤2 log₁₀ of baseline after 12 weeks of therapy. The aim of this study was to investigate whether the probability of obtaining sustained viral (SVR) response is similar when HCV-RNA is undetectable or is present at ≤2 log₁₀ level after 12 weeks of therapy. Material and methods. Retrospective analysis was carried out in 208 HCV genotype-1 chronic hepatitis patients treated with pegylated interferon and ribavirin with available data on HCV viral load after 12 weeks of therapy and definite data on the results of therapy. Results. Seventy-six (68.5%) out of 111 patients with undetectable HCV-RNA and 4 (11.8%) out of 34 patients with HCV-RNA ≤2 log₁₀ from baseline at week 12 reached SVR (odds ratio 16.29, 95% CI 5.08–67.12; p<0.001). Sixty-three patients did not meet any of these criteria and therapy was discontinued. Conclusions. The “12-week stopping rule” includes two different categories of responders considered candidates for maintained therapy, but the probability of obtaining SVR is very low in patients with HCV-RNA that is still detectable at this time of treatment. We suggest that, in these partial responders, the prolongation of therapy should be decided on an individual basis.     

Diagnostic pitfalls and accuracy of diagnosis in acute abdominal pain

OBJECTIVE: To identify the differential diagnostic difficulties in acute abdominal pain at the emergency department and during hospitalization. MATERIAL AND METHODS: Patients with abdominal pain lasting for up to 7 days were registered during 1997-2000 and re-evaluated one year after discharge (n=2851). RESULTS: Diagnoses with low sensitivity at the emergency department but markedly increased sensitivity at discharge were non-specific abdominal pain with a sensitivity value at the emergency department of 0.43, appendicitis 0.80, gallstones 0.68, constipation 0.74 and peptic ulcer 0.26. Corresponding kappa-values were 0.48, 0.74, 0.84, 0.88 and 0.93, respectively. Malignancy, gynaecological complaints, dyspepsia, urinary tract infection and diverticulitis displayed fairly good concordance between the preliminary and discharge judgements, but the predictive diagnostic value was still low at discharge. Sensitivity values at discharge were 0.40, 0.75, 0.73, 0.77 and 0.83, respectively. Among 479 surgically treated patients, 104 initially received a diagnosis usually not requiring surgery and had a median delay until operation of 22 h (95% CI 30-50 h), compared with 8 h (12-18 h) for referrals. CONCLUSIONS: Non-specific abdominal pain is the main differential diagnostic problem in the emergency department also for diagnoses requiring surgery. Constipation is a diagnostic pitfall and when making this diagnosis a careful re-evaluation is necessary.     

Comparison of the diagnostic accuracy of ischemia-modified albumin and echocardiography in patients with acute chest pain

BACKGROUND:

Several imaging tests and biomarkers have been proposed for the identification of patients with unstable angina among those presenting to the emergency department with acute chest pain. Preliminary data suggest that ischemia-modified albumin (IMA) may represent a potentially useful biomarker in these patients.

OBJECTIVE:

To compare IMA and echocardiography in excluding unstable angina in patients with acute chest pain.

METHODS:

Thirty-three patients (mean [± SD] age 59.8±10.8 years; 28 men) presenting to the emergency department with acute chest pain lasting <3 h suggestive of acute coronary syndrome, with normal or non-diagnostic electrocardiograms, and creatine kinase MB and troponin levels within the normal range, were included in the present study.

RESULTS:

After further diagnostic evaluation, five patients (15.2%) were diagnosed with unstable angina. The sensitivity, specificity, positive predictive value and negative predictive (NPV) value of echocardiography for diagnosing unstable angina was 60.0%, 89.3%, 50.0% and 92.6%, respectively. The area under the ROC curve for diagnosing unstable angina based on the serum IMA levels was 0.193 (95% CI 0.047 to 0.339; P<0.05). Based on ROC curve analysis, serum IMA levels ≥31.95 IU/mL yielded the optimal combination of sensitivity and specificity for diagnosing unstable angina. The sensitivity, specificity, positive predictive value and NPV of serum IMA levels ≥31.95 IU/mL for diagnosing unstable angina was 40.0%, 28.6%, 9.1% and 72.7%, respectively.

CONCLUSIONS:

Measurement of serum IMA levels appears to represent a useful tool for excluding unstable angina in patients presenting to the emergency department with acute chest pain. Moreover, IMA shows an NPV that is comparable with echocardiography.     

Early discharge policy of patients with acute colonic diverticulitis following initial CT scan

Introduction  Diverticulitis develops in 15–20% of individuals with diverticulosis. Severity ranges from mild to severe. Mild diverticulitis is uncomplicated confined per colonic inflammation commonly treated conservatively. Recent literature suggests it could be managed in an outpatient setup. Aims  To determine if patients with mild acute colonic diverticulitis (ACD) on early CT scan can be treated and discharged at an early time. Methodology  Retrospective review of patient’s charts admitted during 2005 with ACD confirmed by CT scan performed within 24 h of admission. Severity of ACD was determined according to CT classification. Results  Forty-two (31 women, 11 men) patients included, mean age 66 years, CT severity classification: 61.9% mild, 7.1% moderate, and 31.0% severe diverticulitis. Patients with mild ACD were discharged safely, had no recurrence of their symptoms, and needed no readmission within 6 months of follow-up. Conclusion  Patients with mild ACD on CT scan performed within 24 h could be safely discharged and treated according to protocols of outpatient management of diverticulitis.     

Prospective evaluation of an EDB-based diagnostic program to be used in patients admitted to hospital with acute chest pain

A recently designed computer based decision support system (DSP),almost exclusively based on case history data, was developedto facilitate immediate differentiation between patients withand without urgent need for coronary care unit (CCU) transferralfrom the emergency room, and additionally to distinguish betweenpatients with and without acute myocardial infarction (MI). One-year's prospective testing in a consecutive series of 1252patients with acute chest pain revealed that the DSP, used inaddition to ECG and clinical examination, demonstrated a sensitivityof 96% in the detection of patients in need of CCU observation(MI-sensitivity of 98%), and a specificity of 56% in excludingpatients who were not in need of CCU observation. The proportionof referrals to the CCU judged to be unnecessary was only 17%of the total number of patients seen in the emergency room.     

A diagnostic study of patients with upper abdominal pain.

197 consecutive, non-acute, medical patients who presented with upper abdominal pain were subjected to a standard programme of investigation. The investigation represents an attempt to supplement general clinical experience with exact data. In about half the patients no cause of the pain was found and a diagnosis of X-ray negative dyspepsia was made by elimination. It is concluded that a special research effort is needed to explain the complaint in this large group of patients. Duodenal ulcer was twice as common as gastric ulcer, and two patients suffered from gastric cancer. The diagnostic value of the symptomatology was analysed, but only the relation of pain to meals was found to be of diagnostic interest. In particular, the probability of duodenal ulcer was low and that of X-ray negative dyspepsia high, if the pain was provoked by eating. The age, sex, and acid production also had diagnostic value.     

A new strategy for the application of CA19-9 in the differentiation of pancreaticobiliary cancer: analysis using a receiver operating characteristic curve

OBJECTIVE: Clinicians might be misled in interpreting an elevated CA19-9 when differentiating pancreaticobiliary cancer from benign clinical conditions such as acute cholangitis or cholestasis, because in these conditions, the concentration of CA19-9 may also be elevated. The aims of our study were to calculate new individual cutoff values for CA19-9 according to clinical situations using a receiver operating characteristic (ROC) curve and to define a new strategy for interpreting CA19-9 in pancreaticobiliary cancer. METHODS: One hundred sixty patients with pancreatic diseases (cancer 90, benign disease 70), 322 patients with biliary tract diseases (biliary cancer 152, benign disease 170), and 20,035 asymptomatic controls were enrolled in the present study. An ROC curve was described by plotting the sensitivity on the y-axis against 1-specificity on the x-axis for each of several cutoff values. RESULTS: The area under the ROC curve was significantly greater for pancreatic cancer than for biliary cancer (p < 0.05). For patients with pancreatic cancer, CA19-9 proved to be useful. At a cutoff value of 37 U/ml, sensitivity and specificity were 76.7% and 87.1%, respectively. For patients with biliary cancer, CA19-9 was not helpful. However, when patients with biliary disease were divided into two groups according to the presence of cholangitis or cholestasis, CA19-9 proved to be more useful for the group without cholangitis or cholestasis than for the group with cholangitis or cholestasis (p < 0.05). In the former group, the sensitivity and specificity of CA19-9 were 77.6% and 83%, respectively, at the cutoff value of 37 U/ml. For the latter group, the sensitivity and specificity of CA19-9 were 74% and 41.5% respectively, whereas the specificity reached 87% at 300 U/ml. CA19-9 in diagnosing pancreatic cancer was useful regardless of accompanying acute pancreatitis or cholestasis. The serum concentration of CA19-9 in asymptomatic individuals was 9.42 +/- 9.95 U/ml. Only 1 of 157 patients with a concentration of CA19-9 above 37 U/ml was found to have gallbladder cancer. The positive and negative predictive values were 0.65% and 0.78%, respectively. CONCLUSIONS: The use of CA19-9 for the differentiation of pancreaticobiliary cancer should be applied individually, depending on the clinical situation.     

Evaluation of symptoms and signs of gallstone disease in patients admitted with upper abdominal pain

We present a prospective evaluation of the diagnostic value of 37 symptoms and signs of gallstones in 192 patients admitted with upper abdominal pain. The study was carried out independently of the examination and treatment by the staff. The routine investigations showed 49 patients with gallstones. Univariate analysis showed that old age, previous similar attacks of pain, previous intolerance to fatty foods, severe (that is, requiring analgetic injections) and radiating pain, and tenderness in the upper right quadrant were significantly more frequent in patients with gallstone disease than in those without. The evaluation showed that the classical signs and symptoms are relatively poor in establishing the diagnosis of gallstone disease, but their absence is a relatively good indicator for excluding the diagnosis. The multivariate analysis showed that the diagnosis of gallstone disease than in those without. The evaluation showed that the classical signs and and symptoms are relatively poor in establishing the diagnosis of gallstone disease, but their absence is a relatively good indicator for excluding the diagnosis. The multivariate analysis showed that the diagnosis of gallstone disease depends, in the main, on intolerance to fatty foods, severe pain, and tenderness in the upper right quadrant. The other classical signs and symptoms depend on and vary mutually with these three.     

Plasma lipoprotein particle concentrations in postmenopausal women with unstable coronary artery disease. Analysis of diagnostic accuracy using receiver operating characteristics

BACKGROUND: The contribution of plasma lipids to cardiovascular risk is usually evaluated by measuring plasma concentrations of total cholesterol, triglycerides and HDL cholesterol, and calculating LDL cholesterol concentration. We investigated plasma concentrations of apolipoproteins and lipoprotein particles in women with unstable coronary artery disease (CAD) to evaluate whether these, better than the routine lipid status, could differentiate women with and without coronary atherosclerosis. METHODS: Blood samples for lipid analyses were collected from 119 angiographically examined postmenopausal 49-79-year-old women with unstable CAD, and from 101 age-matched controls. Mean plasma concentrations were compared and the discriminatory ability of the different variables were tested using receiver operating characteristics (ROC). RESULTS: At coronary angiography 19% had normal vessels and 81% had coronary atherosclerosis. A disturbed triglyceride metabolism was the most pronounced lipid abnormality in women with unstable CAD and coronary atherosclerosis. ROC showed that none of the evaluated variables had a particularly high discriminatory power regarding unstable CAD or coronary atherosclerosis. The ratio cholesterol/HDL cholesterol was best with an ROC area of 0.79. Furthermore, the newer lipid variables, i.e. lipoprotein particles and apolipoproteins, were no better than the traditional variables. CONCLUSION: Lipoprotein changes reflecting a disturbed triglyceride metabolism are most pronounced in women with unstable CAD and coronary atherosclerosis. Lipoprotein particles and apolipoproteins alone were no better than lipids and lipoproteins in separating women with from those without coronary atherosclerosis. Our study does not support the measurement of apolipoproteins and lipoprotein particles on the basis of diagnostic accuracy alone.     

Clinical uncertainty, diagnostic accuracy, and outcomes in emergency department patients presenting with dyspnea

BACKGROUND: Dyspnea is a common complaint in the emergency department (ED) and may be a diagnostic challenge. We hypothesized that diagnostic uncertainty in this setting is associated with adverse outcomes, and amino-terminal pro-B-type natriuretic peptide (NT-proBNP) testing would improve diagnostic accuracy and reduce diagnostic uncertainty. METHODS: A total of 592 dyspneic patients were evaluated from the ProBNP Investigation of Dyspnea in the Emergency Department (PRIDE) study. Managing physicians were asked to provide estimates from 0% to 100%of the likelihood of acutely destabilized heart failure (ADHF). A certainty estimate of either 20% or lower or 80% or higher was classified as clinical certainty, while estimates between 21% and 79% were defined as clinical uncertainty. Associations between clinical uncertainty,hospital length of stay, morbidity, and mortality were examined. The diagnostic value of clinical judgment vs NT-proBNP measurement was compared across categories of clinical certainty. RESULTS: Clinical uncertainty was present in 185 patients (31%), 103 (56%) of whom had ADHF. Patients judged with clinical uncertainty had longer hospital length of stay and increased morbidity and mortality,especially those with ADHF. Receiver operating characteristic analysis of clinical judgment yielded an area under the curve (AUC) of 0.88 in the clinical certainty group and 0.76 in the uncertainty group (P<.001); NT-proBNP testing alone in these same groups had AUCs of 0.96 and 0.91, respectively. The combination of clinical judgment with NT-proBNP testing yielded improvements in AUC. CONCLUSIONS: Among dyspneic patients in the ED, clinical uncertainty is associated with increased morbidity and mortality, especially in those with ADHF.The addition of NT-proBNP testing to clinical judgment may reduce diagnostic uncertainty in this setting.     

136例结肠憩室病的内镜特点及临床分析

目的探讨结肠憩室的内镜及临床特点。方法回顾性分析近9年我院经结肠镜诊断的136例结肠憩室患者的临床资料。结果本组中≥60岁人群中结肠憩室的检出率为62.50%,25～59岁人群中结肠憩室的检出率为36.76%;〈25岁人群中结肠憩室的检出率为0.74%。憩室部位以右半结肠更常见（80.15%）;多发憩室（88/136）较单发憩室（48/136）多见;伴发病中以结肠息肉最多（40.44%）。便秘、腹痛、腹胀、便血为常见临床表现。结论结肠憩室好发于老年人,右半结肠为好发部位,多发较单发常见,容易伴发结肠息肉。结肠镜是诊断本病的最佳方法。     

The diagnoses of patients admitted with acute chest pain but without myocardial infarction

OBJECTIVE: The purpose of this study was to describe the frequencies ofvarious diagnoses in patients admitted with acute chest pain,but without acute myocardial infarction, and to evaluate a non-invasivescreening programme for these patients. PATIENTS: A total of 204 consecutive non-acute myocardial infarction patientswere included. Fifty-six had a definite diagnosis within 48h, whereas 148 patients underwent an examination programme includingpulmonary scintigraphy, echocardiography, exercise electrocardiography,myocardial scintigraphy, Holter monitoring, hyperventilationtest, oesophago-gastro-duodenoscopy, 3 h monitoring of oesophagealpH, oesophageal manometry, Bernstein test, physical examinationof the chest wall and thoracic spine, bronchial histamine provocationtest and ultrasonic examination of the abdomen. RESULTS: According to predefined criteria, 186 patients (91%) had atleast one diagnosis, 144 had one, whereas 39 had two, and threepatients had three diagnoses. In 18 patients no diagnosis wasobtained. The diagnoses belonged mainly to three groups: (1)ischaemic heart disease (n=64); (2) gastro-oesophageal diseases(n=85); (3) chest-wall syndromes (n=58). Less frequent diagnosesincluded pulmonary embolism, pleuritis/pneumonia, lung cancer.aortic stenosis, aortic aneurysm and herpes zoster. CONCLUSIONS: The high risk subset of a non-acute myocardial infarction populationcan be identified by means of a clinical evaluation and non-invasivecardiac examinations. Among the remainder, pulmonary embolism,gastro-oesophageal diseases and chest-wall syndromes shouldbe paid special attention. A careful physical examination ofthe chest wall and an upper endoscopy seems to be the most cost-beneficialexamination to employ in this subset.