Ranking and combining multiple predictors without labeled data

In a broad range of classification and decision-making problems, one is given the advice or predictions of several classifiers, of unknown reliability, over multiple questions or queries. This scenario is different from the standard supervised setting, where each classifier’s accuracy can be assessed using available labeled data, and raises two questions: Given only the predictions of several classifiers over a large set of unlabeled test data, is it possible to (i) reliably rank them and (ii) construct a metaclassifier more accurate than most classifiers in the ensemble? Here we present a spectral approach to address these questions. First, assuming conditional independence between classifiers, we show that the off-diagonal entries of their covariance matrix correspond to a rank-one matrix. Moreover, the classifiers can be ranked using the leading eigenvector of this covariance matrix, because its entries are proportional to their balanced accuracies. Second, via a linear approximation to the maximum likelihood estimator, we derive the Spectral Meta-Learner (SML), an unsupervised ensemble classifier whose weights are equal to these eigenvector entries. On both simulated and real data, SML typically achieves a higher accuracy than most classifiers in the ensemble and can provide a better starting point than majority voting for estimating the maximum likelihood solution. Furthermore, SML is robust to the presence of small malicious groups of classifiers designed to veer the ensemble prediction away from the (unknown) ground truth.Every day, multiple decisions are made based on input and suggestions from several sources, either algorithms or advisers, of unknown reliability. Investment companies handle their portfolios by combining reports from several analysts, each providing recommendations on buying, selling, or holding multiple stocks (1, 2). Central banks combine surveys of several professional forecasters to monitor rates of inflation, real gross domestic product growth, and unemployment (3–6). Biologists study the genomic binding locations of proteins by combining or ranking the predictions of several peak detection algorithms applied to large-scale genomics data (7). Physician tumor boards convene a number of experts from different disciplines to discuss patients whose diseases pose diagnostic and therapeutic challenges (8). Peer-review panels discuss multiple grant applications and make recommendations to fund or reject them (9). The examples above describe scenarios in which several human advisers or algorithms provide their predictions or answers to a list of queries or questions. A key challenge is to improve decision making by combining these multiple predictions of unknown reliability. Automating this process of combining multiple predictors is an active field of research in decision science (cci.mit.edu/research), medicine (10), business (refs. 11 and 12 and www.kaggle.com/competitions), and government (www.iarpa.gov/Programs/ia/ACE/ace.html and www.goodjudgmentproject.com), as well as in statistics and machine learning.Such scenarios, whereby advisers of unknown reliability provide potentially conflicting opinions, or propose to take opposite actions, raise several interesting questions. How should the decision maker proceed to identify who, among the advisers, is the most reliable? Moreover, is it possible for the decision maker to cleverly combine the collection of answers from all of the advisers and provide even more accurate answers?In statistical terms, the first question corresponds to the problem of estimating prediction performances of preconstructed classifiers (e.g., the advisers) in the absence of class labels. Namely, each classifier was constructed independently on a potentially different training dataset (e.g., each adviser trained on his/her own using possibly different sources of information), yet they are all being applied to the same new test data (e.g., list of queries) for which labels are not available, either because they are expensive to obtain or because they will only be available in the future, after the decision has been made. In addition, the accuracy of each classifier on its own training data is unknown. This scenario is markedly different from the standard supervised setting in machine learning and statistics. There, classifiers are typically trained on the same labeled data and can be ranked, for example, by comparing their empirical accuracy on a common labeled validation set. In this paper we show that under standard assumptions of independence between classifier errors their unknown performances can still be ranked even in the absence of labeled data.The second question raised above corresponds to the problem of combining predictions of preconstructed classifiers to form a metaclassifier with improved prediction performance. This problem arises in many fields, including combination of forecasts in decision science and crowdsourcing in machine learning, which have each derived different approaches to address it. If we had external knowledge or historical data to assess the reliability of the available classifiers we could use well-established solutions relying on panels of experts or forecast combinations (11–14). In our problem such knowledge is not always available and thus these solutions are in general not applicable. The oldest solution that does not require additional information is majority voting, whereby the predicted class label is determined by a rule of majority, with all advisers assigned the same weight. More recently, iterative likelihood maximization procedures, pioneered by Dawid and Skene (15), have been proposed, in particular in crowdsourcing applications (16–23). Owing to the nonconvexity of the likelihood function, these techniques often converge only to a local, rather than global, maximum and require careful initialization. Furthermore, there are typically no guarantees on the quality of the resulting solution.In this paper we address these questions via a spectral analysis that yields four major insights:

1. Under standard assumptions of independence between classifier errors, in the limit of an infinite test set, the off-diagonal entries of the population covariance matrix of the classifiers correspond to a rank-one matrix.
2. The entries of the leading eigenvector of this rank-one matrix are proportional to the balanced accuracies of the classifiers. Thus, a spectral decomposition of this rank-one matrix provides a computationally efficient approach to rank the performances of an ensemble of classifiers.
3. A linear approximation of the maximum likelihood estimator yields an ensemble learner whose weights are proportional to the entries of this eigenvector. This represents an efficient, easily constructed, unsupervised ensemble learner, which we term Spectral Meta-Learner (SML).
4. An interest group of conspiring classifiers (a cartel) that maliciously attempts to veer the overall ensemble solution away from the (unknown) ground truth leads to a rank-two covariance matrix. Furthermore, in contrast to majority voting, SML is robust to the presence of a small-enough cartel whose members are unknown.

In addition, we demonstrate the advantages of spectral approaches based on these insights, using both simulated and real-world datasets. When the independence assumptions hold approximately, SML is typically better than most classifiers in the ensemble and their majority vote, achieving results comparable to the maximum likelihood estimator (MLE). Empirically, we find SML to be a better starting point for computing the MLE that consistently leads to improved performance. Finally, spectral approaches are also robust to cartels and therefore helpful in analyzing surveys where a biased subgroup of advisers (a cartel) may have corrupted the data.     

Direction of SMR and Beta Change with Attention in Adults

Introduction. The aim of this study was to clarify the interpretation of sensory-motor rhythm (SMR; 13–15 Hz) and beta (16–20 Hz) changes with respect to attention states.

Method. For this purpose, EEG was recorded from 11 participants during (a) a multiple object tracking task (MOT), which required externally directed attention; (b) the retention phase of a visuo-spatial memory task (VSM), which required internally directed attention and avoidance of sensory distraction; and (c) the waiting intervals between trials, which constituted a no-task-imposed control condition. The 2 active tasks were consecutively presented at 2 difficulty levels (i.e., easy and hard). Two analyses of variance were conducted on EEG log spectral amplitudes in the alpha (8–12 Hz), SMR, and beta bands from F3, F4, C3, C4 and P3, P4.

Results. The first 15 analysis compared the MOT to the VSM by difficulty levels and revealed a significant task effect (p < .0005) but no effect of difficulty. The results showed that externally directed attention (MOT) resulted in lower values than internally directed attention (VSM) in all three bands. The second analysis averaged the difficulty levels together and added the no-task-imposed reference condition. The results again showed a significant task effect that did not interact with site, hemisphere, or, more important, band. Post hoc tests revealed that both MOT and VSM produced significantly smaller means than the no-task-imposed condition. This pattern of log-amplitude means and the lack of task interaction with any other factor indicate that task-induced attention reduces EEG power in the same proportion across the 3 bands and the 6 channels studied.

Conclusions. These results contradict a frequent interpretation concerning the relationship between the brain's aptitude to increase low beta in neurofeedback programs and improved sustain attention capacities.     

A new EEG biomarker of neurobehavioural impairment and sleepiness in sleep apnea patients and controls during extended wakefulness

ObjectiveTo explore the use of detrended fluctuation analysis (DFA) scaling exponent of the awake electroencephalogram (EEG) as a new alternative biomarker of neurobehavioural impairment and sleepiness in obstructive sleep apnea (OSA).MethodsEight patients with moderate–severe OSA and nine non-OSA controls underwent a 40-h extended wakefulness challenge with resting awake EEG, neurobehavioural performance (driving simulator and psychomotor vigilance task) and subjective sleepiness recorded every 2-h. The DFA scaling exponent and power spectra of the EEG were calculated at each time point and their correlation with sleepiness and performance were quantified.ResultsDFA scaling exponent and power spectra biomarkers significantly correlated with simultaneously tested performance and self-rated sleepiness across the testing period in OSA patients and controls. Baseline (8am) DFA scaling exponent but not power spectra were markers of impaired simulated driving after 24-h extended wakefulness in OSA (r = 0.738, p = 0.037). OSA patients had a higher scaling exponent and delta power during wakefulness than controls.ConclusionsThe DFA scaling exponent of the awake EEG performed as well as conventional power spectra as a marker of impaired performance and sleepiness resulting from sleep loss.SignificanceDFA may potentially identify patients at risk of neurobehavioural impairment and assess treatment effectiveness.     

宝石CT能谱成像测定甲状腺含碘量的初步探讨

目的 利用宝石CT能谱成像技术测量正常甲状腺含碘量,计算甲状腺与胸锁乳突肌含碘量的碘比值,为高碘或缺碘性甲状腺疾病诊断提供参考依据.方法 采用美国GE公司生产的宝石CT,对来自潍坊医学院的226例怀疑颈部或颈椎疾病患者进行能谱扫描,扫描范围包括甲状腺,胸锁乳突肌.其中男119例,女107例,年龄18 ～ 77岁,平均年龄(46±17)岁.将扫描数据传至AW 4.4工作站,利用GSI Viewer软件处理,找出甲状腺与胸锁乳突肌的最佳对比噪声比及其所对应的单能量图像,在碘基图像上测量甲状腺左右叶、两侧胸锁乳突肌含碘量,计算二者含碘量比值.结果 甲状腺左右叶总含碘量为(1.5233±0.4318)mg/cm3,其中左叶为(1.5230±0.4271)mg/cm3,右叶为(1.5236±0.4365)mg/cm3,二者比较差异无统计学意义(t=0.0084,P＞0.05).男性甲状腺含碘量为(1.6395±0.4105)mg/cm3,女性为(1.4238±0.3832) mg/cm3,二者比较差异有统计学意义(t=3.4743,P＜0.01).甲状腺与胸锁乳突肌含碘量的碘比值为96.6271±33.2442,其中男性比值为94.6250±37.3621,女性比值为98.0000±29.0737,二者比较差异无统计学意义(t=0.3817,P＞0.05).随年龄增长甲状腺含碘量呈逐渐下降趋势,组间比较差异有统计学意义(F=9.66,P＜ 0.01);其中＜40岁组[(1.7256±0.4631) mg/cm3]甲状腺含碘量高于40～60岁组[(1.4517±0.3643 )mg/cm3]和＞60岁组[(1.4368±0.3465)mg/cm3,q值分别为5.6195、5.4158,P均＜0.0l].结论 宝石CT能谱成像能测定甲状腺含碘量,反映人体碘水平,对高碘或缺碘性甲状腺疾病诊断有重要指导意义.     

黄斑裂孔面积相关参数在预测特发性黄斑裂孔患者术后裂孔闭合形态中的应用

目的　应用黄斑裂孔面积相关参数预测特发性黄斑裂孔(IMH)患者术后裂孔闭合形态的有效性。方法　将2018年6月至2020年12月因IMH行玻璃体切割联合内界膜剥除术患者共47例47眼纳入本研究。所有患者均行最佳矫正视力（BCVA）、频域光学相干断层扫描(SD-OCT)检查。将黄斑裂孔闭合形态分为1型及2型。计算黄斑裂孔指数（MHI）、裂孔直径指数（DHI）、黄斑裂孔愈合指数（MHCI）和裂孔形成因子（HFF）。采用ImageJ软件获得以下参数:裂孔两侧外界膜断端到光感受器脱离起点的曲线距离（m、n）、黄斑裂孔面积（MHA）、裂孔区视网膜面积（MHTA）、囊腔面积（MHCSA）。计算裂孔面积指数（MHAI）、裂孔区视网膜面积指数（MHTAI）、囊腔面积指数（MHCSAI）。术前与术后3个月患者BCVA行非参数秩和检验;将患者各评估参数、术后视力与裂孔闭合形态行Spearman相关性分析;术前各种黄斑裂孔评估参数进行受试者工作特征(ROC)曲线分析。结果　1型黄斑裂孔闭合患者术前BCVA与术后3个月比较差异有统计学意义(P<0.001);2型黄斑裂孔闭合患者术前BCVA与术后3个月比较差异无统计学意义(P=0.09)。IMH患者黄斑裂孔闭合形态与术后BCVA（r=0.57,P=0.000 3）、MHI（r=-0.64,P<0.000 1）、MHCI（r=-0.67,P<0.000 1）、HFF（r=-0.66,P<0.000 1）、MHAI（r=0.70,P<0.000 1)均呈显著相关;与MHTAI（r=-0.48,P=0.04）、MHSCAI（r=-0.49,P=0.04）均呈弱相关;与DHI（r=0.35,P=0.42）无相关性。MHI、DHI、HFF、MHCI、MHAI、MHTAI、MHCSAI的AUC分别为0．921、0.720、0.929、0.944、0.957、0.803、0.806,其cut-off值分别为>0.35、>0.56、<0.56、>0.84、>0.33、<0.39、<0.23。结论　运用ImageJ软件对IMH患者的SD-OCT结果进行分析发现,MHAI、MHCI是预测黄斑裂孔患者术后裂孔闭合形态的最重要指标。     

Effects of a novel NMDA receptor antagonist on experimental stroke quantitatively assessed by spectral EEG and infarct volume

Abstract

We used spectral electroencephalographic (EEC) analysis to demonstrate the physiological effect of focal brain ischaemia induced by permanent occlusion of the right middle cerebral artery in rats. A significant shift to lower frequency EEC activity occurred relative to the baseline power spectrum within one hour following the occlusion. Spectral EEG analysis also revealed a cerebroprotective effect of a noncompetitive N-methyl-D-aspartate receptor antagonist, CNS 1102, administered 15 min post-occlusion. Animals treated with this NMDA antagonist exhibited only 26.5% of the slowing in the ischaemic hemisphere compared to aminals given a placebo. Post-mortem analysis conducted 24 h later also revealed reduced infarction volumes for the treated animals, there was a highly significant correlation between the extent of spectral EEG slowing during the initial development of the infarction and subsequent lesion size. These results suggest that spectral EEG analysis may be useful in the early evaluation of experimental and perhaps human stroke and for monitoring the effects of cerebroprotective therapies. [Neurol Res 1994; 16: 443-448]     

杜仲不同提取物对帕金森病小鼠的治疗作用及谱效关系研究

目的探究杜仲不同提取物对帕金森病小鼠的治疗作用及其超高效液相色谱(UPLC)分析与治疗帕金森病的谱效关系。方法通过小鼠爬杆实验、脑内纹状体多巴胺(DA)含量,观察杜仲不同梯度的乙醇提取物对帕金森病小鼠的治疗作用;采用超高效液相色谱-四级杆飞行时间质谱(UPLC-Q-TOF/MS)对杜仲不同提取物进行分析;结合爬杆实验及DA水平结果,采用偏最小二乘法回归(PLSR)分析建立其谱效关系,明确杜仲治疗帕金森病的药效成分。结果杜仲50%、75%乙醇提取物均能明显缩短小鼠爬杆时间;杜仲75%乙醇提取物能显著增加小鼠脑内纹状体DA水平;PLSR分析结果显示,杜仲中杜仲醇苷、鹅掌楸苷、5-羟甲基糠醛、咖啡酸与爬杆实验结果和纹状体多巴胺含量密切相关。结论杜仲乙醇提取物具有抗帕金森病作用,其中75%乙醇提取物效果最显著;杜仲醇苷、鹅掌楸苷、5-羟甲基糠醛、咖啡酸可能是杜仲治疗帕金森病的主要有效成分。