Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark

PurposeSurgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill.MethodsTo achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment.ResultsF1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team).ConclusionSurgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.     

Semi-supervised learning with progressive unlabeled data excavation for label-efficient surgical workflow recognition

Surgical workflow recognition is a fundamental task in computer-assisted surgery and a key component of various applications in operating rooms. Existing deep learning models have achieved promising results for surgical workflow recognition, heavily relying on a large amount of annotated videos. However, obtaining annotation is time-consuming and requires the domain knowledge of surgeons. In this paper, we propose a novel two-stage Semi-Supervised Learning method for label-efficient Surgical workflow recognition, named as SurgSSL. Our proposed SurgSSL progressively leverages the inherent knowledge held in the unlabeled data to a larger extent: from implicit unlabeled data excavation via motion knowledge excavation, to explicit unlabeled data excavation via pre-knowledge pseudo labeling. Specifically, we first propose a novel intra-sequence Visual and Temporal Dynamic Consistency (VTDC) scheme for implicit excavation. It enforces prediction consistency of the same data under perturbations in both spatial and temporal spaces, encouraging model to capture rich motion knowledge. We further perform explicit excavation by optimizing the model towards our pre-knowledge pseudo label. It is naturally generated by the VTDC regularized model with prior knowledge of unlabeled data encoded, and demonstrates superior reliability for model supervision compared with the label generated by existing methods. We extensively evaluate our method on two public surgical datasets of Cholec80 and M2CAI challenge dataset. Our method surpasses the state-of-the-art semi-supervised methods by a large margin, e.g., improving 10.5% Accuracy under the severest annotation regime of M2CAI dataset. Using only 50% labeled videos on Cholec80, our approach achieves competitive performance compared with full-data training method.     

Computer assisted radiation Therapy

Purpose

Surgical process models (SPMs) have recently been created for situation-aware computer-assisted systems in the operating room. One important challenge in this area is the automatic acquisition of SPMs. The purpose of this study is to present a new method for the automatic detection of low-level surgical tasks, that is, the sequence of activities in a surgical procedure, from microscope video images only. The level of granularity that we addressed in this work is symbolized by activities formalized by triplets <action, surgical tool, anatomical structure> .

Methods

Using the results of our latest work on the recognition of surgical phases in cataract surgeries, and based on the hypothesis that most activities occur in one or two phases only, we created a light-weight ontology, formalized as a hierarchical decomposition into phases and activities. Information concerning the surgical tools, the areas where tools are used and three other visual cues were detected through an image-based approach and combined with the information of the current surgical phase within a knowledge-based recognition system. Knowing the surgical phases before the activity, recognition allows supervised classification to be adapted to the phase. Multiclass Support Vector Machines were chosen as a classification algorithm.

Results

Using a dataset of 20 cataract surgeries, and identifying 25 possible pairs of activities, a frame-by-frame recognition rate of 64.5 % was achieved with the proposed system.

Conclusions

The addition of human knowledge to traditional bottom-up approaches based on image analysis appears to be promising for low-level task detection. The results of this work could be used for the automatic indexation of post-operative videos.     

Sensor-based surgical activity recognition in unconstrained environments

Abstract

Introduction : Automatic surgical activity recognition in the operating room (OR) is mandatory to enable assistive surgical systems to manage the information presented to the surgical team. Therefore the purpose of our study was to develop and evaluate an activity recognition model. Material and methods : The system was conceived as a hierarchical recognition model which separated the recognition task into activity aspects. The concept used radio frequency identification (RFID) for instrument recognition and accelerometers to infer the performed surgical action. Activity recognition was done by combining intermediate results of the aspect recognition. A basic scheme of signal feature generation, clustering and sequence learning was replicated in all recognition subsystems. Hidden Markov models (HMM) were used to generate probability distributions over aspects and activities. Simulated functional endoscopic sinus surgeries (FESS) were used to evaluate the system. Results and discussion: The system was able to detect surgical activities with an accuracy of 95%. Instrument recognition performed best with 99% accuracy. Action recognition showed lower accuracies with 81% due to the high variability of surgical motions. All stages of the recognition scheme were evaluated. The model allows distinguishing several surgical activities in an unconstrained surgical environment. Future improvements could push activity recognition even further.     

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.     

手术室提高手术服务效率的管理实践与体会

目的探讨提高手术室服务效率的管理举措,以提高手术室护理质量及手术服务效率。方法强化手术室流程管理,改革护理人员排班模式,完善护理绩效考核机制。结果加强手术室管理后,每日手术台次、每日房间手术台次增加,手术衔接时间缩短;用药交接核查、手术患者转运交接、手术护理记录等护理质量明显提高;手术医生、手术护士、患者对护理工作满意率显著提升。结论规范手术室工作流程,实施安全高效的手术转运,通过绩效考核改革调动各级护士的工作积极性,可以显著提高手术服务质量及其效率。     

基于表面肌电图手势动作意图识别的系统综述

目的 系统综述基于表面肌电图的手势动作意图识别的研究进展。方法 通过检索中国知网、万方数据库、PubMed、Web of Science,搜集基于表面肌电图的手势动作意图识别的实验研究,检索时限为建库至2021年12月。根据实验内容和质量筛选文献,并对其分类方法和其他影响因素进行总结性分析。结果 共返回文献735篇,最终纳入25篇,发表时间主要集中于2012年至2021年,研究对象为正常受试者或截肢者,分类模型包含传统机器学习模型和深度学习模型,其他影响因素包含采集方式、噪声干扰和滑动窗口大小。结论 目前基于表面肌电信号的传统机器学习模型已得到成熟应用,深度学习模型的手势识别技术具有很大潜力。受试者的个体差异、手势分类的实时性需求和肌电设备的稳定性需求仍有待解决。     

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis as well as very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is typically well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both low-level feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin, e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition.     

Automated video-based assessment of surgical skills for training and evaluation in medical schools

Purpose

Routine evaluation of basic surgical skills in medical schools requires considerable time and effort from supervising faculty. For each surgical trainee, a supervisor has to observe the trainees in person. Alternatively, supervisors may use training videos, which reduces some of the logistical overhead. All these approaches however are still incredibly time consuming and involve human bias. In this paper, we present an automated system for surgical skills assessment by analyzing video data of surgical activities.

Method

We compare different techniques for video-based surgical skill evaluation. We use techniques that capture the motion information at a coarser granularity using symbols or words, extract motion dynamics using textural patterns in a frame kernel matrix, and analyze fine-grained motion information using frequency analysis.

Results

We were successfully able to classify surgeons into different skill levels with high accuracy. Our results indicate that fine-grained analysis of motion dynamics via frequency analysis is most effective in capturing the skill relevant information in surgical videos.

Conclusion

Our evaluations show that frequency features perform better than motion texture features, which in-turn perform better than symbol-/word-based features. Put succinctly, skill classification accuracy is positively correlated with motion granularity as demonstrated by our results on two challenging video datasets.

     

Impact of microcirculatory video quality on the evaluation of sublingual microcirculation in critically ill patients

We aimed to assess the impact of image quality on microcirculatory evaluation with sidestream dark-field (SDF) videomicroscopy in critically ill patients and explore factors associated with low video quality. This was a retrospective analysis of a single-centre prospective observational study. Videos of the sublingual microcirculation were recorded using SDF videomicroscopy in 100 adult patients within 12 h from admittance to the intensive care unit and every 24 h until discharge/death. Parameters of vessel density and perfusion were calculated offline for small vessels. For all videos, a quality score (?12 = unacceptable, 1 = suboptimal, 2 = optimal) was assigned for brightness, focus, content, stability, pressure and duration. Videos with a total score ≤8 were deemed as unacceptable. A total of 2455 videos (853 triplets) was analysed. Quality was acceptable in 56 % of videos. Lower quality was associated with worse microvascular density and perfusion. Unreliable triplets (≥1 unacceptable or missing video, 65 % of total) showed lower vessel density, worse perfusion and higher flow heterogeneity as compared to reliable triplets (p < 0.001). Quality was higher among triplets collected by an extensively-experienced investigator or in patients receiving sedation or mechanical ventilation. Perfused vessel density was higher in patients with Glasgow Coma Scale (GCS) ≤8 (18.9 ± 4.5 vs. 17.0 ± 3.9 mm/mm² in those with GCS >8, p < 0.001) or requiring mechanical ventilation (18.0 ± 4.5 vs. 17.2 ± 3.8 mm/mm² in not mechanically ventilated patients, p = 0.059). We concluded that SDF video quality depends on both the operator’s experience and patient’s cooperation. Low-quality videos may produce spurious data, leading to an overestimation of microvascular alterations.     

Towards clinical applicability and computational efficiency in automatic cranial implant design: An overview of the AutoImplant 2021 cranial implant design challenge

Cranial implants are commonly used for surgical repair of craniectomy-induced skull defects. These implants are usually generated offline and may require days to weeks to be available. An automated implant design process combined with onsite manufacturing facilities can guarantee immediate implant availability and avoid secondary intervention. To address this need, the AutoImplant II challenge was organized in conjunction with MICCAI 2021, catering for the unmet clinical and computational requirements of automatic cranial implant design. The first edition of AutoImplant (AutoImplant I, 2020) demonstrated the general capabilities and effectiveness of data-driven approaches, including deep learning, for a skull shape completion task on synthetic defects. The second AutoImplant challenge (i.e., AutoImplant II, 2021) built upon the first by adding real clinical craniectomy cases as well as additional synthetic imaging data. The AutoImplant II challenge consisted of three tracks. Tracks 1 and 3 used skull images with synthetic defects to evaluate the ability of submitted approaches to generate implants that recreate the original skull shape. Track 3 consisted of the data from the first challenge (i.e., 100 cases for training, and 110 for evaluation), and Track 1 provided 570 training and 100 validation cases aimed at evaluating skull shape completion algorithms at diverse defect patterns. Track 2 also made progress over the first challenge by providing 11 clinically defective skulls and evaluating the submitted implant designs on these clinical cases. The submitted designs were evaluated quantitatively against imaging data from post-craniectomy as well as by an experienced neurosurgeon. Submissions to these challenge tasks made substantial progress in addressing issues such as generalizability, computational efficiency, data augmentation, and implant refinement. This paper serves as a comprehensive summary and comparison of the submissions to the AutoImplant II challenge. Codes and models are available at https://github.com/Jianningli/Autoimplant_II.     

手术室腔镜管理中细节管理的应用效果分析

目的 研究手术室腔镜管理中的细节管理措施,旨在分析减少腔镜损坏的管理方法和效果.方法 选择本院手术室自2015年1~3月实施的1200台手术为观察组,对手术室的腔镜器械进行细节管理,将2014年10~12月的1200例手术作为对照组,将观察组手术在器械清洗消毒合格率、器械完好率及器械准备差错的发生率等方面与细节管理实施前的1200例手术进行对比,同时观察进行手术的医师对于细节管理实施的满意度.结果 细节管理实施前后,两组手术在器械清洗消毒合格率、器械完好率及器械准备差错的发生率等方面存在显著差异,观察组明显优于对照组(P<0.05),具有统计学意义.结论 在手术室通过对术中腔镜进行有效的细节管理,可显著提高手术中的护理质量,提高手术腔镜器械的完好率及器械清洗消毒合格率,有效降低器械准备差错的发生并显著增加手术医师对手术护理的满意度,因此,细节管理在手术室实施的意义重大.     

失效模式与效应分析在提升手术室软式内镜风险管理中的应用

目的：探讨失效模式与效应分析（FMEA）在提升手术室软式内镜风险管理中的应用效果。方法：选择我院2021年1月1日-12月1日自动内镜清洗消毒机（AER）清洗消毒后的内镜共827镜次设为FMEA实施前组;选择2022年1月1日-12月1日AER清洗消毒后的内镜共844镜次设为FMEA实施后组。运用FMEA工具,对手术室软式内镜清洗操作程序中的各个环节进行分析,通过计算风险优先指数（RPN）值查找每个环节中的高危因素,并对RPN＞125的关键环节制定并实施改进方案。比较FMEA实施前后失效模式RPN值、内镜消毒合格率、内镜故障维修率、手术患者等待时间、手术室护士AER使用熟练度以及医生满意度的变化。结果：FMEA实施后,经过筛选并整改的5个项目的RPN值均下降至＜125分;总体的RPN值由1247.24分下降至475.05分,RPN值下降率为61.91%;比较FMEA实施前后内镜消毒合格率,两者无统计学差异（P>0.05）;FMEA实施后手术患者等待时间为10.35±1.57min,低于实施前的12.38±1.91min（P＜0.05）;手术室护士AER操作考得分为92.50±3.92分,高于实施前的80.49±5.64分（P＜0.05）;医生满意度为93.87±2.16分,高于实施前的83.07±3.10分（P＜0.05）;比较FMEA实施前后手术室软式内镜故障维修率,两者无统计学差异（P>0.05）。结论：通过FMEA干预,优化手术室软式内镜清洗流程,加强手术室护士专业知识技能培训与考核,使手术室护士软式内镜清洗操作更加熟练,提高清洗效率与质量,减少手术患者的等待时间,提升医生满意度。     

Persistent and automatic intraoperative 3D digitization of surfaces under dynamic magnifications of an operating microscope

One of the major challenges impeding advancement in image-guided surgical (IGS) systems is the soft-tissue deformation during surgical procedures. These deformations reduce the utility of the patient’s preoperative images and may produce inaccuracies in the application of preoperative surgical plans. Solutions to compensate for the tissue deformations include the acquisition of intraoperative tomographic images of the whole organ for direct displacement measurement and techniques that combines intraoperative organ surface measurements with computational biomechanical models to predict subsurface displacements. The later solution has the advantage of being less expensive and amenable to surgical workflow. Several modalities such as textured laser scanners, conoscopic holography, and stereo-pair cameras have been proposed for the intraoperative 3D estimation of organ surfaces to drive patient-specific biomechanical models for the intraoperative update of preoperative images. Though each modality has its respective advantages and disadvantages, stereo-pair camera approaches used within a standard operating microscope is the focus of this article. A new method that permits the automatic and near real-time estimation of 3D surfaces (at 1 Hz) under varying magnifications of the operating microscope is proposed. This method has been evaluated on a CAD phantom object and on full-length neurosurgery video sequences (∼1 h) acquired intraoperatively by the proposed stereovision system. To the best of our knowledge, this type of validation study on full-length brain tumor surgery videos has not been done before. The method for estimating the unknown magnification factor of the operating microscope achieves accuracy within 0.02 of the theoretical value on a CAD phantom and within 0.06 on 4 clinical videos of the entire brain tumor surgery. When compared to a laser range scanner, the proposed method for reconstructing 3D surfaces intraoperatively achieves root mean square errors (surface-to-surface distance) in the 0.28–0.81 mm range on the phantom object and in the 0.54–1.35 mm range on 4 clinical cases. The digitization accuracy of the presented stereovision methods indicate that the operating microscope can be used to deliver the persistent intraoperative input required by computational biomechanical models to update the patient’s preoperative images and facilitate active surgical guidance.     

Preface

Purpose  

Today’s operating room is equipped with different devices supporting the surgeon. Due to the lack of common interfaces between devices, an integrated support of the surgical workflow is missing. In the field of implantation, a smooth exchange of preoperatively planned data between devices is of great interest. Additionally, the availability of standardized preoperative data would facilitate the documentation, especially with regard to Electronic Health Records.     

Vision-based online recognition of surgical activities

Purpose

Surgical processes are complex entities characterized by expressive models and data. Recognizable activities define each surgical process. The principal limitation of current vision-based recognition methods is inefficiency due to the large amount of information captured during a surgical procedure. To overcome this technical challenge, we introduce a surgical gesture recognition system using temperature-based recognition.

Methods

An infrared thermal camera was combined with a hierarchical temporal memory and was used during surgical procedures. The recordings were analyzed for recognition of surgical activities. The image sequence information acquired included hand temperatures. This datum was analyzed to perform gesture extraction and recognition based on heat differences between the surgeon’s warm hands and the colder background of the environment.

Results

The system was validated by simulating a functional endoscopic sinus surgery, a common type of otolaryngologic surgery. The thermal camera was directed toward the hands of the surgeon while handling different instruments. The system achieved an online recognition accuracy of 96 % with high precision and recall rates of approximately 60 %.

Conclusion

Vision-based recognition methods are the current best practice approaches for monitoring surgical processes. Problems of information overflow and extended recognition times in vision-based approaches were overcome by changing the spectral range to infrared. This change enables the real-time recognition of surgical activities and provides online monitoring information to surgical assistance systems and workflow management systems.