Developing a virtual reality simulation system for preoperative planning of thoracoscopic thoracic surgery

BackgroundVideo-assisted thoracoscopic surgery (VATS) has become a standard approach for the treatment of lung cancer. However, its minimally invasive nature limits the field of view and reduces tactile feedback. These limitations make it vital that surgeons thoroughly familiarize themselves with the patient’s anatomy preoperatively. We have developed a virtual reality (VR) surgical navigation system using head-mounted displays (HMD). The aim of this study was to investigate the potential utility of this VR simulation system in both preoperative planning and intraoperative assistance, including support during thoracoscopic sublobar resection.MethodsThree-dimensional (3D) polygon data derived from preoperative computed tomography data was loaded into BananaVision software developed at Colorado State University and displayed on an HMD. An interactive 3D reconstruction image was created, in which all the pulmonary structures could be individually imaged. Preoperative resection simulations were performed with patient-individualized reconstructed 3D images.ResultsThe 3D anatomic structure of pulmonary vessels and a clear vision into the space between the lesion and adjacent tissues were successfully appreciated during preoperative simulation. Surgeons could easily evaluate the real patient’s anatomy in preoperative simulations to improve the accuracy and safety of actual surgery. The VR software and HMD allowed surgeons to visualize and interact with real patient data in true 3D providing a unique perspective.ConclusionsThis initial experience suggests that a VR simulation with HMD facilitated preoperative simulation. Routine imaging modalities combined with VR systems could substantially improve preoperative planning and contribute to the safety and accuracy of anatomic resection.     

Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods

Background

Augmented reality (AR) technology is used to reconstruct three-dimensional (3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.

Navigated liver surgery: State of the art and future perspectives

BackgroundIn recent years, the development of digital imaging technology has had a significant influence in liver surgery. The ability to obtain a 3-dimensional (3D) visualization of the liver anatomy has provided surgery with virtual reality of simulation 3D computer models, 3D printing models and more recently holograms and augmented reality (when virtual reality knowledge is superimposed onto reality). In addition, the utilization of real-time fluorescent imaging techniques based on indocyanine green (ICG) uptake allows clinicians to precisely delineate the liver anatomy and/or tumors within the parenchyma, applying the knowledge obtained preoperatively through digital imaging. The combination of both has transformed the abstract thinking until now based on 2D imaging into a 3D preoperative conception (virtual reality), enhanced with real-time visualization of the fluorescent liver structures, effectively facilitating intraoperative navigated liver surgery (augmented reality).Data sourcesA literature search was performed from inception until January 2021 in MEDLINE (PubMed), Embase, Cochrane library and database for systematic reviews (CDSR), Google Scholar, and National Institute for Health and Clinical Excellence (NICE) databases.ResultsFifty-one pertinent articles were retrieved and included. The different types of digital imaging technologies and the real-time navigated liver surgery were estimated and compared.ConclusionsICG fluorescent imaging techniques can contribute essentially to the real-time definition of liver segments; as a result, precise hepatic resection can be guided by the presence of fluorescence. Furthermore, 3D models can help essentially to further advancing of precision in hepatic surgery by permitting estimation of liver volume and functional liver remnant, delineation of resection lines along the liver segments and evaluation of tumor margins. In liver transplantation and especially in living donor liver transplantation (LDLT), 3D printed models of the donor's liver and models of the recipient's hilar anatomy can contribute further to improving the results. In particular, pediatric LDLT abdominal cavity models can help to manage the largest challenge of this procedure, namely large-for-size syndrome.     

Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery

Background

We applied a new concept of “image overlay surgery” consisting of the integration of virtual reality (VR) and augmented reality (AR) technology, in which dynamic 3D images were superimposed on the patient’s actual body surface and evaluated as a reference for surgical navigation in gastrointestinal, hepatobiliary and pancreatic surgery.

Methods

We carried out seven surgeries, including three cholecystectomies, two gastrectomies and two colectomies. A Macintosh and a DICOM workstation OsiriX were used in the operating room for image analysis. Raw data of the preoperative patient information obtained via MDCT were reconstructed to volume rendering and projected onto the patient's body surface during the surgeries. For accurate registration, OsiriX was first set to reproduce the patient body surface, and the positional coordinates of the umbilicus, left and right nipples, and the inguinal region were fixed as physiological markers on the body surface to reduce the positional error.

Results

The registration process was non-invasive and markerlesss, and was completed within 5 min. Image overlay navigation was helpful for 3D anatomical understanding of the surgical target in the gastrointestinal, hepatobiliary and pancreatic anatomies. The surgeon was able to minimize movement of the gaze and could utilize the image assistance without interfering with the forceps operation, reducing the gap from the VR. Unexpected organ injury could be avoided in all procedures. In biliary surgery, the projected virtual cholangiogram on the abdominal wall could advance safely with identification of the bile duct. For early gastric and colorectal cancer, the small tumors and blood vessels, which usually could not be found on the gastric serosa by laparoscopic view, were simultaneously detected on the body surface by carbon dioxide-enhanced MDCT. This provided accurate reconstructions of the tumor and involved lymph node, directly linked with optimization of the surgical procedures.

Conclusions

Our non-invasive markerless registration using physiological markers on the body surface reduced logistical efforts. The image overlay technique is a useful tool when highlighting hidden structures, giving more information.     

Artificial intelligence assisted display in thoracic surgery: development and possibilities

In this golden age of rapid development of artificial intelligence (AI), researchers and surgeons realized that AI could contribute to healthcare in all aspects, especially in surgery. The popularity of low-dose computed tomography (LDCT) and the improvement of the video-assisted thoracoscopic surgery (VATS) not only bring opportunities for thoracic surgery but also bring challenges on the way forward. Preoperatively localizing lung nodules precisely, intraoperatively identifying anatomical structures accurately, and avoiding complications requires a visual display of individuals’ specific anatomy for surgical simulation and assistance. With the advance of AI-assisted display technologies, including 3D reconstruction/3D printing, virtual reality (VR), augmented reality (AR), and mixed reality (MR), computer tomography (CT) imaging in thoracic surgery has been fully utilized for transforming 2D images to 3D model, which facilitates surgical teaching, planning, and simulation. AI-assisted display based on surgical videos is a new surgical application, which is still in its infancy. Notably, it has potential applications in thoracic surgery education, surgical quality evaluation, intraoperative assistance, and postoperative analysis. In this review, we illustrated the current AI-assisted display applications based on CT in thoracic surgery; focused on the emerging AI applications in thoracic surgery based on surgical videos by reviewing its relevant researches in other surgical fields and anticipate its potential development in thoracic surgery.     

Computer-assisted surgery

The broad range of Computer Assisted Surgery (CAS) represents the integration of computer technology in surgical procedures for presurgical planning, guiding or manipulation. Surgical robots and surgical endoscopic navigation are the most challenging applications to urology. A surgical robot is defined as a computer-controlled manipulator with artificial sensing which can be programmed to move, and position tools to carry out surgical tasks. In urology, robots have been tested in two areas: endourology and laparoscopy. Surgical navigation allows the surgeon to process data from pre- and intraoperative sources, aiming at purification and presentation of the most relevant information. Image-guided systems (IGS), augmented reality (AR) and navigation in endoscopic soft tissue surgery represent the three main topics of surgical urological navigation. IGS involve matching the coordinates from medical imaging (preoperative registration) with coordinates from the patient in the operating room (registration and updating images). IGS have become the standard of care in providing navigational assistance during neurosurgery, offering subsurface and functional information to the surgeon.     

Uses of a dedicated 3D reconstruction software with augmented and mixed reality in planning and performing advanced liver surgery and living donor liver transplantation (with videos)

The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics, virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality. In this article, we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation. Furthermore, we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.     

Augmented reality (AR) and virtual reality (VR) applied in dentistry

The OSCE is a reliable evaluation method to estimate the preclinical examination of dental students. The most ideal assessment for OSCE is used the augmented reality simulator to evaluate. This literature review investigated a recently developed in virtual reality (VR) and augmented reality (AR) starting of the dental history to the progress of the dental skill. As result of the lacking of technology, it needs to depend on other device increasing the success rate and decreasing the risk of the surgery. The development of tracking unit changed the surgical and educational way. Clinical surgery is based on mature education. VR and AR simultaneously affected the skill of the training lesson and navigation system. Widely, the VR and AR not only applied in the dental training lesson and surgery, but also improved all field in our life.     

Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion

Background

New technologies can considerably improve preoperative planning, enhance the surgeon’s skill and simplify the approach to complex procedures. Augmented reality techniques, robot assisted operations and computer assisted navigation tools will become increasingly important in surgery and in residents’ education.

Methods

We obtained 3D reconstructions from simple spiral computed tomography (CT) slides using OsiriX, an open source processing software package dedicated to DICOM images. These images were then projected on the patient's body with a beamer fixed to the operating table to enhance spatial perception during surgical intervention (augmented reality).

Results

Changing a window's deepness level allowed the surgeon to navigate through the patient's anatomy, highlighting regions of interest and marked pathologies. We used image overlay navigation for laparoscopic operations such cholecystectomy, abdominal exploration, distal pancreas resection and robotic liver resection.

Conclusions

Augmented reality techniques will transform the behaviour of surgeons, making surgical interventions easier, faster and probably safer. These new techniques will also renew methods of surgical teaching, facilitating transmission of knowledge and skill to young surgeons.     

A Heads-Up Display for Diabetic Limb Salvage Surgery: A View Through the Google Looking Glass

Although the use of augmented reality has been well described over the past several years, available devices suffer from high cost, an uncomfortable form factor, suboptimal battery life, and lack an app-based developer ecosystem. This article describes the potential use of a novel, consumer-based, wearable device to assist surgeons in real time during limb preservation surgery and clinical consultation. Using routine intraoperative, clinical, and educational case examples, we describe the use of a wearable augmented reality device (Google Glass; Google, Mountain View, CA). The device facilitated hands-free, rapid communication, documentation, and consultation. An eyeglass-mounted screen form factor has the potential to improve communication, safety, and efficiency of intraoperative and clinical care. We believe this represents a natural progression toward union of medical devices with consumer technology.     

Navigation and image-guided HBP surgery: a review and preview

Image‐guided surgery and navigation have resulted from convergent developments in radiology, teletransmission, and computer science. Patient selection and preoperative planning in hepatobiliary‐pancreatic (HBP) surgery rely on preoperative imaging. The operative procedure is finally led by the fusion of additional information gained by the palpating hand and intraoperative ultrasound. Despite advances in reducing morbidity and mortality, decisions are often hardly quantifiable and are restricted to super‐specialists in HBP surgery. New developments in computed tomography (CT) and magnetic resonance imaging (MRI) technology have led to the possibility of the volumetric prediction of liver resections. These data can be shared via telemedicine and used for simulation and training. Three‐dimensional (3D) reconstructions have led to a better topologic understanding of tumor‐vascular tree relations in the individual patient. With the increasing use of ablative procedures and laparoscopy, intraoperative imaging and navigation will hold increasing significance for the HBP surgeon. Flat screen monitors adjacent to the surgical field present computer‐generated 3D virtual liver resection proposals which can be transferred into the real liver. The main obstacles in HBP navigation are the flexibility and mobility of the target organ. Intrahepatic and surface markers seem to be mandatory for computer‐navigated surgery. The first feasibility studies are promising.     

Intraoperative endoscopy: An important adjunct to gastrointestinal surgery

Flexible endoscopy has become an increasingly important skill for gastrointestinal (GI) surgeons, and there is no setting more important than the intraoperative setting for surgeons to employ endoscopic techniques during the course of surgical procedures performed on the GI tract. Endoscopic confirmation of pathology before initiating surgery, intraoperative anastomotic evaluation and margin assessment, and combined laparoscopic-endoscopic approaches to patient care are just a few examples emphasizing the need for surgeons to perform GI endoscopy as a routine adjunct to foregut, bariatric, and colorectal procedures. Intraoperative endoscopy adds value in the operating room and holds the promise of improved surgical outcomes by providing useful clinical information important to point-of-service decision making that allows surgeons to address technical concerns before they manifest as postoperative complications.     

Innovative surgical approaches for hepatocellular carcinoma

Hepatocellular carcinoma(HCC)is the sixth most common cancer worldwide,with an increasing diffusion in Europe and the United States.The management of such a cancer is continuously progressing and the objective of this paper is to evaluate innovation in the surgical treatment of HCC.In this review,we will analyze the modern concept of preoperative management,the role of laparoscopic and robotic surgery,the intraoperative use of three dimensional models and augmented reality,as well as the potential application of fluorescence.     

The role of intraoperative two-dimensional echocardiography in the assessment of thoracic aorta pathology

The role of intraoperative two-dimensional echocardiography is discussed in 15 consecutive patients with thoracic aorta pathology undergoing cardiac surgery. A 5 MHz mechanical scanner was used before and immediately after cardiopulmonary bypass. In 5 patients intraoperative two-dimensional studies revealed crucial morphologic information which, consequently, had a marked influence on their planned surgical procedure. In 3 patients the findings provided additional information whereas in the remaining patients the intraoperative echocardiographic findings confirmed the preoperative diagnosis. Following surgery the adequacy of cardiac repair was assessed and, in one patient, epicardial echocardiography indicated the necessity for reoperation. The application of intraoperative two-dimensional echocardiography leads to a better understanding of the pathology involved and facilitates a more appropriate decision concerning the surgical procedure.     

Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery

Background

We have reported the utility of an image display system using augmented reality (AR) technology in hepatobiliary surgery under laparotomy. Among several procedures, we herein report a system using a novel short rigid scope and stereo-scope, both designed specifically for open abdominal navigation surgery, and their clinical application for hepatobiliary and pancreatic surgery.

Methods

The 3D reconstructed images were obtained from preoperative computed tomography data. In our specialized operating room, after paired-point matching registration, the reconstructed images are overlaid onto the operative field images captured by the short rigid scopes. The scopes, which are compact and sterilizable, can be used in the operative field. The stereo-scope provides depth information. Eight patients underwent operations using this system, including hepatectomy in two, distal pancreatectomy in three, and pancreaticoduodenectomy in three patients. The stereo-scope was used in five patients.

Results

All eight operations were performed safely using the novel short rigid scopes, and stereo images were acquired in all five patients for whom the stereo-scope was used. The scopes were user friendly, and the intraoperative time requirement for our system was reduced compared with the conventional method.

Conclusions

The novel short rigid scope and stereo-scope seem to be suitable for clinical use in open abdominal navigation surgery. In hepatobiliary and pancreatic surgery, our novel system may improve the safety, accuracy and efficiency of operations.     

Artificial Intelligence and Machine Learning in Cardiac Electrophysiology

Cardiac electrophysiology requires the processing of several patient-specific data points in real time to provide an accurate diagnosis and determine an optimal therapy. Expanding beyond the traditional tools that have been used to extract information from patient-specific data, machine learning offers a new set of advanced tools capable of revealing previously unknown data patterns and features. This new tool set can substantially improve the speed and level of confidence with which electrophysiologists can determine patient-specific diagnoses and therapies. The ability to process substantial amounts of data in real time also paves the way to novel techniques for data collection and visualization. Extended realities such as virtual and augmented reality can now enable the real-time visualization of 3-dimensional images in space. This enables improved preprocedural planning and intraprocedural interventions. Machine learning supplemented with novel visualization technologies could substantially improve patient care and outcomes by helping physicians to make more informed patient-specific decisions. This article presents current applications of machine learning and their use in cardiac electrophysiology.     

Efficacy of virtual reality therapy in ideomotor apraxia rehabilitation: A case report

Rationale:We report the possible therapeutic efficacy of immersive virtual reality (VR) rehabilitation for the treatment of ideomotor apraxia in a patient with stroke.Patient concerns:A 56-year-old man with sudden weakness of his left side caused by right frontal, parietal, and corpus callosal infarction was transferred to rehabilitation medicine center for intensive rehabilitation. Although his left-sided weakness had almost subsided 10 days after the onset of symptoms, he presented difficulty using his left hand and required assistance in most activities of daily living.Diagnoses:Ideomotor apraxia in a patient with right hemispheric infarction.Interventions:VR content was displayed to the study participants using a head-mounted display that involved catching of moving fish in the sea by grasping. Before and after of rehabilitative intervention including VR, functional measurements incorporating the Test of Upper Limb Apraxia (TULIA) were conducted. To directly compare therapeutic potencies under different conditions, success rates of consecutive grasping gesture performance were observed in VR, conventional occupational therapy setting, and augmented reality intervention.Outcomes:The patient demonstrated remarkable amelioration of apraxic symptoms while performing the task in the VR environment. At 1 and 3 months after the training, he showed significant improvement in most functions, and the TULIA score increased to 176 from 121 at the initiation of therapy. The number of successful grasps during 30 trials of each grasp trial was 28 in VR, 8 in the occupational therapy setting, and 20 in augmented reality.Lessons:This case report suggests the possible therapeutic efficacy of immersive VR training as a rehabilitative measure for ideomotor apraxia.     

Virtual liver resection: computer-assisted operation planning using a three-dimensional liver representation

In liver surgery, understanding the complicated liver structures and a detailed evaluation of the functional liver remnant volume are essential to perform safe surgical procedures. Recent advances in imaging technology have enabled operation planning using three-dimensional (3D) image–processing software. Virtual liver resection systems provide (1) 3D imaging of liver structures, (2) detailed volumetric analyses based on portal perfusion, and (3) quantitative estimates of the venous drainage area, enabling the investigation of uncharted fields that cannot be examined using a conventional two-dimensional modality. The next step in computer-assisted liver surgery is the application of a virtual hepatectomy to real-time operations. However, the need for a precise alignment between the preoperative imaging data and the intraoperative situation remains to be adequately addressed, since the liver is subject to deformation and respiratory movements during the surgical procedures. We expect that the practical application of a navigation system for transferring the preoperative planning to real-time operations could make liver surgery safer and more standardized in the near future.     

Interactive virtual assembling in augmented reality

In this paper, a computer-aided methodology based on augmented reality to manipulate and interactively assemble virtual objects is presented. The developed system, based on both hardware and software implementation, allows the user to be immersed in an augmented scene (real world and virtual objects) by means of an head-mounted display and interact with the virtual components using a glove with sensors. The author presents the implementation of the system facing the problems concerning the interpretation of user’s intent to grasp and move the objects and the computer-aided assembly based on a novel approach concerning active features dynamic recognition. An example of implementation is herein reported and discussed.     

Early experience with a patient‐specific virtual surgical simulation for rehearsal of endoscopic skull‐base surgery

Background

With the help of contemporary computer technology it is possible to create a virtual surgical environment (VSE) for training. This article describes a patient‐specific virtual rhinologic surgical simulation platform that supports rehearsal of endoscopic skull‐base surgery. We also share our early experience with select cases.

Methods

A rhinologic VSE was developed, featuring a highly efficient direct 3‐dimensional (3D) volume renderer with simultaneous stereoscopic feedback during surgical manipulation of the virtual anatomy, as well as high‐fidelity haptic feedback. We conducted a retrospective analysis on 10 patients who underwent various forms of sinus and ventral skull‐base surgery to assess the ability of the rhinologic VSE to replicate actual intraoperative findings.

Results

In all 10 cases, the simulation experience was realistic enough to perform dissections in a similar manner as in the actual surgery. Excellent correlation was found in terms of surgical exposure, anatomical features, and the locations of pathology.

Conclusion

The current rhinologic VSE shows sufficient realism to allow patient‐specific surgical rehearsal of the sinus and ventral skull base. Further validation studies are needed to assess the benefits of performing patient‐specific rehearsal.