Guidance

Mixed reality can be a powerful tool for image-guided medical procedures, as it allows clinicians to visualize the medical imaging data in 3D directly aligned to the patient. Specific project areas are detailed below.

Transcranial Magnetic Stimulation (TMS)

Transcranial magnetic stimulation (TMS) is a non-invasive brain stimulation method that is an important treatment option for depression and other neuropsychiatric diseases. TMS uses magnetic fields to stimulate nerve cells in the brain and treatment efficacy depends on accurate positioning of the TMS coil. Image-based neuronavigation has been shown to improve treatment efficacy but is rarely used in the clinic due to the added technological burden. To promote the use of image-based neuronavigation and unleash the true potential of TMS for therapies and research studies, we developed an easy-to-use, standalone mixed reality treatment environment that integrates instrument tracking and targeting information including medical imaging data, TMS coil positioning feedback and stimulation response information all within a single, head-mounted display.

Breast Surgical Planning

One quarter of women who undergo breast lumpectomy to treat early-stage breast cancer in the United States require a second surgery because of concerns over incomplete tumor removal. This creates additional burdens for both patients and surgeons and has led to more women choosing mastectomy operations. We proposed projecting a 3D “hologram” based on breast MRI images directly onto the patient using the Microsoft HoloLens 2 to support surgeons during intraoperative planning. We developed a fully on-device, workflow-compatible solution for target localization and visualization, employing embedded reflectors instead of surface markers to provide internal anchors that remain stable despite tissue deformation. A pilot patient study has demonstrated that the system can be used in real surgical settings and resulted in a reduced re-excision rate. By providing accurate and robust localization along with intuitive 3D visualization, this approach holds promise for enhancing surgical precision and enabling
broader adoption of AR across soft-tissue procedures.

Ultrasound

Conventional handheld 2D ultrasound imaging is commonly used to non-invasively visualize anatomy and blood flow, and provide live guidance for a wide range of clinical procedures, including vascular access and biopsies. However, clinical ultrasound users currently must receive training and gain a significant amount of experience (often 1-2 years) to become proficient—to adaptively recognize structures, adjust and optimize probe placement, and mentally map the image shown on the monitor to the patient’s reference frame for interventional guidance. For this reason, we are developing a real-time ultrasound display through a wireless augmented-reality (AR) headset, to provide the user with instant registration of images to the patient and intuitive guidance of interventional tasks.

Past Projects

Craniotomy

Skull base surgery is often used to remove cancerous and non-cancerous growths from inside the head. While many surgeries can be done endoscopically via the nose or mouth, some surgeries require the surgeon to insert a hole or large window into the skull bone. Given the variable nature of the head anatomy and the location of vital structures underneath the bone, it would be helpful to identify the exact location of important anatomy before bone removal to avoid either unnecessary bleeding or additional time-consuming bone removal. Together with the department of Otolaryngology we are developing see-through AR applications that allow the surgeon to “see through” the skull and locate internal anatomy to facilitate planning and improve the safety of the surgical procedure.

Orthopedic Surgery

Femoral Acetabular Impingement (FAI) is a condition in which abnormal bony contact at the hip joint resulting from abnormal bone growth causes pain. Hip arthroscopy surgery is done to remove the excess bone and reduce pain, but it is a challenging procedure due to the limited field of view of the arthroscopic camera. The goal of this project is to incorporate a mixed reality overlay to guide the surgeon to resect the excess bone confidently and accurately.

Relevant Publication Highlights

November 14, 2020

Evaluation of Different Visualization Techniques for Perception-Based Alignment in Medical AR

Many augmented reality (AR) applications require the alignment of virtual objects to the real world; this is particularly important in medical AR scenarios where medical imaging information may be displayed directly on a patient and is used to identify the exact locations of specific anatomical structures within the body.
November 13, 2020

Landmark-Based Mixed-Reality Perceptual Alignment of Medical Imaging Data

Accurate alignment of virtual and real content is important for applications where the virtual rendering of medical imaging data is used to guide the medical procedure such as a surgery.
May 3, 2020

Application of Holographic Augmented Reality for External Approaches to the Frontal Sinus

External approaches to the frontal sinus such as osteoplastic flaps are challenging because they require blind entry into the sinus, posing risks of injury to the brain or orbit.
March 16, 2020

Comparison of Head Pose Tracking Methods for Mixed-Reality Neuronavigation

Repetitive Transcranial Magnetic Stimulation (rTMS) is an important treatment option for medication resistant depression.
April 29, 2019

Mixed-Reality Guidance for Brain Stimulation Treatment of Depression

Depression affects more than 16 million American adults, more than half of whom do not respond to medication.
March 30, 2018

MR Imaging of Magnetic Ink Patterns via Off-Resonance Sensitivity

The magnetic susceptibility effects of magnetic ink in MRI can be exploited to visualize magnetic ink patterns placed on the skin.
October 30, 2017

A Mixed-Reality System for Breast Surgical Planning

We have developed a mixed-reality system that projects a 3D “hologram” of images from a preoperative breast MRI onto a patient using the Microsoft HoloLens.