Mixed reality can be a powerful tool for image-guided medical procedures, as it allows clinicians to visualize the medical imaging data in 3D directly aligned to the patient.  Specific project areas are detailed below.

Transcranial magnetic stimulation (TMS) is a non-invasive brain stimulation method that is an important treatment option for depression and other neuropsychiatric diseases. TMS uses magnetic fields to stimulate nerve cells in the brain and treatment efficacy depends on accurate positioning of the TMS coil. Image-based neuronavigation has been shown to improve treatment efficacy but is rarely used in the clinic due to the added technological burden. To promote the use of image-based neuronavigation and unleash the true potential of TMS for therapies and research studies, we are developing an easy-to-use, mixed reality treatment environment that integrates targeting information including medical imaging data, TMS coil positioning feedback and stimulation response information within a single, head-mounted display.

One quarter of women who undergo breast lumpectomy to treat early-stage breast cancer in the United States undergo a repeat surgery due to concerns that residual tumor was left behind. This has led to a significant increase in women choosing mastectomy operations in the United States. We have developed a mixed-reality system that projects a 3D “hologram” of images from a breast MRI onto a patient using the first-generation Microsoft HoloLens. The goal of this system is to reduce the number of repeated surgeries by improving surgeons' ability to determine tumor extent. We are conducting a pilot study in patients with palpable tumors that tests a surgeon's ability to accurately identify the tumor location via mixed-reality visualization during surgical planning.

Skull base surgery is often used to remove cancerous and non-cancerous growths from inside the head. While many surgeries can be done endoscopically via the nose or mouth, some surgeries require the surgeon to insert a hole or large window into the skull bone. Given the variable nature of the head anatomy and the location of vital structures underneath the bone, it would be helpful to identify the exact location of important anatomy before bone removal to avoid either unnecessary bleeding or additional time-consuming bone removal. Together with the department of Otolaryngology we are developing see-through AR applications that allow the surgeon to “see through” the skull and locate internal anatomy to facilitate planning and improve the safety of the surgical procedure.

Conventional handheld 2D ultrasound imaging is commonly used to non-invasively visualize anatomy and blood flow, and provide live guidance for a wide range of clinical procedures, including vascular access and biopsies.  However, clinical ultrasound users currently must receive training and gain a significant amount of experience (often 1-2 years) to become proficient—to adaptively recognize structures, adjust and optimize probe placement, and mentally map the image shown on the monitor to the patient’s reference frame for interventional guidance. For this reason, we are developing a real-time ultrasound display through a wireless augmented-reality (AR) headset, to provide the user with instant registration of images to the patient and intuitive guidance of interventional tasks.

Femoral Acetabular Impingement (FAI) is a condition in which abnormal bony contact at the hip joint resulting from abnormal bone growth causes pain. Hip arthroscopy surgery is done to remove the excess bone and reduce pain, but it is a challenging procedure due to the limited field of view of the arthroscopic camera. The goal of this project is to incorporate a mixed reality overlay to guide the surgeon to resect the excess bone confidently and accurately.