Mixed reality is a powerful tool for 3D visualization of medical imaging data.  It allows users to view and interact with anatomical structures in 3D, and also enables collaboration, as multiple people can view the same virtual content simultaneously.  Specific project areas are detailed below.

Lung nodule localization can be particularly difficult during minimally invasive thoracic surgery. Mixed reality can assist surgeons in planning their approach to localizing and resecting small lung nodules by allowing them to view and interact with medical imaging data effectively and more intuitively in 3D.  We have developed a mixed-reality visualization tool employing the first-generation Microsoft HoloLens that allows the user to (1) view a patient’s original preoperative imaging, (2) manipulate a 3D rendering of that patient’s chest anatomy, and (3) simulate lung deflation and surgical instrument placement. 

Mixed reality is a useful tool in education by allowing multiple users to perceive and interact with the same 3D virtual content. This can be either done in person where multiple people in the same room share a common experience or, in times where working from home has become increasingly common, with people at different locations interacting in the same virtual environment. The same technology can also be used as an ideal medical collaboration tool, since it allows multiple medical caregivers at different locations to examine and interact with the same medical imaging data or by allowing clinicians to remotely shadow or advice on clinical procedures. We are developing mixed reality education and collaboration tools that provide a better experience for both teachers and students during medical training and that make collaboration between researchers and clinicians easier and more efficient.  Click here to see a video of a tool developed in the IMMERS lab for a shared anatomy experience.

While the MR volumes and images show the inside of the human body, visualization and image evaluation of these images is done separately on flat displays. This separation between the display and the human body can make it difficult for students to learn human anatomy and for medical professionals to simultaneously evaluate interior and exterior features and how they relate to each other. In our research we are developing different virtual rendering methods to visualize medical imaging data on mixed reality displays and test how these technologies compare against conventional 2D visualization methods.  Example applications include the brain and kidneys.