Landmark-Based Mixed-Reality Perceptual Alignment of Medical Imaging Data and Accuracy Validation in Living Subjects

Accurate alignment of virtual and real content is important for applications where the virtual rendering of medical imaging data is used to guide the medical procedure such as a surgery. In 2D augmented rearlity (AR) applications, the alignment accuracy can be directly measured on the 2D screen by measuring the distance between virtual rendering and real patient in pixels. In 3D see-through medical AR applications, the alignment procedure and validation measurements are more complicated, because alignment accuracy depends both on the optical system of the AR device as well as on the user’s individual perception.

In this paper we present an approach for landmark-based alignment, validation and accuracy measurement of a 3D AR overlay of medical images on the real-world subject. This is done by performing an initial MRI of a subject’s head, an AR alignment task of the virtual rendering of the head MRI data to the subject’s real-world head using virtual fiducials, and a second MRI scan to test the accuracy of the AR alignment task. The accuracy measurements showed that the current alignment method already provides a very good accuracy for low-risk medical procedures that require an accuracy of less than 5mm such as transcranial magnetic stimulation treatment.

Leuze C, Sathyanarayana S, Daniel B, McNab J. Landmark-based mixed-reality perceptual alignment of medical imaging data and accuracy validation in living subjects. 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).