Abstract
Surgeons use information from multiple sources when making surgical decisions. These include volumetric datasets (such as CT, PET, MRI, and their variants), 2D datasets (such as endoscopic videos), and vector-valued datasets (such as computer simulations). Presenting all the information to the user in an effective manner is a challenging problem. In this paper, we present a visualization approach that displays the information from various sources in a single coherent view. The system allows the user to explore and manipulate volumetric datasets, display analysis of dataset values in local regions, combine 2D and 3D imaging modalities and display results of vector-based computer simulations. Several interaction methods are discussed: in addition to traditional interfaces including mouse and trackers, gesture-based natural interaction methods are shown to control these visualizations with real-time performance. An example of a medical application (medialization laryngoplasty) is presented to demonstrate how the combination of different modalities can be used in a surgical setting with our approach.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Correa CD: Visualizing what lies inside. SIGGRAPH Comput Graph 43(2):1–6, 2009
Luo H, Mittal R, Zheng X, Bielamowicz S, Walsh R, Hahn J: An immersed-boundary method for flow–structure interaction in biological systems with application to phonation. J Comput Phys 227(22):9303–9332, 2008
Perrin DP, Vasilyev NV, Novotny P, Stoll J, Howe RD, Dupont PE, et al: Image guided surgical interventions. Curr Probl Surg 46(9):730–766, 2009
Peters TM, Cleary K: Image-guided interventions: technology and applications. Springer, New York, 2008
Peters TM: Image-guidance for surgical procedures. Phys Med Biol 14:R505, 2006
Kruger J, Schneider J, Westermann R: ClearView: an interactive context preserving hotspot visualization technique. IEEE Trans Visual Comput Graph 12(5):941–948, 2006
Svakhine N, Ebert DS, Stredney D: Illustration motifs for effective medical volume illustration. IEEE Comput Graph Appl 25(3):31–39, 2005
Bier EA, Stone MC, Pier K, Buxton W and DeRose TD: Toolglass and magic lenses: The see-through interface. In: Proceedings of the 20th annual conference on Computer graphics and interactive techniques, Anaheim, CA, 1993, pp 73–80
Wang L, Zhao Y, Mueller K and Kaufman A: The magic volume lens: an interactive focus+context technique for volume rendering. pp 367–374
Viega J, Conway MJ, Williams G and Pausch R: 3D magic lenses. In: Proceedings of the 9th annual ACM symposium on User interface software and technology, Seattle, Washington, United States, 1996
Bruckner S and Gröller ME: VolumeShop: an interactive system for direct volume illustration. In: IEEE Visualization, 2005, pp 671–678
Burger K, Kruger J, Westermann R: Direct Volume Editing. IEEE Trans Visual Comput Graph 14(6):1388–1395, 2008
Kirmizibayrak C: Interactive volume visualization and editing methods for surgical applications. Ph.D. Dissertation, Department of Computer Science, The George Washington University, Washington, DC, 2011
Zhang Z: Iterative point matching for registration of free-form curves and surfaces. Int J Comput Vision 13(2):119–152, 1994
Gerasimov P, Fernando R and Green S: Shader Model 3.0: using vertex textures, white paper DA-01373-001_v00, NVIDIA, 2004
Scheuermann T and Hensley: Efficient histogram generation using scattering on GPUs. In: Proceedings of the 2007 symposium on Interactive 3D graphics and games, Seattle, Washington, 2007, pp 33–37
Li W, Fan Z, Wei X and Kaufman A: GPU-Based flow simulation with complex boundaries. GPU Gems II, 2003
Yim Y, Wakid M, Kirmizibayrak C, Bielamowicz S and Hahn JK: Registration of 3D CT data to 2D endoscopic image using a gradient mutual information based viewpoint matching for image-guided Medialization Laryngoplasty, JCSE 4(4):368–387, 2010
Studholme C, Hill DLG, Hawkes DJ: An overlap invariant entropy measure of 3D medical image alignment. Pattern Recogn 32(1):71–86, 1999
Johnson R, O’Hara K, Sellen A, Cousins C and Criminisi A: Exploring the potential for touchless interaction in image-guided interventional radiology. In: Proceedings of the 2011 annual conference on Human factors in computing systems, Vancouver, BC, Canada, 2011, pp 3323–3332
Kirmizibayrak C, Radeva N, Wakid M, Philbeck J, Sibert J and Hahn J: Evaluation of gesture based interfaces for medical volume visualization tasks. In: Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, Hong Kong, China, 2011, pp 69–74
Helferty JP, Sherbondy AJ, Kiraly AP, Higgins WE: Computer-based system for the virtual-endoscopic guidance of bronchoscopy. Comp Vis Image Understand 108(1–2):171–187, 2007
OpenNI Organization. “OpenNI User Guide,” 05/01, 2011; http://www.openni.org/documentation
Jin G, Baek N, Hahn JK, Bielamowicz S, Mittal R, Walsh R: Image guided medialization laryngoplasty. Computer Animation and Virtual Worlds 20(1):67–77, 2009
Bielamowicz S: Perspectives on medialization laryngoplasty. Otolaryngolog Clin North America 37(1):139, 2004
Luo H, Mittal R and Bielamowicz SA: Analysis of flow-structure interaction in the larynx during phonation using an immersed-boundary method. J Acoust Soc Am vol. 126:no. 2, 2009
Acknowledgment
This work was supported by NIH grant R01 DC007125 to develop computer-based tools for medialization laryngoplasty. The authors would like to thank Dr. Steven Bielamowicz for his help and informative discussions for application of our methods to otolaryngology, and Dr. Rajat Mittal and Qian Xue for providing the CFD simulation datasets.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kirmizibayrak, C., Yim, Y., Wakid, M. et al. Interactive Visualization and Analysis of Multimodal Datasets for Surgical Applications. J Digit Imaging 25, 792–801 (2012). https://doi.org/10.1007/s10278-012-9461-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10278-012-9461-y