[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/1654059.1654075acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

Terascale data organization for discovering multivariate climatic trends

Published: 14 November 2009 Publication History

Abstract

Current visualization tools lack the ability to perform full-range spatial and temporal analysis on terascale scientific datasets. Two key reasons exist for this shortcoming: I/O and postprocessing on these datasets are being performed in suboptimal manners, and the subsequent data extraction and analysis routines have not been studied in depth at large scales. We resolved these issues through advanced I/O techniques and improvements to current query-driven visualization methods. We show the efficiency of our approach by analyzing over a terabyte of multivariate satellite data and addressing two key issues in climate science: time-lag analysis and drought assessment. Our methods allowed us to reduce the end-to-end execution times on these problems to one minute on a Cray XT4 machine.

References

[1]
T. Peterka, H. Yu, R. Ross, and K.-L. Ma, "Parallel volume rendering on the IBM Blue Gene/P," in EGPGV '08: Proceedings of the Eurographics Symposium on Parallel Graphics and Visualization, April 2008, pp. 73--80.
[2]
"MODIS," http://modis.gsfc.nasa.gov.
[3]
D. Silver and X. Wang, "Tracking and visualizing turbulent 3d features," IEEE Transactions on Visualization and Computer Graphics, vol. 3, no. 2, pp. 129--141, 1997.
[4]
W. E. Lorensen and H. E. Cline, "Marching cubes: A high resolution 3d surface construction algorithm," in Proceedings of ACM SIGGRAPH, 1987, pp. 163--169.
[5]
M. Glatter, C. Mollenhour, J. Huang, and J. Gao, "Scalable data servers for large multivariate volume visualization," IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 5, pp. 1291--1298, 2006.
[6]
L. Gosink, J. C. Anderson, E. W. Bethel, and K. I. Joy, "Query-driven visualization of time-varying adaptive mesh refinement data," IEEE Transactions on Visualization and Computer Graphics, vol. 14, no. 6, 2008.
[7]
K. Stockinger, J. Shalf, K. Wu, and E. Bethel, "Query-driven visualization of large data sets," in VIS '05: Proceedings of the IEEE Visualization Conference, October 2005, pp. 167--174.
[8]
K.-L. Ma, C. Wang, H. Yu, and A. Tikhonova, "In situ processing and visualization for ultrascale simulations," Journal of Physics, vol. 78, June 2007, (Proceedings of the SciDAC 2007 Conference).
[9]
"Lustre," http://www.lustre.org.
[10]
W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitzberg, W. Saphir, and M. Snir, MPI-The Complete Reference: Volume 2 - The MPI Extensions. Cambridge, MA, USA: MIT Press, 1998.
[11]
R. Thakur, W. Gropp, and E. Lusk, "Data sieving and collective I/O in ROMIO," in Proceedings of the 7th Symposium on the Frontiers of Massively Parallel Computation, 1999, pp. 182--189.
[12]
K.-L. Ma, A. Stompel, J. Bielak, O. Ghattas, and E. J. Kim, "Visualizing large-scale earthquake simulations," in SC '03: Proceedings of the ACM/IEEE Supercomputing Conference, 2003.
[13]
H. Yu, K.-L. Ma, and J. Welling, "A parallel visualization pipeline for terascale earthquake simulations," in SC '04: Proceedings of the ACM/IEEE Supercomputing Conference, November 2004.
[14]
H. Yu, K.-L. Ma, and J. Welling, "I/O strategies for parallel rendering of large time-varying volume data," in EGPGV '04: Proceedings of the Eurographics Symposium on Parallel Graphics and Visualization, June 2004, pp. 31--40.
[15]
J. Lofstead, F. Zheng, S. Klasky, and K. Schwan, "Adaptable, metadata rich IO methods for portable high performance IO," in IPDPS '09: Proceedings of the IEEE International Symposium on Parallel and Distributed Processing, 2009.
[16]
M. Glatter, J. Huang, S. Ahern, J. Daniel, and A. Lu, "Visualizing temporal patterns in large multivariate data using textual pattern matching," IEEE Transactions on Visualization and Computer Graphics, vol. 14, no. 6, pp. 1467--1474, 2008.
[17]
L. Gosink, J. Anderson, W. Bethel, and K. Joy, "Variable interactions in query-driven visualization," IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 6, pp. 1400--1407, 2007.
[18]
K. Stockinger, E. W. Bethel, S. Campbell, E. Dart, and K. Wu, "Detecting distributed scans using high-performance query-driven visualization," in SC '06: Proceedings of the ACM/IEEE Supercomputing Conference, October 2006.
[19]
O. Rübel, Prabhat, K. Wu, H. Childs, J. Meredith, C. G. R. Geddes, E. Cormier-Michel, S. Ahern, G. H. Weber, P. Messmer, H. Hagen, B. Hamann, and E. W. Bethel, "High performance multivariate visual data exploration for extremely large data," in SC '08: Proceedings of the ACM/IEEE Supercomputing Conference, November 2008.
[20]
G. E. Blelloch, C. E. Leiserson, B. M. Maggs, C. G. Plaxton, S. Smith, and M. Zagha, "An experimental analysis of parallel sorting algorithms," Theory of Computing Systems, vol. 31, no. 2, pp. 135--167, 1998.
[21]
M. Meissner, J. Huang, D. Bartz, K. Mueller, and R. Crawfis, "A practical evaluation of the four most popular volume rendering algorithms," in Proceedings of the IEEE/ACM Symposium on Volume Visualization, October 2000.
[22]
F.-Y. Tzeng and K.-L. Ma, "Intelligent feature extraction and tracking for visualizing large-scale 4d flow simulations," in SC '05: Proceedings of the ACM/IEEE Supercomputing Conference, November 2005.
[23]
R. Ross, T. Peterka, H.-W. Shen, K.-L. Ma, H. Yu, and K. Moreland, "Visualization and parallel I/O at extreme scale," Journal of Physics, vol. 125, July 2008, (Proceedings of the SciDAC 2008 Conference).
[24]
R. Rew and G. Davis, "NetCDF: An interface for scientific data access," IEEE Computer Graphics and Applications, vol. 10, no. 4, pp. 76--82, 1990.
[25]
J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale, "Parallel netCDF: A high-performance scientific I/O interface," in SC '03: Proceedings of the ACM/IEEE Supercomputing Conference, 2003.
[26]
G. Memik, M. T. Kandemir, W.-K. Liao, and A. Choudhary, "Multicollective I/O: A technique for exploiting inter-file access patterns," ACM Transactions on Storage, vol. 2, no. 3, pp. 349--369, 2006.
[27]
W. Yu, J. S. Vetter, and S. Oral, "Performance characterization and optimization of parallel I/O on the Cray XT," in IPDPS '08: Proceedings of the IEEE International Symposium on Parallel and Distributed Processing, 2008, pp. 1--11.
[28]
"IOR," http://www.cs.sandia.gov/Scalable_IO/ior.html.
[29]
Y. Gu, J. F. Brown, J. P. Verdin, and B. Wardlow, "A five-year analysis of MODIS NDVI and NDWI for grassland drought assessment over the Central Great Plains of the United States," Geophysical Research Letters, vol. 34, 2007.
[30]
B. Gao, "NDWI -- A normalized difference water index for remote sensing of vegetation liquid water from space," Remote Sensing of Environment, vol. 58, no. 3, pp. 257--266, December 1996.
[31]
L. Wang, J. Qu, and X. Xiong, "A seven year analysis of water related indices for Georgia drought assessment over the 2007 wildfire regions," IEEE International Geoscience and Remote Sensing Symposium, 2008.
[32]
C. Liu and J. Wu, "Crop drought monitoring using MODIS NDDI over mid-territory of China," IEEE International Geoscience and Remote Sensing Symposium, 2008.
[33]
T. Tadessee, B. D. Wardlow, and J. H. Ryu, "Identifying time-lag relationships between vegetation condition and climate to produce vegetation outlook maps and monitor drought," 22nd Conference on Hydrology, 2008.

Cited By

View all
  • (2017)Towards GPU-Accelerated Web-GIS for Query-Driven Visual ExplorationWeb and Wireless Geographical Information Systems10.1007/978-3-319-55998-8_8(119-136)Online publication date: 22-Mar-2017
  • (2014)Scalable computation of stream surfaces on large scale vector fieldsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2014.87(1008-1019)Online publication date: 16-Nov-2014
  • (2013)Hierarchical I/O scheduling for collective I/OProceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2013.30(211-218)Online publication date: 13-May-2013
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
SC '09: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis
November 2009
778 pages
ISBN:9781605587448
DOI:10.1145/1654059
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2009

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. MODIS
  2. parallel I/O
  3. query-driven visualization
  4. temporal data analysis

Qualifiers

  • Research-article

Funding Sources

Conference

SC '09
Sponsor:

Acceptance Rates

SC '09 Paper Acceptance Rate 59 of 261 submissions, 23%;
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2017)Towards GPU-Accelerated Web-GIS for Query-Driven Visual ExplorationWeb and Wireless Geographical Information Systems10.1007/978-3-319-55998-8_8(119-136)Online publication date: 22-Mar-2017
  • (2014)Scalable computation of stream surfaces on large scale vector fieldsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2014.87(1008-1019)Online publication date: 16-Nov-2014
  • (2013)Hierarchical I/O scheduling for collective I/OProceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2013.30(211-218)Online publication date: 13-May-2013
  • (2012)Parallel particle advection and FTLE computation for time-varying flow fieldsProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389079(1-11)Online publication date: 10-Nov-2012
  • (2012)Visualization for the Physical SciencesComputer Graphics Forum10.1111/j.1467-8659.2012.03184.x31:8(2317-2347)Online publication date: 1-Dec-2012
  • (2012)Parallel particle advection and FTLE computation for time-varying flow fieldsProceedings of the 2012 International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2012.93(1-11)Online publication date: 10-Nov-2012
  • (2012)Geometric Quantification of Features in Large Flow FieldsIEEE Computer Graphics and Applications10.1109/MCG.2012.4932:4(46-54)Online publication date: 1-Jul-2012
  • (2011)Efficient I/O for parallel visualizationProceedings of the 11th Eurographics conference on Parallel Graphics and Visualization10.5555/2386230.2386243(81-90)Online publication date: 10-Apr-2011
  • (2011)Simplified parallel domain traversalProceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/2063384.2063397(1-11)Online publication date: 12-Nov-2011
  • (2011)Static correlation visualization for large time-varying volume data2011 IEEE Pacific Visualization Symposium10.1109/PACIFICVIS.2011.5742369(27-34)Online publication date: Mar-2011
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media