Abstract
As one of the main technologies of in situ visualization, data compression plays a key role in solving I/O bottleneck and has been intensively studied. However, existing methods take too much compression time to meet the requirement of in situ processing on computational fluid dynamics (CFD) flow field data. To address this problem, we introduce deep learning into CFD data compression and propose a novel in situ compression method based on generative adversarial network (GAN) in this paper. In specific, the proposed method samples small patches from CFD data and trains a GAN which includes two convolutional neural networks: the discriminative network and the generative network. The discriminative network is responsible for compressing data on compute nodes, while the generative network is used to reconstruct data on visualization nodes. Compared with the existing CFD data compression methods, our method has great advantages in compression time and manages to adjust compression ratio according to acceptable reconstruction effect, showing its applicability for loosely coupled in situ visualization. Extensive experimental results demonstrate the good generalization of the proposed method on many datasets.
Graphical Abstract
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Agustsson E, Tschannen M, Mentzer F, Timofte R, Van Gool L (2018) Generative adversarial networks for extreme learned image compression. arXiv:1804.02958
Alexander L, Jrg S (2006) Variable quality compression of fluid dynamical data sets using a 3d dct technique. Geochem Geophys Geosyst 7(1):1–13. https://doi.org/10.1029/2005GC001017
Ballé J, Laparra V, Simoncelli EP (2016) End-to-end optimized image compression. arXiv:1611.01704
Bauer AC, Abbasi H, Ahrens J, Childs H, Geveci B, Klasky S, Moreland K, O’Leary P, Vishwanath V, Whitlock B, Bethel EW (2016) In situ methods, infrastructures, and applications on high performance computing platforms. Comput Gr Forum 35(3):577–597. https://doi.org/10.1111/cgf.12930
Bengio Y (2009) Learning deep architectures for ai. Found Trends Mach Learn 2(1):1–127. https://doi.org/10.1561/2200000006
Cox D (2016) Syntactically informed text compression with recurrent neural networks. arXiv:1608.02893
Guo X, Li W, Iorio F (2016) Convolutional neural networks for steady flow approximation. In: Proceedings of the 22Nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16. ACM, New York, NY, USA, pp 481–490. https://doi.org/10.1145/2939672.2939738
Guo Y, Liu F, Cai Z, Xiao N, Zhao Z (2018) Edge-based efficient search over encrypted data mobile cloud storage. Sensors 18(4): https://doi.org/10.3390/s18041189
Jiang F, Tao W, Liu S, Ren J, Guo X, Zhao D (2017) An end-to-end compression framework based on convolutional neural networks. IEEE Trans Circ Syst Video Technol, 1–13. https://doi.org/10.1109/TCSVT.2017.2734838
Kress J, Klasky S, Podhorszki N, Choi J, Childs H, Pugmire D (2015) Loosely coupled in situ visualization: a perspective on why it’s here to stay. In: Proceedings of the first workshop on in situ infrastructures for enabling extreme-scale analysis and visualization, ISAV2015. ACM, New York, NY, USA, pp. 1–6. https://doi.org/10.1145/2828612.2828623
Lakshminarasimhan S, Shah N, Ethier S, Klasky S, Latham R, Ross R, Samatova NF (2011) Compressing the incompressible with ISABELA: in-situ reduction of spatio-temporal data. In: Jeannot E, Namyst R, Roman J (eds) Euro-Par 2011 parallel processing. Springer, Berlin, pp 366–379
Lee S, You D (2017) Prediction of laminar vortex shedding over a cylinder using deep learning. arXiv:1712.07854
Li M, Zuo W, Gu S, Zhao D, Zhang D (2017) Learning convolutional networks for content-weighted image compression. arXiv:1703.10553
Li S, Gruchalla K, Potter K, Clyne J, Childs H (2015) Evaluating the efficacy of wavelet configurations on turbulent-flow data. In: 2015 IEEE 5th symposium on large data analysis and visualization (LDAV), pp. 81–89. https://doi.org/10.1109/LDAV.2015.7348075
Lindstrom P (2014) Fixed-rate compressed floating-point arrays. IEEE Trans Vis Comput Gr 20(12):2674–2683. https://doi.org/10.1109/TVCG.2014.2346458
Lindstrom P, Isenburg M (2006) Fast and efficient compression of floating-point data. IEEE Trans Vis Comput Gr 12(5):1245–1250. https://doi.org/10.1109/TVCG.2006.143
Ling J, Kurzawski A, Templeton J (2016) Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J Fluid Mech 807:155–166. https://doi.org/10.1017/jfm.2016.615
Liu F, Li T (2018) A clustering k-anonymity privacy-preserving method for wearable iot devices. Sec Commun Netw 2018:1–8. https://doi.org/10.1155/2018/4945152
Sakai R, Sasaki D, Nakahashi K (2013) Parallel implementation of large-scale cfd data compression toward aeroacoustic analysis. Comput Fluids 80:116–127. https://doi.org/10.1016/j.compfluid.2012.04.020 (selected contributions of the 23rd International Conference on Parallel Fluid Dynamics ParCFD2011)
Salloum M, Fabian N, Hensinger DM, Templeton JA (2015) Compressed sensing and reconstruction of unstructured mesh datasets. arXiv:1508.06314
Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117. https://doi.org/10.1016/j.neunet.2014.09.003
Schmidhuber J, Heil S (1996) Sequential neural text compression. IEEE Trans Neural Netw 7(1):142–146. https://doi.org/10.1109/72.478398
Theis L, Bethge M (2015) Generative image modeling using spatial lstms. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates, Inc., Red Hook, pp 1927–1935
Theis L, Shi W, Cunningham A, Huszár F (2017) Lossy Image compression with compressive autoencoders. arXiv:1703.00395
Toderici G, O’Malley SM, Hwang SJ, Vincent D, Minnen D, Baluja S, Covell M, Sukthankar R (2015) Variable rate image compression with recurrent neural networks. arXiv:1511.06085
van den Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. arXiv:1601.06759
Vogler P, Rist U (2017) Wavelet-based compression of volumetric cfd data sets. In: Resch MM, Bez W, Focht E, Gienger M, Kobayashi H (eds) Sustained simulation performance 2017. Springer International Publishing, Cham, pp 123–136
Welch TA (1984) A technique for high-performance data compression. Computer 17(6):8–19. https://doi.org/10.1109/MC.1984.1659158
Zeyen M, Ahrens J, Hagen H, Heitmann K, Habib S (2017) Cosmological particle data compression in practice. In: Proceedings of the in situ infrastructures on enabling extreme-scale analysis and visualization, ISAV’17. ACM, New York, NY, USA, pp. 12–16. https://doi.org/10.1145/3144769.3144776
Acknowledgements
The authors wish to thank Dr. Dong Sun for his guidance in CFD. This work was supported in part by the National Key Research and Development Program of China (#2016YFB0200701, #2018YFB0203904 and #2016YFB1000302), National Nature Science Foundation of China (#U1611261) and the Program for Guangdong Introducing Innovative and Enterpreneurial Teams (#2016ZT06D211).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Liu, Y., Wang, Y., Deng, L. et al. A novel in situ compression method for CFD data based on generative adversarial network. J Vis 22, 95–108 (2019). https://doi.org/10.1007/s12650-018-0519-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12650-018-0519-x