More Web Proxy on the site http://driver.im/

survey

A Systematic Survey of Regularization and Normalization in GANs

Authors:

Muhammad Usman,

Bin LiAuthors Info & Claims

ACM Computing Surveys, Volume 55, Issue 11

Article No.: 232, Pages 1 - 37

https://doi.org/10.1145/3569928

Published: 09 February 2023 Publication History

Abstract

Generative Adversarial Networks (GANs) have been widely applied in different scenarios thanks to the development of deep neural networks. The original GAN was proposed based on the non-parametric assumption of the infinite capacity of networks. However, it is still unknown whether GANs can fit the target distribution without any prior information. Due to the overconfident assumption, many issues remain unaddressed in GANs training, such as non-convergence, mode collapses, and gradient vanishing. Regularization and normalization are common methods of introducing prior information to stabilize training and improve discrimination. Although a handful number of regularization and normalization methods have been proposed for GANs, to the best of our knowledge, there exists no comprehensive survey that primarily focuses on objectives and development of these methods, apart from some incomprehensive and limited-scope studies. In this work, we conduct a comprehensive survey on the regularization and normalization techniques from different perspectives of GANs training. First, we systematically describe different perspectives of GANs training and thus obtain the different objectives of regularization and normalization. Based on these objectives, we propose a new taxonomy. Furthermore, we compare the performance of the mainstream methods on different datasets and investigate the applications of regularization and normalization techniques that have been frequently employed in state-of-the-art GANs. Finally, we highlight potential future directions of research in this domain. Code and studies related to the regularization and normalization of GANs in this work are summarized at https://github.com/iceli1007/GANs-Regularization-Review.

References

[1]

Jonas Adler and Sebastian Lunz. 2018. Banach wasserstein GAN. In Advances in Neural Information Processing Systems. 6754–6763.

[2]

Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. 2020. Image Generators with Conditionally-Independent Pixel Synthesis. (2020). arxiv:cs.CV/2011.13775

[3]

Martin Arjovsky and Léon Bottou. 2017. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862 (2017).

[4]

Martin Arjovsky and Léon Bottou. 2017. Towards Principled Methods for Training Generative Adversarial Networks. (2017). arxiv:stat.ML/1701.04862

[5]

Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein GAN. arXiv preprint arXiv:1701.07875 (2017).

[6]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).

[7]

Gulcin Baykal and Gozde Unal. 2020. DeshuffleGAN: A self-supervised GAN to improve structure learning. arXiv preprint arXiv:2006.08694 (2020).

[8]

Vineeth S. Bhaskara, Tristan Aumentado-Armstrong, Allan D. Jepson, and Alex Levinshtein. 2022. GraN-GAN: Piecewise gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3821–3830.

[9]

Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of the International Conference on Computational Statistics. Springer, 177–186.

[10]

Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018).

[11]

Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. 2016. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093 (2016).

[12]

Fabio M. Carlucci, Antonio D’Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. 2019. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2229–2238.

[13]

Huanhuan Chen. 2008. Diversity and Regularization in Neural Network Ensembles. Ph.D. Dissertation. University of Birmingham.

[14]

Huanhuan Chen, Peter Tino, and Xin Yao. 2009. Probabilistic classification vector machines. IEEE Transactions on Neural Networks 20, 6 (2009), 901–914.

Digital Library

[15]

Huanhuan Chen, Peter Tiňo, and Xin Yao. 2013. Efficient probabilistic classification vector machine with incremental basis function selection. IEEE Transactions on Neural Networks and Learning Systems 25, 2 (2013), 356–369.

[16]

Huanhuan Chen and Xin Yao. 2009. Regularized negative correlation learning for neural network ensembles. IEEE Transactions on Neural Networks 20, 12 (2009), 1962–1979.

Digital Library

[17]

Huanhuan Chen and Xin Yao. 2010. Multiobjective neural network ensembles based on regularized negative correlation learning. IEEE Transactions on Knowledge and Data Engineering 22, 12 (2010), 1738–1751.

Digital Library

[18]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning. PMLR, 1597–1607.

[19]

Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. 2019. Self-supervised GANs via auxiliary rotation loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12154–12163.

[20]

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020).

[21]

Yuanqi Chen, Ge Li, Cece Jin, Shan Liu, and Thomas Li. 2020. SSD-GAN: Measuring the realness in the spatial and spectral domains. arXiv preprint arXiv:2012.05535 (2020).

[22]

Zhuo Chen, Chaoyue Wang, Bo Yuan, and Dacheng Tao. 2020. Puppeteergan: Arbitrary portrait animation with semantic-aware appearance transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13518–13527.

[23]

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8789–8797.

[24]

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. 2020. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8188–8197.

[25]

Gustavo H. de Rosa and João P. Papa. 2021. A survey on text generation using generative adversarial networks. Pattern Recognition 119 (2021), 108098.

Digital Library

[26]

Ugur Demir and Gozde Unal. 2018. Patch-based image inpainting with generative adversarial networks. arXiv preprint arXiv:1803.07422 (2018).

[27]

Terrance DeVries and Graham W. Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017).

[28]

Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems 34 (2021), 8780–8794.

[29]

Carl Doersch, Abhinav Gupta, and Alexei A. Efros. 2015. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision. 1422–1430.

Digital Library

[30]

Ricard Durall, Margret Keuper, and Janis Keuper. 2020. Watch your up-convolution: CNN based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7890–7899.

[31]

Lawrence C. Evans. 1997. Partial differential equations and Monge-Kantorovich mass transfer. Current Developments in Mathematics 1997, 1 (1997), 65–126.

[32]

William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M. Dai, Shakir Mohamed, and Ian Goodfellow. 2017. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. arXiv preprint arXiv:1710.08446 (2017).

[33]

Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan O. Arik, Larry S. Davis, and Tomas Pfister. 2019. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. arXiv preprint arXiv:1910.07153 (2019).

[34]

Nan Gao, Hao Xue, Wei Shao, Sichen Zhao, Kyle Kai Qin, Arian Prabowo, Mohammad Saiedur Rahaman, and Flora D. Salim. 2022. Generative adversarial networks for spatio-temporal data: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 13, 2 (2022), 1–25.

Digital Library

[35]

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018).

[36]

Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. 2018. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv:1802.10551 (2018).

[37]

Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. 2019. AutoGAN: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3224–3234.

[38]

Abel Gonzalez-Garcia, Joost Van De Weijer, and Yoshua Bengio. 2018. Image-to-image translation for cross-domain disentanglement. In Advances in Neural Information Processing Systems. 1287–1298.

[39]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672–2680.

Digital Library

[40]

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C. Courville. 2017. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems. 5767–5777.

Digital Library

[41]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9729–9738.

[42]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. arXiv preprint arXiv:1706.08500 (2017).

[43]

R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 (2018).

[44]

R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. (2019). arxiv:stat.ML/1808.06670

[45]

Arthur E. Hoerl and Robert W. Kennard. 1970. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12, 1 (1970), 55–67.

[46]

Junyuan Hong, Yang Li, and Huanhuan Chen. 2019. Variant Grassmann manifolds: A representation augmentation method for action recognition. ACM Transactions on Knowledge Discovery from Data (TKDD) 13, 2 (2019), 1–23.

Digital Library

[47]

Zhanxuan Hu, Feiping Nie, Rong Wang, and Xuelong Li. 2020. Low rank regularization: A review. Neural Networks 136 (2021), 218–232.

[48]

Rui Huang, Wenju Xu, Teng-Yok Lee, Anoop Cherian, Ye Wang, and Tim Marks. 2020. FX-GAN: Self-supervised GAN learning via feature exchange. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 3194–3202.

[49]

Xun Huang and Serge Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision. 1501–1510.

[50]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).

[51]

Kamran Javed, Nizam Ud Din, Seho Bae, and Juneho Yi. 2019. Image unmosaicing without location information using stacked GAN. IET Computer Vision 13, 6 (2019), 588–594.

[52]

Jongheon Jeong and Jinwoo Shin. 2021. Training GANs with Stronger Augmentations via Contrastive Discriminator. (2021). arxiv:cs.LG/2103.09742

[53]

Bingbing Jiang, Chang Li, Maarten De Rijke, Xin Yao, and Huanhuan Chen. 2019. Probabilistic feature selection and classification vector machine. ACM Transactions on Knowledge Discovery from Data 13, 2 (2019), 1–27.

Digital Library

[54]

Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. 2021. Deceive D: Adaptive pseudo augmentation for GAN training with limited data. Advances in Neural Information Processing Systems 34 (2021), 21655–21667.

[55]

Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: A key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018).

[56]

Animesh Karnewar and Oliver Wang. 2020. MSG-GAN: Multi-scale gradients for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7799–7808.

[57]

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).

[58]

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. 2020. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems 33 (2020), 12104–12114.

[59]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4401–4410.

[60]

Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. 2017. On convergence and stability of GANs. arXiv preprint arXiv:1705.07215 (2017).

[61]

Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. 2019. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1920–1929.

[62]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90.

Digital Library

[63]

Anders Krogh and John A. Hertz. 1992. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems. 950–957.

Digital Library

[64]

Jan Kukačka, Vladimir Golkov, and Daniel Cremers. 2017. Regularization for deep learning: A taxonomy. arXiv preprint arXiv:1710.10686 (2017).

[65]

Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. 2018. The GAN landscape: Losses, architectures, regularization, and normalization. (2018).

[66]

Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. 2018. A large-scale study on regularization and normalization in GANs. arXiv preprint arXiv:1807.04720 (2018).

[67]

Lan Lan, Lei You, Zeyang Zhang, Zhiwei Fan, Weiling Zhao, Nianyin Zeng, Yidong Chen, and Xiaobo Zhou. 2020. Generative adversarial networks and its applications in biomedical informatics. Frontiers in Public Health 8 (2020), 164.

[68]

Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. 2019. Rethinking data augmentation: Self-supervision and self-distillation. arXiv preprint arXiv:1910.05872 (2019).

[69]

Hsin-Ying Lee, Hung-Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Singh, and Ming-Hsuan Yang. 2020. Drit++: Diverse image-to-image translation via disentangled representations. International Journal of Computer Vision 128, 10 (2020), 2402–2417.

Digital Library

[70]

Kwot Sin Lee, Ngoc-Trung Tran, and Ngai-Man Cheung. 2020. InfoMax-GAN: Improved adversarial image generation via information maximization and contrastive learning. arXiv preprint arXiv:2007.04589 (2020).

[71]

Minhyeok Lee and Junhee Seok. 2020. Regularization methods for generative adversarial networks: An overview of recent studies. arXiv preprint arXiv:2005.09165 (2020).

[72]

Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. 2017. On the limitations of first-order approximation in GAN dynamics. arXiv preprint arXiv:1706.09884 (2017).

[73]

Ziqiang Li, Rentuo Tao, Hongjing Niu, Mingdao Yue, and Bin Li. 2021. Interpreting the latent space of GANs via correlation analysis for controllable concept manipulation. In 2020 25th International Conference on Pattern Recognition (ICPR’21). 1942–1948. DOI:

[74]

Ziqiang Li, Rentuo Tao, Jie Wang, Fu Li, Hongjing Niu, Mingdao Yue, and Bin Li. 2021. Interpreting the latent space of GANs via measuring decoupling. IEEE Transactions on Artificial Intelligence 2, 1 (2021), 58–70. DOI:

[75]

Ziqiang Li, Chaoyue Wang, Heliang Zheng, Jing Zhang, and Bin Li. 2022. FakeCLR: Exploring contrastive learning for solving latent discontinuity in data-efficient GANs. arXiv preprint arXiv:2207.08630 (2022).

[76]

Ziqiang Li, Xintian Wu, Beihao Xia, Jing Zhang, Chaoyue Wang, and Bin Li. 2022. A comprehensive survey on data-efficient GANs in image generation. arXiv preprint arXiv:2204.08329 (2022).

[77]

Ziqiang Li, Pengfei Xia, Xue Rui, Yanghui Hu, and Bin Li. 2021. Are high-frequency components beneficial for training of generative adversarial networks. arXiv preprint arXiv:2103.11093 (2021).

[78]

Ziqiang Li, Pengfei Xia, Rentuo Tao, Hongjing Niu, and Bin Li. 2022. A new perspective on stabilizing GANs training: Direct adversarial training. IEEE Transactions on Emerging Topics in Computational Intelligence (2022), 1–12. DOI:

[79]

Jae Hyun Lim and Jong Chul Ye. 2017. Geometric GAN. arXiv preprint arXiv:1705.02894 (2017).

[80]

Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. 2020. Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations.

[81]

Kanglin Liu, Wenming Tang, Fei Zhou, and Guoping Qiu. 2019. Spectral regularization for combating mode collapse in GANs. In Proceedings of the IEEE International Conference on Computer Vision. 6382–6390.

[82]

Shengfei Lyu, Xing Tian, Yang Li, Bingbing Jiang, and Huanhuan Chen. 2019. Multiclass probabilistic classification vector machine. IEEE Transactions on Neural Networks and Learning Systems 31, 10 (2019), 3906–3919.

[83]

Anton Mallasto, Guido Montúfar, and Augusto Gerolin. 2019. How well do WGANs estimate the Wasserstein metric? arXiv preprint arXiv:1910.03875 (2019).

[84]

Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2794–2802.

[85]

Roy Mathias. 1990. The spectral norm of a nonnegative matrix. Linear Algebra Appl. 139 (1990), 269–284.

[86]

Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. 2018. Which training methods for GANs do actually converge? arXiv preprint arXiv:1801.04406 (2018).

[87]

Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2017. The numerics of GANs. In Advances in Neural Information Processing Systems. 1825–1835.

[88]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018).

[89]

Takeru Miyato and Masanori Koyama. 2018. cGANs with projection discriminator. arXiv preprint arXiv:1802.05637 (2018).

[90]

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 8 (2018), 1979–1993.

[91]

Youssef Mroueh, Tom Sercu, and Vaibhava Goel. 2017. McGAN: Mean and covariance feature matching GAN. arXiv preprint arXiv:1702.08398 (2017).

[92]

Vaishnavh Nagarajan and J. Zico Kolter. 2017. Gradient descent GAN optimization is locally stable. In Advances in Neural Information Processing Systems. 5585–5595.

[93]

Weili Nie and Ankit Patel. 2019. Towards a better understanding and regularization of GAN training dynamics. arXiv preprint arxiv:1806.09235 (2019).

[94]

Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. 2016. F-GAN: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems. 271–279.

[95]

Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B. Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. 2018. Is generator conditioning causally related to GAN performance? arXiv preprint arXiv:1802.08768 (2018).

[96]

Takehiko Ohkawa, Naoto Inoue, Hirokatsu Kataoka, and Nakamasa Inoue. 2020. Augmented cyclic consistency regularization for unpaired image-to-image translation. arXiv preprint arXiv:2003.00187 (2020).

[97]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).

[98]

Trevor Park and George Casella. 2008. The Bayesian Lasso. J. Amer. Statist. Assoc. 103, 482 (2008), 681–686.

[99]

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2337–2346.

[100]

Parth Patel, Nupur Kumari, Mayank Singh, and Balaji Krishnamurthy. 2021. LT-GAN: Self-supervised GAN with latent transformation detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3189–3198.

[101]

Henning Petzka, Asja Fischer, and Denis Lukovnicov. 2017. On the regularization of Wasserstein GANs. arXiv preprint arXiv:1709.08894 (2017).

[102]

Guo-Jun Qi. 2020. Loss-sensitive generative adversarial networks on Lipschitz densities. International Journal of Computer Vision 128, 5 (2020), 1118–1140.

Digital Library

[103]

Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. 2019. Mirrorgan: Learning text-to-image generation by redescription. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1505–1514.

[104]

Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andy Brock, Jeff Donahue, Timothy Lillicrap, and Pushmeet Kohli. 2020. Training generative adversarial networks by solving ordinary differential equations. Advances in Neural Information Processing Systems 33 (2020), 5599–5609.

[105]

Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2020. Encoding in style: A stylegan encoder for image-to-image translation. arXiv preprint arXiv:2008.00951 (2020).

[106]

Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. 2017. Stabilizing training of generative adversarial networks through regularization. In Advances in Neural Information Processing Systems. 2018–2028.

[107]

Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, and Kevin Murphy. 2020. XGAN: Unsupervised image-to-image translation for many-to-many mappings. In Domain Adaptation for Visual Understanding. Springer, 33–49.

[108]

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training GANs. In Advances in Neural Information Processing Systems. 2234–2242.

[109]

Tim Salimans and Durk P. Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems. 901–909.

[110]

Axel Sauer, Katja Schwarz, and Andreas Geiger. 2022. StyleGAN-XL: Scaling Stylegan to large diverse datasets. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings. 1–10.

Digital Library

[111]

Florian Schäfer and Anima Anandkumar. 2019. Competitive gradient descent. Advances in Neural Information Processing Systems 32 (2019).

[112]

Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. 2020. Interpreting the latent space of GANs for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9243–9252.

[113]

Yong-Goo Shin, Yoon-Jae Yeo, and Sung-Jea Ko. 2019. Simple yet effective way for improving the performance of GAN. arXiv preprint arXiv:1911.10979 (2019).

[114]

Nripendra Kumar Singh and Khalid Raza. 2021. Medical image generation using generative adversarial networks: A review. Health Informatics: A Computational Perspective in Healthcare. Studies in Computational Intelligence, R. Patgiri, A. Biswas, and P. Roy (Eds.). Vol. 932, Springer, Singapore.

[115]

Rodrigo G. F. Soares, Huanhuan Chen, and Xin Yao. 2012. Semisupervised classification with cluster regularization. IEEE Transactions on Neural Networks and Learning Systems 23, 11 (2012), 1779–1792.

[116]

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685 (2020).

[117]

Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. 2017. VeeGAN: Reducing mode collapse in GANs using implicit variational learning. In Advances in Neural Information Processing Systems. 3308–3318.

[118]

Jan Stanczuk, Christian Etmann, Lisa Maria Kreusser, and Carola-Bibiane Schonlieb. 2021. Wasserstein GANs work because they fail (to approximate the Wasserstein distance). arXiv preprint arXiv:2103.01678 (2021).

[119]

Jianlin Su. 2018. GAN-QP: A novel GAN framework without gradient vanishing and Lipschitz constraint. arXiv preprint arXiv:1811.07296 (2018).

[120]

Jianlin Su. 2018. Training generative adversarial networks via turing test. arXiv preprint arXiv:1810.10948 (2018).

[121]

Rentuo Tao, Ziqiang Li, Renshuai Tao, and Bin Li. 2019. ResAttr-GAN: Unpaired deep residual attributes learning for multi-domain face image translation. IEEE Access 7 (2019), 132594–132608.

[122]

Dávid Terjék. 2019. Virtual adversarial Lipschitz regularization. arXiv preprint arXiv:1907.05681 (2019).

[123]

Khoat Than and Nghia Vu. 2021. Generalization of GANs under Lipschitz continuity and data augmentation. arXiv preprint arXiv:2104.02388 (2021).

[124]

Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. 2019. Improving generalization and stability of generative adversarial networks. arXiv preprint arXiv:1902.03984 (2019).

[125]

Chunwei Tian, Xuanyu Zhang, Jerry Chun-Wen Lin, Wangmeng Zuo, and Yanning Zhang. 2022. Generative adversarial networks for image super-resolution: A survey. arXiv preprint arXiv:2204.13620 (2022).

[126]

Michael E. Tipping. 2001. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research 1, (June2001), 211–244.

Digital Library

[127]

Ngoc-Trung Tran, Viet-Hung Tran, Bao-Ngoc Nguyen, Linxiao Yang, et al. 2019. Self-supervised GAN: Analysis and improvement with multi-class minimax game. In Advances in Neural Information Processing Systems. 13253–13264.

[128]

Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Cheung. 2020. Towards good practices for data augmentation in GAN training. arXiv preprint arXiv:2006.05338 (2020).

[129]

Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. 2021. Regularizing generative adversarial networks under limited data. arXiv preprint arXiv:2104.03310 (2021).

[130]

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016).

[131]

Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using Dropconnect. In International Conference on Machine Learning. 1058–1066.

Digital Library

[132]

Chaoyue Wang, Chaohui Wang, Chang Xu, and Dacheng Tao. 2017. Tag disentangled generative adversarial networks for object image re-rendering. In International Joint Conference on Artificial Intelligence (IJCAI’17).

[133]

Chaoyue Wang, Chang Xu, Chaohui Wang, and Dacheng Tao. 2018. Perceptual adversarial networks for image-to-image transformation. IEEE Transactions on Image Processing 27, 8 (2018), 4066–4079.

[134]

Chaoyue Wang, Chang Xu, Xin Yao, and Dacheng Tao. 2019. Evolutionary generative adversarial networks. IEEE Transactions on Evolutionary Computation 23, 6 (2019), 921–934.

Digital Library

[135]

Yi Wang, Ying-Cong Chen, Xiangyu Zhang, Jian Sun, and Jiaya Jia. 2020. Attentive normalization for conditional image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5094–5103.

[136]

Yuanhao Wang, Guodong Zhang, and Jimmy Ba. 2019. On solving minimax optimization locally: A follow-the-ridge approach. arXiv preprint arXiv:1910.07512 (2019).

[137]

Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. 2022. Diffusion-GAN: Training GANs with diffusion. arXiv preprint arXiv:2206.02262 (2022).

[138]

Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. 2018. Improving the improved training of Wasserstein GANs: A consistency term and its dual effect. arXiv preprint arXiv:1803.01541 (2018).

[139]

Jie Wen, Xiaozhao Fang, Yong Xu, Chunwei Tian, and Lunke Fei. 2018. Low-rank representation with adaptive graph regularization. Neural Networks 108 (2018), 83–96.

[140]

Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan, Qiong Cao, Yibo Yang, and Xinge You. 2021. View Vertically: A hierarchical network for trajectory prediction via fourier spectrums. arXiv preprint arXiv:2110.07288 (2021).

[141]

Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. 2018. Wasserstein divergence for GANs. In Proceedings of the European Conference on Computer Vision. 653–668.

Digital Library

[142]

Yuxin Wu and Kaiming He. 2018. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV’18). 3–19.

Digital Library

[143]

Yue Wu, Pan Zhou, Andrew Gordon Wilson, Eric P. Xing, and Zhiting Hu. 2020. Improving GAN training with probability ratio clipping and sample reweighting. arXiv preprint arXiv:2006.06900 (2020).

[144]

Yi-Lun Wu, Hong-Han Shuai, Zhi-Rui Tam, and Hong-Yu Chiu. 2021. Gradient normalization for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6373–6382.

[145]

Beihao Xia, Conghao Wong, Qinmu Peng, Wei Yuan, and Xinge You. 2022. CSCNet: Contextual semantic consistency network for trajectory prediction in crowded spaces. Pattern Recognition 126 (2022), 108552.

Digital Library

[146]

Sitao Xiang and Hao Li. 2017. On the effects of batch and weight normalization in generative adversarial networks. arXiv preprint arXiv:1704.03971 (2017).

[147]

Yuanbo Xiangli, Yubin Deng, Bo Dai, Chen Change Loy, and Dahua Lin. 2020. Real or not real, that is the question. arXiv preprint arXiv:2002.05512 (2020).

[148]

Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 (2019).

[149]

Minkai Xu, Zhiming Zhou, Guansong Lu, Jian Tang, Weinan Zhang, and Yong Yu. 2021. Towards Generalized Implementation of Wasserstein Distance in GANs. (2021). arxiv:cs.LG/2012.03420

[150]

Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1316–1324.

[151]

Abhay Yadav, Sohil Shah, Zheng Xu, David Jacobs, and Tom Goldstein. 2017. Stabilizing adversarial nets with prediction methods. arXiv preprint arXiv:1705.07364 (2017).

[152]

Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. 2021. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems 34 (2021), 9378–9390.

[153]

Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. 2019. Diversity-sensitive conditional generative adversarial networks. arXiv preprint arXiv:1901.09024 (2019).

[154]

Yasin Yazici, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, and Vijay Chandrasekhar. 2020. Empirical analysis of overfitting and mode drop in GAN training. In 2020 IEEE International Conference on Image Processing (ICIP’20). IEEE, 1651–1655.

[155]

Xin Yi, Ekta Walia, and Paul Babyn. 2019. Generative adversarial network in medical imaging: A review. Medical Image Analysis 58 (2019), 101552.

[156]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5505–5514.

[157]

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference on Computer Vision. 6023–6032.

[158]

Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. 2019. S4l: Self-supervised semi-supervised learning. In Proceedings of the IEEE International Conference on Computer Vision. 1476–1485.

[159]

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017).

[160]

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2018. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318 (2018).

[161]

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. 2017. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 5907–5915.

[162]

Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. 2019. Consistency regularization for generative adversarial networks. arXiv preprint arXiv:1910.12027 (2019).

[163]

Lijun Zhang, Yujin Zhang, and Yongbin Gao. 2018. A Wasserstein GAN model with the total variational regularization. arXiv preprint arXiv:1812.00810 (2018).

[164]

Zhihong Zhang, Yangbin Zeng, Lu Bai, Yiqun Hu, Meihong Wu, Shuai Wang, and Edwin R. Hancock. 2019. Spectral bounding: Strictly satisfying the 1-Lipschitz property for generative adversarial networks. Pattern Recognition 105 (2020), 107179.

[165]

Huimin Zhao, Jianjie Zheng, Wu Deng, and Yingjie Song. 2020. Semi-supervised broad learning system based on manifold regularization and broad network. IEEE Transactions on Circuits and Systems I: Regular Papers 67, 3 (2020), 983–994.

[166]

Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I. Chang, and Yan Xu. 2021. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103.10428 (2021).

[167]

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. 2020. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems 33 (2020), 7559–7570.

[168]

Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. 2018. Bias and generalization in deep generative models: An empirical study. arXiv preprint arXiv:1811.03259 (2018).

[169]

Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. 2020. Improved consistency regularization for GANs. arXiv preprint arXiv:2002.04724 (2020).

[170]

Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. 2020. Image augmentations for GAN training. arXiv preprint arXiv:2006.02595 (2020).

[171]

Changsheng Zhou, Jiangshe Zhang, and Junmin Liu. 2018. Lp-WGAN: Using Lp-norm normalization to stabilize Wasserstein generative adversarial networks. Knowledge-based Systems 161 (2018), 415–424.

[172]

Sanping Zhou, Fei Wang, Zeyi Huang, and Jinjun Wang. 2019. Discriminative feature learning with consistent attention regularization for person re-identification. In Proceedings of the IEEE International Conference on Computer Vision. 8040–8049.

[173]

Zhiming Zhou, Jiadong Liang, Yuxuan Song, Lantao Yu, Hongwei Wang, Weinan Zhang, Yong Yu, and Zhihua Zhang. 2019. Lipschitz generative adversarial nets. arXiv preprint arXiv:1902.05687 (2019).

[174]

Zhiming Zhou, Jian Shen, Yuxuan Song, Weinan Zhang, and Yong Yu. 2019. Towards efficient and unbiased implementation of Lipschitz continuity in GANs. arXiv preprint arXiv:1904.01184 (2019).

Cited By

LI ZYAO YMA XLV J(2025)Neuro-symbolic systems: a perspective of uncertainty managementSCIENTIA SINICA Informationis10.1360/SSI-2024-016355:1(1)Online publication date: 7-Jan-2025
https://doi.org/10.1360/SSI-2024-0163
Li ZWang CRui XXue CLeng JFu ZLi B(2025)Peer Is Your Pillar: A Data-Unbalanced Conditional GANs for Few-Shot Image GenerationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.348510935:2(1303-1317)Online publication date: Feb-2025
https://doi.org/10.1109/TCSVT.2024.3485109
Wang JWang SZhang Y(2025)Deep learning on medical image analysisCAAI Transactions on Intelligence Technology10.1049/cit2.1235610:1(1-35)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1049/cit2.12356
Show More Cited By

Index Terms

A Systematic Survey of Regularization and Normalization in GANs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Machine learning algorithms
      1. Regularization
    2. Machine learning approaches
      1. Neural networks

Recommendations

Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions

Generative Adversarial Networks (GANs) is a novel class of deep generative models that has recently gained significant attention. GANs learn complex and high-dimensional distributions implicitly over images, audio, and data. However, there exist major ...
GIU-GANs: Global Information Utilization for Generative Adversarial Networks
Abstract
Recently, with the rapid development of artificial intelligence, image generation based on deep learning has advanced significantly. Image generation based on Generative Adversarial Networks (GANs) is a promising study. However, because ...
Lipschitz constrained GANs via boundedness and continuity
Abstract
One of the challenges in the study of generative adversarial networks (GANs) is the difficulty of its performance control. Lipschitz constraint is essential in guaranteeing training stability for GANs. Although heuristic methods such as weight ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 55, Issue 11

November 2023

849 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3572825

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 February 2023

Online AM: 02 November 2022

Accepted: 25 October 2022

Revised: 19 October 2022

Received: 17 June 2021

Published in CSUR Volume 55, Issue 11

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Refereed

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
1,296
Total Downloads

Downloads (Last 12 months)369
Downloads (Last 6 weeks)56

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

LI ZYAO YMA XLV J(2025)Neuro-symbolic systems: a perspective of uncertainty managementSCIENTIA SINICA Informationis10.1360/SSI-2024-016355:1(1)Online publication date: 7-Jan-2025
https://doi.org/10.1360/SSI-2024-0163
Li ZWang CRui XXue CLeng JFu ZLi B(2025)Peer Is Your Pillar: A Data-Unbalanced Conditional GANs for Few-Shot Image GenerationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.348510935:2(1303-1317)Online publication date: Feb-2025
https://doi.org/10.1109/TCSVT.2024.3485109
Wang JWang SZhang Y(2025)Deep learning on medical image analysisCAAI Transactions on Intelligence Technology10.1049/cit2.1235610:1(1-35)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1049/cit2.12356
Shao PLei C(2025)Design of Intelligent Structural Sensors for Civil Engineering Using Intelligent Digital Modeling TechnologySmart Infrastructures in the IoT Era10.1007/978-3-031-72509-8_79(943-953)Online publication date: 3-Jan-2025
https://doi.org/10.1007/978-3-031-72509-8_79
Nguyen DVullikanti ASalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Differentially private exact recovery for stochastic block modelsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693606(37798-37839)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693606
Mentzas GFikardos MLepenioti KApostolou D(2024)Exploring the landscape of trustworthy artificial intelligenceIntelligent Decision Technologies10.3233/IDT-24036618:2(837-854)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.3233/IDT-240366
Pintz MBecker DMock M(2024)PARMA: a Platform Architecture to enable Automated, Reproducible, and Multi-party Assessments of AI TrustworthinessProceedings of the 2nd International Workshop on Responsible AI Engineering10.1145/3643691.3648585(20-27)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643691.3648585
Lalor JAbbasi AOketch KYang YForsgren N(2024)Should Fairness be a Metric or a Model? A Model-based Framework for Assessing Bias in Machine Learning PipelinesACM Transactions on Information Systems10.1145/364127642:4(1-41)Online publication date: 22-Mar-2024
https://dl.acm.org/doi/10.1145/3641276
Kou T(2024)From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility GapProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658951(1002-1013)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658951
Ding HHuang NWu YCui X(2024)LEGAN: Addressing Intraclass Imbalance in GAN-Based Medical Image Augmentation for Improved Imbalanced Data ClassificationIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.339685373(1-14)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3396853
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents