More Web Proxy on the site http://driver.im/

research-article

MGAM: Mixed-Gaussian Attention Module for Convolutional Neural Networks

Authors:

Yongsheng LiangAuthors Info & Claims

AIMLR '23: Proceedings of the 2023 Asia Conference on Artificial Intelligence, Machine Learning and Robotics

Article No.: 13, Pages 1 - 5

https://doi.org/10.1145/3625343.3625358

Published: 11 November 2023 Publication History

Abstract

Abstract—Spatial attention holds great potential for enhancing the performance of convolutional neural networks (CNNs). This technique aims to select important spatial regions by generating attention masks across spatial domains. However, the majority of existing spatial attention works rely on convolutional blocks to compute attention weights, disregarding the modeling of spatial key positions. To address this limitation, this paper proposes a Mixed-Gaussian Attention Module (MGAM). The MGAM generates spatial attention masks that follow mixed Gaussian distribution, effectively highlighting the most discriminative semantic features and minimizing the influence of background noise while maintaining low model complexity. Our method based on ResNet50 achieves a 1.34% improvement on the CIFAR100 benchmark and a 1% improvement on the ImageNet-200 benchmark.

References

[1]

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.

Digital Library

[2]

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.

[3]

S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, vol. 28, 2015.

Digital Library

[4]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

[5]

K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask´ r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.

[6]

J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132– 7141.

[7]

F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3156–3164.

[8]

J. Park, S. Woo, J.-Y. Lee, and I. S. Kweon, “Bam: Bottleneck attention module,” arXiv preprint arXiv:1807.06514, 2018.

[9]

L. Yang, R.-Y. Zhang, L. Li, and X. Xie, “Simam: A simple, parameter-free attention module for convolutional neural networks,” in International conference on machine learning. PMLR, 2021, pp. 11863–11874.

[10]

J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi, “Gather-excite: Exploiting feature context in convolutional neural networks,” Advances in neural information processing systems, vol. 31, 2018.

[11]

S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.

Digital Library

[12]

A. Krizhevsky, G. Hinton, “Learning multiple layers of features from tiny images,” 2009.

[13]

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015.

Digital Library

[14]

L. Itti, C. Koch, and E. Niebur, “A model of saliencybased visual attention for rapid scene analysis,” IEEE Transactions on pattern analysis and machine intelligence, vol. 20, no. 11, pp. 1254–1259, 1998.

Digital Library

[15]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510– 4520.

[16]

Z. Qin, P. Zhang, F. Wu, and X. Li, “Fcanet: Frequency channel attention networks,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 783–792.

[17]

Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu, “Eca-net: Efficient channel attention for deep convolutional neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11534–11542.

[18]

Y. Shi, L. Yang, W. An, X. Zhen, and L. Wang, “Parameter-free channel attention for image classification and super-resolution,” arXiv preprint arXiv:2303.11055, 2023.

[19]

C. Niu, J. Zhang, G. Wang, and J. Liang, “Gatcluster: Self-supervised gaussian-attention network for image clustering,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16. Springer, 2020, pp. 735–751.

[20]

“pytorch,” http://pytorch.org/. Accessed July 25, 2022.

[21]

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 618–626.

Cited By

Xiao CLi YLiang YMao Q(2024)SRAM: Spectral Residual Attention Module for Enhancing Shallow Features2024 9th International Conference on Intelligent Computing and Signal Processing (ICSP)10.1109/ICSP62122.2024.10743975(1011-1017)Online publication date: 19-Apr-2024
https://doi.org/10.1109/ICSP62122.2024.10743975

Index Terms

MGAM: Mixed-Gaussian Attention Module for Convolutional Neural Networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

An efficient mixed attention module
Abstract
Recently, the application of attention mechanisms in convolutional neural networks (CNNs) has become a hot area in computer vision. Most existing methods focus on channel attention or spatial attention. Some mixed attention usually achieves better ...

For recent attention methods to increase performance by increasing complexity, we provide an efficient mixed attention that aggregates channel information and spatial information through a learnable combinatorial formulation. In this way, the modelling ...
MSANet: Multi-scale attention networks for image classification
Abstract
The classification of images based on the principles of human vision is a major task in the field of computer vision. It is a common method to use multi-scale information and attention mechanism to obtain better classification performance. The ...
An Attention Module for Convolutional Neural Networks
Artificial Neural Networks and Machine Learning – ICANN 2021
Abstract
Attention mechanism has been regarded as an advanced technique to capture long-range feature interactions and to boost the representation capability for convolutional neural networks. However, we found two ignored problems in current attentional ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

AIMLR '23: Proceedings of the 2023 Asia Conference on Artificial Intelligence, Machine Learning and Robotics

September 2023

133 pages

ISBN:9798400708312

DOI:10.1145/3625343

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

AIMLR 2023

AIMLR 2023: 2023 Asia Conference on Artificial Intelligence, Machine Learning and Robotics

September 15 - 17, 2023

Bangkok, Thailand

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
44
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xiao CLi YLiang YMao Q(2024)SRAM: Spectral Residual Attention Module for Enhancing Shallow Features2024 9th International Conference on Intelligent Computing and Signal Processing (ICSP)10.1109/ICSP62122.2024.10743975(1011-1017)Online publication date: 19-Apr-2024
https://doi.org/10.1109/ICSP62122.2024.10743975

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents