More Web Proxy on the site http://driver.im/

research-article

SRI-Net: : Similarity retrieval-based inference network for light field salient object detection

Authors:

Chenggang YanAuthors Info & Claims

Volume 90, Issue C

https://doi.org/10.1016/j.jvcir.2022.103721

Published: 01 February 2023 Publication History

Abstract

The cutting-edge RGB saliency models are prone to fail for some complex scenes, while RGB-D saliency models are often affected by inaccurate depth maps. Fortunately, light field images can provide a sufficient spatial layout depiction of 3D scenes. Therefore, this paper focuses on salient object detection of light field images, where a Similarity Retrieval-based Inference Network (SRI-Net) is proposed. Due to various focus points, not all focal slices extracted from light field images are beneficial for salient object detection, thus, the key point of our model lies in that we attempt to select the most valuable focal slice, which can contribute more complementary information for the RGB image. Specifically, firstly, we design a focal slice retrieval module (FSRM) to choose an appropriate focal slice by measuring the foreground similarity between the focal slice and RGB image. Secondly, in order to combine the original RGB image and the selected focal slice, we design a U-shaped saliency inference module (SIM), where the two-stream encoder is used to extract multi-level features, and the decoder is employed to aggregate multi-level deep features. Extensive experiments are conducted on two widely used light field datasets, and the results firmly demonstrate the superiority and effectiveness of the proposed SRI-Net.

Highlights

•

We deploy the FSRM to explicitly dig complementary information from all focal slices.

•

The similarity computation is utilized to retrieve the most valuable focal slice.

•

To aggregate the retrieved focal slice and the RGB image, we deploy the two-stream SIM to extract and fuse the deep features of them.

References

[1]

Yang X., Li S., Ma J., Liu H., Yan J., Multi-frame co-saliency spatio-temporal regularization correlation filters for object tracking, J. Vis. Commun. Image Represent. 81 (2021).

[2]

Bai C., Chen J.-n., Huang L., Kpalma K., Chen S., Saliency-based multi-feature modeling for semantic image retrieval, J. Vis. Commun. Image Represent. 50 (2018) 199–204.

[3]

Hu Z., Liu Z., Li G., Ye L., Zhou L., Wang Y., Weakly supervised instance segmentation using multi-stage erasing refinement and saliency-guided proposals ordering, J. Vis. Commun. Image Represent. 73 (2020).

[4]

Fang Y., Chen Z., Lin W., Lin C.-W., Saliency detection in the compressed domain for adaptive image retargeting, IEEE Trans. Image Process. 21 (9) (2012) 3888–3901.

[5]

Fang Y., Lin W., Chen Z., Tsai C.-M., Lin C.-W., A video saliency detection model in compressed domain, IEEE Trans. Circuits Syst. Video Technol. 24 (1) (2013) 27–38.

Digital Library

[6]

Itti L., Koch C., Niebur E., A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach. Intell. 20 (11) (1998) 1254–1259.

Digital Library

[7]

Zhu W., Liang S., Wei Y., Sun J., Saliency optimization from robust background detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2814–2821.

[8]

Jiang H., Wang J., Yuan Z., Wu Y., Zheng N., Li S., Salient object detection: A discriminative regional feature integration approach, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 2083–2090.

[9]

Tong N., Lu H., Ruan X., Yang M.-H., Salient object detection via bootstrap learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1884–1892.

[10]

LeCun Y., Bottou L., Bengio Y., Haffner P., Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278–2324.

[11]

Long J., Shelhamer E., Darrell T., Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.

[12]

Qin X., Zhang Z., Huang C., Gao C., Dehghan M., Jagersand M., Basnet: Boundary-aware salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 7479–7489.

[13]

Fan D.-P., Zhai Y., Borji A., Yang J., Shao L., BBS-Net: RGB-D salient object detection with a bifurcated backbone strategy network, in: Proceedings of the European Conference on Computer Vision, Springer, 2020, pp. 275–292.

[14]

Zhou X., Fang H., Liu Z., Zheng B., Sun Y., Zhang J., Yan C., Dense attention-guided cascaded network for salient object detection of strip steel surface defects, IEEE Trans. Instrum. Meas. (2021).

[15]

Shokri M., Harati A., Taba K., Salient object detection in video using deep non-local neural networks, J. Vis. Commun. Image Represent. 68 (2020).

[16]

Zhang M., Li J., Wei J., Piao Y., Lu H., Memory-oriented decoder for light field salient object detection, Adv. Neural Inf. Process. Syst. 32 (2019).

[17]

Piao Y., Rong Z., Zhang M., Lu H., Exploit and replace: An asymmetrical two-stream architecture for versatile light field saliency detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 11865–11873.

[18]

Zhang Y., Chen G., Chen Q., Sun Y., Deforges O., Zhang L., Learning synergistic attention for light field salient object detection, in: Proceedings of the British Machine Vision Conference, 2021, pp. 1–14.

[19]

Chen Q., Liu Z., Zhang Y., Fu K., Zhao Q., Du H., RGB-D salient object detection via 3D convolutional neural networks, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2021, pp. 1063–1071.

[20]

Borji A., Cheng M.-M., Hou Q., Jiang H., Li J., Salient object detection: A survey, Comput. Vis. Media 5 (2) (2019) 117–150.

[21]

Zhou T., Fan D.-P., Cheng M.-M., Shen J., Shao L., RGB-D salient object detection: A survey, Comput. Vis. Media (2021) 1–33.

[22]

Fu K., Jiang Y., Ji G.-P., Zhou T., Zhao Q., Fan D.-P., Light field salient object detection: A review and benchmark, Comput. Vis. Media (2022) 1–26.

[23]

Achanta R., Hemami S., Estrada F., Susstrunk S., Frequency-tuned salient region detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 1597–1604.

[24]

Liu T., Yuan Z., Sun J., Wang J., Zheng N., Tang X., Shum H.-Y., Learning to detect a salient object, IEEE Trans. Pattern Anal. Mach. Intell. 33 (2) (2010) 353–367.

Digital Library

[25]

Zhu D., Luo Y., Dai L., Shao X., Zhou Q., Itti L., Lu J., Salient object detection via a local and global method based on deep residual network, J. Vis. Commun. Image Represent. 54 (2018) 1–9.

[26]

Wu Z., Su L., Huang Q., Cascaded partial decoder for fast and accurate salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3907–3916.

[27]

Zhou X., Shen K., Liu Z., Gong C., Zhang J., Yan C., Edge-aware multiscale feature integration network for salient object detection in optical remote sensing images, IEEE Trans. Geosci. Remote Sens. (2021).

[28]

Zhao J.-X., Liu J.-J., Fan D.-P., Cao Y., Yang J., Cheng M.-M., EGNet: Edge guidance network for salient object detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 8779–8788.

[29]

Chen S., Tan X., Wang B., Lu H., Hu X., Fu Y., Reverse attention-based residual network for salient object detection, IEEE Trans. Image Process. 29 (2020) 3763–3776.

Digital Library

[30]

Desingh K., Krishna K.M., Rajan D., Jawahar C., Depth really matters: Improving Visual Salient Region detection with depth, in: Proceedings of the British Machine Vision Conference, 2013, pp. 1–11.

[31]

Zhu C., Li G., Wang W., Wang R., An innovative salient object detection using center-dark channel prior, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1509–1515.

[32]

Liu N., Zhang N., Han J., Learning selective self-mutual attention for RGB-D saliency detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 13756–13765.

[33]

Fan D.-P., Lin Z., Zhang Z., Zhu M., Cheng M.-M., Rethinking RGB-D salient object detection: Models, data sets, and large-scale benchmarks, IEEE Trans. Neural Netw. Learn. Syst. 32 (5) (2020) 2075–2089.

[34]

Pang Y., Zhang L., Zhao X., Lu H., Hierarchical dynamic filtering network for rgb-d salient object detection, in: Proceedings of the European Conference on Computer Vision, Springer, 2020, pp. 235–252.

[35]

Zhang J., Fan D.-P., Dai Y., Anwar S., Saleh F.S., Zhang T., Barnes N., Uc-net: Uncertainty inspired rgb-d saliency detection via conditional variational autoencoders, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 8582–8591.

[36]

Fu K., Fan D.-P., Ji G.-P., Zhao Q., JL-DCF: Joint learning and densely-cooperative fusion framework for RGB-D salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 3052–3062.

[37]

Li C., Cong R., Piao Y., Xu Q., Loy C.C., RGB-D salient object detection with cross-modality modulation and selection, in: Proceedings of the European Conference on Computer Vision, Springer, 2020, pp. 225–241.

[38]

Zhang M., Fei S.X., Liu J., Xu S., Piao Y., Lu H., Asymmetric two-stream architecture for accurate RGB-D saliency detection, in: Proceedings of the European Conference on Computer Vision, Springer, 2020, pp. 374–390.

[39]

Zhang M., Ren W., Piao Y., Rong Z., Lu H., Select, supplement and focus for RGB-D saliency detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 3472–3481.

[40]

Liu D., An P., Ma R., Zhan W., Huang X., Yahya A.A., Content-based light field image compression method with Gaussian process regression, IEEE Trans. Multimed. 22 (4) (2019) 846–859.

Digital Library

[41]

Liu D., Huang Y., Wu Q., Ma R., An P., Multi-angular epipolar geometry based light field angular reconstruction network, IEEE Trans. Comput. Imaging 6 (2020) 1507–1522.

[42]

Li N., Ye J., Ji Y., Ling H., Yu J., Saliency detection on light field, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2806–2813.

Digital Library

[43]

Zhang J., Wang M., Gao J., Wang Y., Zhang X., Wu X., Saliency detection with a deeper investigation of light field, in: Proceedings of the International Joint Conference on Artificial Intelligence, 2015, pp. 2212–2218.

[44]

Li N., Sun B., Yu J., A weighted sparse coding framework for saliency detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 5216–5223.

[45]

Zhang J., Liu Y., Zhang S., Poppe R., Wang M., Light field saliency detection with deep convolutional networks, IEEE Trans. Image Process. 29 (2020) 4421–4434.

[46]

Xingjian S., Chen Z., Wang H., Yeung D.-Y., Wong W.-K., Woo W.-c., Convolutional LSTM network: A machine learning approach for precipitation nowcasting, in: Advances in Neural Information Processing Systems, 2015, pp. 802–810.

[47]

Zhang M., Ji W., Piao Y., Li J., Zhang Y., Xu S., Lu H., LFNet: Light field fusion network for salient object detection, IEEE Trans. Image Process. 29 (2020) 6276–6287.

[48]

Jiang Y., Zhang W., Fu K., Zhao Q., MEANet: Multi-modal edge-aware network for light field salient object detection, Neurocomputing 491 (2022) 78–90.

[49]

Liu S., Huang D., et al., Receptive field block net for accurate and fast object detection, in: Proceedings of the European Conference on Computer Vision, 2018, pp. 385–400.

Digital Library

[50]

He K., Zhang X., Ren S., Sun J., Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

[51]

Hu J., Shen L., Sun G., Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.

Digital Library

[52]

Hou Q., Cheng M.-M., Hu X., Borji A., Tu Z., Torr P.H., Deeply supervised salient object detection with short connections, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3203–3212.

[53]

De Boer P.-T., Kroese D.P., Mannor S., Rubinstein R.Y., A tutorial on the cross-entropy method, Ann. Oper. Res. 134 (1) (2005) 19–67.

[54]

Wang T., Piao Y., Li X., Zhang L., Lu H., Deep learning for light field saliency detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 8838–8848.

[55]

Zhang J., Wang M., Lin L., Yang X., Gao J., Rui Y., Saliency detection on light field: A multi-cue approach, ACM Trans. Multimed. Comput. Commun. Appl. 13 (3) (2017) 1–22.

[56]

Paszke A., Gross S., Massa F., Lerer A., Bradbury J., Chanan G., Killeen T., Lin Z., Gimelshein N., Antiga L., et al., Pytorch: An imperative style, high-performance deep learning library, Adv. Neural Inf. Process. Syst. 32 (2019) 8026–8037.

[57]

Graves A., Generating sequences with recurrent neural networks, 2013, arXiv preprint arXiv:1308.0850.

[58]

Fan D.-P., Gong C., Cao Y., Ren B., Cheng M.-M., Borji A., Enhanced-alignment measure for binary foreground map evaluation, in: Proceedings of the International Joint Conference on Artificial Intelligence, 2018, pp. 698–704.

Digital Library

[59]

Fan D.-P., Cheng M.-M., Liu Y., Li T., Borji A., Structure-measure: A new way to evaluate foreground maps, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 4548–4557.

[60]

Simonyan K., Zisserman A., Very deep convolutional networks for large-scale image recognition, 2014, arXiv preprint arXiv:1409.1556.

Index Terms

SRI-Net: Similarity retrieval-based inference network for light field salient object detection
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Machine learning
2. Information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

A Two-Stage Bayesian Integration Framework for Salient Object Detection on Light Field

Unique visual features of 4D light field data have been shown to affect detection of salient objects. Nevertheless, only a few studies explore it yet. In this study, several helpful visual features extracted from light field data are fused in a two-stage ...
LFBCNet: Light Field Boundary-aware and Cascaded Interaction Network for Salient Object Detection
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

In light field imaging techniques, the abundance of stereo spatial information aids in improving the performance of salient object detection. In some complex scenes, however, applying the 4D light field boundary structure to discriminate salient objects ...
Focal Perception Transformer for Light Field Salient Object Detection
Pattern Recognition and Computer Vision
Abstract
Recently light field saliency object detection (LFSOD) has attracted increasing attention due to the significant improvements in challenging scenes using rich light field cues. While many works have significantly progressed in this field, a deeper ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Visual Communication and Image Representation

Journal of Visual Communication and Image Representation Volume 90, Issue C

Feb 2023

604 pages

ISSN:1047-3203

Issue’s Table of Contents

Elsevier Inc.

Publisher

Academic Press, Inc.

United States

Publication History

Published: 01 February 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents