More Web Proxy on the site http://driver.im/

research-article

Revisiting Performance Measures for Cross-Modal Hashing

Authors:

Mingyong LiAuthors Info & Claims

ICMR '22: Proceedings of the 2022 International Conference on Multimedia Retrieval

Pages 536 - 544

https://doi.org/10.1145/3512527.3531363

Published: 27 June 2022 Publication History

Abstract

Recently, cross-modal hashing has attracted much attention due to its low storage cost and fast query speed. Mean Average Precision (MAP) is the most widely used performance measure for cross-modal hashing. However, we found that the MAP scores do not fully reflect the quality of the top-K results for cross-modal retrieval because it neglects multi-label information and overlooks the label semantic hierarchy. In view of this, we propose a new performance measure named Normalized Weighted Discounted Cumulative Gains (NWDCG) by extending Normalized Discounted Cumulative Gains (NDCG) using co-occurrence probability matrix. To verify the effectiveness of NWDCG, we conduct extensive experiments using three popular cross-modal hashing schemes over two publically available datasets.

Supplementary Material

MP4 File (ICMR22-modfp059.mp4)

Presentation video

Download
26.89 MB

References

[1]

2009. Mean Average Precision. In Encyclopedia of Database Systems, Ling Liu and M. Tamer Ö zsu (Eds.). Springer US, 1703. https://doi.org/10.1007/978-0--387--39940--9_3032

[2]

Wenming Cao, Wenshuo Feng, Qiubin Lin, Guitao Cao, and Zhihai He. 2020. A Review of Hashing Methods for Multimodal Retrieval. IEEE Access, Vol. 8 (2020), 15377--15391. https://doi.org/10.1109/ACCESS.2020.2968154

[3]

Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009. NUS-WIDE: a real-world web image database from National University of Singapore. In Proceedings of the 8th ACM International Conference on Image and Video Retrieval, CIVR 2009, Santorini Island, Greece, July 8--10, 2009, Sté phane Marchand-Maillet and Yiannis Kompatsiaris (Eds.). ACM. https://doi.org/10.1145/1646396.1646452

Digital Library

[4]

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Networks. CoRR, Vol. abs/1406.2661 (2014). showeprint[arXiv]1406.2661 http://arxiv.org/abs/1406.2661

[5]

Wen Gu, Xiaoyan Gu, Jingzi Gu, Bo Li, Zhi Xiong, and Weiping Wang. 2019. Adversary Guided Asymmetric Hashing for Cross-Modal Retrieval. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, ICMR 2019, Ottawa, ON, Canada, June 10--13, 2019, Abdulmotaleb El-Saddik, Alberto Del Bimbo, Zhongfei Zhang, Alexander G. Hauptmann, K. Selcc uk Candan, Marco Bertini, Lexing Xie, and Xiao-Yong Wei (Eds.). ACM, 159--167. https://doi.org/10.1145/3323873.3325045

Digital Library

[6]

Di Hu, Feiping Nie, and Xuelong Li. 2019. Deep Binary Reconstruction for Cross-Modal Hashing. IEEE Trans. Multim., Vol. 21, 4 (2019), 973--985. https://doi.org/10.1109/TMM.2018.2866771

[7]

Mark J. Huiskes and Michael S. Lew. 2008. The MIR flickr retrieval evaluation. In Proceedings of the 1st ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2008, Vancouver, British Columbia, Canada, October 30--31, 2008, Michael S. Lew, Alberto Del Bimbo, and Erwin M. Bakker (Eds.). ACM, 39--43. https://doi.org/10.1145/1460096.1460104

Digital Library

[8]

Kalervo Jarvelin and Jaana Kekalainen. 2002 a. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., Vol. 20, 4 (2002), 422--446. https://doi.org/10.1145/582415.582418

Digital Library

[9]

Kalervo Jarvelin and Jaana Kekalainen. 2002 b. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., Vol. 20, 4 (2002), 422--446. https://doi.org/10.1145/582415.582418

Digital Library

[10]

Qing-Yuan Jiang and Wu-Jun Li. 2017. Deep Cross-Modal Hashing. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21--26, 2017. IEEE Computer Society, 3270--3278. https://doi.org/10.1109/CVPR.2017.348

[11]

Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, and Dacheng Tao. 2018. Self-Supervised Adversarial Hashing Networks for Cross-Modal Retrieval. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18--22, 2018. Computer Vision Foundation / IEEE Computer Society, 4242--4251. https://doi.org/10.1109/CVPR.2018.00446

[12]

Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang. 2015. Semantics-preserving hashing for cross-view retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7--12, 2015. IEEE Computer Society, 3864--3872. https://doi.org/10.1109/CVPR.2015.7299011

[13]

Song Liu, Shengsheng Qian, Yang Guan, Jiawei Zhan, and Long Ying. 2020. Joint-modal Distribution-based Similarity Hashing for Large-scale Unsupervised Deep Cross-modal Retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25--30, 2020, Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 1379--1388. https://doi.org/10.1145/3397271.3401086

Digital Library

[14]

Wei Liu, Cun Mu, Sanjiv Kumar, and Shih-Fu Chang. 2014. Discrete Graph Hashing. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8--13 2014, Montreal, Quebec, Canada, Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (Eds.). 3419--3427. https://proceedings.neurips.cc/paper/2014/hash/f63f65b503e22cb970527f23c9ad7db1-Abstract.html

[15]

Claudia Roda and Julie Thomas. 2006. Attention aware systems: Theories, applications, and research agenda. Comput. Hum. Behav., Vol. 22, 4 (2006), 557--587. https://doi.org/10.1016/j.chb.2005.12.005

[16]

Shupeng Su, Zhisheng Zhong, and Chao Zhang. 2019. Deep Joint-Semantics Reconstructing Hashing for Large-Scale Unsupervised Cross-Modal Retrieval. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 3027--3035. https://doi.org/10.1109/ICCV.2019.00312

[17]

Changchang Sun, Xuemeng Song, Fuli Feng, Wayne Xin Zhao, Hao Zhang, and Liqiang Nie. 2019. Supervised Hierarchical Cross-Modal Hashing. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21--25, 2019, Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 725--734. https://doi.org/10.1145/3331184.3331229

Digital Library

[18]

Di Wang, Xinbo Gao, Xiumei Wang, and Lihuo He. 2015. Semantic Topic Multimodal Hashing for Cross-Media Retrieval. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25--31, 2015, Qiang Yang and Michael J. Wooldridge (Eds.). AAAI Press, 3890--3896. http://ijcai.org/Abstract/15/546

[19]

Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013. A Theoretical Analysis of NDCG Type Ranking Measures. In Proceedings of the 26th Annual Conference on Learning Theory (Proceedings of Machine Learning Research, Vol. 30), Shai Shalev-Shwartz and Ingo Steinwart (Eds.). PMLR, Princeton, NJ, USA, 25--54. https://proceedings.mlr.press/v30/Wang13.html

[20]

Gengshen Wu, Zijia Lin, Jungong Han, Li Liu, Guiguang Ding, Baochang Zhang, and Jialie Shen. 2018. Unsupervised Deep Hashing via Binary Latent Factor Models for Large-scale Cross-modal Retrieval. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13--19, 2018, Stockholm, Sweden, Jérôme Lang (Ed.). ijcai.org, 2854--2860. https://doi.org/10.24963/ijcai.2018/396

[21]

Yanzhao Xie, Yu Liu, Yangtao Wang, Lianli Gao, Peng Wang, and Ke Zhou. 2020. Label-Attended Hashing for Multi-Label Image Retrieval. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, Christian Bessiere (Ed.). ijcai.org, 955--962. https://doi.org/10.24963/ijcai.2020/133

Cited By

Huang JKang PHan NChen YFang XGao HZhou G(2023)Two-Stage Asymmetric Similarity Preserving Hashing for Cross-Modal RetrievalIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.328398436:1(429-444)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3283984

Index Terms

Revisiting Performance Measures for Cross-Modal Hashing
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Retrieval effectiveness

Recommendations

Unsupervised Deep Cross-Modal Hashing by Knowledge Distillation for Large-scale Cross-modal Retrieval
ICMR '21: Proceedings of the 2021 International Conference on Multimedia Retrieval

Cross-modal hashing (CMH) maps heterogeneous multiple modality data into compact binary code to achieve fast and flexible retrieval across different modalities, especially in large-scale retrieval. As the data don't need a lot of manual annotation, ...
Semi-supervised cross-modal hashing via modality-specific and cross-modal graph convolutional networks
Highlights
- MCGCN for the first time builds cross-modal graph and jointly learns modality-specific and modality-shared features for semi-supervised cross-modal hashing.
- MCGCN provides a three-channel network architecture, including two modality-...
Abstract
Cross-modal hashing maps heterogeneous multimedia data into Hamming space for retrieving relevant samples across modalities, which has received great research interests due to its rapid retrieval and low storage cost. In real-world applications, ...
Discrete Similarity Preserving Hashing for Cross-modal Retrieval
Artificial Intelligence and Security
Abstract
Hashing methods have attracted great attention for cross-modal retrieval due to the low memory requirement and fast computation. Cross-modal hashing methods aim to transform the data from different modalities into a common Hamming space. However, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '22: Proceedings of the 2022 International Conference on Multimedia Retrieval

June 2022

714 pages

ISBN:9781450392389

DOI:10.1145/3512527

General Chairs:
Vincent Oria
New Jersey Institute of Technology, USA
,
Maria Luisa Sapino
Università degli Studi di Torino, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Brigitte Kerhervé
Université du Québec à Montréal, Canada
,
Program Chairs:
Wen-Huang Cheng
National Yang Ming Chao Tung University, Taiwan
,
Ichiro Ide
Nagoya University, Japan
,
Vivek Singh
Rutgers University, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

State Key Laboratory of Computer Architecture (ICTCAS)
the Fundamental Research Funds for the Central Universities
NSF of Shanghai

Conference

ICMR '22

Sponsor:

SIGMM

ICMR '22: International Conference on Multimedia Retrieval

June 27 - 30, 2022

NJ, Newark, USA

Acceptance Rates

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
154
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang JKang PHan NChen YFang XGao HZhou G(2023)Two-Stage Asymmetric Similarity Preserving Hashing for Cross-Modal RetrievalIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.328398436:1(429-444)Online publication date: 8-Jun-2023
https://dl.acm.org/doi/10.1109/TKDE.2023.3283984

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents