More Web Proxy on the site http://driver.im/

Article

Visual & textual fusion for region retrieval: from both fuzzy matching and bayesian reasoning aspects

Authors:

Hongxun YaoAuthors Info & Claims

MIR '07: Proceedings of the international workshop on Workshop on multimedia information retrieval

Pages 159 - 168

https://doi.org/10.1145/1290082.1290106

Published: 24 September 2007 Publication History

Abstract

This paper presents a novel visual & textual information fusion framework for region-based image retrieval. We explore the issue of linguistic-integrated region retrieval from both Bayesian Reasoning and Fuzzy Region Matching aspects. Firstly, to associate textual information with image regions, we present a region-based soft annotation strategy. Our method automatically labels each image region with multiple keywords, each of which is assigned a confidence factor to indicate its annotation accuracy. In annotation classifier training, we adopt a pairwise coupling (PWC) SVM bagging network to address the problems of sample insufficiency and sample asymmetry. Consequently, in image retrieval, we fuse regions. visual & textual information to rank image similarities at perceptual level. Two fusion schemes are explored in proposed framework: 1. Semantic-Supervised Integrated Region Matching (SSIRM); 2. Keyword-Integrated Bayesian Reasoning (KIBR). SSIRM is a keyword-integrated fuzzy region matching strategy, which is adopted in the case that the query image is pre-annotated; KIBR is adopted in the case that the query image is non-annotated or poorly-annotated, which supports both query-by-example and query-by-keyword based on statistical text-image translation model. Finally, in relevance feedback (RF) learning, we exploit a unified visual & textual learning algorithm to precisely capture users' retrieval intention. Superior annotation, retrieval (over IRM) and RF performances (Both over IRM + SVM at region-level and SVM & ALSVM & ABSVM at global-level) are presented in our experiments, which demonstrate the efficiency of proposed fusion framework to bridge the semantic gap.

References

[1]

A. W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R.Jain, " Content-Based Image Retrieval at the End of the Early Years", IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.22, No.12, Dec. 2000, pp.1349--1380.

Digital Library

[2]

R. C. Veltkamp, M. Tanase, "Content-Based Image Retrieval Systems: A Survey", Technical report UU-CS-2000-34, Dep. of Computing Science, Utrecht University, Oct. 2000. 34.

[3]

Y. Gong, H. J. Zhang and T. C. Chua, "An image database system with content capturing and fast image indexing abilities," Proc. International Conference on Multimedia Computing and Systems, Boston, 14-19 May 1994, pp.121--130.

[4]

Z. Su, H.-J. Zhang, and S. Li, "Relevance Feedback in Content-Based Image Retrieval: Bayesian Framework, Feature Subspaces and Progressive Learning," IEEE Trans. on Image Processing, vol.12, no.3, 2003.8.

Digital Library

[5]

D. Tao, X. Tang, et. al. "Asymmetric Bagging and Random Subspace for Support Vector Machines-Based Relevance Feedback in Image Retrieval," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.28, no.7, July, 2006.

Digital Library

[6]

J. Li, J. Z. Wang, "Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.25, no.9, pp. 1075--1088, Sep. 2003.

Digital Library

[7]

P. Duygulu, K. Barnard, N. de Freitas, and D. Forsyth, "Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary," European Conference of Computer Vision, vol. 4, pp. 97--112, 2002.

Digital Library

[8]

Ye Lu, Hongjiang Zhang, Liu Wenyin et. al., "Joint Semantic and Feature Based Image Retrieval Using Relevance Feedback," IEEE Trans. on Multimedia, vol.5, no.3, pp.339--347, Sep. 2003.

Digital Library

[9]

J. He, M. Li, H.-J. Zhang, Hanghang Tong, Changshui Zhang, "Mean Version Space: A New Active Learning Method for Content-Based Image Retrieval," ACM SIG MM Workshop on Multimedia Information Retrieval, pp.15--23, 2004.

Digital Library

[10]

S. Tong, and E. Chang, "Support Vector Machine Active Learning for Image Retrieval," ACM SIG MM, Pp.107--118, 2001.

Digital Library

[11]

R.M. Haralick, K. Shanmugam, and I. Dinstein, "Texture features for image classification," IEEE Trans. on System, Man, Cybernetics, Vol. 3, pp. 610--621, November 1973.

[12]

K.-S. Goh, E. Y. Chang, B. Li, "Using One-Class and Two-Class SVMs for Multiclass Image Annotation," IEEE Trans. on Knowledge and Data Engineering, vol. 17, no. 10, pp. 1333--1346, Oct., 2005.

Digital Library

[13]

Md. Mahmudur Rahman, P. Bhattacharya, B. C. Desai, "A Framework for Medical Image Retrieval using Machine Learning & Statistical Similarity Matching Techniques with Relevance Feedback," IEEE Trans. on Information Technology in Biomedicine, Vol.11, Issue 1, pp.58--69, Jan. 2007.

Digital Library

[14]

T. Wu, C. J. Lin, and R. C. Weng, "Probability Estimates for Multi-Class Classification by Pairwise Coupling," International Journal on Machine Learning Research, vol.10 (5), pp.975--1005, 2004.

Digital Library

[15]

E. Chang, K. Goh et. al., "CBSA: Content-Based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machines," IEEE Trans. on Circuits and Systems for Video Technology, Vol.13, Issue 1, pp.26--38, Jan.2003.

Digital Library

[16]

F. Jing, M. Li et. al., "A Unified Framework for Image Retrieval Using Keyword and Visual Features," IEEE Trans. on Image Processing, Vol.14, No.7, pp.979--989, July 2000.

Digital Library

[17]

Y. Chen, and J.Z. Wang, "A Region-Based Fuzzy Feature Matching Approach to Content-Based Image Retrieval," IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.24, No.9, Sep. 2002

Digital Library

[18]

R. Y, T. Huang, Mehrotra S et. al., "Relevance Feedback: A Power Tool for Interactive Content-based Image Retrieval," IEEE Trans. Circuits and Systems for Video Technology, 1998. 8(5): 644--655

Digital Library

[19]

S. Liapis, G. Tziritas, "Color and Textual Image Retrieval Using Chromaticity Histograms and Wavelet Frames," IEEE Transactions on Multimedia, Vol.6, No.5, October, 2004, pp.676--686.

Digital Library

[20]

R. Brunelli and O. Mich, "Image Retrieval by Example," IEEE Trans. on Multimedia, Vol.2, No.3, September, 2000, pp.164--171.

Digital Library

[21]

G. Aggarwal, A. T. V., and S. Ghosal, "An Image Retrieval System With Automatic Query Modification," IEEE Trans. on Multimedia, Vol.4, No.2, June 2002, pp. 201--214.

Digital Library

[22]

K.-M. Lee, and W. N. Street, "Cluster-Driven Refinement for Content-Based Digital Image Retrieval," IEEE Trans. on Multimedia, Vol.6, No.6, December 2004, pp.817--827.

Digital Library

[23]

B.C. Ko, H. Byun, "FRIP: A Region-Based Image Retrieval Tool Using Automatic Image Segmentation and Stepwise Boolean AND Matching," IEEE Trans. on Multimedia, Vol.7, No.1, February 2005, pp.105--113.

Digital Library

[24]

D. Tao, X. Tang, X. Li, and Y. Rui, "Direct Kernel Biased Discriminant Analysis: A New Content-Based Image Retrieval Relevance Feedback Algorithm," IEEE Trans. on Multimedia, Vol.8, No. 4, August 2004, pp.716--727.

Digital Library

[25]

Y. Lu, H.-J. Zhang, L. Wenyin, and C. Hu, "Joint Semantic and Feature Based Image Retrieval Using Relevance Feedback," IEEE Trans. on Multimedia, Vol.5, No.3, September 2003, pp.339--347.

Digital Library

[26]

A. Del Bimbo, E. Vicario, "Weighted Walkthroughs between Extended Entities for Retrieval by Spatial Arrangement," IEEE Trans. on Multimedia, Vol.5, No.1, pp.52--70, 2003.

Digital Library

[27]

T. Wang, Y. Rui, J.-G. Sun, "Constraint based Region Matching for Image Retrieval", International Journal of Computer Vision, 56(1/2/3), pp.37--45, 2004.

Digital Library

[28]

J. Aamores, N. Sebe, P. Radeva, T. Gevers, A. Smeulders, "Boosting Contextual Information in Content-Based Image Retrieval," ACM SIG MM Workshop on Multimedia Information Retrieval, pp.31--39, 2004.

Digital Library

[29]

I. J. Cox, M. L. Miller, T. P. Minka, T. V. Papathomas, and P. N. Yianilos, "The Bayesian Image Retrieval System, PicHunter: Theory, Implementation, and Psychophysical Experiments," IEEE Trans. on Image Processing, Vol.9, No.1, pp.20--37, 2000.

Digital Library

[30]

R. Zhang and Z. Zhang, "Hidden Semantic Concept Discovery in Region Based Image Retrieval, Computer Vision and Pattern Recognition, Washington, DC, USA, Vol.2, pp.996--1001,June 2004.

Digital Library

[31]

X. S. Zhou, T. S. Huang, "Relevance Feedback in Image Retrieval: A Comprehensive Review," International Journal of Multimedia Systems, Vol.8, pp 536--544, 2003.

[32]

J. Huang, S. R. Kumar, M. Mitra, W. J. Zhu, and R. Zabin, "Image Indexing using Color Correlogram," Computer Vision and Pattern Recognition, January, 1997, pp.762--768.

Digital Library

[33]

S. Tong and E. Chang, "Support vector machine active learning for image retrieval," ACM SIG Multimedia, pp.107--118, 2001.

Digital Library

[34]

H. Xie and A. Ortega, "An User Preference Information Based Kernel for SVM Active Learning in Content-based Image Retrieval," ACM SIG MM Workshop on Multimedia Information Retrieval, pp.1--6, 2005.

Digital Library

Cited By

Zhiqiang ZHua SYun WZou Q(2016)Annotation-retrieval reinforcement by visual cognition modeling on manifoldNeurocomputing10.1016/j.neucom.2015.07.162215:C(150-159)Online publication date: 26-Nov-2016
https://dl.acm.org/doi/10.1016/j.neucom.2015.07.162
Liu XYao HJi RXu PSun X(2013)Bidirectional-isomorphic manifold learning at image semantic understanding & representationMultimedia Tools and Applications10.1007/s11042-011-0947-264:1(53-76)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.1007/s11042-011-0947-2
Cao LJi RLiu WGao YDuan LMen C(2012)Weakly supervised topic grouping of YouTube search results2012 19th IEEE International Conference on Image Processing10.1109/ICIP.2012.6467502(2885-2888)Online publication date: Sep-2012
https://doi.org/10.1109/ICIP.2012.6467502
Show More Cited By

Index Terms

Visual & textual fusion for region retrieval: from both fuzzy matching and bayesian reasoning aspects
1. Information systems
  1. Information retrieval

Recommendations

A Novel Retrieval Refinement and Interaction Pattern by Exploring Result Correlations for Image Retrieval
Adaptive Multimedial Retrieval: Retrieval, User, and Semantics

Efficient retrieval of image database that contains multiple predefined categories (e.g. medical imaging databases, museum painting collections) poses significant challenges and commercial prospects. By exploring category correlations of retrieval ...
Region-Based Image Clustering and Retrieval Using Fuzzy Similarity and Relevance Feedback
ICCEE '08: Proceedings of the 2008 International Conference on Computer and Electrical Engineering

This paper proposes an interactive approach for region-based image clustering and retrieval. By performing clustering before image retrieval, the search space can be reduced to those clusters that are close to the query target. First, the image is ...
Leveraging non-relevant images to enhance image retrieval performance
MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia

Inherent subjectivity in user's perception of an image has motivated the use of relevance feedback (RF) in the image desigined output's retrieval process. RF techniques interactively determine the user's query concept, given the user's relevance ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MIR '07: Proceedings of the international workshop on Workshop on multimedia information retrieval

September 2007

343 pages

ISBN:9781595937780

DOI:10.1145/1290082

General Chairs:
James Z. Wang
The Pennsylvania State University, USA
,
Nozha Boujemaa
INRIA Rocquencourt, France
,
Program Chairs:
Alberto Del Bimbo
University of Florence, Italy
,
Jia Li
The Pennsylvania State University, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 September 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

MM07

Sponsor:

MM07: The 15th ACM International Conference on Multimedia 2007

September 24 - 29, 2007

Bavaria, Augsburg, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
411
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhiqiang ZHua SYun WZou Q(2016)Annotation-retrieval reinforcement by visual cognition modeling on manifoldNeurocomputing10.1016/j.neucom.2015.07.162215:C(150-159)Online publication date: 26-Nov-2016
https://dl.acm.org/doi/10.1016/j.neucom.2015.07.162
Liu XYao HJi RXu PSun X(2013)Bidirectional-isomorphic manifold learning at image semantic understanding & representationMultimedia Tools and Applications10.1007/s11042-011-0947-264:1(53-76)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.1007/s11042-011-0947-2
Cao LJi RLiu WGao YDuan LMen C(2012)Weakly supervised topic grouping of YouTube search results2012 19th IEEE International Conference on Image Processing10.1109/ICIP.2012.6467502(2885-2888)Online publication date: Sep-2012
https://doi.org/10.1109/ICIP.2012.6467502
Li JLiu FSong CCui J(2010)A novel learning for image retrieval based on both keyword feature and instance feedback2010 3rd International Congress on Image and Signal Processing10.1109/CISP.2010.5646961(1561-1565)Online publication date: Oct-2010
https://doi.org/10.1109/CISP.2010.5646961
Gelernter JCao DLu RFink ECarbonell J(2009)Creating and visualizing fuzzy document classificationProceedings of the 2009 IEEE international conference on Systems, Man and Cybernetics10.5555/1732323.1732438(672-679)Online publication date: 11-Oct-2009
https://dl.acm.org/doi/10.5555/1732323.1732438
Liu XYao HJi RXu PSun XGao WRui YHanjalic AXu CSteinbach EEl Saddik AZhou M(2009)What is a complete set of keywords for image description & annotation on the webProceedings of the 17th ACM international conference on Multimedia10.1145/1631272.1631369(613-616)Online publication date: 23-Oct-2009
https://dl.acm.org/doi/10.1145/1631272.1631369
Liu XJi RYao HXu PSun XLiu TLew Mdel Bimbo ABakker E(2008)Cross-media manifold learning for image retrieval & annotationProceedings of the 1st ACM international conference on Multimedia information retrieval10.1145/1460096.1460121(141-148)Online publication date: 30-Oct-2008
https://dl.acm.org/doi/10.1145/1460096.1460121

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents