More Web Proxy on the site http://driver.im/

article

Combining intra-image and inter-class semantics for consumer image retrieval

Authors:

Jesse S. JinAuthors Info & Claims

Pattern Recognition, Volume 38, Issue 6

Pages 847 - 864

https://doi.org/10.1016/j.patcog.2004.11.002

Published: 01 June 2005 Publication History

Abstract

Unconstrained consumer photos pose great challenge for content-based image retrieval. Unlike professional images or domain-specific images, consumer photos vary significantly. More often than not, the objects in the photos are ill-posed, occluded, and cluttered with poor lighting, focus and exposure. In this paper, we propose a cascading framework for combining intra-image and inter-class similarities in image retrieval, motivated from probabilistic Bayesian principles. Support vector machines are employed to learn local view-based semantics based on just-in-time fusion of color and texture features. A new detection-driven block-based segmentation algorithm is designed to extract semantic features from images. The detection-based indexes also serve as input for support vector learning of image classifiers to generate class-relative indexes. During image retrieval, both intra-image and inter-class similarities are combined to rank images. Experiments using query-by-example on 2400 genuine heterogeneous consumer photos with 16 semantic queries show that the combined matching approach is better than matching with single index. It also outperformed the method of combining color and texture features by 55% in average precision.

References

[1]

K. Rodden, K. Wood, How people manage their digital photos? in: Proceedings of the ACM CHI 2003, 2003.

[2]

Smeulders, A.W.M., Content-based image retrieval at the end of the early years. IEEE Trans. on PAMI. v22 i12. 1349-1380.

[3]

Y.L. Lu, et al., A unified framework for semantics and feature based relevance feedback in image retrieval systems, in: Proceedings of the ACM Multimedia'2000, 2000, pp. 31-37.

[4]

W. Liu, Y. Sun, H. Zhang, MiAlbum-A system for home photo management using the semi-automatic image annotation approach, in: Proceedings of the ACM Multimedia, 2000, pp. 479-480.

[5]

Y. Wu, Q. Tian, T.S. Huang, Discriminant-EM algorithm with application to image retrieval, in: Proceedings of the CVPR'2000, 2000, pp. 1222-1227.

[6]

C. Town, D. Sinclair, Content-based image retrieval using semantic visual categories, Technical Report 2000.14, AT&T Research, Cambridge, 2000.

[7]

M.R. Naphade, et al., A framework for moderate vocabulary semantic visual concept detection, in: Proceedings of the IEEE ICME, 2003, pp. 437-440.

[8]

M.R. Naphade, J.R. Smith, Learning visual models of semantic concepts, in: Proceedings of the IEEE ICIP 2003, 2003.

[9]

A. Amir, et al., The IBM semantic concept detection framework. Slides presented at TRECVID Workshop 2003, http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html, 2003.

[10]

C. Lin, B. Tseng, J. Smith, VideoAnnEx: IBM MPEG-7 annotation tool for multimedia indexing and concept learning, in: Proceedings of the IEEE ICME 2003, 2003.

[11]

P. Duygulu, et al., Object recognition as machine translation: learning a lexicon for a fixed image vocabulary, in: Proceedings of the ECCV'2002, 2002, pp. IV: 97-112.

[12]

K. Barnard, D. Forsyth, Learning the semantics of words and pictures, in: Proceedings of the ICCV, 2001, pp. 408-415.

[13]

Barnard, K., Matching words and pictures. J. Mach. Learning Res. v3. 1107-1135.

[14]

Li, J. and Wang, J.Z., Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Trans. on PAMI. v25 i10. 1-14.

[15]

K. Barnard, et al., The effects of segmentation of feature choices in a translation model of object recognition, in: Proceedings of the CVPR, 2003.

[16]

B. Bradshaw, Semantic based image retrieval: a probabilistic approach, in: Proceedings of the ACM Multimedia, 2000, pp. 167-176.

[17]

P. Lipson, E. Grimson, P. Sinha, Configuration base scene classification and image indexing, in: Proceedings of the CVPR'97, 1997, pp. 1007-1013.

[18]

M. Szummer, R.W. Picard, Indoor-outdoor image classification, in: Proceedings of the IEEE International Work on Content-based Access of Image and Video Databases, January 1998, pp. 42-51.

[19]

Vailaya, A., Bayesian framework for hierarchical semantic classification of vacation images. IEEE Trans. on Image Process. v10 i1. 117-130.

[20]

Naphade, M.R., Kozintsev, I.V. and Huang, T.S., A factor graph framework for semantic video indexing. IEEE Trans. on CSVT. v12 i1. 40-52.

[21]

K. Tieu, P. Viola, Boosting image retrieval, in: Proceedings of the CVPR'2000, 2000, pp. 1228-1235.

[22]

Kapur, J.N. and Kesava, H.K., Entropy Optimization Principles with Applications. 1992. Academic Press, New York.

[23]

Robertson, S.E. and Sparck Jones, K., Relevance weighting of search terms. J. Am. Soc. Info. Sci. v27. 129-146.

[24]

J.H. Lim, Explicit query formulation with visual keywords, in: Proceedings of the ACM Multimedia'2000, 2000, pp. 407-409.

[25]

Lim, J.H., Building visual vocabulary for image indexation and query formulation. Pattern Analysis and Applications (Special Issue on Image Indexation). v4 i2/3. 125-139.

[26]

J.H. Lim, J.S. Jin, Learning consumer photo categories for semantic retrieval, in: Proceedings of the IJCAI'2003, 2003, pp. 1413-1414.

[27]

J.H. Lim, J.S. Jin, Support regions and images for photo event retrieval, in: Proceedings of the IEEE ICIP'2003, II, 2003, pp. 515-518.

[28]

Lim, J.H., Tian, Q. and Mulhem, P., Home photo content modeling for personalized event-based retrieval. IEEE Multimedia. v10 i4. 28-37.

[29]

S. Kumar, A.C. Loui, M. Hebert, Probabilistic classification of image regions using an observation-constrained generative approach, First International Workshop on Generative-Model-Based Vision, June 2002.

[30]

Adams, W.H., Semantic indexing of multimedia content using visual audio, and text cues. Eurasip J. Appl. Signal Process. v2003 i2. 170-185.

[31]

W.H.M. Hsu, S.F. Chang, Generative, discriminative, and ensemble learning on multi-modal perceptual fusion toward news video story segmentation, Proceedings of the IEEE ICME 2004, Taipei, Taiwan, 27-30 June 2004.

[32]

Manjunath, B.S. and Ma, W.Y., Texture features for browsing and retrieval of image data. IEEE Transactions on PAMI. v18 i8. 837-842.

[33]

M. Ortega, et al., Supporting similarity queries in MARS, in: Proceedings of the ACM Multimedia, 1997, pp. 403-413.

[34]

Sung, K.K. and Poggio, T., Example-based learning for view-based human face detection. IEEE Trans. on PAMI. v20 i1. 39-51.

[35]

J.R. Smith, M.R. Naphade, A.P. Natsev, Multimedia semantic indexing using model vectors, in: Proceedings of the IEEE ICME, 2003, pp. II 445-448.

[36]

P.C. Papageorgiou, M. Oren, T. Poggio, A general framework for object detection, in: Proceedings of the ICCV, 1997, pp. 555-562.

[37]

Bishop, C.M., Neural Networks for Pattern Recognition. 1995. Clarendon Press, Oxford.

[38]

Swain, M.J. and Ballard, D.N., . Color indexing. Intl. J. Comput. Vision. v7 i1. 11-32.

[39]

C.E. Brodley, et al., Content-based retrieval from medical image databases: a synergy of human interaction, machine learning and computer vision, in: Proceedings of the AAAI, 1999, pp. 760-767.

[40]

A. Mojsilovic, J. Gomes, Semantic based categorization, browsing and retrieval in medical image databases, in: Proceedings of the IEEE ICIP, 2002.

[41]

Wang, J.Z., Li, J. and Wiederhold, G., SIMPLIcity: semantics-sensitive integrated matching for picture libraries. . IEEE Transactions on PAMI. v23 i9. 947-963.

[42]

Joachims, T., Making large-scale SVM learning practical. In: Scholkopf, B., Burges, C., Smola, A. (Eds.), Advances in Kernel Methods-Support Vector Learning, MIT-Press, Cambridge, MA.

[43]

J.R. Smith, et al., Integrating features, models, and semantics for TREC video, in: NIST Special Publication 500-200: The Tenth Text REtrieval Conference (TREC 2001), 2001, pp. 240-249.

[44]

T. Maenpaa, M. Pietikainen, J. Viertola, Separating color and pattern information for color texture discrimination, in: Proceedings of the ICPR 2002, August 11-15, Quebec City, Canada, 2002, pp. 668-671.

[45]

Johnson, R.A. and Wichern, D.W., Applied Multivariate Statistical Analysis. 1988. Prentice-Hall, Englewood Cliffs, NJ.

[46]

Smith, J.R. and Chang, S.-F., Integrated spatial and feature image query. Multimedia Systems. v7 i2. 129-140.

[47]

Chen, Y. and Wang, J.Z., A region-based fuzzy feature matching approach to content-based image retrieval. IEEE Trans. on PAMI. v24 i9. 1252-1267.

Cited By

Chang CWang HLi C(2009)Semantic analysis of real-world images using support vector machineExpert Systems with Applications: An International Journal10.1016/j.eswa.2009.03.04136:7(10560-10569)Online publication date: 1-Sep-2009
https://dl.acm.org/doi/10.1016/j.eswa.2009.03.041
Han DLi WLi Z(2008)Semantic image classification using statistical local spatial relations modelMultimedia Tools and Applications10.1007/s11042-008-0203-639:2(169-188)Online publication date: 1-Sep-2008
https://dl.acm.org/doi/10.1007/s11042-008-0203-6
Banerjee MKundu M(2007)Image retrieval using fuzzy relevance feedback and validation with MPEG-7 content descriptorsProceedings of the 2nd international conference on Pattern recognition and machine intelligence10.5555/1781034.1781054(144-152)Online publication date: 18-Dec-2007
https://dl.acm.org/doi/10.5555/1781034.1781054
Show More Cited By

Combining intra-image and inter-class semantics for consumer image retrieval
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
2. Information systems
  1. Information retrieval

Recommendations

A unified framework for semantics and feature based relevance feedback in image retrieval systems
MULTIMEDIA '00: Proceedings of the eighth ACM international conference on Multimedia

The relevance feedback approach to image retrieval is a powerful technique and has been an active research direction for the past few years. Various ad hoc parameter estimation techniques have been proposed for relevance feedback. In addition, methods ...
Lire: lucene image retrieval: an extensible java CBIR library
MM '08: Proceedings of the 16th ACM international conference on Multimedia

LIRe (Lucene Image Retrieval) is a light weight open source Java library for content based image retrieval. It provides common and state of the art global image features and offers means for indexing and retrieval. Due to the fact that it is based on a ...
Image Retrieval System for Web: Webscope-CBIR
DEXA '00: Proceedings of the 11th International Workshop on Database and Expert Systems Applications

With the widespread usage of images on Web, the retrieval need for images has been increasing rapidly. But traditional text retrieval system is not fit for the application of image retrieval. We therefore designed Webscope-CBIR, an image retrieval ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 38, Issue 6

June, 2005

162 pages

ISSN:0031-3203

Issue’s Table of Contents

Copyright © Pattern Recognition Society © 2005.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 June 2005

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chang CWang HLi C(2009)Semantic analysis of real-world images using support vector machineExpert Systems with Applications: An International Journal10.1016/j.eswa.2009.03.04136:7(10560-10569)Online publication date: 1-Sep-2009
https://dl.acm.org/doi/10.1016/j.eswa.2009.03.041
Han DLi WLi Z(2008)Semantic image classification using statistical local spatial relations modelMultimedia Tools and Applications10.1007/s11042-008-0203-639:2(169-188)Online publication date: 1-Sep-2008
https://dl.acm.org/doi/10.1007/s11042-008-0203-6
Banerjee MKundu M(2007)Image retrieval using fuzzy relevance feedback and validation with MPEG-7 content descriptorsProceedings of the 2nd international conference on Pattern recognition and machine intelligence10.5555/1781034.1781054(144-152)Online publication date: 18-Dec-2007
https://dl.acm.org/doi/10.5555/1781034.1781054
Banerjee MKundu M(2007)Image Retrieval Using Fuzzy Relevance Feedback and Validation with MPEG-7 Content DescriptorsPattern Recognition and Machine Intelligence10.1007/978-3-540-77046-6_18(144-152)Online publication date: 18-Dec-2007
https://dl.acm.org/doi/10.1007/978-3-540-77046-6_18
Lim JChevallet J(2005)VisMedProceedings of the Second Asia conference on Asia Information Retrieval Technology10.1007/11562382_7(84-96)Online publication date: 13-Oct-2005
https://dl.acm.org/doi/10.1007/11562382_7

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents