research-article

Content-based image retrieval using a combination of visual features and eye tracking data

Authors:

Zhen Liang,

Hong Fu,

Yun Zhang,

Zheru Chi,

Dagan FengAuthors Info & Claims

ETRA '10: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications

Pages 41 - 44

https://doi.org/10.1145/1743666.1743675

Published: 22 March 2010 Publication History

Get Access

Abstract

Image retrieval technology has been developed for more than twenty years. However, the current image retrieval techniques cannot achieve a satisfactory recall and precision. To improve the effectiveness and efficiency of an image retrieval system, a novel content-based image retrieval method with a combination of image segmentation and eye tracking data is proposed in this paper. In the method, eye tracking data is collected by a non-intrusive table mounted eye tracker at a sampling rate of 120 Hz, and the corresponding fixation data is used to locate the human's Regions of Interest (hROIs) on the segmentation result from the JSEG algorithm. The hROIs are treated as important informative segments/objects and used in the image matching. In addition, the relative gaze duration of each hROI is used to weigh the similarity measure for image retrieval. The similarity measure proposed in this paper is based on a retrieval strategy emphasizing the most important regions. Experiments on 7346 Hemera color images annotated manually show that the retrieval results from our proposed approach compare favorably with conventional content-based image retrieval methods, especially when the important regions are difficult to be located based on visual features.

References

[1]

Carson, C., Thomas, M., Belongie, S., Hellerstein, J. M., and Malik, J. 1999. Blobworld: A System for Region-Based Image Indexing and Retrieval. Proc. Visual Information Systems, 509--516.

Digital Library

Google Scholar

[2]

Deng, Y., and Manjunath, B. 2001. Unsupervised Segmentation of Color-Texture Regions in Images and Videos. IEEE Trans. Pattern Anal. Mach. Intell., vol. 23(8), 800--810.

Digital Library

Google Scholar

[3]

Fu, H., Chi, Z., and Feng. D. 2006. Attention-Driven Image Interpretation with Application to Image Retrieval. Pattern Recognition, Vol. 39(9), 1604--1621.

Digital Library

Google Scholar

[4]

De Graef, P., Christiaens, D., and d'Ydewalle, G. 1990. Perceptual Effects of Scene Context on Object Recognition. Psychological Research, Vol. 52, 317--329.

Crossref

Google Scholar

[5]

Henderson, J. M., and Hollingworth, A. 1998. Eye Movements During Scene Viewing: An Overview. in: Eye Guidance While Reading and While Watching Dynamic Scenes, Underwood, G. (Ed.). Elsevier Science, Amsterdam, 269--293.

Google Scholar

[6]

Lv, Q., Charikar, M., and Li. K. 2004. Image Similarity Search With Compact Data Structure. In Proceedings of The ACM International Conference on Information and Knowledge Management, 208--217.

Digital Library

Google Scholar

[7]

Marques, Q., Mayron, L., Borba, G., and Gamba, H. 2006. Using Visual Attention to Extract Regions of Interest in the Context of Image Retrieval. In Proceedings of ACM Annual Southeast Regional Conference, 638--643.

Digital Library

Google Scholar

[8]

Parkhurst, D. J., and Niebur, E. 2003. Scene Content Selected by Active vision. Spatial Vision, Vol. 16(2), 125--154.

Crossref

Google Scholar

[9]

Paschos, M. 2001. Perceptually Uniform Color Spaces for Color Texture Analysis: An Empirical Evaluation. IEEE Trans. Image Process, Vol. 10(6), 932--937.

Digital Library

Google Scholar

[10]

Rutishauser, U., Walther, D., Koch, C., and Perona, P. 2004. Is Bottom-Up Attention Useful for Object Recognition? Cvpr2004, Vol. 2, 37--44.

Digital Library

Google Scholar

[11]

Smeulders, A. W. M., Worring, M., and Santini, S. 2002. Content-Based Image Retrieval At the End of the Early Years. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 22(12), 1349--1380.

Digital Library

Google Scholar

[12]

Spekreijse, H. 2000. Pre-attentive and Attentive Mechanism in Vision. Perceptual Organization and Dysfunction. Vision Search, Vol. 40, 1179--1638.

Google Scholar

[13]

Tsai, C. F., McGarry, K., and Tait, J. 2003. Image Classification Using Hybrid Neural Network, In Proceedings of The ACM SIGIR Conference on Research and Development in Information Retrieaval, 431--432.

Digital Library

Google Scholar

[14]

Wang, X. Y., Hu. F. L. and Yang, H. Y. 2006. A Novel Regions-of-Interest Based Image Retrieval Using Multiple Features. In Proceedings of The Multi-Media Modeling International Conference, Vol. 1, 377--380.

Google Scholar

Cited By

View all

Joos LJäckl BKeim DFischer MPeska LLokoč JGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Known-Item Search in Video: An Eye Tracking-Based StudyProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658119(311-319)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658119
Li QHou W(2019)Using Experts’ Perceptual Skill for Dermatological Image SegmentationIntelligent Systems and Applications10.1007/978-3-030-29513-4_86(1199-1208)Online publication date: 24-Aug-2019
https://doi.org/10.1007/978-3-030-29513-4_86
Vrochidis SPatras IKompatsiaris I(2017)Gaze movement-driven random forests for query clustering in automatic video annotationMultimedia Tools and Applications10.5555/3048787.304883776:2(2861-2889)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.5555/3048787.3048837
Show More Cited By

Index Terms

Content-based image retrieval using a combination of visual features and eye tracking data
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment
    2. Information retrieval query processing

Recommendations

Eye movement as an interaction mechanism for relevance feedback in a content-based image retrieval system
ETRA '10: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications

Relevance feedback (RF) mechanisms are widely adopted in Content-Based Image Retrieval (CBIR) systems to improve image retrieval performance. However, there exist some intrinsic problems: (1) the semantic gap between high-level concepts and low-level ...
Content-based image retrieval based on eye-tracking
COGAIN '18: Proceedings of the Workshop on Communication by Gaze Interaction

To improve the performance of an image retrieval system, a novel content-based image retrieval (CBIR) framework with eye tracking data based on an implicit relevance feedback mechanism is proposed in this paper. Our proposed framework consists of three ...
Content-Based Image Retrieval Based on ROI Detection and Relevance Feedback

Content-based image retrieval is an important research topic in computer vision. We present a new method that combines region of interest (ROI) detection and relevance feedback. The ROI based approach is more accurate in describing the image content ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ETRA '10: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications

March 2010

353 pages

ISBN:9781605589947

DOI:10.1145/1743666

Conference Chairs:
Carlos Hitoshi Morimoto
University of Sao Paulo, Brazil
,
Howell Istance
De Montfort University, UK
,
Program Chairs:
Aulikki Hyrskykari
University of Tampere, Finland
,
Qiang Ji
Rensselaer Polytechnic Institute

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 March 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research Grants Council, University Grants Committee, Hong Kong

Conference

ETRA '10

Sponsor:

ETRA '10: Eye Tracking Research and Applications

March 22 - 24, 2010

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
609
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Joos LJäckl BKeim DFischer MPeska LLokoč JGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Known-Item Search in Video: An Eye Tracking-Based StudyProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658119(311-319)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658119
Li QHou W(2019)Using Experts’ Perceptual Skill for Dermatological Image SegmentationIntelligent Systems and Applications10.1007/978-3-030-29513-4_86(1199-1208)Online publication date: 24-Aug-2019
https://doi.org/10.1007/978-3-030-29513-4_86
Vrochidis SPatras IKompatsiaris I(2017)Gaze movement-driven random forests for query clustering in automatic video annotationMultimedia Tools and Applications10.5555/3048787.304883776:2(2861-2889)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.5555/3048787.3048837
Liang ZFu HLo WXu B(2017)Eye-tracking Aided Digital Training System for Strabismus TherapyJournal of Advances in Information Technology10.12720/jait.8.1.57-62(57-62)Online publication date: 2017
https://doi.org/10.12720/jait.8.1.57-62
Xin Liu Xianzhong Zhou Tianqi Ji Han Bai Huaxiong Li (2017)Combining eye movements for semantic image classification2017 IEEE 14th International Conference on Networking, Sensing and Control (ICNSC)10.1109/ICNSC.2017.8000186(761-766)Online publication date: May-2017
https://doi.org/10.1109/ICNSC.2017.8000186
Zhou XGao XWang JYu HWang ZChi Z(2017)Eye tracking data guided feature selection for image classificationPattern Recognition10.1016/j.patcog.2016.09.00763:C(56-70)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1016/j.patcog.2016.09.007
Cao HLi YAllen CPhinney MShyu C(2016)Visual Reasoning Indexing and Retrieval Using In-memory Computing2016 IEEE Second International Conference on Multimedia Big Data (BigMM)10.1109/BigMM.2016.83(17-24)Online publication date: Apr-2016
https://doi.org/10.1109/BigMM.2016.83
Vrochidis SPatras IKompatsiaris I(2016)Gaze movement-driven random forests for query clustering in automatic video annotationMultimedia Tools and Applications10.1007/s11042-015-3221-176:2(2861-2889)Online publication date: 22-Jan-2016
https://doi.org/10.1007/s11042-015-3221-1
Li QTian MLiu JSun J(2016)An implicit relevance feedback method for CBIR with real-time eye trackingMultimedia Tools and Applications10.1007/s11042-015-2873-175:5(2595-2611)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1007/s11042-015-2873-1
Chen ZFu HLo WChi Z(2015)Eye-Tracking Aided Digital System for Strabismus Diagnosis2015 IEEE International Conference on Systems, Man, and Cybernetics10.1109/SMC.2015.403(2305-2309)Online publication date: Oct-2015
https://doi.org/10.1109/SMC.2015.403
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Eye movement as an interaction mechanism for relevance feedback in a content-based image retrieval system

Content-based image retrieval based on eye-tracking

Content-Based Image Retrieval Based on ROI Detection and Relevance Feedback