More Web Proxy on the site http://driver.im/

article

WhittleSearch: Interactive Image Search with Relative Attribute Feedback

Authors:

Adriana Kovashka,

Kristen GraumanAuthors Info & Claims

International Journal of Computer Vision, Volume 115, Issue 2

Pages 185 - 210

https://doi.org/10.1007/s11263-015-0814-0

Published: 01 November 2015 Publication History

Abstract

We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image sought. For example, perusing image results for a query "black shoes", the user might state, "Show me shoe images like these, but sportier." Offline, our approach first learns a set of ranking functions, each of which predicts the relative strength of a nameable attribute in an image (e.g., sportiness). At query time, the system presents the user with a set of exemplar images, and the user relates them to his/her target image with comparative statements. Using a series of such constraints in the multi-dimensional attribute space, our method iteratively updates its relevance function and re-ranks the database of images. To determine which exemplar images receive feedback from the user, we present two variants of the approach: one where the feedback is user-initiated and another where the feedback is actively system-initiated. In either case, our approach allows a user to efficiently "whittle away" irrelevant portions of the visual feature space, using semantic language to precisely communicate her preferences to the system. We demonstrate our technique for refining image search for people, products, and scenes, and we show that it outperforms traditional binary relevance feedback in terms of search speed and accuracy. In addition, the ordinal nature of relative attributes helps make our active approach efficient--both computationally for the machine when selecting the reference images, and for the user by requiring less user interaction than conventional passive and active methods.

References

[1]

Berg, T., Berg, A. & Shih, J. (2010). Automatic attribute discovery and characterization from noisy web data. In: Proceedings of the European Conference on Computer Vision (ECCV).

[2]

Biswas, A. & Parikh, D. (2013). Simultaneous active learning of classifiers and attributes via relative feedback. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]

Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P. & Belongie, S. (2010). Visual recognition with humans in the loop. In: Proceedings of the European Conference on Computer Vision (ECCV).

[4]

Cox, I., Miller, M., Minka, T., Papathomas, T., & Yianilos, P. (2000). The Bayesian image retrieval system, PicHunter: Theory, implementation and psychophysical experiments. IEEE Transactions on Image Processing, 9(1), 20-37.

Digital Library

[5]

Douze, M., Ramisa, A., Schmid, C. (2011). Combining attributes and fisher vectors for efficient image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]

Farhadi, A., Endres, I., Hoiem, D., Forsyth, D. (2009). Describing objects by their attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]

Ferecatu, M., Geman, D. (2007). Interactive search for image categories by mental matching. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[8]

Flickner, M., Sawhney, H., Nilback, W., Ashley, J., Huang, Q., Dom, B., et al. (1995). Query by image and video content: The QBIC system. IEEE Computer, 28(9), 23-32.

Digital Library

[9]

Geman, D. & Jedynak, B. (1998). Model-based classification trees. IEEE Transactions on Information Theory, 47(3), 1075-1082.

[10]

Iqbal, Q. & Aggarwal, J. K. (2002) CIRES: A system for content-based retrieval in digital image libraries. In: Proceedings of the International Conference on Control, Automation, Robotics and Vision.

[11]

Jayaraman, D., Sha, F. & Grauman, K. (2014). Decorrelating semantic visual attributes by resisting the urge to share. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]

Joachims, T. (2002). Optimizing search engines using clickthrough data. In: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).

[13]

Joachims, T. (2006). Training linear SVMs in linear time. In: Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD).

[14]

Kekalainen, J., & Jarvelin, K. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422-446.

[15]

Kovashka, A. & Grauman, K. (2013a). Attribute adaptation for personalized image search. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[16]

Kovashka, A. & Grauman, K. (2013b). Attribute pivots for guiding relevance feedback in image search. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[17]

Kovashka, A., Vijayanarasimhan, S. & Grauman, K. (2011). Actively selecting annotations among objects and attributes. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[18]

Kovashka, A., Parikh, D. & Grauman, K. (2012). Whittle search: Image search with relative attribute feedback. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]

Kulkarni, P., Sharma, G., Zepeda, J. & Chevallier, L. (2014). Transfer learning via attributes for improved on-the-fly classification. In: Proceedings of the Winter Conference on Applications of Computer Vision (WACV).

[20]

Kumar, N., Belhumeur, P. & Nayar, S. (2008). Facetracer: A search engine for large collections of images with faces. In: Proceedings of the European Conference on Computer Vision (ECCV).

[21]

Kumar, N., Berg, A. C., Belhumeur, P. N. & Nayar, S. K. (2009). Attribute and simile classifiers for face verification. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[22]

Kurita, T., Kato, T. (1993). Learning of personal visual impression for image database systems. In: Proceedings of the International Conference on Document Analysis and Recognition (ICDAR).

[23]

Lampert, C., Nickisch, H. & Harmeling, S. (2009). Learning to detect unseen object classes by between-class attribute transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]

Li, B., Chang, E. & Li, C. S. (2001). Learning image query concepts via intelligent sampling. In: Proceedings of the International Conference on Multimedia and Expo (ICME).

[25]

Ma, W. & Manjunath, B. (1997). NeTra: A toolbox for navigating large image databases. In: Proceedings of the International Conference on Image Processing (ICIP).

[26]

MacArthur, S. D., Brodley, C. E. & Shyu, C. R. (2000). Relevance feedback decision trees in content-based image retrieval. In: Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries.

[27]

Maji, S. (2012). Discovering a lexicon of parts and attributes. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshop on Parts and Attributes.

[28]

Mensink, T., Verbeek, J. & Csurka, G. (2011). Learning structured prediction models for interactive image labeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]

Naphade, M., Smith, J., Tesic, J., Chang, S. F., Hsu, W., Kennedy, L., et al. (2006). Large-scale concept ontology for multimedia. IEEE Transactions on Multimedia, 13(3), 86-91.

Digital Library

[30]

Oliva, A., & Torralba, A. (2001). Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision (IJCV), 42(3), 145-175.

Digital Library

[31]

Parikh, D., & Grauman, K. (2011a) Interactively building a discriminative vocabulary of nameable attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]

Parikh, D., & Grauman, K. (2011b) Relative Attributes. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[33]

Parikh, D., & Grauman, K. (2013) Implied feedback: Learning nuances of user behavior in image search. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[34]

Parkash, A., & Parikh, D. (2012) Attributes for classifier feedback. In: Proceedings of the European Conference on Computer Vision (ECCV).

[35]

Patterson, G., Xu, C., Su, H., & Hays, J. (2014). The SUN attribute database: Beyond Categories for deeper scene understanding. International Journal of Computer Vision (IJCV), 108(1-2), 59-81.

Digital Library

[36]

Platt, J. C. (1999) Probabilistic output for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers 10(3), 61-74.

[37]

Rasiwasia, N., Moreno, P., & Vasconcelos, N. (2007). Bridging the gap: Query by semantic example. IEEE Transactions on Multimedia, 9(5), 923-938.

Digital Library

[38]

Rastegari, M., Parikh, D., Diba, A. & Farhadi, A. (2013). Multi-attribute queries: To merge or not to merge? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]

Rui, Y., Huang, T., Ortega, M., & Mehrotra, S. (1998). Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Transactions on Circuits and Video Technology, 8(5), 644-655.

Digital Library

[40]

Saleh, B., Farhadi, A. & Elgammal, A. (2013). Object-centric anomaly detectionbyattribute-based reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]

Scheirer, W., Kumar, N., Belhumeur, P. & Boult, T. (2012). Multi-attribute spaces: Calibration for attribute fusion and similarity search. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]

Siddiquie, B., Feris, R. & Davis, L. (2011). Image ranking and retrieval based on multi-attribute queries. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]

Smith, J., Naphade, M. & Natsev, A. (2003). Multimedia semantic indexing using model vectors. In: Proceedings of the International Conference on Multimedia and Expo (ICME).

[44]

Sznitman, R., & Jedynak, B. (2010). Active testing for face detection and localization. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 32(10), 1914-1920.

Digital Library

[45]

Tieu, K. & Viola, P. (2000). Boosting image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]

Tong, S. & Chang, E. (2001). Support vector machine active learning for image retrieval. In: Proceedings of the ACM International Conference on Multimedia.

[47]

Vijayanarasimhan, S. & Kapoor, A. (2010). Visual recognition and detection under bounded computational resources. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]

Wah, C. & Belongie, S. (2013). Attribute-based detection of unfamiliar classes with humans in the loop. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]

Wah, C., Van Horn, G., Branson, S., Maji, S., Perona, P. & Belongie, S. (2014). Similarity comparisons for interactive fine-grained categorization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]

Wang, X., Liu, K. & Tang, X. (2011). Query-specific visual semantic spaces for web image re-ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]

Wang, Y. & Mori, G. (2010). A discriminative latent model of object classes and attributes. In: Proceedings of the European Conference on Computer Vision (ECCV).

[52]

Zavesky, E. & Chang, S. F. (2008). Cu-Zero: Embracing the Frontier of interactive visual search for informed users. In: Proceedings of the ACM International Conference on Multimedia Information Retrieval.

[53]

Zhang, C., & Chen, T. (2002). An active learning framework for content based information retrieval. IEEE Transactions on Multimedia, 4(2), 260-268.

Digital Library

[54]

Zhou, X., & Huang, T. (2003). Relevance feedback in image retrieval: A comprehensive review. ACM Multimedia Systems, 8(6), 536-544.

Cited By

Cote MBranzan Albu A(2024)Attribute-based document image retrievalInternational Journal on Document Analysis and Recognition10.1007/s10032-023-00447-627:1(57-71)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10032-023-00447-6
Baldrati ABertini MUricchio TDel Bimbo A(2023)Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based FeaturesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361759720:3(1-24)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3617597
Zhang YJi ZPang YLi X(2023)Consensus Knowledge Exploitation for Partial Query Based Image RetrievalIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.328150733:12(7900-7913)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3281507
Show More Cited By

Recommendations

Attribute Pivots for Guiding Relevance Feedback in Image Search
ICCV '13: Proceedings of the 2013 IEEE International Conference on Computer Vision

In interactive image search, a user iteratively refines his results by giving feedback on exemplar images. Active selection methods aim to elicit useful feedback, but traditional approaches suffer from expensive selection criteria and cannot predict in ...
WhittleSearch: Image search with relative attribute feedback
CVPR '12: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

We propose a novel mode of feedback for image search, where a user describes which properties of exemplar images should be adjusted in order to more closely match his/her mental model of the image(s) sought. For example, perusing image results for a ...
Spoken Attributes: Mixing Binary and Relative Attributes to Say the Right Thing
ICCV '13: Proceedings of the 2013 IEEE International Conference on Computer Vision

In recent years, there has been a great deal of progress in describing objects with attributes. Attributes have proven useful for object recognition, image search, face verification, image description, and zero-shot learning. Typically, attributes are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Computer Vision

International Journal of Computer Vision Volume 115, Issue 2

November 2015

142 pages

ISSN:0920-5691

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 November 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cote MBranzan Albu A(2024)Attribute-based document image retrievalInternational Journal on Document Analysis and Recognition10.1007/s10032-023-00447-627:1(57-71)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s10032-023-00447-6
Baldrati ABertini MUricchio TDel Bimbo A(2023)Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based FeaturesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/361759720:3(1-24)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3617597
Zhang YJi ZPang YLi X(2023)Consensus Knowledge Exploitation for Partial Query Based Image RetrievalIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.328150733:12(7900-7913)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3281507
Jaiswal ALiu HFrommholz IHasibi FFang YAizawa A(2021)Semantic Hilbert Space for Interactive Image RetrievalProceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3471158.3472253(307-315)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3471158.3472253
Baldrati ABertini MUricchio TDel Bimbo A(2021)Conditioned Image Retrieval for Fashion using Contrastive Learning and CLIP-based FeaturesProceedings of the 3rd ACM International Conference on Multimedia in Asia10.1145/3469877.3493593(1-5)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1145/3469877.3493593
Liu SZhou XJiang XWu HShi Y(2021)Face Shows Your Intention: Visual Search Based on Full-face Gaze Estimation with Channel-spatial AttentionProceedings of the 2021 5th International Conference on Innovation in Artificial Intelligence10.1145/3461353.3461362(76-81)Online publication date: 5-Mar-2021
https://dl.acm.org/doi/10.1145/3461353.3461362
Fang YXiao ZZhang WHuang YWang LBoujemaa NGeman D(2021)Attribute Prototype Learning for Interactive Face RetrievalIEEE Transactions on Information Forensics and Security10.1109/TIFS.2021.305927416(2593-2607)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1109/TIFS.2021.3059274
Ahmed SYanikoglu B(2021)Relative Attribute Classification with Deep-RankSVMPattern Recognition. ICPR International Workshops and Challenges10.1007/978-3-030-68790-8_51(659-671)Online publication date: 10-Jan-2021
https://dl.acm.org/doi/10.1007/978-3-030-68790-8_51
Shimizu EFisher MParis SMcCann JFatahalian KIqbal SMacLean KChevalier FMueller S(2020)Design AdjectivesProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415866(261-278)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379337.3415866
Grauman KCaverlee JHu XLalmas MWang W(2020)Computer Vision for FashionProceedings of the 13th International Conference on Web Search and Data Mining10.1145/3336191.3372192(3-3)Online publication date: 20-Jan-2020
https://dl.acm.org/doi/10.1145/3336191.3372192
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents