Just Accepted
Interactive Garment Recommendation with User in the Loop
Authors: 
Federico Becattini, Xiaolin Chen, Andrea Puccia, Haokun Wen, Xuemeng Song, Liqiang Nie, 
Alberto Del BimboAuthors Info & Claims
Accepted on 13 October 2024
Abstract
Recommending fashion items often leverages rich user profiles and makes targeted suggestions based on past history and previous purchases. In this paper, we work under the assumption that no prior knowledge is given about a user. We propose to build a user profile on the fly by integrating user reactions as we recommend complementary items to compose an outfit. We present a reinforcement learning agent capable of suggesting appropriate garments and ingesting user feedback so to improve its recommendations and maximize user satisfaction. To train such a model, we resort to a proxy model to be able to simulate having user feedback in the training loop. We experiment on the IQON3000 fashion dataset and we find that a reinforcement learning-based agent becomes capable of improving its recommendations by taking into account personal preferences. Furthermore, such task demonstrated to be hard for non-reinforcement models, that cannot exploit exploration during training.
References
[1]
Lorenzo Agnolucci, Alberto Baldrati, Marco Bertini, and Alberto Del Bimbo. 2024. iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval. arXiv preprint arXiv:2405.02951 (2024).
[2]
Alberto Baldrati, Davide Morelli, Marcella Cornia, Marco Bertini, and Rita Cucchiara. 2024. Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing. arXiv preprint arXiv:2403.14828 (2024).
[3]
Federico Becattini, Giuseppe Becchi, Andrea Ferracani, Alberto Del Bimbo, Liliana Lo Presti, Giuseppe Mazzola, Marco La Cascia, Federico Cunico, Andrea Toaiari, Marco Cristani, et al. 2022. I-MALL An Effective Framework for Personalized Visits. Improving the Customer Experience in Stores. In Proceedings of the 1st Workshop on Multimedia Computing towards Fashion Recommendation. 11–19.
[4]
Federico Becattini, Lavinia De Divitiis, Claudio Baecchi, and Alberto Del Bimbo. 2023. Fashion recommendation based on style and social events. Multimedia Tools and Applications (2023), 1–16.
[5]
Federico Becattini, Federico Palai, and Alberto Del Bimbo. 2022. Understanding Human Reactions Looking at Facial Microexpressions With an Event Camera. IEEE Transactions on Industrial Informatics 18, 12 (2022), 9112–9121.
[6]
Federico Becattini, Xuemeng Song, Claudio Baecchi, Shi-Ting Fang, Claudio Ferrari, Liqiang Nie, and Alberto Del Bimbo. 2021. PLM-IPE: A Pixel-Landmark Mutual Enhanced Framework for Implicit Preference Estimation. In ACM Multimedia Asia. 1–5.
[7]
Federico Becattini, Federico Maria Teotini, and Alberto Del Bimbo. 2023. Transformer-based Graph Neural Networks for Outfit Generation. IEEE Transactions on Emerging Topics in Computing (2023).
[8]
Wolmer Bigi, Claudio Baecchi, and Alberto Del Bimbo. 2020. Automatic interest recognition from posture and behaviour. In Proceedings of the 28th ACM International Conference on Multimedia. 2472–2480.
[9]
Yanbei Chen, Shaogang Gong, and Loris Bazzani. 2020. Image Search with Text Feedback by Visiolinguistic Attention Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2998–3008.
[10]
Zeyu Cui, Zekun Li, Shu Wu, Xiaoyu Zhang, and Liang Wang. 2019. Dressing as a Whole: Outfit Compatibility Learning Based on Node-wise Graph Neural Networks. In The World Wide Web Conference. ACM, 307–317.
[11]
Lavinia De Divitiis, Federico Becattini, Claudio Baecchi, and Alberto Del Bimbo. 2021. Garment recommendation with memory augmented neural networks. In International conference on pattern recognition. Springer, 282–295.
[12]
Lavinia De Divitiis, Federico Becattini, Claudio Baecchi, and Alberto Del Bimbo. 2021. Style-based outfit recommendation. In 2021 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 1–4.
[13]
Lavinia De Divitiis, Federico Becattini, Claudio Baecchi, and Alberto Del Bimbo. 2023. Disentangling features for fashion recommendation. ACM Transactions on Multimedia Computing, Communications and Applications 19, 1s (2023), 1–21.
[14]
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. 2020. A theoretical analysis of deep Q-learning. In Learning for dynamics and control. PMLR, 486–489.
[15]
Chunbin Gu, Jiajun Bu, Zhen Zhang, Zhi Yu, Dongfang Ma, and Wei Wang. 2021. Image Search with Text Feedback by Deep Hierarchical Attention Mutual Information Maximization. In Proceedings of the International ACM Conference on Multimedia. ACM, 4600–4609.
[16]
Weili Guan, Xuemeng Song, Haoyu Zhang, Meng Liu, Chung-Hsing Yeh, and Xiaojun Chang. 2022. Bi-directional Heterogeneous Graph Hashing towards Efficient Outfit Recommendation. In Proceedings of the ACM International Conference on Multimedia. ACM, 268–276.
[17]
Weili Guan, Haokun Wen, Xuemeng Song, Chun Wang, Chung-Hsing Yeh, Xiaojun Chang, and Liqiang Nie. 2022. Partially Supervised Compatibility Modeling. IEEE Trans. Image Process. 31 (2022), 4733–4745.
[18]
Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogério Schmidt Feris. 2018. Dialog-based Interactive Image Retrieval. In Proceedings of the Conference on Neural Information Processing Systems. 676–686.
[19]
Xintong Han, Zuxuan Wu, Phoenix X. Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S. Davis. 2017. Automatic Spatially-Aware Fashion Concept Discovery. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 1472–1480.
[20]
Xintong Han, Zuxuan Wu, Yu-Gang Jiang, and Larry S. Davis. 2017. Learning Fashion Compatibility with Bidirectional LSTMs. In Proceedings of the International ACM Conference on Multimedia. ACM, 1078–1086.
[21]
Seungmin Lee, Dongwan Kim, and Bohyung Han. 2021. CoSMo: Content-Style Modulation for Image Retrieval With Text Feedback. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, 802–812.
[22]
Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning 8 (1992), 293–321.
[23]
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning Convolutional Neural Networks for Graphs. In Proceedings of the International Conference on Machine Learning. JMLR.org, 2014–2023.
[24]
Anwesan Pal, Sahil Wadhwa, Ayush Jaiswal, Xu Zhang, Yue Wu, Rakesh Chada, Pradeep Natarajan, and Henrik I Christensen. 2023. FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory. arXiv preprint arXiv:2308.10170 (2023).
[25]
Giovanna Scaramuzzino, Federico Becattini, and Alberto Del Bimbo. 2023. Attribute disentanglement with gradient reversal for interactive fashion retrieval. Pattern Recognition Letters 172 (2023), 203–212.
[26]
Karin Sevegnani, Arjun Seshadri, Tian Wang, Anurag Beniwal, Julian J. McAuley, Alan Lu, and Gerard Medioni. 2022. Contrastive Learning for Interactive Recommendation in Fashion. CoRR abs/2207.12033 (2022).
[27]
Dandan Sha, Daling Wang, Xiangmin Zhou, Shi Feng, Yifei Zhang, and Ge Yu. 2016. An Approach for Clothing Recommendation Based on Multiple Image Attributes. In Web-Age Information Management International Conference. Springer, 272–285.
[28]
Xuemeng Song, Fuli Feng, Jinhuan Liu, Zekun Li, Liqiang Nie, and Jun Ma. 2017. NeuroStylist: Neural Compatibility Modeling for Clothing Matching. In Proceedings of the International ACM Conference on Multimedia. ACM, 753–761.
[29]
Xuemeng Song, Xianjing Han, Yunkai Li, Jingyuan Chen, Xin-Shun Xu, and Liqiang Nie. 2019. GP-BPR: Personalized Compatibility Modeling for Clothing Matching. In Proceedings of the ACM International Conference on Multimedia. ACM, 320–328.
[30]
Xuemeng Song, Xianjing Han, Yunkai Li, Jingyuan Chen, Xin-Shun Xu, and Liqiang Nie. 2019. GP-BPR: Personalized compatibility modeling for clothing matching. In Proceedings of the 27th ACM International Conference on Multimedia. 320–328.
[31]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[32]
Mariya I. Vasileva, Bryan A. Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David A. Forsyth. 2018. Learning Type-Aware Embeddings for Fashion Compatibility. In Proceedings of the European Conference on Computer Vision. Springer, 405–421.
[33]
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. 2019. Composing Text and Image for Image Retrieval - an Empirical Odyssey. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 6439–6448.
[34]
Haokun Wen, Xuemeng Song, Xin Yang, Yibing Zhan, and Liqiang Nie. 2021. Comprehensive Linguistic-Visual Composition Network for Image Retrieval. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1369–1378.
[35]
Haokun Wen, Xian Zhang, Xuemeng Song, Yinwei Wei, and Liqiang Nie. 2023. Target-Guided Composed Image Retrieval. In Proceedings of the ACM International Conference on Multimedia. ACM, 915–923.
[36]
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogério Feris. 2021. Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 11307–11317.
[37]
Wei Wu, Jiawei Liu, Kecheng Zheng, Qibin Sun, and Zhengjun Zha. 2022. Temporal Complementarity-Guided Reinforcement Learning for Image-to-Video Person Re-Identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 7309–7318.
[38]
Yaxiong Wu, Craig Macdonald, and Iadh Ounis. 2022. Multi-Modal Dialog State Tracking for Interactive Fashion Recommendation. In Proceedings of the ACM Conference on Recommender Systems. ACM, 124–133.
[39]
Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2020. Self-Supervised Reinforcement Learning for Recommender Systems. In Proceedings of the International ACM SIGIR conference on research and development in Information Retrieval. ACM, 931–940.
[40]
Chen Yanai, Adir Solomon, Gilad Katz, Bracha Shapira, and Lior Rokach. 2022. Q-Ball: Modeling Basketball Games Using Deep Reinforcement Learning. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 8806–8813.
[41]
Wenhui Yu, Huidi Zhang, Xiangnan He, Xu Chen, Li Xiong, and Zheng Qin. 2018. Aesthetic-based Clothing Recommendation. In Proceedings of the World Wide Web Conference on World Wide Web. ACM, 649–658.
[42]
Yifei Yuan and Wai Lam. 2021. Conversational fashion image retrieval via multiturn natural language feedback. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 839–848.
[43]
Gangjian Zhang, Shikui Wei, Huaxin Pang, and Yao Zhao. 2021. Heterogeneous Feature Fusion and Cross-modal Alignment for Composed Image Retrieval. In Proceedings of the International ACM Conference on Multimedia. ACM, 5353–5362.
[44]
Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. 2017. Memory-augmented attribute manipulation networks for interactive fashion search. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 1520–1528.
[45]
Haifeng Zhao and Xingjun Wang. 2019. Bi-Group Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the International Conference on Computer Science and Software Engineering. ACM, 35–39.
[46]
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A Deep Reinforcement Learning Framework for News Recommendation. In Proceedings of the World Wide Web Conference on World Wide Web. ACM, 167–176.
Index Terms
- Interactive Garment Recommendation with User in the Loop
Recommendations
Personalized clothing recommendation combining user social circle and fashion style consistency
With the rapid expansion of social networks and fashion websites, clothing recommendation has attracted more attention of researchers, since various web data bring opportunities for recommender systems to solve the problems of cold start and sparsity. ...
Preference-based user rating correction process for interactive recommendation systems
In most of the recommendation systems, user rating is an important user activity that reflects their opinions. Once the users return their ratings about items the systems have suggested, the user ratings can be used to adjust the recommendation ...
Garment Recommendation with Memory Augmented Neural Networks
Pattern Recognition. ICPR International Workshops and ChallengesAbstractFashion plays a pivotal role in society. Combining garments appropriately is essential for people to communicate their personality and style. Also different events require outfits to be thoroughly chosen to comply with underlying social clothing ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
EISSN:1551-6865
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 02 November 2024
Accepted: 13 October 2024
Revised: 21 June 2024
Received: 09 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 66Total Downloads
- Downloads (Last 12 months)66
- Downloads (Last 6 weeks)66
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in