[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3583780.3614773acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

AdaMCT: Adaptive Mixture of CNN-Transformer for Sequential Recommendation

Published: 21 October 2023 Publication History

Abstract

Sequential recommendation (SR) aims to model users' dynamic preferences from a series of interactions. A pivotal challenge in user modeling for SR lies in the inherent variability of user preferences. An effective SR model is expected to capture both the long-term and short-term preferences exhibited by users, wherein the former can offer a comprehensive understanding of stable interests that impact the latter. To more effectively capture such information, we incorporate locality inductive bias into the Transformer by amalgamating its global attention mechanism with a local convolutional filter, and adaptively ascertain the mixing importance on a personalized basis through layer-aware adaptive mixture units, termed as AdaMCT. Moreover, as users may repeatedly browse potential purchases, it is expected to consider multiple relevant items concurrently in long-/short-term preferences modeling. Given that softmax-based attention may promote unimodal activation, we propose the Squeeze-Excitation Attention (with sigmoid activation) into SR models to capture multiple pertinent items (keys) simultaneously. Extensive experiments on three widely employed benchmarks substantiate the effectiveness and efficiency of our proposed approach. Source code is available at https://github.com/juyongjiang/AdaMCT.

References

[1]
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural news recommendation with long-and short-term user representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 336--345.
[2]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
[3]
Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior sequence transformer for e-commerce recommendation in alibaba. In Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data. 1--4.
[4]
Tianwen Chen and Raymond Chi-Wing Wong. 2019. Session-based recommendation with local invariance. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 994--999.
[5]
Sung Min Cho, Eunhyeok Park, and Sungjoo Yoo. 2020. MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation. In Fourteenth ACM Conference on Recommender Systems. 515--520.
[6]
Evangelia Christakopoulou and George Karypis. 2016. Local item-item models for top-n recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. 67--74.
[7]
Qiang Cui, Shu Wu, Qiang Liu, Wen Zhong, and Liang Wang. 2018. MV-RNN: A multi-view recurrent neural network for sequential recommendation. IEEE Transactions on Knowledge and Data Engineering, Vol. 32, 2 (2018), 317--331.
[8]
Robin Devooght and Hugues Bersini. 2017. Long and short-term recommendations with recurrent neural networks. In Proceedings of the 25th conference on user modeling, adaptation and personalization. 13--21.
[9]
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021. Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth. arXiv preprint arXiv:2103.03404 (2021).
[10]
Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. 2020. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Transactions on Information Systems (TOIS), Vol. 39, 1 (2020), 1--42.
[11]
Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. 2020. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Learning. PMLR, 3800--3809.
[12]
Jiayan Guo, Peiyan Zhang, Chaozhuo Li, Xing Xie, Yan Zhang, and Sunghun Kim. 2022. Evolutionary Preference Learning via Graph Nested GRU ODE for Session-based Recommendation. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 624--634.
[13]
Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yinglong Wang, Jun Ma, and Mohan Kankanhalli. 2019. Attentive long short-term preference modeling for personalized product search. ACM Transactions on Information Systems (TOIS), Vol. 37, 2 (2019), 1--27.
[14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[15]
Ruining He, Wang-Cheng Kang, and Julian McAuley. 2017a. Translation-based recommendation. In Proceedings of the eleventh ACM conference on recommender systems. 161--169.
[16]
Ruining He and Julian McAuley. 2016. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 191--200.
[17]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017b. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. 173--182.
[18]
Yun He, Yin Zhang, Weiwen Liu, and James Caverlee. 2020. Consistency-Aware Recommendation for User-Generated Item List Continuation. In Proceedings of the 13th International Conference on Web Search and Data Mining. 250--258.
[19]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based recommendations with recurrent neural networks. In Proceedings of Fourth International Conference on Learning Representations (ICLR'16).
[20]
Yupeng Hou, Binbin Hu, Zhiqiang Zhang, and Wayne Xin Zhao. 2022. Core: simple and effective session-based recommendation within consistent representation space. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 1796--1801.
[21]
Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Tianchi Yang. 2018b. Local and global information fusion for top-n recommendation in heterogeneous information network. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 1683--1686.
[22]
Jie Hu, Li Shen, and Gang Sun. 2018a. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7132--7141.
[23]
Liang Hu, Longbing Cao, Shoujin Wang, Guandong Xu, Jian Cao, and Zhiping Gu. 2017. Diversifying Personalized Recommendation with User-session Context. In IJCAI. 1858--1864.
[24]
Xiaowen Huang, Shengsheng Qian, Quan Fang, Jitao Sang, and Changsheng Xu. 2020. Meta-path Augmented Sequential Recommendation with Contextual Co-attention Network. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 16, 2 (2020), 1--24.
[25]
Mingi Ji, Weonyoung Joo, Kyungwoo Song, Yoon-Yeong Kim, and Il-Chul Moon. 2020. Sequential Recommendation with Relation-Aware Kernelized Self-Attention. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4304--4311.
[26]
Juyong Jiang, Yingtao Luo, Jae Boum Kim, Kai Zhang, and Sunghun Kim. 2021. Sequential Recommendation with Bidirectional Chronological Augmentation of Transformer. arXiv preprint arXiv:2112.06460 (2021).
[27]
Kyeongpil Kang, Junwoo Park, Wooyoung Kim, Hojung Choe, and Jaegul Choo. 2019. Recommender system using sequential and global preference via attention mechanism and topic modeling. In Proceedings of the 28th ACM international conference on information and knowledge management. 1543--1552.
[28]
Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM). IEEE, 197--206.
[29]
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer, Vol. 42, 8 (2009), 30--37.
[30]
Chaozhuo Li, Bochen Pang, Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. 2021. Adsgnn: Behavior-graph augmented relevance modeling in sponsored search. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 223--232.
[31]
Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining. 322--330.
[32]
Jing Lin, Weike Pan, and Zhong Ming. 2020. FISSA: fusing item similarity models with self-attention networks for sequential recommendation. In Fourteenth ACM Conference on Recommender Systems. 130--139.
[33]
Huafeng Liu, Liping Jing, Jingxuan Wen, Zhicheng Wu, Xiaoyi Sun, Jiaqi Wang, Lin Xiao, and Jian Yu. 2020a. Deep Global and Local Generative Model for Recommendation. In Proceedings of The Web Conference 2020. 551--561.
[34]
Huiting Liu, Huimin Liu, Qiang Ji, Peng Zhao, and Xindong Wu. 2020b. Collaborative deep recommendation with global and local item correlations. Neurocomputing, Vol. 385 (2020), 278--291.
[35]
Qiang Liu, Shu Wu, Diyi Wang, Zhaokang Li, and Liang Wang. 2016b. Context-aware sequential recommendation. In 2016 IEEE 16th International Conference on Data Mining (ICDM). IEEE, 1053--1058.
[36]
Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016a. Predicting the next location: A recurrent model with spatial and temporal contexts. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30.
[37]
Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018. STAMP: short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1831--1839.
[38]
Mi Luo, Fei Chen, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Jiashi Feng, and Zhenguo Li. 2020. Metaselector: Meta-learning for recommendation with user-level adaptive model selection. In Proceedings of The Web Conference 2020. 2507--2513.
[39]
Yingtao Luo, Qiang Liu, and Zhaocheng Liu. 2021. STAN: Spatio-Temporal Attention Network for Next Location Recommendation. In Proceedings of The Web Conference.
[40]
Chen Ma, Peng Kang, and Xue Liu. 2019. Hierarchical gating networks for sequential recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 825--833.
[41]
Chen Ma, Liheng Ma, Yingxue Zhang, Jianing Sun, Xue Liu, and Mark Coates. 2020. Memory Augmented Graph Neural Networks for Sequential Recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 5045--5052.
[42]
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 43--52.
[43]
Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International Conference on Data Mining. IEEE, 995--1000.
[44]
Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web. 811--820.
[45]
Jun Song, Yueyang Wang, Siliang Tang, Yin Zhang, Zhigang Chen, Zhongfei Zhang, Tong Zhang, and Fei Wu. 2020. Local-Global Memory Neural Network for Medication Prediction. IEEE transactions on neural networks and learning systems (2020).
[46]
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management. 1441--1450.
[47]
Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, and Xia Hu. 2021. Sparse-interest network for sequential recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 598--606.
[48]
Jiaxi Tang and Ke Wang. 2018a. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 565--573.
[49]
Jiaxi Tang and Ke Wang. 2018b. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 565--573.
[50]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
[51]
Jianling Wang, Kaize Ding, Liangjie Hong, Huan Liu, and James Caverlee. 2020. Next-item Recommendation with Sequential Hypergraphs. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1101--1110.
[52]
Yiqi Wang, Chaozhuo Li, Zheng Liu, Mingzheng Li, Jiliang Tang, Xing Xie, Lei Chen, and Philip S Yu. 2022. An adaptive graph pre-training framework for localized collaborative filtering. ACM Transactions on Information Systems, Vol. 41, 2 (2022), 1--27.
[53]
Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack. 2020. SSE-PT: Sequential recommendation via personalized transformer. In Fourteenth ACM Conference on Recommender Systems. 328--337.
[54]
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-based recommendation with graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 346--353.
[55]
Ruobing Xie, Yalong Wang, Rui Wang, Yuanfu Lu, Yuanhang Zou, Feng Xia, and Leyu Lin. 2022. Long short-term temporal meta-learning in online recommendation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 1168--1176.
[56]
Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. 2019a. Graph Contextualized Self-Attention Network for Session-based Recommendation. In IJCAI, Vol. 19. 3940--3946.
[57]
Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Jiajie Xu, Victor S Sheng S. Sheng, Zhiming Cui, Xiaofang Zhou, and Hui Xiong. 2019b. Recurrent convolutional neural network for sequential recommendation. In The World Wide Web Conference. 3398--3404.
[58]
Yong Xu, Jiahui Chen, Chao Huang, Bo Zhang, Hao Xing, Peng Dai, and Liefeng Bo. 2020. Joint Modeling of Local and Global Behavior Dynamics for Session-Based Recommendation. In ECAI 2020. IOS Press, 545--552.
[59]
Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016. A dynamic recurrent model for next basket recommendation. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 729--732.
[60]
Feng Yu, Yanqiao Zhu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2020. TAGNN: Target attentive graph neural networks for session-based recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1921--1924.
[61]
Lu Yu, Chuxu Zhang, Shangsong Liang, and Xiangliang Zhang. 2019b. Multi-order attentive ranking model for sequential recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 5709--5716.
[62]
Zeping Yu, Jianxun Lian, Ahmad Mahmoody, Gongshen Liu, and Xing Xie. 2019a. Adaptive User Modeling with Long and Short-Term Preferences for Personalized Recommendation. In IJCAI. 4213--4219.
[63]
Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M Jose, and Xiangnan He. 2019. A simple convolutional generative network for next item recommendation. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 582--590.
[64]
Peiyan Zhang, Jiayan Guo, Chaozhuo Li, Yueqi Xie, Jae Boum Kim, Yan Zhang, Xing Xie, Haohan Wang, and Sunghun Kim. 2023. Efficiently leveraging multi-level user intent for session-based recommendation via atten-mixer network. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining. 168--176.
[65]
Peiyan Zhang and Sunghun Kim. 2023. A Survey on Incremental Update for Neural Recommender Systems. arXiv preprint arXiv:2303.02851 (2023).
[66]
Qinzhe Zhang, Jia Wu, Hong Yang, Weixue Lu, Guodong Long, and Chengqi Zhang. 2016. Global and local influence-based social recommendation. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. 1917--1920.
[67]
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), Vol. 52, 1 (2019), 1--38.
[68]
Yi Zhao, Chaozhuo Li, Jiquan Peng, Xiaohan Fang, Feiran Huang, Senzhang Wang, Xing Xie, and Jibing Gong. 2023. Beyond the Overlapping Users: Cross-Domain Recommendation via Adaptive Anchor Link Learning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1488--1497.
[69]
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 5941--5948.

Cited By

View all
  • (2025)Let long-term interests talk: An disentangled learning model for recommendation based on short-term interests generationInformation Processing & Management10.1016/j.ipm.2024.10399762:2(103997)Online publication date: Mar-2025
  • (2025)Implicit local–global feature extraction for diffusion sequence recommendationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109471139(109471)Online publication date: Jan-2025
  • (2025)A computational intelligence-based approach for detecting drowsiness in sleep apnea patients using multimodal bio-signalsIntelligent Computing Techniques in Biomedical Imaging10.1016/B978-0-443-15999-2.00010-4(185-199)Online publication date: 2025
  • Show More Cited By

Index Terms

  1. AdaMCT: Adaptive Mixture of CNN-Transformer for Sequential Recommendation

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management
    October 2023
    5508 pages
    ISBN:9798400701245
    DOI:10.1145/3583780
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. CNNs
    2. sequential recommendation
    3. transformer

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CIKM '23
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

    Upcoming Conference

    CIKM '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)305
    • Downloads (Last 6 weeks)18
    Reflects downloads up to 13 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Let long-term interests talk: An disentangled learning model for recommendation based on short-term interests generationInformation Processing & Management10.1016/j.ipm.2024.10399762:2(103997)Online publication date: Mar-2025
    • (2025)Implicit local–global feature extraction for diffusion sequence recommendationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109471139(109471)Online publication date: Jan-2025
    • (2025)A computational intelligence-based approach for detecting drowsiness in sleep apnea patients using multimodal bio-signalsIntelligent Computing Techniques in Biomedical Imaging10.1016/B978-0-443-15999-2.00010-4(185-199)Online publication date: 2025
    • (2024)TCGC: Temporal Collaboration-Aware Graph Co-Evolution Learning for Dynamic RecommendationACM Transactions on Information Systems10.1145/368747043:1(1-27)Online publication date: 27-Aug-2024
    • (2024)DNS-Rec: Data-aware Neural Architecture Search for Recommender SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688117(591-600)Online publication date: 8-Oct-2024
    • (2024)CMG: A Causality-enhanced Multi-view Graph Model for Stock Trend PredictionProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679886(3699-3703)Online publication date: 21-Oct-2024
    • (2024)Modeling User Fatigue for Sequential RecommendationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657802(996-1005)Online publication date: 10-Jul-2024
    • (2024)Diffusion Recommendation with Implicit Sequence InfluenceCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3651951(1719-1725)Online publication date: 13-May-2024
    • (2024)Is Contrastive Learning Necessary? A Study of Data Augmentation vs Contrastive Learning in Sequential RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645661(3854-3863)Online publication date: 13-May-2024
    • (2024)Enhancement in Context-Aware Recommender Systems – A Systematic Review2024 Second International Conference on Emerging Trends in Information Technology and Engineering (ICETITE)10.1109/ic-ETITE58242.2024.10493725(1-13)Online publication date: 22-Feb-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media