What Song Am I Thinking Of?
Pages 418 - 432
Abstract
Information Need (IN) is a complex phenomenon due to the difficulty experienced when realising and formulating it into a query format. This leads to a semantic gap between the IN and its representation (e.g., the query). Studies have investigated techniques to bridge this gap by using neurophysiological features. Music Information Retrieval (MIR) is a sub-field of IR that could greatly benefit from bridging the gap between IN and query, as songs present an acute challenge for IR systems. A searcher may be able to recall/imagine a piece of music they wish to search for but still need to remember key pieces of information (title, artist, lyrics) used to formulate a query that an IR system can process. Although, if a MIR system could understand the imagined song, it may allow the searcher to satisfy their IN better. As such, in this study, we aim to investigate the possibility of detecting pieces from Electroencephalogram (EEG) signals captured while participants “listen” to or “imagine” songs. We employ six machine learning models on the publicly available data set, OpenMIIR. In the model training phase, we devised several experiment scenarios to explore the capabilities of the models to determine the potential effectiveness of Perceived and Imagined EEG song data in a MIR system. Our results show that, firstly, we can detect perceived songs using the recorded brain signals, with an accuracy of 62.0% (SD 5.4%). Furthermore, we classified imagined songs with an accuracy of 60.8% (SD 13.2%). Insightful results were also gained from several experiment scenarios presented within this paper. Overall, the encouraging results produced by this study are a crucial step towards information retrieval systems capable of interpreting INs from the brain, which can help alleviate the semantic gap’s negative impact on information retrieval.
References
[1]
Allegretti, M., Moshfeghi, Y., Hadjigeorgieva, M., Pollick, F.E., Jose, J.M., Pasi, G.: When relevance judgement is happening? An EEG-based study. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 719–722 (2015)
[2]
Belkin NJ Anomalous states of knowledge as a basis for information retrieval Can. J. Inf. Sci. 1980 5 1 133-143
[3]
Belkin, N.J., Oddy, R.N., Brooks, H.M.: Ask for information retrieval: Part II. Results of a design study. J. Doc. (1982)
[4]
Belkin, N.J., Oddy, R.N., Brooks, H.M.: Ask for information retrieval: Part I. Background and theory. J. Doc. 38, 61–71 (1997)
[5]
Chen JX, Jiang DM, and Zhang YN A hierarchical bidirectional GRU model with attention for EEG-based emotion classification IEEE Access 2019 7 118530-118540
[6]
Chen, X., et al.: Web search via an efficient and effective brain-machine interface. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1569–1572 (2022)
[7]
Cole, C.: A theory of information need for information retrieval that connects information to knowledge. J. Am. Soc. Inf. Sci. Technol. 62(7), 1216–1231 (2011)., https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.21541
[8]
Cole C A theory of information need for information retrieval that connects information to knowledge J. Am. Soc. Inf. Sci. Technol. 2011 62 7 1216-1231
[9]
Davis III, K.M., Kangassalo, L., Spapé, M., Ruotsalo, T.: BrainSourcing: crowdsourcing recognition tasks via collaborative brain-computer interfacing. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2020)
[10]
Eugster, M.J., et al.: Predicting term-relevance from brain signals. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2014, pp. 425–434. Association for Computing Machinery, New York (2014).
[11]
Gao ZK, Li YL, Yang YX, and Ma C A recurrence network-based convolutional neural network for fatigue driving detection from EEG Chaos: Interdisc. J. Nonlinear Sci. 2019 29 11 113126
[12]
Gramfort, A., et al.: MEG and EEG data analysis with MNE-python. Front. Neurosci. 267 (2013)
[13]
Gwizdka J, Hosseini R, Cole M, and Wang S Temporal dynamics of eye-tracking and EEG during reading and relevance decisions J. Am. Soc. Inf. Sci. 2017 68 10 2299-2312
[14]
Hofmann, T.: Collaborative filtering via gaussian probabilistic latent semantic analysis. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 259–266 (2003)
[15]
Hsu, J.L., Zhen, Y.L., Lin, T.C., Chiu, Y.S.: Personalized music emotion recognition using electroencephalography (EEG). In: 2014 IEEE International Symposium on Multimedia, pp. 277–278 (2014).
[16]
Huigen E, Peper A, and Grimbergen C Investigation into the origin of the noise of surface electrodes Med. Biol. Eng. Compu. 2002 40 3 332-338
[17]
Ingwersen P and Järvelin K The Turn: Integration of Information Seeking and Retrieval in Context 2005 Heidelberg Springer
[18]
Jacucci G et al. Integrating neurophysiologic relevance feedback in intent modeling for information retrieval J. Am. Soc. Inf. Sci. 2019 70 9 917-930
[19]
Jordan KG Emergency EEG and continuous EEG monitoring in acute ischemic stroke J. Clin. Neurophysiol. 2004 21 5 341-352
[20]
Kangassalo, L., Spapé, M., Jacucci, G., Ruotsalo, T.: Why do users issue good queries? neural correlates of term specificity. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 375–384 (2019)
[21]
Kangassalo L, Spapé M, Ravaja N, and Ruotsalo T Information gain modulates brain activity evoked by reading Sci. Rep. 2020 10 1 1-10
[22]
Karameh, F.N., Dahleh, M.A.: Automated classification of EEG signals in brain tumor diagnostics. In: Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No. 00CH36334), vol. 6, pp. 4169–4173. IEEE (2000)
[23]
Kauppi JP et al. Towards brain-activity-controlled information retrieval: decoding image relevance from meg signals Neuroimage 2015 112 288-298
[24]
Keshavarz H Human information behaviour and design, development and evaluation of information retrieval systems Program 2008 42 4 391-401
[25]
Kingphai K and Moshfeghi Y Longo L and Leva MC On EEG preprocessing role in deep learning effectiveness for mental workload classification Human Mental Workload: Models and Applications 2021 Cham Springer 81-98
[26]
Kuhlthau CC Inside the search process: information seeking from the user’s perspective J. Am. Soc. Inf. Sci. 1991 42 5 361-371
[27]
Maior, H.A., Ramchurn, R., Martindale, S., Cai, M., Wilson, M.L., Benford, S.: fNIRS and neurocinematics. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–6 (2019)
[28]
Michalkova, D., Parra-Rodriguez, M., Moshfeghi, Y.: Information need awareness: an EEG study. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2022, pp. 610–621. Association for Computing Machinery, New York (2022).
[29]
Midha S, Maior HA, Wilson ML, and Sharples S Measuring mental workload variations in office work tasks using fNIRS Int. J. Hum Comput Stud. 2021 147 102580
[30]
Morita, Y., Huang, H.H., Kawagoe, K.: Towards music information retrieval driven by EEG signals: architecture and preliminary experiments. In: 2013 IEEE/ACIS 12th International Conference on Computer and Information Science (ICIS), pp. 213–217 (2013).
[31]
Moshfeghi, Y.: NeuraSearch: neuroscience and information retrieval. In: CEUR Workshop Proceedings, vol. 2950, pp. 193–194 (2021)
[32]
Moshfeghi Y, Pinto LR, Pollick FE, Jose JM, et al. Serdyukov P et al. Understanding relevance: an fMRI study Advances in Information Retrieval 2013 Heidelberg Springer 14-25
[33]
Moshfeghi, Y., Pollick, F.E.: Search process as transitions between neural states. In: Proceedings of the 2018 World Wide Web Conference, pp. 1683–1692 (2018)
[34]
Moshfeghi Y and Pollick FE Neuropsychological model of the realization of information need J. Am. Soc. Inf. Sci. 2019 70 9 954-967
[35]
Moshfeghi, Y., Triantafillou, P., Pollick, F.: Towards predicting a realisation of an information need based on brain signals. In: The World Wide Web Conference, pp. 1300–1309 (2019)
[36]
Moshfeghi, Y., Triantafillou, P., Pollick, F.E.: Understanding information need: an fMRI study. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2016, pp. 335–344. Association for Computing Machinery, New York (2016).
[37]
Moshfeghi, Y., Triantafillou, P., Pollick, F.E.: Understanding information need: an fMRI study. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 335–344 (2016)
[38]
Ntalampiras S and Potamitis I A statistical inference framework for understanding music-related brain activity IEEE J. Sel. Top. Signal Process. 2019 13 2 275-284
[39]
Paisalnan S, Moshfeghi Y, and Pollick F Neural correlates of realisation of satisfaction in a successful search process Proc. Assoc. Inf. Sci. Technol. 2021 58 1 282-291
[40]
Paisalnan S, Pollick F, Moshfeghi Y, et al. Nicosia G et al. Towards understanding neuroscience of realisation of information need in light of relevance and satisfaction judgement Machine Learning, Optimization, and Data Science 2022 Cham Springer 41-56
[41]
Pang S, Hu X, Cai Z, Gong J, and Zhang M Building change detection from bi-temporal dense-matching point clouds and aerial images Sensors 2018 18 4 966
[42]
Ras ZW and Wieczorkowska A Advances in Music Information Retrieval 2010 Heidelberg Springer
[43]
van Rijsbergen, C.J.: (Invited paper) a new theoretical framework for information retrieval. In: Proceedings of the 9th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 1986, pp. 194–200. Association for Computing Machinery, New York (1986).
[44]
Savolainen R Information need as trigger and driver of information seeking: a conceptual analysis Aslib J. Inf. Manage. 2017 69 2-21
[45]
Schaefer, R.S., Farquhar, J., Blokland, Y., Sadakata, M., Desain, P.: Name that tune: decoding music from the listening brain. NeuroImage 56(2), 843–849 (2011)., https://www.sciencedirect.com/science/article/pii/S1053811910008402,multivariate Decoding and Brain Reading
[46]
Schedl, M., Gómez, E., Urbano, J., et al.: Music information retrieval: recent developments and applications. Found. Trends® Inf. Retrieval 8(2–3), 127–261 (2014)
[47]
Sonawane, D., Miyapuram, K.P., Rs, B., Lomas, D.J.: GuessTheMusic: song identification from electroencephalography response. In: Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD), pp. 154–162 (2021)
[48]
Spape, M., Davis, K., Kangassalo, L., Ravaja, N., Sovijarvi-Spape, Z., Ruotsalo, T.: Brain-computer interface for generating personally attractive images. IEEE Trans. Affect. Comput. 1(1) (2021)
[49]
Stober S Toward studying music cognition with information retrieval techniques: Lessons learned from the OpenMIIR initiative Front. Psychol. 2017 8 1255
[50]
Stober, S.: Toward studying music cognition with information retrieval techniques: lessons learned from the OpenMIIR initiative. Front. Psychol. 8 (2017)., https://www.frontiersin.org/articles/10.3389/fpsyg.2017.01255
[51]
Stober, S., Sternin, A., Owen, A.M., Grahn, J.A.: Towards music imagery information retrieval: introducing the OpenMIIR dataset of EEG recordings from music perception and imagination. In: International Society for Music Information Retrieval Conference (2015)
[52]
Taylor RS The process of asking questions Am. Doc. 1962 13 4 391-396
[53]
Taylor, R.S.: Question-negotiation an information-seeking in libraries. Technical report, Lehigh Univ Bethlehem PA Center for Information Science (1967)
[54]
Taylor RS Question-negotiation and information seeking in libraries Coll. Res. Libr. 1968 76 251-267
[55]
de la Torre-Ortiz, C., Spapé, M.M., Kangassalo, L., Ruotsalo, T.: Brain relevance feedback for interactive image generation. In: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 1060–1070 (2020)
[56]
Tzallas AT, Tsipouras MG, and Fotiadis DI Epileptic seizure detection in EEGs using time-frequency analysis IEEE Trans. Inf Technol. Biomed. 2009 13 5 703-710
[57]
van der Veen V, dutt-Sharma N, Cavallaro L, and Bos H Balzarotti D, Stolfo SJ, and Cova M Memory errors: the past, the present, and the future Research in Attacks, Intrusions, and Defenses 2012 Heidelberg Springer 86-106
[58]
Wissbrock, F.: Information need assessment in information retrieval; beyond lists and queries. In: Proceedings of the 27th German Conference on Artificial Intelligence (2004)
[59]
Ye, Z., et al.: Brain topography adaptive network for satisfaction modeling in interactive information access system. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 90–100 (2022)
[60]
Ye, Z., et al.: Towards a better understanding of human reading comprehension with brain signals. In: Proceedings of the ACM Web Conference 2022, pp. 380–391 (2022)
[61]
Zhang, J., et al.: Global or local: constructing personalized click models for web search. In: Proceedings of the ACM Web Conference 2022, pp. 213–223 (2022)
[62]
Zhao R, Yan R, Wang J, and Mao K Learning to monitor machine health with convolutional bi-directional LSTM networks Sensors 2017 17 2 273
Index Terms
- What Song Am I Thinking Of?
Index terms have been assigned to the content through auto-classification.
Recommendations
GuessTheMusic: Song Identification from Electroencephalography response
CODS-COMAD '21: Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD)The music signal comprises of different features like rhythm, timbre, melody, harmony. Its impact on the human brain has been an exciting research topic for the past several decades. Electroencephalography (EEG) signal enables the non-invasive ...
Chord Progressions Selection Based on Song Audio Features
Hybrid Artificial Intelligent SystemsAbstractA chord progression is an essential building block in music. In the field of music theory is usually assumed that these progressions influence the mood, emotion, genre or other critical aspects of the songs, and also in the perception that they ...
Music similarity and retrieval
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrievalThis tutorial serves as an introductory course to the field of and state-of-the-art in music information retrieval (MIR) and in particular to music similarity estimation which is an essential component of music retrieval. Apart from explaining ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Sep 2023
502 pages
ISBN:978-3-031-53965-7
DOI:10.1007/978-3-031-53966-4
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 15 February 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Jan 2025