More Web Proxy on the site http://driver.im/

poster

The Human-Built Environment-Natural Environment Relation - An Immersive Multisensory Exploration with 'System of a Sound'

Authors:

Rodolfo Ocampo,

Caroline Pegram,

Adrian Schmidt,

Brendan WrightAuthors Info & Claims

IUI '23 Companion: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces

Pages 8 - 11

https://doi.org/10.1145/3581754.3584119

Published: 27 March 2023 Publication History

Abstract

The relationship between human activity, the built environment, such as homes, offices, and schools, and the surrounding natural environment hide interaction patterns that can be better understood. Intelligent user interfaces today use dashboards with tables and figures, missing a way where people can innately relate to this relationship. We explored emotion-oriented data sonification as an experimental way for people to engage with data. Our audio-visual and gestural intelligent user interface, "System of a Sound", takes real-time data streams centred on the location where it is installed and uses a large language model to choose music samples that elicit matching emotions. The samples are combined using an artificial intelligence music engine to compose a live soundscape that reveals interaction patterns between human activity, the built environment, and the surrounding natural environment. This exciting IUI Demo offers conference participants emotion-oriented data sonification for data exploration.

Supplementary Material

MP4 File (soas-video.mp4)

The Human-Built Environment-Natural Environment Relation - An Immersive Multisensory Exploration with 'System of a Sound'

Download
93.17 MB

References

[1]

Andrea Agostinelli 2023. MusicLM: Generating Music From Text. Google Research (2023)

[2]

Josh Andres. 2022. Adaptive human bodies and adaptive built environments for enriching futures. Frontiers in Computer Science 4, (2022). https://www.frontiersin.org/articles/10.3389/fcomp.2022.931973

[3]

Josh Andres, Alex Zafiroglu, Katherine Daniell, Paul Wong, Mina Henein, Xuanying Zhu, Ben Sweeting, Michael Arnold, De Pembrey MacNamara, and Ariela Helfgott. 2022. Cybernetic lenses for designing and living in a complex world. In Extended Abstracts of the 2022 OzCHI Conference on Human Factors in Computing Systems (OZCHI EA ’22). Association for Computing Machinery, New York, NY, USA. (2022) (2022).

Digital Library

[4]

Stewart Brand. 2018. Pace Layering: How Complex Systems Learn and Keep Learning. Journal of Design and Science (January 2018).

[5]

Jean-Pierre Briot and François Pachet. 2020. Deep learning for music generation: challenges and directions. Neural Computing and Applications 32, 4 (2020), 981–993.

Digital Library

[6]

Berry Eggen. 2016. Interactive Soundscapes of the Future Everyday Life. In Peripheral Interaction: Challenges and Opportunities for HCI in the Periphery of Attention, Saskia Bakker, Doris Hausen and Ted Selker (eds.). Springer International Publishing, Cham, 239–251.

[7]

Mara Helmuth and Margaret Schedel. 2022. Links between sonification and generative music. Interdisciplinary Science Reviews 47, 2 (April 2022), 215–242.

[8]

Thomas Hermann and Andy Hunt. 2004. The importance of interaction in sonification. (2004).

[9]

Thomas Hermann, Andy Hunt, and John G. Neuhoff. 2011. The sonification handbook. Logos Verlag Berlin.

[10]

William Jackson, Robert Argen, Nic Bax, Graeme Clark, Sarah Coleman, Ian Cresswell, Kathryn Emmerson, Karen Evans, Mark Hibberd, Emma Johnston, Melita Keywood, Andrew Klekociuk, Richard Mackay, Dan Metcalfe, Helen Murphy, Alex Rankin, David C. Smith, and Barbara Wienecke. 2017. Australia state of the environment 2016: overview. Department of the Environment and Energy (Australia). https://apo.org.au/node/74267

[11]

Minoru Kobayashi and Chris Schmandt. 1997. Dynamic Soundscape: Mapping Time to Space for Audio Browsing. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI ’97), Association for Computing Machinery, New York, NY, USA, 194–201.

Digital Library

[12]

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Such, Kenny Hsu, and Lilian Weng. 2022. Text and Code Embeddings by Contrastive Pre-Training.

[13]

Rodolfo Ocampo, Josh Andres, Adrian Schmidt, Caroline Pegram, Justin Shave, Charlton Hill, Brendan Wright, and Oliver Bown. 2023. Using GPT-3 to achieve semantically relevant data sonificiation for an art installation. International Conference on Evolutionary and Biologically Inspired Music, Sound, Art and Design (2023).

Digital Library

[14]

Amit Singhal. 2001. Modern Information Retrieval: A Brief Overview. IEEE Data Engineering Bulletin. 24.

[15]

Brianna J. Tomlinson, Bruce N. Walker, and Emily B. Moore. 2020. Auditory Display in Interactive Science Simulations: Description and Sonification Support Interaction and Enhance Opportunities for Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), Association for Computing Machinery, New York, NY, USA, 1–12.

Digital Library

[16]

Built environment and health. Australian Institute of Health and Welfare. https://www.aihw.gov.au/reports/australias-health/built-environment-and-health

[17]

Uncanny Valley. https://uncannyvalley.com.au/

[18]

Next-Generation Pose Detection with MoveNet and TensorFlow.js. https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html

Cited By

Aseniero BLee MWang YZhou QShahmansouri NGoldstein R(2024)Experiential Views: Towards Human Experience Evaluation of Designed Spaces using Vision-Language ModelsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650815(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650815
Hutson JHutson PHarper-Nichols MHutson JHutson PHarper-Nichols M(2024)Conclusion: Future Applications, Technologies, and DirectionsSensemaking and Neuroaesthetics10.1007/978-3-031-58045-1_5(243-271)Online publication date: 9-May-2024
https://doi.org/10.1007/978-3-031-58045-1_5

Index Terms

The Human-Built Environment-Natural Environment Relation - An Immersive Multisensory Exploration with 'System of a Sound'

Index terms have been assigned to the content through auto-classification.

Recommendations

Immersive auditory display system 'sound cask': three-dimensional sound field reproduction system based on the boundary surface control principle
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Sound cask was developed to realize the perfect 3D auditory display that creates 3D sound waves around the listener's head just the same as the primary sound field, based on the boundary surface control (BoSC) principle.

If we consider the sound ...
Datascaping: Data Sonification as a Narrative Device in Soundscape Composition
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences

Soundscape composition is an art form that has grown from acoustic ecology and soundscape studies. Current practices foster a wide range of approaches, from the educational and documentary function of the world soundscape project (WSP) to the creation ...
The immersive computer-controlled audio sound theater: history and current trends in multi-modal sound diffusion
SIGGRAPH '09: SIGGRAPH 2009: Talks

Multi-channel sound diffusion has been an essential part of the electronic and computer music performance from the beginnings of the genre in the 1950's. We see early experimentation in sound spatialization in Varèse's use of loudspeaker "paths" in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '23 Companion: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces

March 2023

266 pages

ISBN:9798400701078

DOI:10.1145/3581754

Copyright © 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 March 2023

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

IUI '23

Sponsor:

IUI '23: 28th International Conference on Intelligent User Interfaces

March 27 - 31, 2023

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)73
Downloads (Last 6 weeks)8

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aseniero BLee MWang YZhou QShahmansouri NGoldstein R(2024)Experiential Views: Towards Human Experience Evaluation of Designed Spaces using Vision-Language ModelsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650815(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650815
Hutson JHutson PHarper-Nichols MHutson JHutson PHarper-Nichols M(2024)Conclusion: Future Applications, Technologies, and DirectionsSensemaking and Neuroaesthetics10.1007/978-3-031-58045-1_5(243-271)Online publication date: 9-May-2024
https://doi.org/10.1007/978-3-031-58045-1_5

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents