More Web Proxy on the site http://driver.im/

research-article

Temporal interaction between an artificial orchestra conductor and human musicians

Authors:

Dennis Reidsma,

Pieter BosAuthors Info & Claims

Computers in Entertainment (CIE), Volume 6, Issue 4

Article No.: 53, Pages 1 - 22

https://doi.org/10.1145/1461999.1462005

Published: 24 December 2008 Publication History

Abstract

The Virtual Conductor project concerns the development of the first properly interactive virtual orchestra conductor—a Virtual Human that can conduct a piece of music through interaction with musicians, leading and following them while they are playing. This article describes our motivation for developing such a system; related work in the areas of ambient entertainment and coordinated timing, automatic music processing, virtual humans, and conducting; the design, implementation, and evaluation of the Virtual Conductor, and, finally, contains a discussion of the resulting system and expected developments in the (still ongoing) Virtual Conductor project.

References

[1]

Alonso, M., David, B., and Richard, G. 2004. Tempo and beat estimation of musical signals. In Proceedings International Conference on Music Information Retrieval. 158--163.

[2]

Bailenson, J. and Yee, N. 2005. Digital chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological Science 16, 10 (Oct.), 814--819.

[3]

Bos, P. 2007. Interacting with a virtual conductor. M. S. thesis, Human Media Interaction, University of Twente, NL.

[4]

Brown, J. 1991. Calculation of a constant q spectral transform. J. Acoustical Society of America 89, 1, 425--434.

[5]

Carse, A. 1935. Orchestral Conducting. Augener Ltd.

[6]

Crown, C. L. 1991. Coordinated interpersonal timing of vision and voice as a function of interpersonal attraction. J. Language Social Psychology 10, 1, 29--46.

[7]

Dannenberg, R. and Hu, N. 2003. Polyphonic audio matching for score following and intelligent audio editors. In Proceedings of the Interactional Computer Music Conference. International Computer Music Association, 27--33.

[8]

Dipaola, S. and Arya, A. 2006. Emotional remapping of music to facial animation. In Proceedings of the ACM SIGGRAPH Symposium on Videogames (Sandbox '06). ACM, New York, 143--149.

Digital Library

[9]

Dixon, S. 2003. On the analysis of musical expression in audio signals. In Storage and Retrieval for Media Databases, SPIE vol. 5021. 122--132.

[10]

Dixon, S. 2005. Live tracking of musical performances using on-line time warping. In Proceedings of the 8th International Conference on Digital Audio Effects. 92--97.

[11]

Fuelberth, R. J. V. 2004. The effect of various left-hand conducting gestures on perceptions of anticipated vocal tension in singers. Int. J. Research Choral Singing 2, 1 (Jan.), 27--38.

[12]

Galkin, E. W. 1988. A History off Orchestral Conducting: In Theory and Practice. Stuyvesant, New York.

[13]

Goto, M. 2001. An audio-based real-time beat tracking system for music with or without drumsounds. J. New Music Research 3, 2 (June), 159--171.

[14]

Gouyon, F. and Dixon, S. 2005. A review of automatic rhythm description systems. Computer Music J. 29, 1 (Feb.), 34--54.

Digital Library

[15]

Gouyon, F., Klapuri, A., Dixon, S., Alonso, M., Tzanetakis, G., Uhle, C., and Cano, P. 2006. An experimental comparison of audio tempo induction algorithms. IEEE Trans. Speech Audio Processing 14, 5 (Sept.), 1832--1844.

Digital Library

[16]

Ilmonen, T. and Takala, T. 1999. Conductor following with artificial neural networks. In Proceedings of the International Computer Music Conference (ICMC, Beijing), 367--370.

[17]

Jansen, L., Van Dijk, B., and Retra, J. 2006. Musical multimodal child computer interaction. In Proceedings of the 2006 Conference on Interaction Design and Children (IDC). ACM, New York, 163--164.

Digital Library

[18]

Klapuri, A., Eronen, A., and Astola, J. 2006. Analysis of the meter of acoustic musical signals. IEEE Trans. Speech Audio Processing 14, 1 (Jan.), 342--355.

Digital Library

[19]

Kolesnik, P. and Wanderley, M. 2004. Recognition, analysis and performance with expressive conducting gestures. In Proceedings of the International Computer Music Conference (ICMC).

[20]

Lee, E., Grüll, I., Kiel, H., and Borchers, J. 2006. CONGA: A framework for adaptive conducting gesture analysis. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME, IRCAM, Centre Pompidou, Paris), 260--265.

Digital Library

[21]

Lee, M., Garnett, G., and Wessel, D. 1992. An adaptive conductor follower. In Proceedings of the International Computer Music Conference. International Computer Music Association, 454--455.

[22]

Mancini, M., Bresin, R., and Pelachaud, C. 2007. A virtual head driven by music expressivity. IEEE Trans. Audio, Speech, and Language Processing 15, 6, 1833--1841.

Digital Library

[23]

Michalowski, M. P., Sabanovic, S., and Kozima, H. 2007. A dancing robot for rhythmic social interaction. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), ACM, New York, 89--96.

Digital Library

[24]

Murphy, D., Andersen, T. H., and Jensen, K. 2003. Conducting audio files via computer vision. In Proceedings of the GW03. 529--540.

[25]

Nagaoka, C., Komori, M., and Yoshikawa, S. 2007. Embodied synchrony in conversation. In Conversational Informatics: An Engineering Approach, L. Nishida (ed.), Wiley, 331--351.

[26]

Nakra, T. M. 2000. Inside the conductor's jacket. Ph.D. dissertation, MIT Media Laboratory. Cambridge, MA.

[27]

Nijholt, A., Reidsma, D., Van Welbergen, H., Op Den Akker, H., and Ruttkay, Z. M. 2008. Mutually coordinated anticipatory multimodal interaction. In Proceedings of the COST. Springer, Berlin.

Digital Library

[28]

Pardo, B. and Birmingham, W. P. 2005. Modeling form for on-line following of musical performances. In AAAI. 1018--1023.

Digital Library

[29]

Poggi, I. 2002. The lexicon of the conductor's face. In Language, Vision and Music. John Benjamins, 271--284.

[30]

Prausnitz, F. 1983. Score and Podium: A Complete Guide to Conducting. W. W. Norton, New York.

[31]

Ramseyer, F. and Tschacher, W. 2008. Synchrony in dyadic psychotherapy sessions. In Simultaneity: Temporal Structures and Observer Perspectives. World Scientific, Singapore, Ch. 18.

[32]

Raphael, C. 2004. A hybrid graphical model for aligning polyphonic audio with musical scores. In Proceedings of the International Conference on Music Information Retrieval.

[33]

Reidsma, D., Van Welbergen, H., Poppe, R., Bos, P., and Nijholt, A. 2006. Towards bidirectional dancing interaction. In Lecture Notes in Computer Science 4161, R. Harper et al. (eds.) Springer, Berlin, 1--12.

Digital Library

[34]

Robins, B., Dautenhahn, K., Nehaniv, C. L., Mirza, N. A., Francois, D., and Olsson, L. 2005. Sustaining interaction dynamics and engagement in dyadic child-robot interaction kinesics: Lessons learnt from an exploratory study. In Proceedings of the 14th IEEE International Workshop on Robot and Human Interactive Communication (ROMAN).

[35]

Rudolph, M. 1995. The Grammar of Conducting: A Comprehensive Guide to Baton Technique and Interpretation (3^rd ed.), Schirmer.

[36]

Ruttkay, Z. M., Huang, A., and Eliëns, A. 2003. The conductor: Gestures for embodied agents with logic programming. In Proceedings of the 2nd Hungarian Computer Graphics Conference. 9--16.

[37]

Scheirer, E. D. 1998. Tempo and beat analysis of acoustic musical signals. J. Acoustical Society America 103, 1 (Jan.), 558--601.

[38]

Schögler, B. 1998. Music as a tool in communications research. Nordic J. Music Therapy 7, 1, 40--49.

[39]

Schwarz, D., Orio, N., and Schnell, N. 2004. Robust polyphonic midi score following with hidden markov models. In Proceedings of the International Computer Music Conference (ICMC).

[40]

Segen, J. 1999. Visual conductor. In Proceedings of the ACM SIGGRAPH '99 Conference: Conference abstracts and applications. ACM, New York, 180.

Digital Library

[41]

Seppänen, J., Eronen, A., and Hiipakka, J. 2006. Joint beat and tatum tracking from music signals. In Proceedings of the 7th International Conference on Music Information Retrieval, R. Dannenberg et al. (eds.), 23--28.

[42]

Shiratori, T., Nakazawa, A., and Ikeuchi, K. 2006. Dancing-to-music character animation. Computer Graphics Forum 25, 3 (Sept.), 449--458.

[43]

Skadsem, J. A. 1997. Effect of conductor verbalization, dynamic markings, conductor gesture, and choir dynamic level on singers' dynamic responses. J. Research in Music Education 45, 4, 509--520.

[44]

Suzuki, N., Takeuchi, Y., Ishii, K., and Okada, M. 2003. Effects of echoic mimicry using hummed sounds on human-computer interaction. Speech Communication 40, 4, 559--573.

Digital Library

[45]

Tanaka, H. and Suzuki, H. 2004. Dance interaction with qrio: A case study for non-boring interaction by using an entrainment ensemble model. In Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication (ROMAN), IEEE, 419--424.

[46]

Ter Maat, M., Ebbers, R. M., Reidsma, D., and Nijholt, A. 2008. Beyond the beat: Modeling intentions in a virtual conductor. In Proceedings of the INTETAIN. ACM, New York.

Digital Library

[47]

Tomida, T., Ishihara, A., Ueki, A., Tomari, Y., Fukushima, K., and Inakage, M. 2007. MiXer: The communication entertainment content by using “entrainment phenomenon” and “bio-feedback”. In Advances in Computer Entertainment Technology, M. Inakage et al. (eds.), ACM, New York, 286--287.

Digital Library

[48]

Wang, T.-S., Zheng, N.-N., Li, Y., Xu, Y.-Q., and Shum, H.-Y. 2003. Learning kernel-based HMMs for dynamic sequence synthesis. Graphical Models 65, 4, 206--221.

Digital Library

[49]

Weinberg, G. and Driscoll, S. 2006. Robot-human interaction with an anthropomorphic percussionist. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, New York, 1229--1232.

Digital Library

[50]

Yin, J., Wang, Y., and Hsu, D. 2005. Digital violin tutor: An integrated system for beginning violin learners. In Proceedings of the 13th Annual ACM International Conference on Multimedia. ACM, New York, 976--985.

Digital Library

Cited By

Liu FChen DZhou RYang SXu F(2022)Self-Supervised Music Motion Synchronization Learning for Music-Driven Conducting Motion GenerationJournal of Computer Science and Technology10.1007/s11390-022-2030-z37:3(539-558)Online publication date: 31-May-2022
https://doi.org/10.1007/s11390-022-2030-z
Fujishiro IKobayashi A(2021)[Invited Paper] Ambient Music Co-player: Generating Affective Video in Response to Impromptu Music PerformanceITE Transactions on Media Technology and Applications10.3169/mta.9.29:1(2-12)Online publication date: 2021
https://doi.org/10.3169/mta.9.2
Heggli OKonvalinka IKringelbach MVuust P(2021)A metastable attractor model of self–other integration (MEAMSO) in rhythmic synchronizationPhilosophical Transactions of the Royal Society B: Biological Sciences10.1098/rstb.2020.0332376:1835(20200332)Online publication date: 23-Aug-2021
https://doi.org/10.1098/rstb.2020.0332
Show More Cited By

Index Terms

Recommendations

Human Coded Orchestra: a System for Extemporary Group Singing Performance
ACE '16: Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology

We propose Human Coded Orchestra, a new approach to enable a group of individuals to sing in harmony by using computed directional speakers. The possibilities of musical performance by an untrained group have been explored in the fields of science and ...
Virtual rap dancer: invitation to dance
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing Systems

This paper presents a virtual rap dancer that is able to dance to the beat of music coming in from music recordings, beats obtained from music, voice or other input through a microphone, motion beats detected in the video stream of a human dancer, or ...
A high-fidelity orchestra simulator for individual musicians' practice

We developed the Open Orchestra system to provide individual musicians with a high-fidelity experience of ensemble rehearsal or performance, combined with the convenience and flexibility of solo study. This builds on the theme of an immersive orchestral ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computers in Entertainment

Computers in Entertainment Volume 6, Issue 4

SPECIAL ISSUE: Media Arts (Part II)

December 2008

218 pages

EISSN:1544-3574

DOI:10.1145/1461999

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 December 2008

Accepted: 01 May 2008

Received: 01 March 2008

Published in CIE Volume 6, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
718
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu FChen DZhou RYang SXu F(2022)Self-Supervised Music Motion Synchronization Learning for Music-Driven Conducting Motion GenerationJournal of Computer Science and Technology10.1007/s11390-022-2030-z37:3(539-558)Online publication date: 31-May-2022
https://doi.org/10.1007/s11390-022-2030-z
Fujishiro IKobayashi A(2021)[Invited Paper] Ambient Music Co-player: Generating Affective Video in Response to Impromptu Music PerformanceITE Transactions on Media Technology and Applications10.3169/mta.9.29:1(2-12)Online publication date: 2021
https://doi.org/10.3169/mta.9.2
Heggli OKonvalinka IKringelbach MVuust P(2021)A metastable attractor model of self–other integration (MEAMSO) in rhythmic synchronizationPhilosophical Transactions of the Royal Society B: Biological Sciences10.1098/rstb.2020.0332376:1835(20200332)Online publication date: 23-Aug-2021
https://doi.org/10.1098/rstb.2020.0332
Sasaki KInoue T(2019)Coordinating Real-Time Serial Cooperative Work by Cuing the Order in the Case of Theatrical Performance PracticeMobile Information Systems10.1155/2019/45459172019(1-10)Online publication date: 13-Feb-2019
https://doi.org/10.1155/2019/4545917
Olafsson SKimani EAsadi RBickmore T(2017)That’s a RapIntelligent Virtual Agents10.1007/978-3-319-67401-8_41(325-334)Online publication date: 26-Aug-2017
https://doi.org/10.1007/978-3-319-67401-8_41
Kobayashi AFujishiro I(2016)An Affective Video Generation System Supporting Impromptu Musical Performance2016 International Conference on Cyberworlds (CW)10.1109/CW.2016.11(17-24)Online publication date: Sep-2016
https://doi.org/10.1109/CW.2016.11
Takatsu RMaki YInoue TOkada KShigeno H(2016)Multiple virtual conductors allow amateur orchestra players to perform better and more easily2016 IEEE 20th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD.2016.7566038(486-491)Online publication date: May-2016
https://doi.org/10.1109/CSCWD.2016.7566038
van Welbergen HYaghoubzadeh RKopp S(2014)AsapRealizer 2.0: The Next Steps in Fluent Behavior Realization for ECAsIntelligent Virtual Agents10.1007/978-3-319-09767-1_56(449-462)Online publication date: 2014
https://doi.org/10.1007/978-3-319-09767-1_56
Reidsma DRadha MNijholt A(2014)Mediated Interactions and Musical Expression—A SurveyDigital Da Vinci10.1007/978-1-4939-0536-2_4(79-98)Online publication date: 12-Apr-2014
https://doi.org/10.1007/978-1-4939-0536-2_4
Park TLee ULee BLee HSon SSong SSong JBruckman ACounts SLampe CTerveen L(2013)ExerSyncProceedings of the 2013 conference on Computer supported cooperative work10.1145/2441776.2441823(409-422)Online publication date: 23-Feb-2013
https://dl.acm.org/doi/10.1145/2441776.2441823
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents