Abstract
In this paper we introduce a video post-processing method that enhances the rhythm of a dancing performance, in the sense that the dancing movements are more in time to the beat of the music. The dancing performance as observed in a video is analyzed and segmented into motion intervals delimited by motion beats. We present an image-space method to extract the motion beats of a video by detecting frames at which there is a significant change in direction or motion stops. The motion beats are then synchronized with the music beats such that as many beats as possible are matched with as little as possible time-warping distortion to the video. We show two applications for this cross-media synchronization: one where a given dance performance is enhanced to be better synchronized with its original music, and one where a given dance video is automatically adapted to be synchronized with different music.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Repp, B. H. Musical synchronization. In: Music, Motor Control and the Brain. Altenmuller, E.; Wiesendanger, M.; Keselring, J. Eds. Oxford University Press, 55–76, 2006.
Kim, T.-h.; Park, S. I.; Shin, S. Y. Rhythmicmotion synthesis based on motion-beat analysis. ACM Transactions on Graphics Vol. 22, No. 3, 392–401, 2003.
Shiratori, T.; Nakazawa, A.; Ikeuchi, K. Dancing-tomusic character animation. Computer Graphics Forum Vol. 25, No. 3, 449–458, 2006.
Chu, W.-T.; Tsai, S.-Y. Rhythm of motion extraction and rhythm-based cross-media alignment for dance videos. IEEE Transactions on Multimedia Vol. 14, No. 1, 129–141, 2012.
Flash, T.; Hogan, N. The coordination of arm movements: An experimentally confirmed mathematical model. Journal of Neuroscience Vol. 5, No. 7, 1688–1703, 1985.
Jones, M. R.; Boltz, M. Dynamic attending and responses to time. Psychological Review Vol. 96, No. 3, 459–491, 1989.
Leyvand, T.; Cohen-Or, D.; Dror, G.; Lischinski, D. Digital face beautification. In: Proceedings of the ACM SIGGRAPH 2006 Sketches, Article No. 169, 2006.
Zhou, S.; Fu, H.; Liu, L.; Cohen-Or, D.; Han, X. Parametric reshaping of human bodies in images. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 126, 2010.
Zhou, F.; Torre, F. Canonical time warping for alignment of human behavior. In: Proceedings of the Advances in Neural Information Processing Systems, 2286–2294, 2009.
Zhou, F.; De la Torre, F. Generalized time warping for multi-modal alignment of human motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1282–1289, 2012.
Yoon, J.-C.; Lee, I.-K.; Byun, S. Automated music video generation using multi-level feature-based segmentation. In: Handbook of Multimedia for Digital Entertainment and Arts. Furht, B. Ed. Springer, 385–401, 2009.
Yoon, J.-C.; Lee, I.-K.; Lee, H.-C. Feature-based synchronization of video and background music. In: Advances in Machine Vision, Image Processing, and Pattern Analysis. Lecture Notes in Computer Science, Vol. 4153. Zheng, N.; Jiang, X.; Lan, X. Eds. Springer Berlin Heidelberg, 205–214, 2006.
Jehan, T.; Lew, M.; Vaucelle, C. Cati dance: Self-edited, self-synchronized music video. In: Proceedings of the ACM SIGGRAPH 2003 Sketches & Applications, 1–1, 2003.
Suwajanakorn, S.; Seitz, S. M.; Kemelmacher-Shlizerman, I. Synthesizing Obama: Learning lip sync from audio. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 95, 2017.
Caspi, Y.; Irani, M. Spatio-temporal alignment of sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 24, No. 11, 1409–1424, 2002.
Slaney, M.; Covell, M. FaceSync: A linear operator for measuring synchronization of video facial images and audio tracks. In: Proceedings of the 13th International Conference on Neural Information Processing Systems, 784–790, 2000.
Wang, O.; Schroers, C.; Zimmer, H.; Gross, M.; Sorkine-Hornung, A. VideoSnapping: Interactive synchronization of multiple videos. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 77, 2014.
Lu, S.-P.; Zhang, S.-H.; Wei, J.; Hu, S.-M.; Martin, R. R. Timeline editing of objects in video. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 7, 1218–1227, 2013.
Shiratori, T.; Nakazawa, A.; Ikeuchi, K. Detecting dance motion structure through music analysis. In: Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, 857–862, 2004.
Denman, H.; Doyle, E.; Kokaram, A.; Lennon, D.; Dahyot, R.; Fuller, R. Exploiting temporal discontinuities for event detection and manipulation in video streams. In: Proceedings of the 7th ACM SIGMM International Workshop on Multimedia Information Retrieval, 183–192, 2005.
McKinney, M. F.; Moelants, D.; Davies, M. E. P.; Klapuri, A. Evaluation of audio beat tracking and music tempo extraction algorithms. Journal of New Music Research Vol. 36, No. 1, 1–16, 2007.
Ellis, D. P. W. Beat tracking by dynamic programming. Journal of New Music Research Vol. 36, No. 1, 51–60, 2007.
Patel, A. D.; Iversen, J. R.; Bregman, M. R.; Schulz, I. Experimental evidence for synchronization to a musical beat in a nonhuman animal. Current Biology Vol. 19, No. 10, 827–830, 2009.
Patel, A. D.; Iversen, J. R.; Bregman, M. R.; Schulz, I. Studying synchronization to a musical beat in nonhuman animals. Annals of the New York Academy of Sciences Vol. 1169, No. 1, 459–469, 2009.
Patel, A. D.; Iversen, J. R.; Bregman, M. R.; Schulz, I.; Schulz, C. Investigating the human-specificity of synchronization to music. In: Proceedings of the 10th International Conference on Music Perception and Cognition, 100–104, 2008.
Author information
Authors and Affiliations
Corresponding author
Additional information
Rachele Bellini received her B.Sc. degree cum laude in digital communication in 2012 and her M.Sc. degree cum laude in computer science in 2015, both from the University of Milan. Since 2014 she is collaborating with Cohen-Or’s group at Tel Aviv University on texturing, image processing, and video analysis. She is currently working as a VFX Developer at Pixomondo LLC (Los Angeles).
Yanir Kleiman obtained his Ph.D. degree from Tel Aviv University in 2016, under the supervision of Prof. Daniel Cohen-Or. He was a post-doctor at the École Polytechnique in France in 2016 and 2017. He is currently a graphics software developer at Double Negative Visual Effects. His research focuses on shape analysis, including shape similarity, correspondence and segmentation, and included work in other domains such as image synthesis, crowd sourcing, and deep learning. Prior to his Ph.D., Yanir had more than 10 years of professional experience as a software developer and visual effects artist.
Daniel Cohen-Or is a professor at the School of Computer Science, Tel Aviv University. He received his B.Sc. (cum laude) degree in mathematics and computer science and his M.Sc. (cum laude) degree in computer science, both from Ben-Gurion University, in 1985 and 1986, respectively. He received his Ph.D. degree from the Department of Computer Science at the State University of New York at Stony Brook in 1991. He received the 2005 Eurographics Outstanding Technical Contributions Award. In 2015, he was named a Thomson Reuters Highly Cited Researcher. Currently, his main interests are in image synthesis, analysis and reconstruction, motion and transformations, and shapes and surfaces.
Electronic supplementary material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Bellini, R., Kleiman, Y. & Cohen-Or, D. Dance to the beat: Synchronizing motion to audio. Comp. Visual Media 4, 197–208 (2018). https://doi.org/10.1007/s41095-018-0115-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-018-0115-y