CBIL: Collective Behavior Imitation Learning for Fish from Real Videos
Authors: 
Yifan Wu, 
Zhiyang Dou, Yuko Ishiwaka, Shun Ogawa, Yuke Lou, Wenping Wang, Lingjie Liu, Taku KomuraAuthors Info & Claims
Article No.: 242, Pages 1 - 17
Abstract
Reproducing realistic collective behaviors presents a captivating yet formidable challenge. Traditional rule-based methods rely on hand-crafted principles, limiting motion diversity and realism in generated collective behaviors. Recent imitation learning methods learn from data but often require ground-truth motion trajectories and struggle with authenticity, especially in high-density groups with erratic movements. In this paper, we present a scalable approach, Collective Behavior Imitation Learning (CBIL), for learning fish schooling behavior directly from videos, without relying on captured motion trajectories. Our method first leverages Video Representation Learning, in which a Masked Video AutoEncoder (MVAE) extracts implicit states from video inputs in a self-supervised manner. The MVAE effectively maps 2D observations to implicit states that are compact and expressive for following the imitation learning stage. Then, we propose a novel adversarial imitation learning method to effectively capture complex movements of the schools of fish, enabling efficient imitation of the distribution of motion patterns measured in the latent space. It also incorporates bio-inspired rewards alongside priors to regularize and stabilize training. Once trained, CBIL can be used for various animation tasks with the learned collective motion priors. We further show its effectiveness across different species. Finally, we demonstrate the application of our system in detecting abnormal fish behavior from in-the-wild videos.
References
[1]
I Aoki. 1982. A simulation study on the schooling mechanism in fish. Bull. Jap. Soc. Sci. Fish 48 (1982), 1081.
[2]
Jinseok Bae, Jungdam Won, Donggeun Lim, Cheol-Hui Min, and Young Min Kim. 2023. Pmp: Learning to physically interact with environments using part-wise motion priors. In ACM SIGGRAPH 2023 Conference Proceedings. 1--10.
[3]
Michele Ballerini, Nicola Cabibbo, Raphael Candelier, Andrea Cavagna, Evaristo Cisbani, Irene Giardina, Vivien Lecomte, Alberto Orlandi, Giorgio Parisi, Andrea Procaccini, et al. 2008. Interaction ruling animal collective behavior depends on topological rather than metric distance: Evidence from a field study. Proceedings of the national academy of sciences 105, 4 (2008), 1232--1237.
[4]
Otman Benchekroun, Jiayi Eris Zhang, Siddhartha Chaudhuri, Eitan Grinspun, Yi Zhou, and Alec Jacobson. 2023. Fast complementary dynamics via skinning eigenmodes. arXiv preprint arXiv:2303.11886 (2023).
[5]
Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019. DReCon: data-driven responsive control of physics-based characters. ACM Trans. Graph. 38, 6, Article 206 (nov 2019), 11 pages.
[6]
William Bialek, Andrea Cavagna, Irene Giardina, Thierry Mora, Edmondo Silvestri, Massimiliano Viale, and Aleksandra M. Walczak. 2012. Statistical mechanics for natural flocks of birds. Proceedings of the National Academy of Sciences 109, 13 (2012), 4786--4791. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1118633109
[7]
Daniel S Calovi, Ugo Lopez, Paul Schuhmacher, Hugues Chaté, Clément Sire, and Guy Theraulaz. 2015. Collective response to perturbations in a data-driven fish school model. Journal of The Royal Society Interface 12, 104 (2015), 20141362.
[8]
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2019. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. arXiv:1812.08008 [cs.CV]
[9]
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. 9650--9660.
[10]
Andrea Cavagna, Alessio Cimarelli, Irene Giardina, Giorgio Parisi, Raffaele Santagati, Fabio Stefanini, and Massimiliano Viale. 2010. Scale-free correlations in starling flocks. Proceedings of the National Academy of Sciences 107, 26 (2010), 11865--11870.
[11]
Andrea Cavagna, Irene Giardina, and Tomás S. Grigera. 2018. The physics of flocking: Correlation as a compass from experiments to theory. Physics Reports 728 (2018), 1--62. The physics of flocking: Correlation as a compass from experiments to theory.
[12]
Panayiotis Charalambous, Julien Pettre, Vassilis Vassiliades, Yiorgos Chrysanthou, and Nuria Pelechano. 2023. GREIL-Crowds: Crowd Simulation with Deep Reinforcement Learning and Examples. ACM Trans. Graph. 42, 4, Article 137 (jul 2023), 15 pages.
[13]
Chun-Tse Chien, Rui-Yang Ju, Kuang-Yi Chou, Enkaer Xieerke, and Jen-Shiun Chiang. 2024. YOLOv8-AM: YOLOv8 with Attention Mechanisms for Pediatric Wrist Fracture Detection. arXiv:2402.09329 [cs.CV]
[14]
Yong Shean Chong and Yong Haur Tay. 2017. Abnormal Event Detection in Videos using Spatiotemporal Autoencoder. arXiv:1701.01546 [cs.CV]
[15]
Soon-Jo Chung, Aditya Avinash Paranjape, Philip Dames, Shaojie Shen, and Vijay Kumar. 2018. A survey on aerial swarm robotics. IEEE Transactions on Robotics 34, 4 (2018), 837--855.
[16]
Peishan Cong, Ziyi Wang, Zhiyang Dou, Yiming Ren, Wei Yin, Kai Cheng, Yujing Sun, Xiaoxiao Long, Xinge Zhu, and Yuexin Ma. 2024. LaserHuman: Language-guided Scene-aware Human Motion Generation in Free Environment. arXiv preprint arXiv:2403.13307 (2024).
[17]
Iain D. Couzin, Jens Krause, Nigel R. Franks, and Simon A. Levin. 2005. Effective leadership and decision-making in animal groups on the move. Nature 433, 7025 (01 Feb 2005), 513--516.
[18]
Iain D Couzin, Jens Krause, Richard James, Graeme D Ruxton, and Nigel R Franks. 2002. Collective memory and spatial sorting in animal groups. Journal of theoretical biology 218, 1 (2002), 1--11.
[19]
Anthony I Dell, John A Bender, Kristin Branson, Iain D Couzin, Gonzalo G de Polavieja, Lucas PJJ Noldus, Alfonso Pérez-Escudero, Pietro Perona, Andrew D Straw, Martin Wikelski, et al. 2014. Automated image-based tracking and its application in ecology. Trends in ecology & evolution 29, 7 (2014), 417--428.
[20]
Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. 2023. C· ase: Learning conditional adversarial skill embeddings for physics-based characters. In SIGGRAPH Asia 2023 Conference Papers. 1--11.
[21]
Yusen Feng, Xiyan Xu, and Libin Liu. 2023. MuscleVAE: Model-Based Controllers of Muscle-Actuated Characters. In SIGGRAPH Asia 2023 Conference Papers. 1--11.
[22]
Audrey Filella, Fran çois Nadal, Clément Sire, Eva Kanso, and Christophe Eloy. 2018. Model of Collective Fish Behavior with Hydrodynamic Interactions. Phys. Rev. Lett. 120 (May 2018), 198101. Issue 19.
[23]
Levi Fussell, Kevin Bergamin, and Daniel Holden. 2021. SuperTrack: motion tracking for physically simulated characters using supervised learning. ACM Trans. Graph. 40, 6, Article 197 (dec 2021), 13 pages.
[24]
Wayne M Getz. 2024. An Information Theoretic Treatment of Animal Movement Tracks. arXiv:2403.16290 [q-bio.PE]
[25]
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. 2022. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5152--5161.
[26]
Yong Guo, Zhiyang Dou, Nan Zhang, Xiyue Liu, Boni Su, Yuguo Li, and Yinping Zhang. 2023. Student close contact behavior and COVID-19 transmission in China's classrooms. PNAS nexus 2, 5 (2023), pgad142.
[27]
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. 2018. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2255--2264.
[28]
Stephen Gustafson, Hemagiri Arumugam, Paul Kanyuk, and Michael Lorenzen. 2016. Mure: fast agent based crowd simulation for vfx and animation. In ACM SIGGRAPH 2016 Talks. 1--2.
[29]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000--16009.
[30]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729--9738.
[31]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.
[32]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs.CV]
[33]
Conor Heins, Beren Millidge, Lancelot Da Costa, Richard P. Mann, Karl J. Friston, and Iain D. Couzin. 2024. Collective behavior from surprise minimization. Proceedings of the National Academy of Sciences 121, 17 (2024), e2320239121. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.2320239121
[34]
James E. Herbert-Read, Andrea Perna, Richard P. Mann, Timothy M. Schaerf, David J. T. Sumpter, and Ashley J. W. Ward. 2011. Inferring the rules of interaction of shoaling fish. Proceedings of the National Academy of Sciences 108, 46 (2011), 18726--18731. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1109355108
[35]
Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems (Barcelona, Spain) (NIPS'16). Curran Associates Inc., Red Hook, NY, USA, 4572--4580.
[36]
Hans A Hofmann, Annaliese K Beery, Daniel T Blumstein, Iain D Couzin, Ryan L Earley, Loren D Hayes, Peter L Hurd, Eileen A Lacey, Steven M Phelps, Nancy G Solomon, et al. 2014. An evolutionary framework for studying mechanisms of social behavior. Trends in ecology & evolution 29, 10 (2014), 581--589.
[37]
Berthold K.P. Horn and Brian G. Schunck. 1981. Determining optical flow. Artificial Intelligence 17, 1 (1981), 185--203.
[38]
Yuko Ishiwaka, Xiao Zeng, Shun Ogawa, Donovan Westwater, Tadayuki Tone, and Masaki Nakada. 2022. DeepFoids: Adaptive Bio-Inspired Fish Simulation with Deep Reinforcement Learning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 18377--18389. https://proceedings.neurips.cc/paper_files/paper/2022/file/74fa9e6bc36aa567fe7cf002b733a30d-Paper-Conference.pdf
[39]
Yuko Ishiwaka, Xiao S. Zeng, Michael Lee Eastman, Sho Kakazu, Sarah Gross, Ryosuke Mizutani, and Masaki Nakada. 2021. Foids: bio-inspired fish simulation for generating synthetic datasets. ACM Trans. Graph. 40, 6, Article 207 (dec 2021), 15 pages.
[40]
Yaroslav Ispolatov. 2016. Collective Behaviour: Computing in fish schools. eLife 5 (jan 2016), e12852.
[41]
Rohit Jena, Changliu Liu, and Katia Sycara. 2021. Augmenting gail with bc for sample efficient imitation learning. In Conference on Robot Learning. PMLR, 80--90.
[42]
Xuebo Ji, Zherong Pan, Xifeng Gao, and Jia Pan. 2024. Text-Guided Synthesis of Crowd Animation. In ACM SIGGRAPH 2024 Conference Papers. 1--11.
[43]
Mingjie Jiang, Anyu Zhou, Runping Chen, Yuqin Yang, Hao Dong, and Wei Wang. 2023. Collective motions of fish originate from balanced local perceptual interactions and individual stochastics. Phys. Rev. E 107 (Feb 2023), 024411. Issue 2.
[44]
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2020. Unity: A General Platform for Intelligent Agents. arXiv:1809.02627 [cs.LG]
[45]
Paul Kanyuk, Leon J. W. Park, and Emily Weihrich. 2015. Headstrong, Hairy, and Heavily Clothed: Animating Crowds of Scotsmen. In ACM SIGGRAPH 2012 Talks (Los Angeles, California) (SIGGRAPH '12). Association for Computing Machinery, New York, NY, USA, Article 52, 1 pages.
[46]
Taekyung Ki, Dongchan Min, and Gyeongsu Chae. 2024. Learning to Generate Conditional Tri-plane for 3D-aware Expression Controllable Portrait Animation. arXiv:2404.00636 [cs.CV]
[47]
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG]
[48]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 [cs.CV]
[49]
Alex Kushleyev, Daniel Mellinger, Caitlin Powers, and Vijay Kumar. 2013. Towards a swarm of agile micro quadrotors. Autonomous Robots 35, 4 (2013), 287--300.
[50]
Jaedong Lee, Jungdam Won, and Jehee Lee. 2018. Crowd simulation by deep reinforcement learning. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games (Limassol, Cyprus) (MIG '18). Association for Computing Machinery, New York, NY, USA, Article 2, 7 pages.
[51]
Kang Hoon Lee, Myung Geol Choi, Qyoun Hong, and Jehee Lee. 2007. Group behavior from video: a data-driven approach to crowd simulation. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (San Diego, California) (SCA '07). Eurographics Association, Goslar, DEU, 109--118.
[52]
Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. 2021. Learning a family of motor skills from a single motion clip. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1--13.
[53]
Yoonsang Lee, Sungeun Kim, and Jehee Lee. 2010. Data-driven biped control. In ACM SIGGRAPH 2010 papers. 1--8.
[54]
Dan Li, Dacheng Chen, Jonathan Goh, and See kiong Ng. 2019. Anomaly Detection with Generative Adversarial Networks for Multivariate Time Series. arXiv:1809.04758 [cs.LG]
[55]
Weizi Li, David Wolinski, Julien Pettré, and Ming C. Lin. 2015. Biologically-inspired visual simulation of insect swarms. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 425--434.
[56]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6--12, 2014, Proceedings, Part V 13. Springer, 740--755.
[57]
Libin Liu, Michiel Van De Panne, and KangKang Yin. 2016. Guided learning of control graphs for physics-based characters. ACM Transactions on Graphics (TOG) 35, 3 (2016), 1--14.
[58]
Libin Liu, KangKang Yin, Michiel Van de Panne, Tianjia Shao, and Weiwei Xu. 2010. Sampling-based contact-rich motion control. In ACM SIGGRAPH 2010 papers. 1--10.
[59]
Xiyue Liu, Zhiyang Dou, Lei Wang, Boni Su, Tianyi Jin, Yong Guo, Jianjian Wei, and Nan Zhang. 2022. Close contact behavior-based COVID-19 transmission and interventions in a subway system. Journal of Hazardous Materials 436 (2022), 129233.
[60]
Qiujing Lu, Yipeng Zhang, Mingjian Lu, and Vwani Roychowdhury. 2022. Action-conditioned On-demand Motion Generation. arXiv:2207.08164 [cs.CV]
[61]
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. 2023a. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10895--10904.
[62]
Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. 2023b. Universal Humanoid Motion Representations for Physics-Based Control. arXiv preprint arXiv:2310.04582 (2023).
[63]
Dhendra Marutho, Sunarna Hendra Handaka, Ekaprana Wijaya, and Muljono. 2018. The Determination of Cluster Number at k-Mean Using Elbow Method and Purity Evaluation on Headline News. In 2018 International Seminar on Application for Technology of Information and Communication. 533--538.
[64]
Xiangfei Meng, Junjun Pan, Hong Qin, and Pu Ge. 2018. Real-time fish animation generation by monocular camera. Computers & Graphics 71 (2018), 55--65.
[65]
Joel W Newbolt, Jun Zhang, and Leif Ristroph. 2019. Flow interactions between uncoordinated flapping swimmers give rise to group cohesion. Proceedings of the National Academy of Sciences 116, 7 (2019), 2419--2424.
[66]
Hiro-Sato Niwa. 1996. Newtonian Dynamical Approach to Fish Schooling. Journal of Theoretical Biology 181, 1 (1996), 47--63.
[67]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023).
[68]
Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. 2023. Synthesizing physically plausible human motions in 3d scenes. arXiv preprint arXiv:2308.09036 (2023).
[69]
Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. 2019a. Learning predict-and-simulate policies from unorganized human motion data. ACM Trans. Graph. 38, 6, Article 205 (nov 2019), 11 pages.
[70]
Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and Jehee Lee. 2019b. Learning predict-and-simulate policies from unorganized human motion data. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--11.
[71]
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. 2018a. Deep-mimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37, 4 (2018), 1--14.
[72]
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. 2022. ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions on Graphics 41, 4 (July 2022), 1--17.
[73]
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018b. SFV: Reinforcement Learning of Physical Skills from Videos. arXiv:1810.03599 [cs.GR]
[74]
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. AMP: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics 40, 4 (July 2021), 1--20.
[75]
Sahithi Podila and Ying Zhu. 2017. Animating escape maneuvers for a school of fish. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (San Francisco, California) (I3D '17). Association for Computing Machinery, New York, NY, USA, Article 18, 2 pages.
[76]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 779--788.
[77]
Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. 2023. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13756--13766.
[78]
Craig W. Reynolds. 1987. Flocks, herds and schools: A distributed behavioral model. SIGGRAPH Comput. Graph. 21, 4 (aug 1987), 25--34.
[79]
David Ryu and Paul Kanyuk. 2007. Rivers of rodents: an animation-centric crowds pipeline for Ratatouille. In ACM SIGGRAPH 2007 Sketches (San Diego, California) (SIGGRAPH '07). Association for Computing Machinery, New York, NY, USA, 65--es.
[80]
Daiki Satoi, Mikihiro Hagiwara, Akira Uemoto, Hisanao Nakadai, and Junichi Hoshino. 2016. Unified motion planner for fishes with various swimming styles. ACM Trans. Graph. 35, 4, Article 80 (jul 2016), 15 pages.
[81]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG]
[82]
Yousuf Soliman, Marcel Padilla, Oliver Gross, Felix Knöppel, Ulrich Pinkall, and Peter Schröder. 2024. Going with the Flow. ACM Transactions on Graphics (TOG) 43, 4 (2024), 1--12.
[83]
Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. 2023. CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings (SIGGRAPH '23). ACM.
[84]
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022. Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022).
[85]
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. arXiv:2203.12602 [cs.CV]
[86]
Siddhartha Verma, Guido Novati, and Petros Koumoutsakos. 2018. Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proceedings of the National Academy of Sciences 115, 23 (2018), 5849--5854.
[87]
Tamás Vicsek and Anna Zafeiris. 2012. Collective motion. Physics reports 517, 3--4 (2012), 71--140.
[88]
Marek Vondrak, Leonid Sigal, Jessica Hodgins, and Odest Jenkins. 2012. Video-based 3D motion capture through biped control. ACM Trans. Graph. 31, 4, Article 27 (jul 2012), 12 pages.
[89]
Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. 2023. Tlcontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135 (2023).
[90]
Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. 2024b. YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information. arXiv:2402.13616 [cs.CV]
[91]
Jingbo Wang, Zhengyi Luo, Ye Yuan, Yixuan Li, and Bo Dai. 2024a. PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios. arXiv preprint arXiv:2404.19722 (2024).
[92]
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. 2022. Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis. arXiv:2205.13001 [cs.CV]
[93]
Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, and Sameh Khamis. 2023. Learning human dynamics in autonomous driving scenarios. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20796--20806.
[94]
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2020. A scalable approach to control diverse behaviors for physically simulated characters. ACM Transactions on Graphics (TOG) 39, 4 (2020), 33--1.
[95]
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2021. Control strategies for physically simulated characters performing two-player competitive sports. ACM Transactions on Graphics (TOG) 40, 4 (2021), 1--11.
[96]
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022. Physics-based character controllers using conditional vaes. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--12.
[97]
Pei Xu, Xiumin Shang, Victor Zordan, and Ioannis Karamouzas. 2023a. Composite Motion Learning with Task Control. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1--16.
[98]
Pei Xu, Kaixiang Xie, Sheldon Andrews, Paul G Kry, Michael Neff, Morgan McGuire, Ioannis Karamouzas, and Victor Zordan. 2023b. AdaptNet: Policy adaptation for physics-based character control. ACM Transactions on Graphics (TOG) 42, 6 (2023), 1--17.
[99]
Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. 2022. Controlvae: Model-based learning of generative controllers for physics-based characters. ACM Transactions on Graphics (TOG) 41, 6 (2022), 1--16.
[100]
Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. 2023. MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations. arXiv preprint arXiv:2310.10198 (2023).
[101]
Ri Yu, Hwangpil Park, and Jehee Lee. 2021. Human dynamics from monocular video with dynamic camera movements. ACM Trans. Graph. 40, 6, Article 208 (dec 2021), 14 pages.
[102]
Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. 2023. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16010--16021.
[103]
Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, and Kayvon Fatahalian. [n. d.]. Learning Physically Simulated Tennis Skills from Broadcast Videos. ACM Trans. Graph. ([n. d.]), 14 pages.
[104]
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2024a. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).
[105]
Nan Zhang, Xueze Yang, Boni Su, and Zhiyang Dou. 2024b. Analysis of SARS-CoV-2 transmission in a university classroom based on real human close contact behaviors. Science of The Total Environment 917 (2024), 170346.
[106]
Yunbo Zhang, Deepak Gopinath, Yuting Ye, Jessica Hodgins, Greg Turk, and Jungdam Won. 2023. Simulation and Retargeting of Complex Multi-Character Interactions. arXiv:2305.20041 [cs.GR]
[107]
Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. 2023. EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Motion Generation. arXiv preprint arXiv:2312.02256 (2023).
[108]
Xin Zhou, Xiangyong Wen, Zhepei Wang, Yuman Gao, Haojia Li, Qianhao Wang, Tiankai Yang, Haojian Lu, Yanjun Cao, Chao Xu, et al. 2022. Swarm of micro flying robots in the wild. Science Robotics 7, 66 (2022), eabm5954.
[109]
Haosheng Zou, Hang Su, Shihong Song, and Jun Zhu. 2018. Understanding human behaviors in crowds by imitating the decision-making process. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (New Orleans, Louisiana, USA) (AAAI'18/IAAI'18/EAAI'18). AAAI Press, Article 937, 8 pages.
Index Terms
- CBIL: Collective Behavior Imitation Learning for Fish from Real Videos
Index terms have been assigned to the content through auto-classification.
Recommendations
Tracking the Race Between Deep Reinforcement Learning and Imitation Learning
Quantitative Evaluation of SystemsAbstractLearning-based approaches for solving large sequential decision making problems have become popular in recent years. The resulting agents perform differently and their characteristics depend on those of the underlying learning approach. Here, we ...
Accounting for patterns of collective behavior in crowd locomotor dynamics for realistic simulations
Transactions on Edutainment VIIDo people in a crowd behave like a set of isolated individuals or like a cohesive group? Studies of crowd modeling usually consider pedestrian behavior either from the point of view of an isolated individual or from that of large swarms. We introduce ...
Soft imitation reinforcement learning with value decomposition for portfolio management
AbstractImitation learning has been recognized as a method to accelerate the training process of deep reinforcement learning agents in search of optimal strategies. Nevertheless, existing imitation learning algorithms have limitations in effectively ...
Highlights- We develop a novel soft imitation reinforcement learning (SIRL) to make good use of expert demonstrations.
- The weight of the behavior cloning loss in the SIRL framework can be updated adaptively when environments change.
- Instead of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 November 2024
Published in TOG Volume 43, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 140Total Downloads
- Downloads (Last 12 months)140
- Downloads (Last 6 weeks)75
Reflects downloads up to 11 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in