[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

Human Motion Forecasting in Dynamic Domain Shifts: A Homeostatic Continual Test-Time Adaptation Framework

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15089))

Included in the following conference series:

  • 269 Accesses

Abstract

Existing motion forecasting models, while making progress, struggle to bridge the gap between the source and target domains. Recent solutions often rely on an unrealistic assumption that the target domain remains stationary. Due to the ever-changing environment, however, the real-world test distribution may experience ongoing/continual shifts over time, leading to catastrophic forgetting and error accumulation when adapting to evolving domains. To solve these challenges, this work introduces HoCoTTA, a framework for homeostatic continual test-time adaptation. It aligns with the knowledge distillation and parameter isolation paradigm, enabling the identification of domain-invariant and domain-specific knowledge, where the former is shared (to be retained) in continual TTA across domains, while the latter needs to be updated. Specifically, we propose a multi-domain homeostasis assessment to estimate the uncertainty of the current model parameter when faced with novel-domain samples. Then, the Fisher information matrix is computed to measure the parameter sensitivity, with larger indicating the domain-sensitive parameter, and vice versa. Moreover, we propose an isolated parameter optimization strategy to update those domain-specific parameters to adapt to the new-domain, while preserving the invariant ones. In our experimental result, HoCoTTA outperforms the state-of-the-art approaches on several benchmarks, especially excelling in addressing continuous domain drifts, achieving a large improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 49.99
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 64.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. CMU Graphics Lab: Carnegie-Mellon motion capture (Mocap) database (2003). http://mocap.cs.cmu.edu

  2. Aliakbarian, S., Saleh, F., Petersson, L., Gould, S., Salzmann, M.: Contextually plausible and diverse 3D human motion prediction. In: ICCV, pp. 11333–11342 (2021)

    Google Scholar 

  3. Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2D human pose estimation: new benchmark and state of the art analysis. In: CVPR, pp. 3686–3693 (2014)

    Google Scholar 

  4. Barsoum, E., Kender, J., Liu, Z.: HP-GAN: probabilistic 3D human motion prediction via GAN. In: CVPR, pp. 1418–1427 (2018)

    Google Scholar 

  5. Bayasi, N., Hamarneh, G., Garbi, R.: Culprit-Prune-Net: efficient continual sequential multi-domain learning with application to skin lesion classification. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 165–175. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_16

    Chapter  Google Scholar 

  6. Brahma, D., Rai, P.: A probabilistic framework for lifelong test-time adaptation. In: CVPR, pp. 3582–3591 (2023)

    Google Scholar 

  7. Cai, Y., et al.: Learning progressive joint propagation for human motion prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV 2020. LNCS, pp. 226–242. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58571-6_14

    Chapter  Google Scholar 

  8. Chen, D., Wang, D., Darrell, T., Ebrahimi, S.: Contrastive test-time adaptation. In: CVPR, pp. 295–305 (2022)

    Google Scholar 

  9. Chen, L., Zhang, Y., Song, Y., Shan, Y., Liu, L.: Improved test-time adaptation for domain generalization. In: CVPR, pp. 24172–24182 (2023)

    Google Scholar 

  10. Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F.: Context-aware human motion prediction. In: CVPR, pp. 6992–7001 (2020)

    Google Scholar 

  11. Cui, Q., Sun, H., Lu, J., Li, B., Li, W.: Meta-auxiliary learning for adaptive human pose prediction. In: AAAI (2023)

    Google Scholar 

  12. Cui, Q., et al.: Test-time personalizable forecasting of 3D human poses. In: ICCV, pp. 274–283 (2023)

    Google Scholar 

  13. Cui, Q., Sun, H., Yang, F.: Learning dynamic relationships for 3D human motion prediction. In: CVPR, pp. 6519–6527 (2020)

    Google Scholar 

  14. Dang, L., Nie, Y., Long, C., Zhang, Q., Li, G.: MSR-GCN: multi-scale residual graph convolution networks for human motion prediction. In: ICCV, pp. 11467–11476 (2021)

    Google Scholar 

  15. Fang, Y., Yap, P.T., Lin, W., Zhu, H., Liu, M.: Source-free unsupervised domain adaptation: a survey. arXiv preprint arXiv:2301.00265 (2022)

  16. Fleuret, F., et al.: Test time adaptation through perturbation robustness. In: NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications (2021)

    Google Scholar 

  17. Fragkiadaki, K., Levine, S., Felsen, P., Malik, J.: Recurrent network models for human dynamics. In: ICCV, pp. 4346–4354 (2015)

    Google Scholar 

  18. Gan, Y., et al.: Decorate the newcomers: visual domain prompt for continual test time adaptation. In: AAAI, vol. 37, pp. 7595–7603 (2023)

    Google Scholar 

  19. Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: ICML, pp. 1180–1189. PMLR (2015)

    Google Scholar 

  20. Gong, T., Jeong, J., Kim, T., Kim, Y., Shin, J., Lee, S.J.: Note: robust continual test-time adaptation against temporal correlation. NeurIPS 35, 27253–27266 (2022)

    Google Scholar 

  21. Gui, L.Y., Wang, Y.X., Liang, X., Moura, J.M.F.: Adversarial geometry-aware human motion prediction. In: ECCV, pp. 786–803 (2018)

    Google Scholar 

  22. Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., Moreno-Noguer, F.: Back to MLP: a simple baseline for human motion prediction. In: WACV, pp. 4809–4819 (2023)

    Google Scholar 

  23. Guo, X., Choi, J.: Human motion prediction via learning local structure representations and temporal dependencies. In: AAAI, pp. 2580–2587 (2019)

    Google Scholar 

  24. Habibie, I., Xu, W., Mehta, D., Pons-Moll, G., Theobalt, C.: In the wild human pose estimation using explicit 2D features and intermediate 3D representations. In: CVPR, pp. 10905–10914 (2019)

    Google Scholar 

  25. Hassan, M., et al.: Stochastic scene-aware motion prediction. In: ICCV, pp. 11374–11384 (2021)

    Google Scholar 

  26. He, Y., Carass, A., Zuo, L., Dewey, B.E., Prince, J.L.: Autoencoder based self-supervised test-time adaptation for medical image analysis. Med. Image Anal. 72, 102136 (2021)

    Google Scholar 

  27. Hu, M., et al.: Fully test-time adaptation for image segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 251–260. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87199-4_24

    Chapter  Google Scholar 

  28. Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE TPAMI 36, 1325–1339 (2014)

    Google Scholar 

  29. Kang, G., Jiang, L., Yang, Y., Hauptmann, A.G.: Contrastive adaptation network for unsupervised domain adaptation. In: CVPR, pp. 4893–4902 (2019)

    Google Scholar 

  30. Kundu, J.N., Gor, M., Babu, R.V.: BiHMP-GAN: bidirectional 3D human motion prediction GAN. In: AAAI, vol. 33, pp. 8553–8560 (2019)

    Google Scholar 

  31. Kundu, J.N., Venkat, N., Babu, R.V., et al.: Universal source-free domain adaptation. In: CVPR, pp. 4544–4553 (2020)

    Google Scholar 

  32. Li, C., Zhang, Z., Sun Lee, W., Hee Lee, G.: Convolutional sequence to sequence model for human dynamics. In: CVPR, pp. 5226–5234 (2018)

    Google Scholar 

  33. Li, J., Seltzer, M.L., Wang, X., Zhao, R., Gong, Y.: Large-scale domain adaptation via teacher-student learning. arXiv preprint arXiv:1708.05466 (2017)

  34. Li, M., Chen, S., Zhang, Z., Xie, L., Tian, Q., Zhang, Y.: Skeleton-parted graph scattering networks for 3D human motion prediction. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol. 13666, pp. 18–36. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20068-7_2

  35. Li, M., Chen, S., Zhao, Y., Zhang, Y., Wang, Y., Tian, Q.: Dynamic multiscale graph neural networks for 3D skeleton based human motion prediction. In: CVPR, pp. 214–223 (2020)

    Google Scholar 

  36. Li, S., Xie, M., Gong, K., Liu, C.H., Wang, Y., Li, W.: Transferable semantic augmentation for domain adaptation. In: CVPR, pp. 11516–11525 (2021)

    Google Scholar 

  37. Li, Y., Li, K., Jiang, S., Zhang, Z., Huang, C., Da Xu, R.Y.: Geometry-driven self-supervised method for 3D human pose estimation. In: AAAI, vol. 34, pp. 11442–11449 (2020)

    Google Scholar 

  38. Liu, J., et al.: ViDA: homeostatic visual domain adapter for continual test time adaptation. arXiv preprint arXiv:2306.04344 (2023)

  39. Liu, J., Yuan, H., Lu, X.M., Wang, X.: Quantum fisher information matrix and multiparameter estimation. J. Phys. A Math. Theor. 53(2), 023001 (2020)

    Article  MathSciNet  Google Scholar 

  40. Liu, Q., Lin, L., Shen, Z., Yang, Z.: Periodically exchange teacher-student for source-free object detection. In: ICCV, pp. 6414–6424 (2023)

    Google Scholar 

  41. Liu, Y., Zhang, W., Wang, J.: Source-free domain adaptation for semantic segmentation. In: CVPR, pp. 1215–1224 (2021)

    Google Scholar 

  42. Liu, Y., Kothari, P., Van Delft, B., Bellot-Gurlet, B., Mordan, T., Alahi, A.: TTT++: when does self-supervised test-time training fail or thrive? NeurIPS 34, 21808–21820 (2021)

    Google Scholar 

  43. Liu, Z., Lyu, K., Wu, S., Chen, H., Hao, Y., Ji, S.: Aggregated multi-GANs for controlled 3D human motion prediction. In: AAAI, pp. 2225–2232 (2021)

    Google Scholar 

  44. Lodagala, V.S., Ghosh, S., Umesh, S.: PADA: pruning assisted domain adaptation for self-supervised speech representations. In: 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 136–143. IEEE (2023)

    Google Scholar 

  45. Ma, T., Nie, Y., Long, C., Zhang, Q., Li, G.: Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In: CVPR, pp. 6437–6446 (2022)

    Google Scholar 

  46. Ma, T., Nie, Y., Long, C., Zhang, Q., Li, G.: Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In: CVPR, pp. 6437–6446 (2022)

    Google Scholar 

  47. Mao, W., Liu, M., Salzmann, M.: Generating smooth pose sequences for diverse human motion prediction. In: ICCV, pp. 13309–13318 (2021)

    Google Scholar 

  48. Mao, W., Liu, M., Salzmann, M., Li, H.: Learning trajectory dependencies for human motion prediction. In: ICCV, pp. 9489–9497 (2019)

    Google Scholar 

  49. Martinez, J., Black, M.J., Romero, J.: On human motion prediction using recurrent neural networks. In: CVPR, pp. 2891–2900 (2017)

    Google Scholar 

  50. Martínez-González, A., Villamizar, M., Odobez, J.M.: Pose transformers (POTR): human motion prediction with non-autoregressive transformers. In: ICCV, pp. 2276–2284 (2021)

    Google Scholar 

  51. Melas-Kyriazi, L., Manrai, A.K.: PixMatch: unsupervised domain adaptation via pixelwise consistency training. In: CVPR, pp. 12435–12445 (2021)

    Google Scholar 

  52. Orbes-Arteaga, M., et al.: Augmentation based unsupervised domain adaptation. arXiv preprint arXiv:2202.11486 (2022)

  53. Ovadia, Y., et al.: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. In: NIPS (2019)

    Google Scholar 

  54. Piergiovanni, A.J., Angelova, A., Toshev, A., Ryoo, M.S.: Adversarial generative grammars for human activity prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 507–523. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_30

    Chapter  Google Scholar 

  55. Roy, S., et al.: Uncertainty-guided source-free domain adaptation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol. 13685, pp. 537–555. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19806-9_31

  56. Ruiz, A.H., Gall, J., Moreno-Noguer, F.: Human motion prediction via spatio-temporal inpainting. In: CVPR, pp. 7134–7143 (2018)

    Google Scholar 

  57. Saadatnejad, S., et al.: A generic diffusion-based approach for 3D human pose prediction in the wild. In: ICRA, pp. 8246–8253. IEEE (2023)

    Google Scholar 

  58. Sanyal, S., Babu, R.V., et al.: Continual domain adaptation through pruning-aided domain-specific weight modulation. In: CVPR, pp. 2456–2462 (2023)

    Google Scholar 

  59. Shin, C., Kim, T., Lee, S., Leey, S.: Test-time adaptation for out-of-distributed image inpainting. In: ICIP (2021)

    Google Scholar 

  60. Sofianos, T., Sampieri, A., Franco, L., Galasso, F.: Space-time-separable graph convolutional network for pose forecasting. In: ICCV, pp. 11209–11218 (2021)

    Google Scholar 

  61. Sójka, D., Cygert, S., Twardowski, B., Trzciński, T.: AR-TTA: a simple method for real-world continual test-time adaptation. In: ICCV, pp. 3491–3495 (2023)

    Google Scholar 

  62. Spall, J.C.: Monte Carlo computation of the fisher information matrix in nonstandard settings. J. Comput. Graph. Stat. 14(4), 889–909 (2005)

    Article  MathSciNet  Google Scholar 

  63. Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: a dataset of whole-body human grasping of objects. In: ECCV (2020)

    Google Scholar 

  64. Tanke, J., et al.: Social diffusion: long-term multiple human motion anticipation. In: ICCV, pp. 9601–9611 (2023)

    Google Scholar 

  65. Tomar, D., Vray, G., Bozorgtabar, B., Thiran, J.P.: TeSLA: test-time self-learning with automatic adversarial augmentation. In: CVPR, pp. 20341–20350 (2023)

    Google Scholar 

  66. Varsavsky, T., Orbes-Arteaga, M., Sudre, C.H., Graham, M.S., Nachev, P., Cardoso, M.J.: Test-time unsupervised domain adaptation. In: Martel, A.L., et al. (eds.) MICCAI 2020, pp. 428–436. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_42

    Chapter  Google Scholar 

  67. Volpi, R., Morerio, P., Savarese, S., Murino, V.: Adversarial feature augmentation for unsupervised domain adaptation. In: CVPR, pp. 5495–5504 (2018)

    Google Scholar 

  68. Wang, D., Shelhamer, E., Liu, S., Olshausen, B., Darrell, T.: Tent: fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 (2020)

  69. Wang, Q., Fink, O., Van Gool, L., Dai, D.: Continual test-time domain adaptation. In: CVPR, pp. 7201–7211 (2022)

    Google Scholar 

  70. Xu, C., Tan, R.T., Tan, Y., Chen, S., Wang, X., Wang, Y.: Auxiliary tasks benefit 3D skeleton-based human motion prediction. In: ICCV, pp. 9509–9520 (2023)

    Google Scholar 

  71. Xu, S., Wang, Y.X., Gui, L.Y.: Diverse human motion prediction guided by multi-level spatial-temporal anchors. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol. 13682, pp. 251–269. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20047-2_15

  72. Yang, S., Wang, Y., Van De Weijer, J., Herranz, L., Jui, S.: Generalized source-free domain adaptation. In: ICCV, pp. 8978–8987 (2021)

    Google Scholar 

  73. Ye, J., Fu, C., Zheng, G., Paudel, D.P., Chen, G.: Unsupervised domain adaptation for nighttime aerial tracking. In: CVPR, pp. 8896–8905 (2022)

    Google Scholar 

  74. Yuan, L., Xie, B., Li, S.: Robust test-time adaptation in dynamic scenarios. In: CVPR, pp. 15922–15932 (2023)

    Google Scholar 

  75. Zhang, M., Levine, S., Finn, C.: MEMO: test time robustness via adaptation and augmentation. NeurIPS 35, 38629–38642 (2022)

    Google Scholar 

  76. Zhang, Y., Black, M.J., Tang, S.: We are more than our joints: predicting how 3D bodies move. In: CVPR, pp. 3372–3382 (2021)

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (62176125, 62306141), and in part by the Natural Science Foundation of Jiangsu Province (BK20220939).

Author information

Authors and Affiliations

  1. Nanjing University of Science and Technology, Nanjing, China

    Qiongjie Cui, Huaijiang Sun, Weiqing Li & Jianfeng Lu

  2. Tianjin AiForward Science and Technology Co., Ltd., Tianjin, China

    Bin Li

Authors
  1. Qiongjie Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huaijiang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weiqing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianfeng Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huaijiang Sun .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cui, Q., Sun, H., Li, W., Lu, J., Li, B. (2025). Human Motion Forecasting in Dynamic Domain Shifts: A Homeostatic Continual Test-Time Adaptation Framework. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15089. Springer, Cham. https://doi.org/10.1007/978-3-031-72751-1_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72751-1_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72750-4

  • Online ISBN: 978-3-031-72751-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation