Abstract
Due to the limitation of training datasets and global motion understanding to optical flow estimation, current methods only focus on local clues and ignore the motion continuity of consecutive frames, resulting in an inconsistent motion problem. After theoretical analysis, we find the multi-frame methods need to pay more attention to the continuity between cost volumes than two-frame methods. Thus, our method is based on a multi-frame framework and introduces extra object coordinates in each frame by the segmentation model to revise the matching pairs in cost volume. Specifically, we introduce a Cost Volume Adaptation Module, including a Bbox Spatial Queries to store coordinates information and a Correlation Query Queue to query the object position of different frames. On the Sintel and KITTI test benchmark, our proposed MemoFlow achieves 1.00 and 1.69 average endpoint error (AEPE) on the clean and final passes and an F1-all error of 4.43%, ranking 1st among all three-frame methods and two-frame methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bao, Wenbo, et al. “KalmanFlow 2.0: Efficient video optical flow estimation via context-aware kalman filtering.” IEEE Transactions on Image Processing 28.9 (2019): 4233-4246
Brox, Thomas, and Jitendra Malik. “Object segmentation by long term analysis of point trajectories.” European conference on computer vision. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010
Butler, D., et al. “MPI-Sintel optical flow benchmark: Supplemental material.” MPI-IS-TR-006, MPI for Intelligent Systems (2012). Citeseer, 2012
Chen, Yonghu, et al. “MFCFlow: A Motion Feature Compensated Multi-Frame Recurrent Network for Optical Flow Estimation.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023
Cheng, Jingchun, et al.“Segflow: Joint learning for video object segmentation and optical flow.” Proceedings of the IEEE international conference on computer vision. 2017
Chin, Toshio M., William Clement Karl, and Alan S. Willsky. “Probabilistic and sequential computation of optical flow using temporal coherence.” IEEE Transactions on Image Processing 3.6 (1994): 773-788
Ding, Mingyu, et al. “Every frame counts: Joint learning of video segmentation and optical flow.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020
Dong, Qiaole, Chenjie Cao, and Yanwei Fu. “Rethinking Optical Flow from Geometric Matching Consistent Perspective.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023
Dosovitskiy, Alexey, et al. “Flownet: Learning optical flow with convolutional networks.” Proceedings of the IEEE international conference on computer vision. 2015
Du, Yunhao, et al. “Strongsort: Make deepsort great again.” IEEE Transactions on Multimedia (2023)
Elad, M., Feuer, A.: Recursive optical flow estimation–adaptive filtering approach. J. Vis. Commun. Image Represent. 9(2), 119–138 (1998)
Geiger, Andreas, et al. “Vision meets robotics: The kitti dataset.” The International Journal of Robotics Research 32.11 (2013): 1231-1237
Huang, Zhaoyang, et al. “Flowformer: A transformer architecture for optical flow.” European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022
Ilg, Eddy, et al. “Flownet 2.0: Evolution of optical flow estimation with deep networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017
Jiang, Shihao, et al. “Learning to estimate hidden motions with global motion aggregation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021
Kondermann, Daniel, et al. “The hci benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2016
Wang, Xianshun, et al. “Camera Parameters Aware Motion Segmentation Network with Compensated Optical Flow.” 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021
Lu, Yongyi, Cewu Lu, and Chi-Keung Tang. “Online video object detection using association LSTM.” Proceedings of the IEEE International Conference on Computer Vision. 2017
Luo, C., Zhan, J., Xue, X., Wang, L., Ren, R., Yang, Q.: Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11139, pp. 382–391. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01418-6_38
Mayer, Nikolaus, et al. “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016
Mohan, R., Valada, A.: Efficientps: Efficient panoptic segmentation. Int. J. Comput. Vision 129(5), 1551–1579 (2021)
Ochs, Peter, and Thomas Brox. “Higher order motion models and spectral clustering.” 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012
Ošep, Aljoša, et al. “Track, then decide: Category-agnostic vision-based multi-object tracking.” 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018
Ren, Zhile, et al. “A fusion approach for multi-frame optical flow estimation.” 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2019
Runz, Martin, Maud Buffier, and Lourdes Agapito. “Maskfusion: Real-time recognition, tracking and reconstruction of multiple moving objects.” 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 2018
Shi, Xiaoyu, et al. “Videoflow: Exploiting temporal cues for multi-frame optical flow estimation.” arXiv preprint arXiv:2303.08340 (2023)
Shi, Xiaoyu, et al. “Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023
Sui, Xiuchao, et al. “Craft: Cross-attentional flow transformer for robust optical flow.” Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 2022
Sun, Deqing, et al. “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018
Sun, Deqing, et al. “Models matter, so does training: An empirical study of cnns for optical flow estimation.” IEEE transactions on pattern analysis and machine intelligence 42.6 (2019): 1408-1423
Sun, Shangkun, et al. “Skflow: Learning optical flow with super kernels.” Advances in Neural Information Processing Systems 35 (2022): 11313-11326
Teed, Z., Deng, J.: RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 402–419. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_24
Wang, Bo, et al. “SplatFlow: Learning Multi-frame Optical Flow via Splatting.” arXiv preprint arXiv:2306.08887 (2023)
Weinzaepfel, Philippe, et al. “DeepFlow: Large displacement optical flow with deep matching.” Proceedings of the IEEE international conference on computer vision. 2013
Wojke, Nicolai, Alex Bewley, and Dietrich Paulus. “Simple online and realtime tracking with a deep association metric.” 2017 IEEE international conference on image processing (ICIP). IEEE, 2017
Wulff, Jonas, and Michael J. Black. “Efficient sparse-to-dense optical flow estimation using a learned basis and layers.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015
Xu, Haofei, et al. “Gmflow: Learning optical flow via global matching.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022
Xu, Ning, et al. “Youtube-vos: A large-scale video object segmentation benchmark.” arXiv preprint arXiv:1809.03327 (2018)
Zhou, Shili, et al. “SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model.” arXiv preprint arXiv:2307.16586 (2023)
Cheng, Ri, et al. “Context-Aware Iteration Policy Network for Efficient Optical Flow Estimation.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 2. 2024
Hu, Jie, et al. “You only segment once: Towards real-time panoptic segmentation.“Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023
Zhang, Chaoning, et al. “Faster segment anything: Towards lightweight sam for mobile applications.” arXiv preprint arXiv:2306.14289 (2023)
Acknowledgements
This work was supported by National Science and Technology Major Project from Minister of Science and Technology, China (2021ZD-0201403), National Natural Science Foundation of China (62103399), Youth Innovation Promotion Association, Chinese Academy of Sciences (2021233, 202324-2), Shanghai Academic Research Leader (22XD1424500).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, M., Shi, W., Zhu, D., Wang, L., Li, J. (2025). MemoFlow: Modifying Explicit Motion of Inconsistency in Optical Flow. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15330. Springer, Cham. https://doi.org/10.1007/978-3-031-78113-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-78113-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-78112-4
Online ISBN: 978-3-031-78113-1
eBook Packages: Computer ScienceComputer Science (R0)