Abstract
Nowadays, deep learning based models like CLIP allow simple design of cross-modal video search systems that are able to solve many tasks considered as highly challenging several years ago. In this paper, we analyze a CLIP based search approach that focuses on situations, where users cannot find proper text queries to describe searched video segments. The approach relies on suggestions of classes for displayed intermediate result sets and thus allows users to realize missing words and ideas to describe video frames. This approach is supported with a preliminary study showing potential of the method. Based on the results, we extend a respected known-item search system for the Video Browser Showdown, where more challenging visual known-item search tasks are planned.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
According to our VBS 2022 log analysis, the general context-aware ranker was almost not used at all anyway, users preferred temporal queries.
References
Amato, G., et al.: VISIONE at video browser showdown 2022. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 543–548. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_52
Hezel, N., Schall, K., Jung, K., Barthel, K.U.: Efficient search and browsing of large-scale video collections with Vibro. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 487–492. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_43
Lokoč, J., Mejzlík, F., Souček, T., Dokoupil, P., Peška, L.: Video search with context-aware ranker and relevance feedback. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13142, pp. 505–510. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98355-0_46
Lokoč, J., et al.: A task category space for user-centric comparative multimedia search evaluations. In: Þór Jónsson, B., et al. (eds.) MMM 2022. LNCS, vol. 13141, pp. 193–204. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98358-1_16
Lokoč, J., Bailer, W., Schoeffmann, K., Münzer, B., Awad, G.: On influential trends in interactive video retrieval: video browser showdown 2015–2017. IEEE Trans. Multimedia 20(12), 3361–3376 (2018). https://doi.org/10.1109/TMM.2018.2830110
Lokoč, J., et al.: Is the reign of interactive search eternal? findings from the video browser showdown 2020. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 17(3) (2021). https://doi.org/10.1145/3445031
Peska, L., Mejzlík, F., Soucek, T., Lokoc, J.: Towards evaluating and simulating keyword queries for development of interactive known-item search systems. In: Gurrin, C., Jónsson, B.Þ., Kando, N., Schöffmann, K., Chen, Y.P., O’Connor, N.E. (eds.) Proceedings of the 2020 on International Conference on Multimedia Retrieval, ICMR 2020, Dublin, Ireland, 8–11 June 2020, pp. 281–285. ACM (2020). https://doi.org/10.1145/3372278.3390726
Radford, A., et al.: Learning transferable visual models from natural language supervision. CoRR abs/2103.00020 (2021). https://arxiv.org/abs/2103.00020
Rossetto, L., et al.: Interactive video retrieval in the age of deep learning-detailed evaluation of VBS 2019. IEEE Trans. Multimedia 23, 243–256 (2020). https://doi.org/10.1109/TMM.2020.2980944
Rossetto, L., Gasser, R., Sauter, L., Bernstein, A., Schuldt, H.: A system for interactive multimedia retrieval evaluations. In: Lokoč, J., et al. (eds.) MMM 2021. LNCS, vol. 12573, pp. 385–390. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67835-7_33
Rossetto, L., Schuldt, H., Awad, G., Butt, A.A.: V3C – a research video collection. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11295, pp. 349–360. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05710-7_29
Truong, Q.T., et al.: Marine video kit: a new marine video dataset for content-based analysis and retrieval. In: Dang-Nguyen, D., et al. (eds.) MMM 2023. LNCS, vol. 13833, pp. xx–yy. Springer, Cham (2023)
Acknowledgment
This work has been supported by Charles University grant SVV-260588.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lokoč, J., Vopálková, Z., Dokoupil, P., Peška, L. (2023). Video Search with CLIP and Interactive Text Query Reformulation. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13833. Springer, Cham. https://doi.org/10.1007/978-3-031-27077-2_50
Download citation
DOI: https://doi.org/10.1007/978-3-031-27077-2_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27076-5
Online ISBN: 978-3-031-27077-2
eBook Packages: Computer ScienceComputer Science (R0)