Abstract
Out-of-context (OOC) detection is a challenging task involving identifying images and texts that are irrelevant to the context in which they are presented. Large vision-language models (LVLMs) are effective at various tasks, including image classification and text generation. However, the extent of their proficiency in multimodal OOC detection tasks is unclear. In this paper, we investigate the ability of LVLMs to detect multimodal OOC and show that these models cannot achieve high accuracy on OOC detection tasks without fine-tuning. However, we demonstrate that fine-tuning LVLMs on multimodal OOC datasets can further improve their OOC detection accuracy. To evaluate the performance of LVLMs on OOC detection tasks, we fine-tune MiniGPT-4 on the NewsCLIPpings dataset, a large dataset of multimodal OOC. Our results show that fine-tuning MiniGPT-4 on the NewsCLIPpings dataset significantly improves the OOC detection accuracy in this dataset. This suggests that fine-tuning can significantly improve the performance of LVLMs on OOC detection tasks.
F. Shalabi and H. Felouat—These authors contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lin, X., Liao, X., Xu, T., Pian, W., Wong, K.: Rumor detection with hierarchical recurrent convolutional neural network. In: Natural Language Processing And Chinese Computing: 8th CCF International Conference, NLPCC 2019, Dunhuang, China, October 9-14, 2019, Proceedings, Part II 8, pp. 338–348 (2019)
Islam, M., Liu, S., Wang, X., Xu, G.: Deep learning for misinformation detection on online social networks: a survey and new perspectives. Soc. Netw. Anal. Min. 10, 1–20 (2020)
Su, W., et al.: Vl-BERT: pre-training of generic visual-linguistic representations. ArXiv Preprint ArXiv:1908.08530 (2019)
Li, L., Yatskar, M., Yin, D., Hsieh, C., Chang, K.: VisualBERT: a simple and performant baseline for vision and language. ArXiv Preprint ArXiv:1908.03557 (2019)
Radford, A., et al.: Learning transferable visual models from natural language supervision. Int. Conf. Mach. Learn., 8748–8763 (2021)
Radford, A., et al.: Language models are unsupervised multitask learners. OpenAI Blog. 1, 9 (2019)
Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Alayrac, J., et al.: Flamingo: a visual language model for few-shot learning. Adv. Neural. Inf. Process. Syst. 35, 23716–23736 (2022)
Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv Preprint ArXiv:2301.12597 (2023)
Chung, H., et al.: Scaling instruction-finetuned language models. ArXiv Preprint ArXiv:2210.11416 (2022)
OpenAI GPT-4 Technical Report (2023)
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N.: Visual ChatGPT: talking, drawing and editing with visual foundation models. ArXiv Preprint ArXiv:2303.04671 (2023)
Yang, Z., et al.: MM-ReAct: prompting ChatGPT for multimodal reasoning and action. ArXiv Preprint ArXiv:2303.11381 (2023)
Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: enhancing vision-language understanding with advanced large language models. ArXiv Preprint ArXiv:2304.10592 (2023)
Singhal, S., Pandey, T., Mrig, S., Shah, R., Kumaraguru, P.: Leveraging intra and inter modality relationship for multimodal fake news detection. Companion Proc. Web Conf. 2022, 726–734 (2022)
Wu, Y., Zhan, P., Zhang, Y., Wang, L., Xu, Z.: Multimodal fusion with co-attention networks for fake news detection. Find. Assoc. Comput. Linguist.: ACL-IJCNLP 2021, 2560–2569 (2021)
Jing, J., Wu, H., Sun, J., Fang, X., Zhang, H.: Multimodal fake news detection via progressive fusion networks. Inf. Process. Manage. 60, 103120 (2023)
Zhang, Y., Trinh, L., Cao, D., Cui, Z., Liu, Y.: Detecting out-of-context multimodal misinformation with interpretable neural-symbolic model. ArXiv Preprint ArXiv:2304.07633 (2023)
Moholdt, E., Khan, S., Dang-Nguyen, D.: Detecting out-of-context image-caption pairs in news: a counter-intuitive method. ArXiv Preprint ArXiv:2308.16611 (2023)
Aneja, S., Bregler, C., Niessner, M.: COSMOS: catching out-of-context image misuse using self-supervised learning. Proc. AAAI Conf. Artif. Intell. 37, 14084–14092 (2023)
Luo, G., Darrell, T., Rohrbach, A.: NewsCLIPpings: automatic generation of out-of-context multimodal media. EMNLP, pp. 6801–6817 (2021)
Huang, M., Jia, S., Chang, M., Lyu, S.: Text-image de-contextualization detection using vision-language models. ICASSP, pp. 8967–8971 (2022)
Zhang, P., et al.: VinVL: revisiting visual representations in vision-language models. CVPR, pp. 5579–5588 (2021)
Shalabi, F., Nguyen, H., Felouat, H., Chang, C., Echizen, I.: Image-text out-of-context detection using synthetic multimodal misinformation. In: 2023 Asia-Pacific Signal And Information Processing Association Annual Summit and Conference (APSIPA ASC) (2023)
Dosovitskiy, A., et al.: An image is worth 16\(\,\times \,\)16 words: transformers for image recognition at scale. ICLR (2021)
Reimers, N., Gurevych, I.: Sentence Embeddings using Siamese BERT-Networks. EMNLP, Sentence-BERT (2019)
Chiang, W., et al.: Vicuna: an open-source chatbot impressing GPT-4 with 90% ChatGPT quality. See https://vicuna.Lmsys.Org (2023). Accessed 14 Apr 2023
Touvron, H., et al.: LLaMA: open and efficient foundation language models. ArXiv Preprint ArXiv:2302.13971 (2023)
Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Comput. Surv. 55, 1–38 (2023)
Hu, E., et al.: LoRA: low-rank adaptation of large language models. ArXiv Preprint ArXiv:2106.09685 (2021)
Kuang, W., et al.: FederatedScope-LLM: a comprehensive package for fine-tuning large language models in federated learning. ArXiv Preprint ArXiv:2309.00363 (2023)
Liu, F., Wang, Y., Wang, T., Ordonez, V.: Visual news: benchmark and challenges in news image captioning. In: EMNLP, pp. 6761–6771 (2021)
Abdelnabi, S., Hasan, R., Fritz, M.: Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources. In: CVPR, pp. 14940–14949 (2022)
Acknowledgments
This work was partially supported by JSPS KAKENHI Grant JP21H04907, and by JST CREST Grants JPMJCR18A6 and JPMJCR20D3, Japan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shalabi, F., Felouat, H., Nguyen, H.H., Echizen, I. (2024). Leveraging Chat-Based Large Vision Language Models for Multimodal Out-of-Context Detection. In: Barolli, L. (eds) Advanced Information Networking and Applications. AINA 2024. Lecture Notes on Data Engineering and Communications Technologies, vol 202. Springer, Cham. https://doi.org/10.1007/978-3-031-57916-5_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-57916-5_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-57915-8
Online ISBN: 978-3-031-57916-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)