More Web Proxy on the site http://driver.im/

research-article

ViQuAE, a Dataset for Knowledge-based Visual Question Answering about Named Entities

Authors:

Olivier Ferret,

Camille Guinaudeau,

Hervé Le Borgne,

Romaric Besançon,

Jose G. Moreno,

Jesús Lovón MelgarejoAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 3108 - 3120

https://doi.org/10.1145/3477495.3531753

Published: 07 July 2022 Publication History

Abstract

Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero- and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are freely available at https://github.com/PaulLerner/ViQuAE.

Supplementary Material

MP4 File (SIGIR22-rs1787.mp4)

In this talk, Paul Lerner presents ViQuAE, a new dataset for Knowledge-based Visual Question Answering about named Entities.

Download
35.82 MB

References

[1]

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://openaccess.thecvf.com/content_cvpr_2018/html/Anderson_ Bottom-Up_and_Top-Down_CVPR_2018_paper.html

[2]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, Santiago, Chile, 2425--2433. https://doi.org/10.1109/ICCV.2015.279

Digital Library

[3]

Elias Bassani. 2022. ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison. In Advances in Information Retrieval (Lecture Notes in Computer Science), Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kjetil Nørvåg, and Vinay Setty (Eds.). Springer International Publishing, Cham, 259--264. https://doi.org/10.1007/978--3-030--99739--7_30

[4]

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs] (Aug. 2021). http://arxiv.org/abs/2108.07258 arXiv: 2108.07258.

[5]

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).

[6]

Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2021. WebQA: Multihop and Multimodal QA. arXiv:2109.00590 [cs] (Sept. 2021). http://arxiv.org/abs/2109.00590

[7]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1870--1879. https://aclanthology.org/P17--1171/

[8]

Christopher Clark and Matt Gardner. 2018. Simple and Effective Multi-Paragraph Reading Comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 845--855. https://doi.org/10. 18653/v1/P18--1078

[9]

Paul Clough, Mark Sanderson, and Henning Müller. 2004. The CLEF Cross Language Image Retrieval Track (ImageCLEF) 2004. In Image and Video Retrieval (Lecture Notes in Computer Science), Peter Enser, Yiannis Kompatsiaris, Noel E. O'Connor, Alan F. Smeaton, and Arnold W. M. Smeulders (Eds.). Springer, Berlin, Heidelberg, 243--251. https://doi.org/10.1007/978--3--540--27814--6_31

[10]

Robert Dale and Nicholas Haddock. 1991. Generating Referring Expressions Involving Relations. In Fifth Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Berlin, Germany. https://aclanthology.org/E91--1028

[11]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248--255. https://doi.org/10.1109/CVPR. 2009.5206848 ISSN: 1063--6919.

[12]

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https: //openaccess.thecvf.com/content_CVPR_2019/html/Deng_ArcFace_Additive_ Angular_Margin_Loss_for_Deep_Face_Recognition_CVPR_2019_paper.html

[13]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171--4186. https://doi.org/10.18653/v1/N19--1423

[14]

The European Parliament. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). http://data.europa.eu/eli/reg/2016/679/oj/eng Legislative Body: EP, CONSIL.

[15]

Paolo Ferragina and Ugo Scaiella. 2010. TAGME: on-the-fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM'10). Association for Computing Machinery, New York, NY, USA, 1625--1628. https://doi.org/10. 1145/1871437.1871689

Digital Library

[16]

R. A. Fisher. 1937. The design of experiments. The design of experiments. 2nd Ed (1937). https://www.cabdirect.org/cabdirect/abstract/19371601600 Publisher: Oliver & Boyd, Edinburgh & London.

[17]

Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378--382. https://doi.org/10.1037/h0031619

[18]

François Gardères and Maryam Ziaeefard. 2020. ConceptBert: Concept-Aware Representation for Visual Question Answering. Findings of the Association for Computational Linguistics: EMNLP 2020 (2020), 10. https://aclanthology.org/2020. findings-emnlp.44/

[19]

Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. 2016. MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition. In Computer Vision -- ECCV 2016 (Lecture Notes in Computer Science), Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 87--102. https://doi.org/10.1007/978--3--319--46487--9_6

[20]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778. https://openaccess.thecvf.com/content_ cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf

[21]

J. Johnson, M. Douze, and H. Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7, 3 (2019), 535--547. https://doi.org/10. 1109/TBDATA.2019.2921572

[22]

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1601--1611. https://doi.org/10.18653/v1/P17--1147

[23]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for OpenDomain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6769--6781. https://www.aclweb.org/anthology/2020.emnlpmain.550

[24]

Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://openaccess.thecvf.com/content_cvpr_2017/html/Kembhavi_Are_ You_Smarter_CVPR_2017_paper.html

[25]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference for Learning Representations. http://arxiv.org/abs/1412.6980

[26]

Emiel Krahmer and Kees van Deemter. 2019. Computational Generation of Referring Expressions: An Updated Survey. In The Oxford Handbook of Reference, B. Abbott and J. Gundel (Eds.). Oxford University Press. https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/ 9780199687305.001.0001/oxfordhb-9780199687305-e-19

[27]

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics 7 (March 2019), 452--466. https://doi.org/10.1162/tacl_ a_00276

[28]

Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Online, 1000--1008. https://aclanthology.org/2021.eacl-main.86

[29]

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A Community Library for Natural Language Processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 175--184. https://aclanthology.org/2021.emnlp-demo.21

[30]

Guohao Li, Hang Su, and Wenwu Zhu. 2017. Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks. arXiv:1712.00733 [cs] (Dec. 2017). http://arxiv.org/abs/1712.00733

[31]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision -- ECCV 2014 (Lecture Notes in Computer Science), David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 740--755. https://doi.org/10.1007/978- 3--319--10602--1_48

[32]

Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A Replication Study of Dense Passage Retriever. arXiv:2104.05740 [cs] (April 2021). http: //arxiv.org/abs/2104.05740

[33]

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3195--3204. https://ieeexplore.ieee.org/document/8953725/

[34]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32 (2019). https://papers. nips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html

[35]

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 2523--2544. https://doi.org/10.18653/v1/2021.naacl-main.200

[36]

Filip Radenovi?, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondej Chum. 2018. Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://openaccess.thecvf.com/content_cvpr_2018/html/Radenovic_ Revisiting_Oxford_and_CVPR_2018_paper.html

[37]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748--8763.

[38]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 2383--2392. https://doi.org/10.18653/v1/D16--1264

[39]

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8821--8831. https://proceedings.mlr.press/v139/ramesh21a.html

[40]

Revanth Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, Alexander Schwing, and Heng Ji. 2021. MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding. (Dec. 2021). https: //arxiv.org/abs/2112.10728v1

[41]

Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Hancock-Beaulieu, and Mike Gatford. 1995. Okapi at TREC-3. In Third Text REtrieval Conference (TREC-3) (NIST Special Publication, Vol. 500--225), Donna K. Harman (Ed.). National Institute of Standards and Technology (NIST), 109--126. https://citeseerx.ist.psu. edu/viewdoc/download?doi=10.1.1.32.9922&rep=rep1&type=pdf

[42]

Shailaja Keyur Sampat, Yezhou Yang, and Chitta Baral. 2020. Visuo-Linguistic Question Answering (VLQA) Challenge. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 4606--4616. https://doi.org/10.18653/v1/2020.findings-emnlp.413

[43]

Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. KVQA: Knowledge-Aware Visual Question Answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8876--8884. https: //144.208.67.177/ojs/index.php/AAAI/article/view/4915

Digital Library

[44]

Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. https://www.cv-foundation.org/openaccess/ content_cvpr_workshops_2014/W15/html/Razavian_CNN_Features_Off-theShelf_2014_CVPR_paper.html

[45]

Mark D. Smucker, James Allan, and Ben Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (CIKM '07). Association for Computing Machinery, New York, NY, USA, 623--632. https://doi.org/10.1145/1321440.1321528

Digital Library

[46]

Rohini K. Srihari, Zhongfei Zhang, and Aibing Rao. 2000. Intelligent Indexing and Semantic Retrieval of Multimodal Documents. Information Retrieval 2, 2 (May 2000), 245--275. https://doi.org/10.1023/A:1009962928226

Digital Library

[47]

Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. MultiModalQA: Complex Question Answering over Text, Tables and Images. In ICLR 2021. https: //openreview.net/forum?id=ee6W5UgQLa

[48]

Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'00). ACM Press, Athens, Greece, 200--207. https://doi.org/10.1145/345508.345577

Digital Library

[49]

Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Henge. 2017. Explicit knowledge-based reasoning for visual question answering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. 1290--1296. https://dl.acm.org/doi/abs/10.5555/3171642.3171825

[50]

Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. 2018. FVQA: Fact-Based Visual Question Answering. IEEE transactions on pattern analysis and machine intelligence 40, 10 (2018), 2413--2427. https://doi.org/10. 1109/TPAMI.2017.2754246

[51]

Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 5878--5882. https://doi.org/10.18653/v1/D19--1599

[52]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. HuggingFace's Transformers: State-of-the-art Natural Language Processing. arXiv:1910.03771 [cs] (July 2020). http://arxiv.org/ abs/1910.03771

[53]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 2369--2380. https://doi.org/10. 18653/v1/D18--1259

[54]

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters 23, 10 (Oct. 2016), 1499--1503. https://doi.org/10. 1109/LSP.2016.2603342 Conference Name: IEEE Signal Processing Letters.

Cited By

Lerner P(2024)Knowledge-based Visual Question Answering about Named EntitiesACM SIGIR Forum10.1145/3642979.364300957:2(1-2)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3642979.3643009
Li KWang YHe YLi YWang YLiu YWang ZXu JChen GLou PWang LQiao Y(2024)MVBench: A Comprehensive Multi-modal Video Understanding Benchmark2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02095(22195-22206)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02095
Ishmam MShovon MMridha MDey N(2024)From image to languageInformation Fusion10.1016/j.inffus.2024.102270106:COnline publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1016/j.inffus.2024.102270
Show More Cited By

Index Terms

ViQuAE, a Dataset for Knowledge-based Visual Question Answering about Named Entities
1. Information systems
  1. Information retrieval

Recommendations

Knowledge-based Visual Question Answering about Named Entities

This thesis is positioned at the intersection of several research fields, Natural Language Processing, Information Retrieval (IR) and Computer Vision, which have unified around representation learning and pre-training methods. We have defined and studied ...
Cross-Modal Retrieval for Knowledge-Based Visual Question Answering
Advances in Information Retrieval
Abstract
Knowledge-based Visual Question Answering about Named Entities is a challenging task that requires retrieving information from a multimodal Knowledge Base. Named entities have diverse visual representations and are therefore difficult to ...
Named entity disambiguation for questions in community question answering

Named entity disambiguation (NED) refers to the task of mapping entity mentions in running texts to the correct entries in a specific knowledge base (e.g., Wikipedia). Although there has been a lot of work on NED for long and formal texts like Wikipedia ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

ANR

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
460
Total Downloads

Downloads (Last 12 months)154
Downloads (Last 6 weeks)10

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lerner P(2024)Knowledge-based Visual Question Answering about Named EntitiesACM SIGIR Forum10.1145/3642979.364300957:2(1-2)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3642979.3643009
Li KWang YHe YLi YWang YLiu YWang ZXu JChen GLou PWang LQiao Y(2024)MVBench: A Comprehensive Multi-modal Video Understanding Benchmark2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02095(22195-22206)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02095
Ishmam MShovon MMridha MDey N(2024)From image to languageInformation Fusion10.1016/j.inffus.2024.102270106:COnline publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1016/j.inffus.2024.102270
Gao ZChen ZCui ERen YWang WZhu JTian HYe SHe JZhu XLu LLu TQiao YDai JWang W(2024)Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performanceVisual Intelligence10.1007/s44267-024-00067-62:1Online publication date: 10-Dec-2024
https://doi.org/10.1007/s44267-024-00067-6
Xie PLiu B(2024)Probing Fundamental Visual Comprehend Capabilities on Vision Language Models via Visual Phrases from Structural DataCognitive Computation10.1007/s12559-024-10351-816:6(3484-3504)Online publication date: 5-Sep-2024
https://doi.org/10.1007/s12559-024-10351-8
Chen ZWang WTian HYe SGao ZCui ETong WHu KLuo JMa ZMa JWang JDong XYan HGuo HHe CShi BJin ZXu CWang BWei XLi WZhang WZhang BCai PWen LYan XDou MLu LZhu XLu TLin DQiao YDai JWang W(2024)How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suitesScience China Information Sciences10.1007/s11432-024-4231-567:12Online publication date: 13-Dec-2024
https://doi.org/10.1007/s11432-024-4231-5
Lerner PFerret OGuinaudeau C(2024)Cross-Modal Retrieval for Knowledge-Based Visual Question AnsweringAdvances in Information Retrieval10.1007/978-3-031-56027-9_26(421-438)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56027-9_26
Wang JLi ZYu JYang LXia REl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Fine-Grained Multimodal Named Entity Recognition and Grounding with a Generative FrameworkProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612322(3934-3943)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612322

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents