VISCOUNTH: A Large-scale Multilingual Visual Question Answering Dataset for Cultural Heritage

To create our VQA dataset, we resorted to a semi-automatic approach that involves the collaboration of expert and non-expert users and the use of text processing and natural language processing techniques to obtain an accurate list of question-answer pairs. We considered a scenario where an image is associated to available knowledge either manually (e.g., artworks in a museum can be associated with their descriptions) or by object recognition (e.g., architectural properties identified by taking pictures), and generated a dataset as a list of question-answer pairs, each one associated to an image, a description and a set of available information items. An instance of question-answer pair is: “Who is the author?”—“The author of the cultural asset is Pierre François Basan.”

Our semi-automatic approach consisted in two main steps. The first part of the process focused on generating a list of question types with associated verbal forms by considering both expert and non-expert perspectives, the latter assessed by surveys. Then, for each question type, we automatically generated a list of question-answer pairs by combining question forms and associated answer templates with information from relevant cultural assets in ArCo, and accurately cleaning the results. This process was performed by an ad hoc tool, developed following a build-and-evaluate iterative process. At each step, we evaluated a sample of the produced dataset to propose new data cleaning rules for improving results. The process ended when the desired accuracy was achieved. Eventually, question-answer pairs from different question types were combined. Next, we first detail our question types generation process, then fully describe the question-answer pairs generation by drawing from question types.

The question types generation process was based on the following two perspectives carried out independently: a domain experts’ perspective, represented by a selection of natural language competency questions (CQs) [41] previously considered to model the ArCo ontology network [13], and a user-centered perspective, represented by a set of questions from mostly non-expert (65 of 104) users, collected through five questionnaires on a set of different images of cultural assets belonging to ArCo (five cultural assets per questionnaire). In the questionnaires, the users were asked to formulate a number of questions (minimum 5, maximum 10) that they considered related to each image presented (questions they would ask if they were enjoying the cultural asset in a museum or a cultural site). In this way, we collected 2,920 questions from a very heterogeneous group of users in terms of age (from 24 to 70 years old and 42 years average age), cultural background and interests. Subsequently, the questions were semi-automatically analyzed and annotated to recognize their semantics, associate them (when possible) with ArCo’s metadata, and create corresponding SPARQL queries for data extraction.

In the clustering process, we grouped user-produced questions into semantic clusters, named question types, with the purpose of grouping together questions that ask for the same information. Clustering was first performed automatically by text analysis and sentence similarity, then validated and corrected manually. The automatic procedure consisted in the following steps. We initially aggregated sentences that resulted to be identical after tokenization, lemmatization, and stop words removal. Then, for each question, we identified the most semantically similar one in the whole set by Sentence-BERT [43] and aggregated sentences whose similarity was above 84% (we found empirically that this value resulted in a low error rate). Eventually, we performed average linkage agglomerative clustering with a similarity threshold of 60%. To prepare for manual validation, we extracted a list of question forms, each one associated to a numerical ID representing the cluster it belongs to. Questions in the same cluster (e.g., “Who is the author?” and “Who made it?”) were placed close to each other. After removing identical sentences, we obtained about \(\text{1,659}\) questions, grouped in 126 clusters. Each question was then manually associated to a textual (human meaningful) ID (e.g., “AUTHOR”) agreed by the annotators and a special “NODATA” ID (about 10%) was introduced for questions that refer to information that is not contained in ArCo. Table 1 gives an overview of the question types generation process, where the effort of users and experts is combined. Each question type is labeled as “Expert” if it comes from the competency questions of ArCo ontology network and has been formulates by the team of experts (counted once in column Mention), “Users” if the question was formulated by non-expert users through the questionnaire, or “Both” if both users and experts proposed such a question (possibly with different verbal forms). At the end of the process, after excluding clusters that refer to unavailable and unusable information, we obtained 43 question types, with 20 of them referred by both users and experts.

In addition, the experts grouped the question types into three categories based on their nature. Most questions (31) were labeled as “contextual,” as it was not possible to find the appropriate answers in the images associated with the question type considered (e.g., “DATING”). Instead, eight question types were defined as “visual” (e.g., “BLACKANDWHITE”), since the answers can be inferred from the images associated to the cultural asset, while for four “mixed” question types the answers derive both from visual and contextual information (e.g., “SUBJECT”). Figure 2 depicts all 43 question types of QA split into this three categories, and some examples of images of cultural assets (i.e., PAINTING, SCULPTURE, PRINT, FRESCO) to which they are associated. Eventually, the experts defined an answer template and a SPARQL query for each question type.

We employed SparqlWrapper⁴ for executing the SPARQL queries and extracting textual data and pictures from ArCo. We removed cultural assets that have zero or more than one associated pictures. For each record of the query results, we generated a question-answer pair by randomly drawing a question verbal form by the set of appropriated verbal forms in the associated question cluster, with the same distribution of the results of the user questionnaires (frequently proposed questions were selected with higher probability), and building the associated answer from the answer template.

Some question verbal forms are appropriate only for specific types of cultural assets (e.g., “Who was it painted by?” makes sense only for paintings). To establish the appropriated verbal forms for a cultural assets, we mapped both question verbal forms and cultural assets with corresponding macro-categories (we defined nine macro-categories, i.e., SCULPTURE, OBJECT, PHOTO, FRESCO, CHURCH, FIND, PRINT, PAINTING, OTHER). Since this information is not available in ArCo, we considered the available textual description of the cultural asset category to build the mapping. Due to the multitude of categories, we performed a filtering and mapping operation to bring the wide range of types back into a small but explanatory set. As a state-of-the-art work on Italian cultural heritage, we took into account the controlled vocabularies defined by the ICCD-MiC,⁵ which also provided the data for ArCo KG [13]. These controlled vocabularies ensure a standardized terminology for the description and cataloging of cultural heritage and help overcome the semantic heterogeneity that is often present in creating such catalogs. First, we filtered the vocabularies’ elements closest to the type of artworks to which users refer in their questions. We mapped each textual description of category with an entry in the controlled vocabularies. As detailed in Reference [12], we used a string matching algorithm that takes as input a list of words from a well-defined taxonomy and a general description in free text and returns the equivalent term from the reference taxonomy.

To improve both the form of the answer itself and its rendering in its context, we adopted two approaches. First, we applied a set of cleaning rules, such as removing data with errors and changing patterns of verbal forms (e.g., from “Baldin, Luigi” to “Luigi Baldin”).⁶ Second, we employed pre-trained language models to improve the form of conversational answers by adapting each sentence to its associated datum (e.g., Italian prepositions and articles have to be chosen according to the gender and number of corresponding nouns or adjectives). To solve this problem, we applied the cloze task of BERT [23] on the generated answers, asking to infer words whose genre and number depend on the specific datum and cannot be previously determined.⁷ Furthermore, we applied a final grammar correction task by automatic translating the sentence from Italian to English and back to Italian by means of a pre-trained language models for translation.⁸

Eventually, we automatically generated the description of each cultural asset by combining the long answers of all associated question-answer pairs, since this information is not available in ArCo.

The generated VQA dataset contains 6.49M question-answer pairs covering cultural assets, 43 question types and 427 verbal forms. The number of question-answer pairs per template ranges from 35 to 576K. Each question-answer pair is associated with the corresponding cultural asset and its information, including its picture, a description and its URI in ArCo. The number of question types associated to each image depends on the cultural asset’s type and ranges from a minimum of 1 to a maximum of 26 question types associated to a certain cultural asset, as in the example of 26 IDs associated to the “PRINT” depicted in Figure 3.

The final dataset is the largest resource available for training and validating VQA models in the cultural heritage domain. It comprises 6,493,867 question-answer pairs, with associated visual, textual and structured information. In Table 2, we report this data in comparison to the AQUA [27] dataset statistics. In contrast to AQUA, we consider a new dimension that incorporates mixed (contextual and visual) question types. Additionally, our dataset is two orders of magnitude larger than AQUA.

We associate each cultural asset in our dataset with a set of question-answer pairs, with both a long conversational answer and a short synthetic answer, an image, a natural language description, its URI in ArCo, the reference ontology class and its type. In addition, we provide information on the text span of the answer in the description, when possible.

We make our dataset available on GitHub.⁹ We also provide two samples in Italian and English of 50 question-answer pairs per question type that we manually evaluated. Results show an overall accuracy of the long answers (percent of correct entries) of 96.6% for the Italian sample, and of 93% for the English one. We also provide statistics that reports, for each question type, its usage, the number of associated question forms, the number of question-answer pairs generated, and the accuracy. The distribution of cultural asset types in the dataset is provided in Figure 4. The most common question type are “TYPE”, “TITLE,” and “MATERIALORTECHNIQUE,” while “EVENTSITE,” “PURPOSE,” and “BLACKANDWHITE” have fewer associated cultural assets. Excluding cultural assets not classified in a specific category (“OTHER”), the macro categories with more elements are “OBJECT” (26%) and “PAINTING” (13%), while the less populated one is “FRESCO” (<1%).

Furthermore, Table 3 shows the breakdown of the number of question-answer pairs by cultural asset type and question type.

To evaluate VQA models on the collected dataset, we follow the standard evaluation setting proposed in Reference [42]. We rely on two metrics, Exact match and Macro-averaged F1 score:

Note that for both metrics, we do not consider articles and punctuations.

In addition, text-based models generate variable length sentences as a subset of the textual description, whereas vision-based models pick a a candidate among a predefined dictionary of possible answers. In both cases, we take the set of words and compare it to the ground truth to compute Exact match and F1 score.

We carry out a quantitative evaluation by first testing off-the-shelf language pre-trained models. We do not expect such models to perform well on visual questions, but we want to assess whether such models can exploit their language understanding to comprehend questions relative to the cultural heritage domain. As detailed in Section 4, we use as text-based models RoBERTa [35] and DistilBert [45]. Both datasets have been pre-trained on SQUAD [42], a reading comprehension dataset with more than 100,000 questions-answer pairs crowd-sourced on a set of Wikipedia articles.

Interestingly, when evaluated on contextual questions, such models perform poorly as can be seen in Table 4. Both models are capable of answering with a certain degree of correctness to a few question categories, namely, “DEDICATION” and “USEFUNCTION,” with DistilBert obtaining good F1 scores on an additional restricted number of categories such as “TECHNIQUE” and “AFFIXEDAUTHOR.” For most of the remaining question categories, we report an F1 close to 0. This suggests the presence of a domain shift between standard question answering datasets (such as SQUAD) and VISCOUNTH. In fact, in art-related question-answers, as well as descriptions, there is often usage of domain specific jargon that is not present in generic text corpora, making the models unable to understand the question or identify the answer within the description.

Nonetheless, although unlikely given the proven capabilities of such pre-trained models, a low F1 could be caused by intrinsic limits in the architectures. To further confirm the presence of a domain shift, rather than some form of model limitation, we fine-tuned the best of the two models, DistilBert, on the VISCOUNTH dataset. This leads to a significant improvement. The model gains on average 54 points of F1-score, obtaining close to perfect results for question types such as “TITLE,” “AUTHOR,” “FOUNDLOCATION,” and “PREPARATORYWORK.” Interestingly, for other categories instead DistilBert still reports low scores, close to zero “HISTORICALINFO,” “DECORATIVEPURPORSE,” “PURPORSE”). These categories however either are less represented in the data as shown in Table 3 or are intrinsically harder. For instance, the “HISTORICALINFO” category presents a high variability in how questions are formulated and frequently asks for generic concepts, which require a high level reasoning on the description content.

We also perform a similar evaluation with the vision-based model LXMERT [51]. However, two issues must be taken into account. First, as in most vision-based models, since they cannot rely on textual descriptions, the VQA task is treated as a classification task. Answering a question corresponds to selecting the most relevant answer among a dictionary of pre-defined words or short sentences. For this reason, the domain shift is much more emphasized: If the dictionary does not contain terms suitable for cultural heritage, then the model will not perform well. Second, whereas a text-based model could answer visual questions if the requested information is also in the description, a vision-based model cannot answer contextual questions in any way. As a consequence, we cannot apply a pre-trained vision-model due to significant differences in the answer dictionary. But even fine-tuning the model on VISCOUNTH leads to an F1-score of 0. To perform such finetuning, we create a new dictionary of answers by filtering the most frequent answers in the training set. More precisely, we selected the answers that appear more than 8 times.

Moving to mixed questions (Table 5), on the one hand, we can observe a similar behaviour for text-based models, although the overall F1-score is much lower, since visual knowledge is required to answer correctly. On the other hand, LXMERT is able to provide correct answers to some of the questions. Notably, for the “MATERIAL” question type, LXMERT surpasses text-based models by a considerable margin, yet it is unable to answer to “MEASUREMENT” questions, contrary to DistilBert.

As expected, for visual questions, we can observe an opposite trend compared to contextual questions. In Table 6, we report the results, showing that LXMERT can provide for almost all question categories a high rate of correct questions. However, after being fine-tuned on VISCOUNTH, DistilBert is capable of addressing questions related to “AFFIXEDTRANSCRIPT” and “BLACKANDWHITE.” This is due to the fact that sometimes the answers can also be found in the textual description.

For most experiments, we report both the macro-averaged F1-score and the Exact Match (EM) metrics. It can be noticed that the F1 score is a relaxation of the EM metric in the sense that it allows an answer to be loosely compared to the ground truth, even when not all words are the same, thus accounting for synonyms or different phrasings.

Finally, we evaluate our combined model. We exploit the question classifier to understand which model is more suitable to address a specific question, without looking at the description nor the image. The BERT-based classifier, described in Section 4, obtains a question classification accuracy of 98.4% on the test set, indicating that it is fully capable of understanding the nature of the questions. We do not include mixed questions in training, and at inference time, we consider the question to be either visual or contextual based on the output of the classifier.

As can be seen from Tables 4, 5, and 6, the model is able to exploit both models to accurately answer visual and contextual questions, with only a slight drop for language-based samples. For mixed questions, our model is able to improve compared to LXMERT but exhibits a drop compared to DistilBert. This confirms that mixed questions indeed pose a challenge yet to be solved in question answering applications.

In Table 7, we report the overall average scores in terms of F1 and Exact Match. The average is computed as the mean of all category scores, i.e., contextual, mixed, and visual together. Our combined model retains the best results, providing a baseline for future work in visual question answering for cultural heritage.

To better understand the challenges in the dataset, we show a breakdown of results divided by question category and type of cultural property in Table 8. We do this only for visual questions, since contextual questions do not exploit visual information. This table shows how the performance of our approach vary depending on the type of artwork. We can observe, as expected, that there is a gap between the score obtained for different types of artwork on specific question classes. As example the question category “CONSERVATION” (that includes questions about the conservation state of the artwork) results easier for prints than sculptures. Vice-versa, the category “AFFIXEDLANGUAGE” (that has questions about the language of the writing attached to the cultural asset) has better results for sculptures. Finally, we can observe that the category “AFFIXEDTRANSCRIPT,” that refers to the text present in the artwork, obtains very low results. This is due to the fact that these kind of questions are very challenging and require the extraction and the understanding of text in images and currently this can be done only with specific networks.

In this section, we provide a qualitative analysis of the answers given by our approach to questions in the VISCOUNTH dataset.

The dataset is divided into three main question types: visual, contextual and mixed. For each type there are multiple question categories, which refer to different types of cultural assets. We thus expect the answers given by our model to be affected by all this aspects. In Figure 5, we show the behaviour of our model in answering different kinds of questions for different types of cultural assets. For contextual questions, we expect that the answer has to be extracted from a natural language description, therefore a language model is sufficient to answer these questions. As we can see in Tables 3 and 4, our model is able to answer the most common contextual questions in the dataset but has lower performance for questions that appear in few examples. In Figure 5, we can observe how our model is able to answer correctly to different categories of contextual questions (“LOCATION, AUTHOR, TITLE, DATING,” etc.) for different types of artworks. For these types of questions, we do not observe different performances for different types of artworks. This is due to the fact that in these cases, our question answering language model is agnostic to visual information, being solely based on textual descriptions.

Confirming the results of Table 5, we observe that our model obtains low performances on mixed questions. This kind of questions result to be very challenging, since they require both visual knowledge and contextual knowledge. For instance, for the “MATERIAL” category, the model should be able to describe the different materials the artworks are made of and learn how to recognize them visually. Our model selects either the vision-based model or the textual-based model to answer a question, hence there is not a specific way to handle this kind of questions, thus leading to a lack of performance.

Regarding visual questions, we can observe from Table 8 that we have a variation in the performances based on the type of artwork for different classes of visual questions. For example, we can observe that the questions of the “SHAPE” category, that refers to the shape of the artwork, as expected, perform better for prints than for sculptures. Moreover, as shown in Figure 5, several artworks contain transcripts and there is a specific question category (“AFFIXEDTRANSCRIPT”) for this detail. Our model obtains very low performance on this question class, since it does not contain a specific trained model for scene text extraction.