Keywords

1 Introduction

Recent research efforts show several interesting convergence points between knowledge graph and computer vision, such as improving object detection with external knowledge graphs [1], scene description with triples [2], knowledge graph completion with visual features [3] and visuo-semantic search [4]. In this paper, we explore their synergy for image user profiling which has not been sufficiently studied so far.

Since several years, knowledge graphs have been leveraged to conduct user profiling through semantic analysis of text and to improve content-based recommendation approaches by providing structured metadata [5,6,7,8,9,10]. Today, a tremendous amount of multimedia data are available on the web and are being produced continuously. Modern websites should be equipped with systems which can understand users’ interests through their interactions with multimedia data and adapt the services accordingly in order to provide a better user experience.

Our main contribution is two novel image user profiling approaches:

  • The first one maps an image to entities which correspond to the objects appearing in the image.

  • The second one maps to entities which are depicted by visually similar images and which exist in the conceptual scope of the dataset within which further personalisation tasks are conducted.

In the rest of the paper, we discuss some related work on image user profiling in Sect. 2, we then present our two image user profiling approaches in Sect. 3, Sect. 4 describes the demonstration given at ESWC 2018 and Sect. 5 concludes the paper.

2 Related Work

We present some existing approaches which create user profiles from images. In [11], the authors try to detect demographic attributes of individual users and group types from the photos posted on photo sharing sites. In [12], the authors derive users’ personalities from pictures posted on Instagram. In [13], the authors introduce a picture-based user elicitation and recommendation method for tourism products. The system creates a user profile which consists of 7 traveller types accompanied with a matching degree. A very similar tool is presented in [14] which maps photos to 17 tourist types. In [15], the photos are mapped to several pre-defined categories such as “leisure”, “art” and “culture”. Different from these existing approaches, the approaches that we propose map images to knowledge graph entities. This choice has been motivated by existing work which has proven the advantages of such semantic user profiling per se and in personalisation systems [5,6,7,8,9,10].

3 Two Novel Image User Profiling Approaches

In this section, we present two image user profiling approaches which map an input image to knowledge graph entities. A user profile contains top-n entities representing things of interest to a user who appreciates the input image. In this paper, we use DBpedia, knowing that other similar large-scale knowledge graphs like Wikidata can also be used.

The first approach consists of mapping an image to entities which correspond to the objects appearing in the image. There are two main steps: object detection and entity linking. For object detection, we use a computer vision tool named “Inception-V3” [16]. Inception-V3 is a convolutional neural network model trained for the ImageNet Large Visual Recognition Challenge using the data from 2012. The model tries to classify entire images into 1000 classes which are WordNet synsets like “gazelle” and “patio, terrace”. At the entity linking step, we map the 1000 synsets to corresponding DBpedia entities. We are completely aware that this is a very basic and obvious approach. We still present it because we did not find it in the state of the art.

The second approach consists of mapping an image to entities which are depicted by visually similar images and which exist in the conceptual scope of the catalogue within which further personalisation tasks are conducted. The conceptual scope is a new notion that we propose in our work. We assume that the user profiling is not an end in itself but should serve further personalisation tasks. The created user profiles should be useful for these further tasks. Given a catalogue of items, we currently consider that its conceptual scope consists of all knowledge graph entities which directly appear in the catalogue. The entities can be obtained by two means: direct item linking and item description linking as presented in [17]. To compute the visual similarity between images, we rely on the penultimate layer outputted by Inception-V3 which is a 2048-dimensional vector. The similarity between two images is determined by the Euclidean distance between their vectors.

Our second approach requires the following steps:

  1. 1.

    We constitute the conceptual scope of the catalogue.

  2. 2.

    We retrieve the images depicting the entities in the conceptual scope (linked by the property “foaf:depiction”).

  3. 3.

    We compute pairwise visual similarity between the input image and the depicting images with the method explained above.

  4. 4.

    We retain the n most similar depicting images and thereafter the entities linked to them.

In Fig. 1, we give an example to illustrate the proposed approaches.

Fig. 1.
figure 1

Example illustrating the two proposed image user profiling approaches

Full size image

4 Demonstration Given at ESWC 2018

In the demonstration given at ESWC 2018, we showcase the second approach which is more advanced. We configured it with a real and recent commercial catalogue of a popular French travel agency. The catalogue contains 1,357 tours which take place in more than 136 countries and regions. The tours are depicted by 11,614 distinct images. The conceptual scope is obtained by item description linking and contains 13,109 DBpedia entities. We provide a Web interface where users can select an image that he/she is interested in and are then shown the profile (top-3 DBpedia entities) corresponding to the selected image (Fig. 2).

Fig. 2.
figure 2

Screenshots of the demo given at ESWC 2018

Full size image

5 Conclusion

In this paper, we explored the synergy between knowledge graph technologies and computer vision tools. We proposed two novel image user profiling approaches which map an image to knowledge graph entities representing the interests of a user who appreciates the image. We described the demonstration given at ESWC 2018 which is configured with a real and recent travel domain dataset. As future work, we plan to evaluate our profiling approaches and to apply them in personalisation tasks.