A Joint Scene Text Recognition and Visual Appearance Model for Protest Issue Classification

Social movements have been a crucial means of political participation in democratic and non-democratic societies. In recent years, the Arab Spring in 2011 as a notable example, such movements aggressively used social media as a platform for organizing protesters. Visual images and texts spread around on the internet played vital roles in bonding and attracting citizens to the movements. Political scientists and sociologists attempted to exploit the vast amount of information on the internet to analyze social movements. Their use of artificial intelligence techniques, however, has been limited. In this paper, we newly introduce a joint scene-text recognition and visual appearance model. Specifically, our model employs a character-level scene text detection and an n-gram character embedding model. Our model can extract semantic information from scene texts including handwritten words on signs and placards, which past automated studies of social movements by social scientists have not yet utilized. By employing such semantic information, our model can, in contrast to past research, classify protest images into semantic categories such as types of protest issues (e.g., gender, race, green). We newly created the Protest Issue Image Dataset consisting of 2,859 images with five protest issue categories to evaluate our model. Our model scored F1 score and provided a significant boost in performance over the conventional visual model used by social scientists, which scored . Our model can be applied to various types of social movement studies, including recent research which examines how different types of protest issues relate to other political and social factors.