[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
BPET: A Unified Blockchain-Based Framework for Peer-to-Peer Energy Trading
Previous Article in Journal
Optimizing Requirements Prioritization for IoT Applications Using Extended Analytical Hierarchical Process and an Advanced Grouping Framework
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Future Internet
Volume 16
Issue 5
10.3390/fi16050161
futureinternet-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses

by
Jieli Chen
1,
Kah Phooi Seng
1,2,*,
Li Minn Ang
3,
Jeremy Smith
4 and
Hanyue Xu
1
1
XJTLU Entrepreneur College (Taicang), Xian Jiaotong-Liverpool University, Taicang 215400, China
2
School of Computer Science, Queensland University of Technology, Brisbane, QLD 4000, Australia
3
School of Engineering and Science, University of Sunshine Coast, Petrie, QLD 4502, Australia
4
Department of Electrical Engineering & Electronics, University of Liverpool, Liverpool L69 3BX, UK
*
Author to whom correspondence should be addressed.
Future Internet 2024, 16(5), 161; https://doi.org/10.3390/fi16050161
Submission received: 22 March 2024 / Revised: 25 April 2024 / Accepted: 3 May 2024 / Published: 7 May 2024
Browse Figures
Figure 1
<p>The proposed multimodal hierarchical graph-based situation awareness (MHGSA) system.</p> "> Figure 2
<p>Proposed multi-branch feature extraction module for constructing node features.</p> "> Figure 3
<p>Structure of the proposed multi-branch feature extraction module for constructing node features from video frames.</p> "> Figure 4
<p>Audio feature extraction module for constructing multi-grained audio event nodes features.</p> "> Figure 5
<p>Multimodal hierarchical graph construction.</p> "> Figure 6
<p>Classifier and loss calculation for MHGSA.</p> "> Figure 7
<p>Fine- and coarse-grained categories of the extracted VGGSound-DR dataset.</p> ">
Versions Notes

Abstract

:
Situational awareness (SA) is crucial in disaster response, enhancing the understanding of the environment. Social media, with its extensive user base, offers valuable real-time information for such scenarios. Although SA systems excel in extracting disaster-related details from user-generated content, a common limitation in prior approaches is their emphasis on single-modal extraction rather than embracing multi-modalities. This paper proposed a multimodal hierarchical graph-based situational awareness (MHGSA) system for comprehensive disaster event classification. Specifically, the proposed multimodal hierarchical graph contains nodes representing different disaster events and the features of the event nodes are extracted from the corresponding images and acoustic features. The proposed feature extraction modules with multi-branches for vision and audio features provide hierarchical node features for disaster events of different granularities, aiming to build a coarse-granularity classification task to constrain the model and enhance fine-granularity classification. The relationships between different disaster events in multi-modalities are learned by graph convolutional neural networks to enhance the system’s ability to recognize disaster events, thus enabling the system to fuse complex features of vision and audio. Experimental results illustrate the effectiveness of the proposed visual and audio feature extraction modules in single-modal scenarios. Furthermore, the MHGSA successfully fuses visual and audio features, yielding promising results in disaster event classification tasks.

1. Introduction

Rapidly sensing and understanding the data generated during a disaster can help in disaster response. On social media, data generated by users in the near future of a disaster are likely to express the situation they are facing as well as disaster-related information, such as the category of the disaster event. Situational awareness (SA) systems are particularly important in this area as they can automate the sensing and understanding of user-generated information to help people perform rapid disaster response [1,2]. However, user-generated information is usually multimodal and typically contains visual, audio, and textual information. Appropriate integration of multimodal features has also been shown in several works to improve the accuracy of data analysis [3,4] and it becomes a challenge to deal with the multimodal information in these messages [5].
The introduction of artificial intelligence has greatly enabled disaster response and situational awareness systems. In disaster response, the application of convolutional neural networks (CNNs) provides a powerful tool to analyze visual information quickly and accurately [6,7,8]. By performing feature extraction and pattern recognition on images of a disaster site, CNNs can help determine the type, scale, and impact of a disaster. However, in the practical application of disaster response, the integration of multimodal data needs to be considered in addition to visual information [9]. The synergistic analysis of multi-source information such as speech data might provide a more comprehensive understanding of the disaster scene.
In previous work, much research has focused on the processing of single-modal information, such as processing only image [6] or text [10] data. Such approaches have been successful in some contexts but they potentially fall short in fully leveraging the correlations between different modal information, limiting the system’s ability to fully understand the overall environment.
In recent years, with the rise of graph learning, multimodal information fusion has become more flexible and efficient. By abstracting information from different modalities into nodes of a graph and using graph convolutional networks (GCNs) for information transfer and fusion, the GCNs can automatically learn the relationships between modalities, enabling the whole system to better understand the association between multimodal information [11]. It is worth noting that the introduction of graph learning does not mean abandoning the deeper mining of each modality. On the contrary, combining graph learning with traditional CNNs allows for a more comprehensive exploitation of the characteristics of each modality, resulting in a more information-rich and robust multimodal feature representation. The introduction of graph learning for modality data [11] can be divided into reconstructing modalities into graph structure [12,13,14] and identifying graph nodes from modalities [15,16,17]. We utilize the latter approach to extract nodes representing different disaster events from multimodal data (visual and audio) and construct a graph using the relationships of the disaster events. This suggests the two following challenges:
  • Providing an effective discriminative representation of multimodal data;
  • Placing demands on the construction of the graph structure to ensure that the network can learn the relationships between the different modalities.
Therefore, in the light of the challenges outlined above, we focus on extracting disaster-related information from paired visual and audio data (usually represented as video) in this work. Our proposed multimodal hierarchical graph-based SA system can classify disaster events at coarse- and fine-grained for primary and advanced sensing based on visual and audio, referring to the multi-level sensing introduced by the three-layer model of SA [18]. The main contributions of this paper are as follows:
  • We propose a multi-branching feature extraction framework that consists of shared convolutional layers and branching convolutional layers for events of specific granularity to provide independent trainable parameters for different granularities during end-to-end joint optimization;
  • We construct an event-relational multimodal hierarchical graph to represent disaster events at different granularities to improve the performance of the system in advanced perception by multilayering the perception of the SA system;
  • We propose a method for multimodal fusion using hierarchical graph representation learning, which enhances relational learning of multimodal data;
  • The proposed MHGSA system is evaluated on datasets and consists of a significant improvement over the unimodal baseline approach.

2. Proposed Methodology

This paper presents the proposed multimodal hierarchical graph-based situational awareness (MHGSA) system for disaster response. It consists of a visual feature extraction module, an audio feature extraction module, and the multimodal hierarchical graph. Figure 1 shows the architecture of the proposed system. This section will first introduce the two multi-branch feature extraction modules for vision and audio and their implementation setups and the subsequent sections will present the proposed methodology for hierarchical graphs, including graph construction, gated graph convolutional neural network, and classification of graph nodes.

2.1. Multi-Branch Featrue Extraction Module

This section describes two feature extraction modules that provide nodes features for hierarchical graphs. Their role is to extract a representation from a given piece of image and its accompanying audio clip suitable for the task of classifying disaster events. We first define a set of fine-grained disaster events and corresponding coarse-grained disaster events, denoted as E f and E c , and the number of events they contain are N and M , respectively.
E c = E c 1 ,   E c 2 , … , E c N ,  
E f = E f 1 , E f 2 , … , E f M ,
and any fine-grained event E f n has only one corresponding coarse-grained event E c m , as follows:
∀ E f n ∈ E f , ∃ E c m ∈ E c   s u c h   t h a t   f E f n = E c m .
f : E f → E c ,
The multi-branch structure aims to provide separate model parameters for E f and E c to cope with end-to-end joint training. This structure has been applied in some work to provide hierarchical (i.e., multi-granularity) image classification [19,20,21,22]. In contrast to the independent branching structure, the multi-branching structure employs a shared convolutional network to extract common visual features in the image, which solves part of the parameter redundancy problem. Using this multi-branch structure, we propose a model derived from EfficientNet [23] for generating nodes feature for the visual and audio outputs, which is designed to work with the event-relational hierarchical graph to model image and acoustic features to features that correspond to different disaster events. As shown in Figure 2, linear layers corresponding to the number of E f and E c provided by the nodes with features of the specified dimensions.

2.1.1. Visual Feature Extraction for Multigranularity

Visual signals, an important part of human perception, are understood by computers in the form of pixels that are intuitive and easy to understand. In social media, visual signals usually appear as images or videos (images with multiple frames), which are usually single-channel (monochrome) or three-channel (RGB). Thanks to convolutional neural networks, RGB images represented as three-dimensional arrays are converted into one-dimensional vector representations through multiple layers of convolution and pooling. A common approach is to use one or more layers of fully connected networks defining appropriate input–output dimensions as classifiers for convolutional neural networks to achieve downstream tasks. In this subsection, we only discuss the feature extraction module before the fully connected layers.
Based on the optimization of depth, width, and resolution, EfficientNet, as a CNN model, provides better classification accuracy with a smaller number of parameters by virtue of efficient parameter settings. Seven such models are proposed in [23], from EfficientNet-b0 to EfficientNet-b7, and their number of parameters gradually increases from 5.3 M to 66 M. In the ImageNet [24] dataset, EfficientNet has a much smaller number of parameters compared to other models with similar accuracy. For example, the classic ResNet50 [25] has about 26M parameters but its performance is lower than EfficientNet-b1, which has no more than 8M parameters. In this paper, we utilize the mobile inverted bottleneck convolution (MBConv) block of EfficientNet as the basis to build the feature extraction module. A single MBConv block contains a classical convolutional block (i.e., a stack of convolutional, batch normalization, and activation layers), a squeeze-and-excitation module [26] to provide a channel attention mechanism, and a 1 × 1 convolutional layer paired with residual connections. As shown in Figure 3, our proposed multi-branch visual feature extraction module employs a stack of convolutional layers in a multi-branching pattern, which is divided into a front segment and a back segment. When the model uses only shared convolutional blocks and one branch convolutional block, the model will converge to the same as EfficienNet-b1.
The convolutional blocks in the front segment will be used as a shared set of convolutional layers V i s u a l C o n v s h a r e d and subsequently, two sets of identical convolutional layers, V i s u a l C o n v p a r a l l e l c and V i s u a l C o n v p a r a l l e l f , in the back segment are constructed for coarse- and fine-grained event classification.
Given a set of videos X = x 1 , x 2 , … , x T from dataset D with its corresponding multi-granularity labels Y = { y f 1 , y c 1 , y f 2 , y c 2 , … , y f T , y c T } , frames F i = { f 1 , f 2 , … , f K } are extracted from i t h video x i . The visual feature extraction module uses shared parameters to process multiple frames in a single video; the feed-forward process of the network can be represented as
V i s u a l S h a r e d F e a t f k = C o n v s h a r e d f k ,
V i s u a l C o a r s e F e a t x i = C o n v p a r a l l e l c V i s u a l S h a r e d F e a t f k ,   k ∈ [ 1 , K ]
V i s u a l F i n e F e a t x i = C o n v p a r a l l e l f V i s u a l S h a r e d F e a t f k ,   k ∈ [ 1 , K ]
In this work, we extracted 10 frames ( K = 10 ) to represent a video. The multi-branching feature extraction module allows each frame to obtain two feature vectors representing coarse and fine granularity for representing the disaster event. Thus, a single video will obtain 2 K feature vectors in order to construct a hierarchical graph for visual features. We assign a linear layer to each convolutional branch to obtain a node representation for each image frame, i.e., F C f and F C c , which is shown in Figure 3. In this way, the node features of a single video in the hierarchical graph can be represented as
N o d e F e a t c   x i V i s u a l = σ W c ⋅ V i s u a l C o a r s e F e a t x i ,
N o d e F e a t f   x i V i s u a l = σ W f ⋅ V i s u a l F i n e F e a t x i .
where W c and W f are the learnable weights of F C c and F C f . σ is an activation function, where the rectified linear unit (ReLU) is employed.
When using the module alone for vision-only disaster event classification, we add the corresponding pooling and fully connected layers at the end of the two convolutional branches to normalize the output categories.
y ^ c = σ W c ⋅ V i s u a l C o a r s e F e a t x i ,
y ^ f = σ W f ⋅ V i s u a l F i n e F e a t x i .

2.1.2. Audio Feature Extraction for Multigranularity

Audio data are typically represented as a sequence of waveforms in the time domain. Our original idea to learn their representation and make them assist visual features for downstream tasks lies in the fact that the occurrence of a disaster event is usually accompanied by a corresponding sound. For example, a fire is usually accompanied by the crackling sound of burning objects or a storm is usually accompanied by the sound of rain or wind. We designed an audio preprocessing algorithm to convert one-dimensional waveforms into two-dimensional audio features. LFCC and CQT as acoustic features are extracted after performing voice active detection (VAD) on the audio.
A Voice Activity Detection (VAD) module was implemented [27] for data preprocessing. This module filters activated speech by calculating the short-time energy and short-time zero-crossing counter of the audio. Suppose there is a segment of audio w i that is paired with x i , as mentioned before, thus:
W = w 1 , w 2 , … , w T .
The energy and zero crossing counter of w i can be calculated as follows:
E n e r g y i = w i 2 ,
Z C C i = s i g n w i − s i g n ( w i − 1 )
By averaging the T -length audio into F segments, the short-time energy and short-time zero-crossing counter can be expressed as follows:
E n e r g y s h o r t − t i m e f = ∑ f ∈ F E n e r g y f
Z C C s h o r t − t i m e f = ∑ f ∈ F Z C C f
Suitable audio clips are filtered through a set threshold with the filtering rule:
E n e r g y f > T h r e s h o l d E n e r g y   ∧ Z C C f < T h r e s h o l d Z C C
LFCC and CQT are extracted on the activated audio clips
L F C C a c t i v e = D C T log 10 ⁡ L i n e a r F r e q F i l t e r B a n k S T F T w t = A c t i v e F r a m e s ,
C Q T a c t i v e = a b s ( C o n s t a n t Q F i l t e r B a n k S T F T w t = A c t i v e F r a m e s ) ,
where L F C C a c t i v e and C Q T a c t i v e are both matrices. They are concatenated to form an image-like 2-channel acoustic feature vector for the subsequent audio feature extraction
A u d i o S h a r e d F e a t w i = C o n c a t ( L F C C a c t i v e , C Q T a c t i v e )
Multiple audio events may be included in a single audio clip. Inspired by [17], we modified the originally shared single linear layer to linear layers containing only one neuron corresponding to the number of events, i.e., F C f i and F C c i , to obtain a node representation of the corresponding events. Therefore, a single audio can provide feature vectors for all events, i.e., E f and E c , and the process can be briefly expressed as follows:
N o d e F e a t c i A u d i o = σ W c i ⋅ W A u d i o ⋅ A u d i o S h a r e d F e a t w i  
N o d e F e a t f j A u d i o = σ W f j ⋅ W A u d i o ⋅ A u d i o S h a r e d F e a t w i
where W c i and W f j is the learnable weights of F C c i and F C f j for the i t h and j t h event in E c and E f . W A u d i o denotes the learnable parameters of the audio feature extraction module. Figure 4 illustrates the structure of the audio feature extraction module.

2.2. Hierarchical Graph for Disaster Event Classification

2.2.1. Hierarchical Graph Construction

A multimodal hierarchical graph consisting of events is built after having all the node features of the fine-grained disaster event   E f and the coarse-grained disaster event E c in visual and audio features. It aims to learn the relationships between different event nodes and update the node features employing the subsequent graph convolutional learning. Inspired by [17], we define a hierarchical graph containing multimodal multi-granularity event nodes. They are all constructed in a fully connected manner. There is a graph G = ( V , E ) , where V and E denote the nodes and edges, respectively. The initial connection of nodes can be represented as follows:
A i , j = 1 ,   i ∈ N ,   j ∈ N
where A is the adjacency matrix of the graph and N denotes the number of nodes in the graph. This type of connection requires each node to be self-connected, so the number of edges contained in a single graph is indicated below:
E = N N + 1 2 ,
where ⋅ denotes the base of a set, i.e., the number of elements in the set.
The hierarchical graph construction is visually portrayed in Figure 5, where blue nodes correspond to coarse-grained event categories, and yellow nodes signify fine-grained ones. Visual features are represented by circular nodes, whereas audio features are denoted by square nodes. The visual and audio features are output by their corresponding feature extraction module and are concatenated outside the module to form node features for the graph construction.
Based on the proposed visual and audio feature extraction modules, we construct a node for each disaster event and assign a node feature for it. In order to independently learn the node features for each type of event, node features are constructed from the corresponding granularity branches using linear layers, as described in Equations (8) and (9) for visual features and Equations (21) and (22) for audio features.
For single-modal hierarchical graphs, i.e., G S i n g l e V i s u a l (visual only) or G S i n g l e A u d i o (audio only), their node features are obtained from the proposed multi-branch feature extraction module for corresponding modality. A single modality hierarchical graph G s i n g l e can be represented as
G S i n g l e m = V C o a r s e m ∪ V F i n e m , E S i n g l e m , m ∈ V i s u a l ,   A u d i o ,
where N i m denotes the node features. This allows subsequent GCN to learn the subordination relationship of coarse-grained and fine-grained events.
The proposed multimodal hierarchical graph G M u l t i , composed using all the event nodes of different modalities, makes it have twice as many nodes as event categories.
G M u l t i = V V i s u a l ∪   V A u d i o ,   E S i n g l e V i s u a l ∪ E S i n g l e A u d i o ∪ E M u l t i V i s u a l ↔ A u d i o ,
where V V i s u a l and V A u d i o denote the nodes from G S i n g l e V i s u a l and G S i n g l e A u d i o , respectively. Regarding them as subgraphs, E M u l t i V i s u a l ↔ A u d i o contains edges connecting them. Employing the gated graph convolutional network, E S i n g l e V i s u a l and E S i n g l e A u d i o can be learned by the previous layers and be passed to the layer for G G l o b a l by adding initialed edges connecting V V i s u a l and V A u d i o .

2.2.2. Graph Convolutional Network for Classification

To learn the relationship between coarse granularity events and fine granularity events, we use the residual gated graph convolutional network (RG-GCN) proposed by Bresson and Laurent [28]. It is known that the vanilla GCN can define the feature vector h i as [29]:
h i l + 1 = G C N h i l ,   { h j l : j → i } = R e L U U l h i l + ∑ j → i V l h j l ,
where l denotes the layer level, h j is a set of unordered feature vectors of all neighboring nodes, and U , V are learnable parameters for the message passing on current node and neighboring nodes. After adding a gating mechanism to the edge [30]:
h i l + 1 = G - G C N h i l ,   { h j l : j → i } = R e L U U l h i l + ∑ j → i φ i j ⨂ V l h j l ,
φ i j = σ U ¯ h i l + V ¯ h j l ,
where φ i j denotes the edge gates it brings two sets of weight parameters U ¯ and V ¯ to learn on edges. σ is the sigmoid activation function and ⊗ is the point-wise multiplication operator. Adding the residual mechanism, the RG-GCN is simply denoted as [25]:
h i l + 1 = R G - G C N h i l ,   { h j l : j → i } = G - G C N h i l ,   { h j l : j → i } + h i l ,
With the depth of graph convolution, each event node can aggregate the features of neighboring event nodes to achieve the update to obtain new own features and the weighting of edges in RG-GCN can model the correlation between different disaster events. For all nodes I i l + 1 in h i l + 1 , as described in Equations (27)–(30), the node features at the l + 1 t h layer will take into account both the features of its own node I i l , the neighboring node I j l , and of the edge e i j l between I i l and I j l at the l t h layer. e i j l is the edge feature weighted by the edge gates φ i j and its unweighted initial value can be obtained in the adjacency matrix A i j described in Equation (23).
We used a three-layer RG-GCN for learning nodes and edges in the hierarchical graph. We classify the learned event node features for classification. In the multimodal graph, we fused event nodes of the same granularity, as follows:
y ^ i = σ W f c ⋅ I i j l   i ∈ C o a r s e , F i n e ,   j ∈ V i s u a l , A u d i o , l = 3

2.2.3. Loss Function

The loss function of MHGSA is based on the cross-entropy loss function weighted summation of coarse and fine granularity. It can be expressed as:
L c = − l o g ( e p y ^ i c o a r s e ∑ j e p j c o a r s e )
L f = − l o g ( e p y ^ i f i n e ∑ j e p j f i n e )
L f i n a l = A c L c + A f L f
where L c and L f denote cross-entropy loss calculated for coarse- and fine- granularity and L f i n a l denotes the final loss. p y ^ i represents the i t h output logits and p j represents the j t h element of the logits. The value A denotes the loss weight of contributing to the loss function.
As shown in Figure 6, two linear layers are employed to classify the coarse- and fine-grained events from the event nodes of the multimodal hierarchical graph. The logits y ^ output from the linear layers is then used to calculate the error between the true label y utilizing the Equations (32)–(34). Fine-grained classification with more event categories will be used as the final classification result.
The weights assigned to the losses of different granularity classification tasks determine how much a branch contributes to the final loss function. When training with the employment of Adam [31] as the optimizer, the network of branches involved in that classification will not be updated when the weight of a classification task is equal to 0; on the contrary, when the weight is equal to 1, the weights in the network will be involved in training. This allows controlling their relative values to influence how well different branches are trained as well as how fast they are optimized. Our original intention was to add coarse granularity classification as an auxiliary task to the original classification task to better train the shared convolutional blocks. Therefore, the weight of the coarse granularity is the same as that of the fine granularity in the early stage of training, e.g., [0.5, 0.5]. In the later stages of training, the weight of fine granularity is gradually increased and the weight of coarse granularity is decreased until the weight of coarse granularity is decreased to 0, to improve the model’s ability to classify fine granularity.

3. Experiments and Results

In this section, we analyze the performance of the MHGSA system through two main research questions: (1) whether the proposed multibranch structural feature extraction modules for vision and audio lead to better model performance in the unimodal case; and (2) whether the multimodal hierarchical maps lead to effective feature fusion and provide additional performance in the multimodal case. In the following subsections, we provide a detailed description of the experimental datasets and experimental setup and present our experimental results and analysis.

3.1. Experiment Datasets

In order to evaluate each module in the MHGSA system in a more comprehensive way, we introduced VGGSound [32] datasets in the experimental part to evaluate different modules. It is worth mentioning that the datasets typically have only one granularity of event labeling. In order to fit the multi-branching structure, we classified the labels of the dataset with coarse granularity. VGGSound [32], an audio–visual dataset containing more than 300 classes, consists of more than 200 k ten-second video clips, totaling about 560 h of memory. We extracted 12 disaster-related classes from the dataset and categorized them into three coarse-grained categories. The extracted dataset contains a total of 9289 video clips, with 50 video clips in each class for testing.

3.2. Details of Implementation

We built the MHGSA system using the PyTorch (Menlo Park, CA, USA) framework with hardware specifications of Intel (Santa Clara, CA, USA) Core i9 CPU and Nvidia (Santa Clara, CA, USA) GTX 4090 GPU. To minimize losses, an Adam optimizer with 64 batches and an initial learning rate of 1 × 10 − 3 was used. To dynamically adjust the learning rate, we defined a decay rate of 0.1, which was activated when the validation set metrics did not improve within 5 epochs. We performed several training and testing sessions using different random seeds to obtain more balanced results, which made the experiment more credible.

3.3. Experiments on the MHGSA System

The MHGSA system, based on the two previously mentioned multi-branch feature extraction modules for different modalities, uses a residual gated graph convolutional neural network based on hierarchical graphs as multimodal feature fusion. We extracted 12 categories from the original VGGSound dataset and categorized them into three categories as coarse granularity labels (Figure 7). We referred the extracted dataset as VGGSound for Disaster Response (VGGSound-DR).
We conducted three experiments to compare the vision-only, audio-only, and multimodal cases. ResNet50 [25], EfficientNet-b1 [23] ResNet3D [33], and PANNs [34] are used as baselines to explore the performance of our models. It is worth mentioning that the proposed models are constructed differently in terms of classifiers due to the use of a multi-branch structure. We refer to the proposed feature extractor without graph learning as MHSA and to the model that uses the proposed graph-based approach as MHGSA. In addition, the comparison of MHSA and MHGSA demonstrates the effectiveness of the graph-based approach in the multimodal case. The average accuracy and its standard deviation, the best accuracy, the total number of trainable parameters, and the average inference time for one video clip are shown to reveal the performance of models.
As indicated in Table 1, in the visual-only mode (VO), the best classification accuracy of the MHSA-VO using 10 frames as input and fully connected layers as classifiers is higher than that of EfficientNet-b1 (57.2%), ResNet-50 (54.8%), and R3D-18 (58.5%), at 65.3%, and the number of parameters is much lower than that of ResNet-50 and R3D-18. This suggests that the multi-branch structure is effectively utilized to the coarse-grained classification task to improve the model’s performance on the fine-grained classification task. Multi-frame input significantly improves the performance of the model. MHSA-VO improves accuracy by more than 5% when using multiple frames as input over the single-frame case. To reduce the computational cost, we first trained MHSA-VO using single-frame inputs and froze the trained parameters for multi-frame training. Thus, the multi-frame input does not affect the number of parameters but only the inference time. The MHGSA-VO adds hierarchical graph construction and three layers of RG-GCN for classification compared to MHSA-VO, which provides a slight improvement in accuracy, at 66.7%. This indicates that introducing a single-modal event relational hierarchical graph in the visual-only mode contributes to the model’s performance in disaster event classification. It is worth mentioning that MHSA-VO serves as a subset of MHGSA-VO; the training process of MHGSA-VO introduces and freezes the pre-training parameters of MHSA-VO, i.e., it only trains the graph-related parameters.
In audio-only mode (AO), as shown in Table 2, MHGSA-AO achieved the highest accuracy of 67.8%. It improved 0.6% over MHSA-AO without graph learning and 2.7%, 2.0%, and 15.2% over ResNet-50, EfficientNet-b1, and PANNs, respectively. This indicates that hierarchical graph construction and learning play a positive role when applied to concepts with multiple granularities.
In multimodal mode, we assign a baseline model to vision and audio features and concatenate their feature vectors as the input of the fully connected layer classifier. We used this approach to construct ResNet-50-MM and EfficienNet-b1-MM as baseline models. On the other hand, we used both visual and audio feature extraction modules and compared fully connected layers (MHSA) and RG-GCN (MHGSA) as classifiers. As shown in Table 3, the highest results were achieved by the MHGSA, obtaining 77.6% accuracy with 34M parameters. The graph-based model outperforms the traditional fully connected layer by about 4M higher number of parameters in the multimodal mode, proving the effectiveness of multimodal hierarchical graph learning. Different from the visual-only and audio-only cases, the introduction of a graph-based approach improves more significantly in the multimodal case. This also confirms our idea of constructing a multimodal hierarchical graph, i.e., modeling the relationship between events across modalities using a graph learning approach, in order to improve the model’s ability to discriminate between different events. We can see the improved performance of the model when using multimodal information. Compared to VO and AO, all models show a substantial increase in accuracy with a doubled number of parameters in the multimodal mode, with the highest increases being in MHGSA at 10.9% and 9.8%.
From the perspective of situation awareness, more comprehensive perception and deeper understanding are important goals. In contrast to baseline models, the proposed MHGSA uses audio–visual modalities to obtain a more comprehensive perception, whereas multi-granularity classification allows the model to have a richer understanding of the environment. The classification results of the events with different granularities provide a richer understanding of the environment, enabling disaster response. Table 4 illustrates the precision metrics for every coarse- and fine-grained category evaluated from the MHGSA in different cases. The results from the coarse-grained categories intuitively show that the visual and audio modalities have different representational capabilities in different categories. For example, visual-only models are much more precise in the ‘Nature Disasters’ category than audio-only models, which is the opposite in the ‘Disaster Alerts’ category. For the fine-grained categories, the multimodal model obtained the highest accuracy in the vast majority of the categories, especially the ‘rocket launch’ category, which obtained 92.5% with less than 75% precision for both VO and AO. This indicates the rationality of building a multimodal situational awareness system, i.e., using different modalities for complementary information.
For inference time, the introduction of multi-branching structure and graph learning implies more computational consumption, so MHGSA-VO, MHGSA-AO, and MHGSA take about 46.1 ms, 74.7 ms, and 149.3 ms, respectively, in recognizing the disaster category of a single video clip, which is higher than the baseline model. However, it could be negligible in comparison to the length of the video clip (10 s). The inference time may vary with the size of the number of frame samples K . In addition, as mentioned previously, the training process of MHGSA involves multiple stages. The feature extractors of the corresponding modalities need to be trained in advance to obtain a reasonable representation of the multimodal data. This may imply a more tedious training process when more modalities are introduced. A potential challenge is to introduce textual information, including plain text and text that appears in visual and audio, to obtain more comprehensive information for disaster event recognition. In addition, graph learning approaches may help to introduce modal information with multiple views and features.

4. Conclusions

In this work, we present a multimodal hierarchical graph-based situational awareness system (MHGSA) for rapid disaster response. The multimodal feature extraction module in this system employs a multi-branching architecture that allows it to provide independent parameters for coarse and fine granularity branches to improve model performance in end-to-end joint optimization. In addition, a multimodal hierarchical graph construction method is used for visual and audio feature fusion. The proposed MHGSA has been validated against several disaster-related datasets and has achieved promising results by outperforming the baseline model in both unimodal and multimodal scenarios.

Author Contributions

Conceptualization, K.P.S. and L.M.A.; Methodology, J.C.; Software, J.C.; Investigation, J.C. and H.X.; Writing—original draft, J.C.; Writing—review & editing, K.P.S., L.M.A. and J.S.; Supervision, K.P.S. and J.S.; Project administration, H.X. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

The data used in this study is publicly available in [32].

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Priya, S.; Bhanu, M.; Dandapat, S.K.; Ghosh, K.; Chandra, J. TAQE: Tweet Retrieval-Based Infrastructure Damage Assessment During Disasters. IEEE Trans. Comput. Soc. Syst. 2020, 7, 389–403. [Google Scholar] [CrossRef]
  2. Sakaki, T.; Okazaki, M.; Matsuo, Y. Earthquake Shakes Twitter Users: Real-Time Event Detection by Social Sensors. In Proceedings of the 19th International Conference on World Wide Web, Raleigh, NC, USA, 26–30 April 2010; Association for Computing Machinery: New York, NY, USA, 2010; pp. 851–860. [Google Scholar]
  3. Chu, L.; Zhang, Y.; Li, G.; Wang, S.; Zhang, W.; Huang, Q. Effective Multimodality Fusion Framework for Cross-Media Topic Detection. IEEE Trans. Circuits Syst. Video Technol. 2016, 26, 556–569. [Google Scholar] [CrossRef]
  4. Blandfort, P.; Patton, D.; Frey, W.R.; Karaman, S.; Bhargava, S.; Lee, F.-T.; Varia, S.; Kedzie, C.; Gaskell, M.B.; Schifanella, R.; et al. Multimodal Social Media Analysis for Gang Violence Prevention. In Proceedings of the International AAAI Conference on Web and Social Media, Palo Alto, CA, USA, 25–28 June 2018. [Google Scholar]
  5. Zhou, H.; Yin, H.; Zheng, H.; Li, Y. A Survey on Multi-Modal Social Event Detection. Knowl. Based Syst. 2020, 195, 105695. [Google Scholar] [CrossRef]
  6. Muhammad, K.; Khan, S.; Elhoseny, M.; Hassan Ahmed, S.; Wook Baik, S. Efficient Fire Detection for Uncertain Surveillance Environment. IEEE Trans. Ind. Inform. 2019, 15, 3113–3122. [Google Scholar] [CrossRef]
  7. Muhammad, K.; Khan, S.; Palade, V.; Mehmood, I.; de Albuquerque, V.H.C. Edge Intelligence-Assisted Smoke Detection in Foggy Surveillance Environments. IEEE Trans. Ind. Inform. 2020, 16, 1067–1075. [Google Scholar] [CrossRef]
  8. Kyrkou, C.; Theocharides, T. EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2020, 13, 1687–1699. [Google Scholar] [CrossRef]
  9. Ramachandran, U.; Hong, K.; Iftode, L.; Jain, R.; Kumar, R.; Rothermel, K.; Shin, J.; Sivakumar, R. Large-Scale Situation Awareness with Camera Networks and Multimodal Sensing. Proc. IEEE 2012, 100, 878–892. [Google Scholar] [CrossRef]
  10. Fan, C.; Wu, F.; Mostafavi, A. A Hybrid Machine Learning Pipeline for Automated Mapping of Events and Locations From Social Media in Disasters. IEEE Access 2020, 8, 10478–10490. [Google Scholar] [CrossRef]
  11. Ektefaie, Y.; Dasoulas, G.; Noori, A.; Farhat, M.; Zitnik, M. Multimodal Learning with Graphs. Nat. Mach Intell. 2023, 5, 340–350. [Google Scholar] [CrossRef]
  12. Alam, F.; Joty, S.; Imran, M. Graph Based Semi-Supervised Learning with Convolution Neural Networks to Classify Crisis Related Tweets. In Proceedings of the International AAAI Conference on Web and Social Media, Palo Alto, CA, USA, 25–28 June 2018. [Google Scholar]
  13. Luo, C.; Song, S.; Xie, W.; Shen, L.; Gunes, H. Learning Multi-Dimensional Edge Feature-Based AU Relation Graph for Facial Action Unit Recognition. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, Vienna, Austria, 23–29 July 2022; pp. 1239–1246. [Google Scholar]
  14. Liu, Q.; Xiao, L.; Yang, J.; Wei, Z. CNN-Enhanced Graph Convolutional Network with Pixel- and Superpixel-Level Feature Fusion for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2021, 59, 8657–8671. [Google Scholar] [CrossRef]
  15. Sui, L.; Guan, X.; Cui, C.; Jiang, H.; Pan, H.; Ohtsuki, T. Graph Learning Empowered Situation Awareness in Internet of Energy with Graph Digital Twin. IEEE Trans. Ind. Inform. 2023, 19, 7268–7277. [Google Scholar] [CrossRef]
  16. Zheng, S.; Zhu, Z.; Liu, Z.; Guo, Z.; Liu, Y.; Yang, Y.; Zhao, Y. Multi-Modal Graph Learning for Disease Prediction. IEEE Trans. Med. Imaging 2022, 41, 2207–2216. [Google Scholar] [CrossRef] [PubMed]
  17. Hou, Y.; Song, S.; Yu, C.; Wang, W.; Botteldooren, D. Audio Event-Relational Graph Representation Learning for Acoustic Scene Classification. IEEE Signal Process. Lett. 2023, 30, 1382–1386. [Google Scholar] [CrossRef]
  18. Endsley, M.R. Toward a Theory of Situation Awareness in Dynamic Systems. Hum Factors 1995, 37, 32–64. [Google Scholar] [CrossRef]
  19. Liu, X.; Zhang, L.; Li, T.; Wang, D.; Wang, Z. Dual Attention Guided Multi-Scale CNN for Fine-Grained Image Classification. Inf. Sci. 2021, 573, 37–45. [Google Scholar] [CrossRef]
  20. Won, C.S. Multi-Scale CNN for Fine-Grained Image Recognition. IEEE Access 2020, 8, 116663–116674. [Google Scholar] [CrossRef]
  21. Qiu, Z.; Hu, M.; Zhao, H. Hierarchical Classification Based on Coarse- to Fine-Grained Knowledge Transfer. Int. J. Approx. Reason. 2022, 149, 61–69. [Google Scholar] [CrossRef]
  22. Zhu, X.; Bain, M. B-CNN: Branch Convolutional Neural Network for Hierarchical Classification. arXiv 2017, arXiv:1709.09890. [Google Scholar]
  23. Tan, M.; Le, Q.V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019. [Google Scholar]
  24. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 248–255. [Google Scholar]
  25. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
  26. Hu, J.; Shen, L.; Sun, G. Squeeze-and-Excitation Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 7132–7141. [Google Scholar]
  27. Wei, J.; Zhang, Q.; Ning, W. Self-Supervised Learning Representation for Abnormal Acoustic Event Detection Based on Attentional Contrastive Learning. Digit. Signal Process. 2023, 142, 104199. [Google Scholar] [CrossRef]
  28. Bresson, X.; Laurent, T. Residual Gated Graph ConvNets. arXiv 2018, arXiv:1711.07553. [Google Scholar]
  29. Kipf, T.N.; Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. arXiv 2017, arXiv:1609.02907. [Google Scholar]
  30. Marcheggiani, D.; Titov, I. Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling. arXiv 2017, arXiv:1703.04826. [Google Scholar]
  31. Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv 2017, arXiv:1412.6980. [Google Scholar]
  32. Chen, H.; Xie, W.; Vedaldi, A.; Zisserman, A. Vggsound: A Large-Scale Audio-Visual Dataset. In Proceedings of the ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 4–8 May 2020; pp. 721–725. [Google Scholar]
  33. Tran, D.; Wang, H.; Torresani, L.; Ray, J.; LeCun, Y.; Paluri, M. A Closer Look at Spatiotemporal Convolutions for Action Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 6450–6459. [Google Scholar]
  34. Kong, Q.; Cao, Y.; Iqbal, T.; Wang, Y.; Wang, W.; Plumbley, M.D. PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition. IEEE/ACM Trans. Audio Speech Lang. Process. 2020, 28, 2880–2894. [Google Scholar] [CrossRef]
Futureinternet 16 00161 g001
Figure 1. The proposed multimodal hierarchical graph-based situation awareness (MHGSA) system.
Figure 1. The proposed multimodal hierarchical graph-based situation awareness (MHGSA) system.
Futureinternet 16 00161 g001
Futureinternet 16 00161 g002
Figure 2. Proposed multi-branch feature extraction module for constructing node features.
Figure 2. Proposed multi-branch feature extraction module for constructing node features.
Futureinternet 16 00161 g002
Futureinternet 16 00161 g003
Figure 3. Structure of the proposed multi-branch feature extraction module for constructing node features from video frames.
Figure 3. Structure of the proposed multi-branch feature extraction module for constructing node features from video frames.
Futureinternet 16 00161 g003
Futureinternet 16 00161 g004
Figure 4. Audio feature extraction module for constructing multi-grained audio event nodes features.
Figure 4. Audio feature extraction module for constructing multi-grained audio event nodes features.
Futureinternet 16 00161 g004
Futureinternet 16 00161 g005
Figure 5. Multimodal hierarchical graph construction.
Figure 5. Multimodal hierarchical graph construction.
Futureinternet 16 00161 g005
Futureinternet 16 00161 g006
Figure 6. Classifier and loss calculation for MHGSA.
Figure 6. Classifier and loss calculation for MHGSA.
Futureinternet 16 00161 g006
Futureinternet 16 00161 g007
Figure 7. Fine- and coarse-grained categories of the extracted VGGSound-DR dataset.
Figure 7. Fine- and coarse-grained categories of the extracted VGGSound-DR dataset.
Futureinternet 16 00161 g007
Table 1. Experiment on VGGSound-DR in visual-only mode.
Table 1. Experiment on VGGSound-DR in visual-only mode.
ModeModelAvg. Acc. (S.D.)Best Acc.Avg. Time (ms/Video)Params
Visual-OnlyResNet-5054.1 (±0.5)%54.8%13.126M
Eff.Net-b157.2 (±0.6)%58.0%18.28M
R3D-18 *58.2 (±0.2)%58.5%10.433M
MHSA-VO59.4 (±0.5)%60.0%18.815M
MHSA-VO *64.9 (±0.4)%65.3%37.815M
MHGSA-VO *66.2 (±0.4)%66.7%46.117M
* Model using multiple frames.
Table 2. Experiment on VGGSound-DR in audio-only mode.
Table 2. Experiment on VGGSound-DR in audio-only mode.
ModeModelAvg. Acc. (S.D.)Best Acc.Avg. Time (ms/Video)Params
Audio-OnlyResNet-5064.8 (±0.2)%65.1%49.226M
Eff.Net-b165.3 (±0.3)%65.8%52.68M
PANNs51.9 (±0.5)%52.6%42.675M
MHSA-AO66.7 (±0.4)%67.2%52.815M
MHGSA-AO67.3 (±0.4)%67.8%74.717M
Table 3. Experiment on VGGSound-DR in multimodal mode.
Table 3. Experiment on VGGSound-DR in multimodal mode.
ModeModelAvg. Acc. (S.D.)Best Acc.Avg. Time (ms/Video)Params
Multi-ModalResNet-50-MM69.1 (±0.4)%69.7%61.452M
Eff.Net-b1-MM71.3 (±0.5)%71.9%74.317M
MHSA *75.9 (±0.5)%76.5%91.130M
MHGSA *77.3 (±0.3)%77.6%149.334M
* Model using multiple frames on a visual path.
Table 4. Classification precision of MHGSA for coarse-grained and fine-grained categories in visual-only, audio-only, and multimodal cases.
Table 4. Classification precision of MHGSA for coarse-grained and fine-grained categories in visual-only, audio-only, and multimodal cases.
Coarse-GrainedVOAOMMFine-GrainedVOAOMM
Natural Disasters94.486.197.4Hail91.784.292.3
Thunder56.781.287.1
Tornado Roaring95.053.772.1
Volcano explosion71.257.480.4
Conflicts77.279.691.1Cap gun shooting50.774.175.4
Machine gun shooting72.380.087.0
Missile launch73.465.192.5
Disaster Alerts90.396.298.8Ambulance siren33.348.744.2
Civil defense siren55.789.691.7
Fire truck siren67.449.165.9
Police car (siren)49.244.051.0
Smoke detector beeping86.091.797.8
Overall88.488.796.4Overall67.068.378.2
Highest results are highlighted in bold.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Chen, J.; Seng, K.P.; Ang, L.M.; Smith, J.; Xu, H. AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses. Future Internet 2024, 16, 161. https://doi.org/10.3390/fi16050161

AMA Style

Chen J, Seng KP, Ang LM, Smith J, Xu H. AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses. Future Internet. 2024; 16(5):161. https://doi.org/10.3390/fi16050161

Chicago/Turabian Style

Chen, Jieli, Kah Phooi Seng, Li Minn Ang, Jeremy Smith, and Hanyue Xu. 2024. "AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses" Future Internet 16, no. 5: 161. https://doi.org/10.3390/fi16050161

APA Style

Chen, J., Seng, K. P., Ang, L. M., Smith, J., & Xu, H. (2024). AI-Empowered Multimodal Hierarchical Graph-Based Learning for Situation Awareness on Enhancing Disaster Responses. Future Internet, 16(5), 161. https://doi.org/10.3390/fi16050161

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop