Hierarchical semantic interaction-based deep hashing network for cross-modal retrieval

Deep cross-modal hashing

In these years, deep learning has been widely used in cross-modal retrieval tasks due to its appealing performance in various computer vision applications such as Vasan et al. (2020), Dwivedi et al. (2021), Bhattacharya et al. (2020), Gadekallu et al. (2020), Jalil Piran et al. (2020), Jalil Piran, Islam & Suh (2018), and Joshi et al. (2018). It obtains hash representations and hash function learning in an optimal end-to-end framework which also demonstrates the robustness. One of the most typical is deep cross-modal hashing (DCMH) (Jiang & Li, 2017), which firstly applies the deep learning architecture to cross-modal hashing retrieval. The self-constraint and attention-based hashing network (SCAHN) (Wang et al., 2020a) explores the hash representations of intermediate layers in an adaptive attention matrix. The correlation hashing network (CHN) (Cao et al., 2016) adopts the triplet loss measured by cosine distance to reveal the semantic relationship between instances and acquires high-ranking hash codes. Pairwise relationship guided deep hashing (PRDH) (Yang et al., 2017) leverages pairwise instances as input for each modality where supervised information is fully explored to measure the distance of intra- and inter-modality, respectively. Cross-modal hamming hashing (CMHH) (Cao et al., 2018) learns high-quality hash representations and hash codes with a well-designed focal loss and a quantization loss. Although these algorithms mentioned above have obtained high performance in CMH tasks, they ignore the rich spatial information from intermediate layers, which is essential to the modality-invariant hash representations learning procedure.

Multi-label similarity learning

In the real-world scenario or benchmark datasets, instances are always related to multiple labels. Thus multi-label learning has attracted more and more attention in various applications. However, most existing CMH methods adopt single label constraints to measure the similarity among intra- and inter-modality instances. Self-supervised adversarial hashing (SSAH) (Li et al., 2018), which uses an independent network to learn multi-label representations, and thus the semantic correlations are preserved. However, it only takes the multi-label information to supervise the label network training, and the original images or text are still measured by single-label. Improved deep hashing network (IDHN) (Zhang et al., 2019) introduces pairwise similarity metrics to fit the multi-label instances applications. In contrast, this method concentrates on the single modality hashing retrieval. Different from these methods that apply multi-label information, our HSIDHN employs both single-label and multi-label constraint to learn more robust hash representations. Significantly, the Maximum Mean Discrepancy (MMD) is adopted as the multi-label calculation criterion. To our knowledge, the HSIDHN is the first method using MMD in the deep CMH framework.

Problem definition

We use uppercase letters to represent matrices, such as X, and lowercase letters representing vectors, such as y. The transpose of G are denoted as G^T, and sign function sign(·) is defined as:

(1) $$\begin{array}{c}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(x)=\{\begin{array}{cc}1\hfill & x\ge 0\hfill \\ -1\hfill & x<0\hfill \end{array}\hfill \end{array}$$

We assume the training dataset $O={\left\{o_i\right\}}_{i=1}^N$ with N instances, which all of them have label information and image-text modality feature vectors. The ith training instance is denoted as $o_i=\left(v_i,t_i\right)$, where v_i ∈ $R^{d_v}$ and t_i ∈ $R^{d_t}$ denote the d_v and d_t dimensional feature vectors of image and text respectively. Moreover, the label based semantic similarity matrix is defined as $S_{\ast }=\left\{S_{\ast }^{vt},S_{\ast }^{vv},S_{\ast }^{tt}\right\}$, where $S_{\ast }^{vv}=\left\{S_{ij}^{vv}|i,j=1,2,\dots ,N\right\}\in R^{N\times N}$ and $S_{\ast }^{tt}=\left\{S_{ij}^{tt}|i,j=1,2,\dots ,N\right\}\in R^{N\times N}$ denote the intra-modality similarity matrix of image and text, $S_{\ast }^{vt}=\left\{S_{ij}^{vt}|i,j=1,2,\dots ,N\right\}\in R^{N\times N}$ denotes the inter-modality similarity matrix between image and text. S_* means the “hard” similarity and “soft” similarity when * = h or * = r.

Given the training datasets O and S, the main objective of our proposed HSIDHN is to learn two modality discriminative hash functions h^(v)(v) and h^(t)(t) for image and text modalities, which can map features vectors into a compact binary space and preserve relationship and correlation among instances. The learning framework can be roughly divided into two parts, hash representations learning section and hash function learning section. Therefore, $F=\left\{f_{v_i}|i=1,2,\cdots ,N\right\}\in R^{N\times c}$ and $G=\left\{g_{t_i}|i=1,2,\cdots ,N\right\}\in R^{N\times c}$ are used to denote the learned hash representations of image-modality and text-modality. Besides, $B=\left\{B_i|i=1,2,\cdots ,N\right\}\in R^{N\times c}$ is the projection of the final hash codes from F and G by simply using a sign function B = sign(F + G).

The architecture of our proposed HSIDHN which consists of two parts. One component is the backbone network used to extract hash representations. The other one is the Bidirectional Bi-linear Interaction (BBI) module used to capture the hierarchical semantic correlation of each modality data from different levels.

Network framework of HSIDHN

For most cross-modal hashing retrieval methods, the multi-level and multi-scale information cannot be fully explored. Thus, it may limit the invariance and discrimination of the final learned hash representations. In this paper, we propose a novel Hierarchical Semantic Interaction-based Deep Hashing Network (HSIDHN) for large-scale cross-modal retrieval, where a multi-level and multi-scale interaction based network and bidirectional bi-linear interaction module are used to explicitly specifics spatial and semantic information. The general architecture of our proposed HSIDHN is shown in Fig. 1.

In terms of the multi-level and multi-scale hash representations generation, HSIDHN contains double end-to-end network to learn hash functions and hash representations from text and image modality. The deep feature extraction procedure is conducted on Resnet (He et al., 2016), and pair-wise pairs of images and text are applied as input for the Image Network and Text Network. For the Text Network, the bag-of-words (BoW) vector policy has been widely adopted to extract features from Text Networks since Jiang & Li (2017). However, it is inappropriate to learn rich features demanded by the hash functions learning procedure because of BoW vectors’ sparsity. To solve this issue, a multi-scale operation is leveraged by multiple pooling layers, and the vectors are resized by bi-linear-interpolation. Finally, these vectors are concatenated together and fed to the text network, which consequently is helpful to construct semantic correlation for the text. Both image and text networks generate multi-level feature information from mid-layers by exploring an adaptive average pooling. Motivated by SPPNet (Purkait, Zhao & Zach, 2017), the multi-scale fusion structure is also applied to hash representations from each layer to obtain rich spatial information. Therefore, the semantic relevance and correlation from different layers can be fully explored to enhance the invariance of hash representations for both image and text modality. The whole architecture is shown in Fig. 2.

As different layers of the network have complementary hash representations. Thus the interaction among different layers may help to learn discriminative hash representations. A bidirectional bi-linear integration module integrates the multi-scaled multi-level hash representations from intermediate layers to learn more robust hash representations. The bidirectional bi-linear integration policy has two main procedures, the bottom-up and top-down progress. The bottom-up procedure allows shadow activation, which always being covered by the top layers to accumulate slowly. The top-down operation can take advantage of contextual and spatial information. Therefore, the combination of bottom-up and top-down interaction policy could generate better hash representations. We assume there are K multi-scaled and multi-levels hash representations generated from Resnet (He et al., 2016), and the f^(k) and g^(k) are output from K^th block.

Feeding an instance to the network, the output feature map is X ∈ R^c, where c is the dimension of X, and Z ∈ R^o is the bi-linear representation with dimension o. For Z_i in $z=\left[z_1,z_2,\cdots ,z_o\right]$, the bi-linear pooling interaction can be defined as:

(2) $$z_i=I(\mathrm{x},\mathrm{x})={\mathrm{x}}^T{W}_i\mathrm{x}$$where W_i is the weighted projection matrix need to be learned, I(x, x) is the interaction function. According to Rendle (2010), the weighted projection matrix in Eq. (2) can be rewritten by factorizing as:

(3) $$z_i=I(\mathrm{x},\mathrm{x})={\mathrm{x}}^T{U}_i{V}_i^T\mathrm{x}=U_i^T\mathrm{x}\circ V_i^T\mathrm{x}$$where U_i ∈ R^c and V_i ∈ R^c. And consequently, the output features z is calculated as:

(4) $$\mathbf{z}=P^T\left(U^T\mathbf{x}\circ V^T\mathbf{x}\right)$$where U, V and P are projection matrices.

The proposed bidirectional bi-linear integration policy aims to explore the interaction between intermediate layers. Taking the image modality as an example, we firstly select two layers f_i and f_j from multi-scaled multi-level hash representations. Hence, inspired by Yu et al. (2018), the Eq. (4) can be rewritten as:

(5) $$\mathbf{z}=P^T\left(U^T{\mathbf{f}}_{\mathbf{i}}\circ V^T{\mathbf{f}}_{\mathbf{j}}\right)$$where P,U,V are projection matrices. To reduce number of parameters, the bi-linear pooling is divided into two stages, which can be formulated as:

(6) $$\mathbf{P}\mathbf{B}=U^T{f}_i\circ V^T{f}_j$$

(7) $$\mathbf{P}\mathbf{L}=P^{T}PB$$

Thus, the interaction between two layers can be defined as:

(8) $$Z==P^T\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}\left(\mathrm{P}\mathrm{B}\left(X_1\right)\circ \mathrm{P}\mathrm{B}\left(X_2\right)\right)$$

(9) $$=P^T\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}\left(I\left(X_1,X_2\right)\right)$$

In this paper, the interaction is applied on multi-layer and the representation of each layers is defined as:

(10) $$Z_v=BI\left(f_1,f_2,f_3\right)=P^T\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left[I\left(f_1,f_2\right),I\left(f_1,f_3\right),I\left(f_2,f_3\right)\right]$$

(11) $$\begin{array}{c}{Z}_t=BI\left(g_1,g_2,g_3\right)\end{array}=P^T\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left[I\left(g_1,g_2\right),I\left(g_1,g_3\right),I\left(g_2,g_3\right)\right]$$where f₁, f₂, f₃ and g₁, g₂, g₃ are hash representations from different layers of image and text modality, and concat denotes the concatenation operation. However, the bi-linear pooling operation from one direction may lead to a vanishing gradient problem. This is because parameters from intermediate layers update faster than the end. Thus, the bidirectional bi-linear integration policy can be written as:

(12) $$\begin{array}{l}{Z}_v=BBI\left(f_1,f_2,f_3\right)=P^T\text{concat}\left[Z_1,Z_2\right]\\ =P^T\text{concat}[\underset{\text{bottom-top}}{\underbrace{I\left(\cdots I\left(I\left(f_1,f_2\right),f_3\right)\right)}},\underset{\text{top-down}}{\underbrace{I\left(I\left(I\left(f_3,f_2\right),f_1\right)\cdots ,f_1\right)}}],\end{array}$$

(13) $$\begin{array}{l}{Z}_t=BBI\left(g_1,g_2,g_3\right)=P^T\text{concat}\left[Z_1,Z_2\right]\\ =P^T\text{concat}[\underset{\text{bottom-top}}{\underbrace{I\left(\cdots I\left(I\left(g_1,g_2\right),g_3\right)\right)}},\underset{\text{top-down}}{\underbrace{I\left(I\left(I\left(g_3,g_2\right),g_1\right)\cdots ,g_1\right)}}],\end{array}$$where f₁, f₂, f₃ and g₁, g₂, g₃ are hash representations from different layers of image and text modality and concat denotes the concatenation operation. And Z₁ and Z₂ are the multi-layer interaction from bottom-up and top-down procedure.

Optimization

By aggregating the Eqs. (20), (29) and (30), we get the general objective function as:

(31) $$\underset{B,{\theta }_x,{\theta }_y}{min}\mathcal{L}=\underset{B,{\theta }_x,{\theta }_y}{min}\left({\mathcal{L}}_h+\gamma {\mathcal{L}}_r+\beta {\mathcal{L}}_q\right)$$where θ_x and θ_y are network parameters of image and text, and B is the learned binary codes. γ and β are hyper-parameters to control each part’s weights in the general objective function. We adopt an alternating optimization algorithm, and some parameters are fixed while other parameters are optimized.

Fix B, optimize θ_x and θ_y

The back propagation (BP) algorithm is adopted to update parameters θ_Dx, θ_Dy by descending gradients:

(32) $$\theta \leftarrow \theta -\eta \cdot {\mathrm{\nabla }}_{\theta }{\displaystyle \frac{1}{n}\mathcal{L}}$$

Fix θ_x and θ_y, optimize B

As the θ_x and θ_y is fixed, the optimization of binary codes B can be defined as:

(33) $$\begin{array}{l}\underset{B}{\min }tr\left(B^T(\eta (F+G))\right)=\eta {\displaystyle \sum _{i,j}{B}_{ij}}\left(F_{ij}+G_{ij}\right)\\ s.t.B\in {\{-1,+1\}}^{c\times N}\end{array}$$

which can be formulated as:

(34) $$\begin{array}{c}B=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}(\eta (F+G))\hfill \end{array}$$

Datasets

MIRFlickr-25K (Huiskes & Lew, 2008): There are 25,000 instances images in the MIRFlickr-25K collected from Flickr with several textual descriptions. Following the standard experimental settings proposed in DCMH (Jiang & Li, 2017), 20,015 data are samples are leveraged with less than 24 distinct labels. A 1,386-dimensional BoW vector is generated for each text description.

NUS-WIDE (Chua et al., 2009): There are 269,468 image-text instances pair belonging to 81 categories collected from real-world web datasets. Each textual description for image instance is represented by a 1,000-dimensional binary vector. In this paper, 21 of the most frequently used categories are chosen with 190,421 images and related text.

We randomly select 10,000 and 10,500 instances from MIRFlickr-25K and NUS-WIDE as the training set to reduce the computational cost. Meanwhile, we randomly choose 2,000 and 2,100 samples as the query set for MIRFlickr-25k and NUS-WIDE, respectively. The remained data are leveraged as a retrieval set after the query set is selected. Images are normalized before inputting to the network. The details of dataset division are summarized in Table 1.

Implementation details

Our HSIDHN is implemented using the Pytorch (Paszke et al., 2019) framework and performed on one TITAN Xp GPU server. In the end-to-end framework, Resnet-34 is applied as the backbone network. For the bidirectional bi-linear interaction module, the last three parts of hash representations are integrated to enhance the capability hash representations, respectively. Moreover, the multi-scale fusion of text is applied on pooling sizes of 1, 5, 10, 15, 30. The image network parameters initialization is pre-trained on the ImageNet (Russakovsky et al., 2015) dataset, and the network for text-modality is initialized by Normal distribution $N\left(\mu ,{\sigma }^2\right)$ with μ = 0 and σ = 0.1. Learning rate is initialized in 10^−1.1 and gradually decays to 10^−6.1, and the mini-batch size is 128. Besides, we use the SGD as our optimization for image and text networks.

Evaluation and baselines

To measure CMH methods’ performance, we adopt hamming ranking as the retrieval protocol, which sorting instances by hamming distance. In this paper, the PR Curves and Mean Average Precision (MAP) (Liu et al., 2014) are leveraged as the evaluation criteria for HSIDHN.

The HSIDHN is compared with several baseline methods including SCM (Zhang & Li, 2014), DCMH (Jiang & Li, 2017), CMHH (Cao et al., 2018), PRDH (Yang et al., 2017), CHN (Cao et al., 2016), SepH (Lin et al., 2015) and SSAH (Li et al., 2018). Table 2 and Table 3 illustrates the MAP results of HSIDHN and other methods in different lengths of hash codes. Fig. 3 and Fig. 4 demonstrate the PR curves of different length of hash codes conducted on MIRFlickr-25K and NUS-WIDE. From the result, we can get the following observations and analysis.

• HSIDHN dramatically exceeds other methods on different lengths of hash codes in consideration of the MAP, which reveals the advantages of the multi-scale and multi-level interaction module. It is worth nothing that HSIDHN outperform DCMH by 9.8–3.9% and 17.77–12.93% in terms of MAP for Image-query-Text and Text-query-Image tasks on MIRFLICKR-25K and NUS-WIDE. This is mainly because that the multi-scale process could explore different receptive field of input data, where information with different size could be fully used. Additionally, the hierarchical feature interaction could explore the useful specific feature from different layer and integrated them to enhance the capability of final hash representations.

• The high performance of HSIDHN is partly because the semantic relation and correlation from different intermediate layers are explored by bidirectional bi-linear interaction module. Besides, the multi-scale fusion could further make full use of leverage spatial information.

• There is a kind of imbalance between the performance of image-query-text and text-query-image in almost all the other baselines. However, this phenomenon could be effectively avoided in HSIDHN. This is mainly due to the dual-similarity measurement, which can be sufficient to unify the image-modality and text-modality in the latent common space.

• All deep CMH methods, including DCMH, CHN, PRDH, CMHHH, and SSAH obtain higher performance than other shadow hashing methods such as SePH and SCM. This demonstrates the effectiveness and efficiency of deep neural networks in hash representations and hash function learning, which is more robust than the non-deep neural network methods. Thus, deep neural network based deep hashing methods could obtain better performance.

Ablation study

In this section, the importance of each component of HSIDHN is validated. To evaluate the effect of the different modules, the settings of experimental are defined as:

• HSIDHN-SIM is designed by replacing the dual-similarity by single hamming distance measurement.

• HSIDHN-BBI is designed by removing the interaction between layers, and the final hash representations are generated from the final layer of the network.

The results of the ablation study are shown in Table 4. Firstly, there is no doubt that the dual-similarity measurement is better than the hamming-based distance. This is mainly due to that the fine-grained dual-similarity could better preserve the semantic relationship. Moreover, the performance experiences a significant drop when the BBI policy is removed. This may partly because the BBI policy can explore the more robust hash representations from intermediate layers of networks.

Time complexity

The Eq. (31) is taken as the final loss function to train. Each term of the Eq. (31) is MSE loss or max log-likelihood loss which are general in cross-modal retrieval applications. A server with a Titan Xp card is leveraged to train. For the whole HSIDHN, the training and validation procedure need around 28 h for MIRFLICKR-25K and 53 h for NUS-WIDE. The proposed HSIDHN have a fast convergence rate than other deep hashing methods, as the introduction of bidirectional bi-linear interaction and dual-similarity measurement.

Limitation of HSIDHN and future work

Although the appealing performance has been obtained in the HSIDHN framework, there are still some limitations. Firstly, the network architecture, especially the multi-scale and multi-level features extraction process, requires huge GPU memory to train. The model compression might be the possible solution to solve it. Secondly, the performance of text-query-image is not as significant as image-query-text. This is partly because of the sparsity of features learning from text modality. Some pretraining model is the possible way to learn higher quality features from the original text.