Deep CNN Combined With Relevance Feedback for Trademark Image Retrieval

4.1 Feature Vector Database Generation

4.1.1 Datasets

FlickrLogos-27¹, FlickrLogos-32¹, and FlickrLogos-32 PLUS Datasets are used for evaluating the experiments. The details are depicted in Table 1. The openly accessible FlickrLogos-32 database includes images that are nontrademark images. The FlickrLogos-32 PLUS dataset is generated by eliminating these nontrademark images and including a number of trademark images.

Table 1:

Trademark Databases Used for Experimentation.

Database	Total no. of images	Trademark category
Publicly accessible: FlickrLogos-27 databasea	1080	27
Publicly accessible: FlickrLogos-32 databasea	8240	32
Generated: FlickrLogos-32 PLUS database	8550	38

1. ^ahttp://www.multimedia-computing.de/flickrlogos/data/

Figure 2 shows example images of trademarks utilized in the implementation commencing with the FlickrLogos-32 PLUS database. The low-level features of an image such as shape, color, and texture, express the visual content of the image [10]. Shape, color, and texture feature extraction are implemented to retain the database feature’s vector of the trademark images. Circularity and Fourier descriptor methods are executed for shape, color moments, color histogram, and color correlogram practices that are employed to express color, and the Haar wavelet and Gabor wavelet are used for texture.

Figure 2:

Example Trademarks From FlickrLogos-32 PLUS Database.

4.2 Feature Set Optimization

The optimized feature set is obtained by executing the PSO method, which is given to the SOM for training. The retrieval of image schemes includes various features of database depiction comprising unnecessary features not functional for the retrieval process. Hence, optimization of features is needed to eliminate these unnecessary features from the database features. The search room contains the particle’s present coordinate, in the structure of optimization search. The best solution is acknowledged by retaining the trace of every particle’s coordinates within the crisis area. This value is referred to as P_best. The top value attained by any particle among the entire particles in the area is known as Q_best. The k^th particle can be represented as C_k = (C_k1, C_k2, ………… C_kn), and the velocity, i.e. the particle’s rate of progress, is designated as T_k = (T_k1, T_k2 ………… T_kn) particle and alters its positions and velocity through the subsequent equations:

where

T_k (v) and T_k (v + 1) denote the velocity of the k^th particle on the iteration v and v + 1, respectively;

C_k (v) and C_k (v + 1) denote the k^th particle position on the iteration v and v + 1, respectively;

d₁ and d₂ depict the cognitive acceleration coefficient and social acceleration coefficient; and a₁ and a₂ being random number ranges between (0, 1).

PSO is a transformative computational method showing swarm intelligence and optimizing feature set; however, it does not ensure optimization [1]. Consequently, the clustering of these features is executed utilizing SOM. The RF is implemented using some clusters near the query, identified by averaging their features.

4.3 Cluster Creation Through the Self-Organizing Map

The planning of a dataset as an input, Sⁿ, on top of a typical two-dimensional node array is defined by SOM. The parametric reference vector nk∈Sn is linked with each node k in the plan.

The proposed work projected the node array onto a rectangular pattern. The input is mapped by evaluating every input vector v against the nk:P, and the top correspondent is determined. Euclidean distance is used for comparing each input vector v∈Sn with the nk:P. The winning node e is computed as follows:

Here, v is mapped through e, comparative to the component values n_k. Topographically closer nodes within the array exhibit learning through a similar input. The updation formula is:

where the discrete-time coordinate is t, and the region-defining function is h_e,k. The random values are used for the nk:P in the early phase [14].

4.4 Relevance Feedback

The query modification-based RF is used in the proposed work and integrated with the DCNN. Query modification is a hybrid through the improved query point (IQP), query expansion (QE), and query modification (QM) methods. The testing is done by executing 10 RF iterations. The positive image query points of every iteration are clustered by employing the k-means clustering algorithm. Each cluster is allotted a number. The feedback information from the user is maintained as log records in the following tables, and the patterns found repeatedly are mined in the next query.

1. Initial log table – includes the query image, relevant images, query point, and iteration number.

2. Query point table – includes the query image and query point.

3. Navigation action table – includes the query image, cluster number, and iteration number.

   This table is created as follows:

   1. Using the results of retrieval for each iteration, the clustering of query points is done through the k-means algorithm and allocated a number.

   2. The cluster number and iteration number are stored in this table.

   3. A pattern for every query is constructed using these clusters. For instance, for a query image, the retrieval result images exist as 1st iterations: cluster 3, 2nd iterations: cluster 2, and 3rd iterations: cluster 1; then, the pattern is N31, N22, and N13. The mining is done using these sequential patterns through the Apriori algorithm for mining.

4. Log division table – encloses the query point and relevant image

The initial log table holds the information related to every feedback after each RF iteration. The navigation patterns are mined using the navigation action table. The relevant images are searched through the query point table and log division table. The sequential patterns obtained are used for constructing the navigation pattern tree. The sequential pattern is depicted through every branch of this tree. The query for the entire chronological patterns is exploited as the seed of the tree called the query kernel.

The following steps describe the search process:

1. The modified query point is obtained by averaging the features of images marked as positive.

2. The nearest query kernels are obtained to acquire the similar chronological patterns.

3. These similar trees of navigation pattern are utilized to get the neighboring leaf nodes.

4. The query points denoted by “t”, which are relevant, are obtained from the collection of the nearest leaf nodes.

5. The “n” significant images are obtained.

The suggested method implements step 1 for generating the IQP. Steps “2–5” are executed for QE. The QM process is implemented by obtaining new feature weights utilizing the positive image features taken from each feedback.

4.5 DCNN in Relevance Feedback

This paper proposes a CBIR framework for trademark images by integrating DCNN with an RF mechanism. The relevant images from the log records maintained in the above tables are given as input to the DCNN for training. The CaffeNet model [4] is utilized; an execution of the AlexNet model [15] is used for experimentation. The model includes eight neural network layers, which are trained; the initial five are convolutional, and the remaining three are fully connected. The max-pooling layers trail the 1st, 2nd, and 5th convolutional layers, whereas the ReLU nonlinearity [f(y) = max(0; y)] is employed to each fully connected and convolutional layer, excluding the last one indicated by FC8. The seventh neural network layer representations indicated by FC7 are extracted through the CaffeNet model. The CaffeNet model parameters are used to initialize the entire layers of the latest model, excluding the FC8, substituted by a modified classification layer depicting the given dataset labels. There are no changes up to the convolutional layers, which are initially a fully connected layer indicated by FC6; FC7 and FC8 layers are updated using error backpropagation. The feature representations are extracted through the FC7 layer’s activations having a feature dimension of 4096.

In the first RF iteration, the user gives feedback as relevant or irrelevant images to the system. This feedback information is utilized to train the DCNN model regarding the feature representation of relevant and irrelevant images. Subsequently, the feedback information in the next RF iterations is utilized in the framework, and the CNN model weights are updated, with the intention that relevant images move toward the query depiction, whereas the irrelevant ones deviates from the same. The representations acquired through a CNN model for a collection of images as input are modifiable by changing the model weights. The FC7 parameters are trained on positive/relevant and negative/ irrelevant images. The regression task during this training is executed through Euclidean loss. The query modification in RF is, thus, combined with DCNN for training upon relevant and irrelevant images.

Let the feature illustration for the query image that appeared in the FC7 layer is be s∈P4096×1.

Y+={Yp∈P4096×1,p=1,…,M} is the feature representation set of M images marked as relevant by the user in the feedback. Y−={Yq∈P4096×1,q=1,…,N} denotes the feature representation set of N irrelevant images marked by the user.

These relevant and irrelevant illustrations of the images are modified using retraining capabilities of the neural network. The following equations are utilized to deduce the modified target representations of the positive/relevant and negative/irrelevant images.

The above optimization problems are solved through gradient descent. The objective function’s first order gradients τ⁺ and τ⁻ are calculated as follows:

and

As a result, the update rules for the n^th iteration are devised as:

The preferred distance from the query representation is monitored by the parameter β∈[0,0.5]. The above feature representations are used for the n + 1 iteration as targets within the FC7 layer. The neural network and the DCNN regression task is devised and given further training through backpropagation. Resulting in the generation of positive/relevant and negative/irrelevant images representations nearer to the targets y^{(m + 1)}. Hence, the positive/relevant images come nearer to the query image within the FC7 layer, and the negative/irrelevant ones move far from the query. Accordingly, the RF feedback is incorporated by inputting the query and the feedback information of the images to the CNN input layer and getting the modified representations for FC7. This procedure is executed for every RF iteration; the CNN model is initialized with the preceding iteration’s parameters and re-trained for the modified set of positive/relevant and negative/irrelevant images [18]. The above process is executed by the system from the start for every new query.

The algorithmic steps for the suggested framework can be given as follows:

Input: Database Images from FlickrLogos-32 PLUS dataset and Query Image

Output: The nine most relevant images against the given query are retrieved

5.1 Implementation Setup

The dataset used for testing the framework are FlickrLogos-27, FlickrLogos-32, and FlickrLogos-32 PLUS database. The optimization of the database features from the overall 598 to the best features as 315,4615 to the selected number of best features as 2310 and 5034 number of features to the best 2519 number of features is done through the PSO algorithm for the FlickrLogos-27, FlickrLogos-32, and FlickrLogos-32 PLUS database, respectively. These optimized features are fed to the SOM for learning. The classes/clusters are formed by the SOM using these optimized features. The top three most relevant clusters are recognized by the system against a given query by averaging the image features within the cluster and the Euclidean distance between them. These three clusters are searched for relevant images.

First, 20 relevant/similar images are returned by the system based on similarity. For every query, 10 RF iterations are executed. For every RF iteration, user feedback is taken upon two images per feedback, as relevant or irrelevant. The CNN layers are initialized through the CaffeNet model parameters, except the FC8, which is replaced by a modified classification layer describing the dataset labels. The proposed work used 10 as relevant and 10 as irrelevant images to refine the model. The value of parameter β is set as 0.2, and the model training is done for one epoch for every RF iteration. The final output images retrieved are the most relevant nine images.

5.2 Analysis of Time Complexity

The PSO algorithm’s time complexity is O (2nd + n² + 2nd); here, n represents the particle number, and the search space’s dimension is d. For the SOM algorithm, it is given by O (s²); where s denotes the input sample size. The RF algorithm possesses the time and memory cost linear to the dimension of the total features [25]. Therefore its time complexity is calculated as:

Here, P = ∑a=1mpa, where p_a represents the a^th feature’s dimension.

All the convolutional layers in the DCNN have time complexity as:

where i represents the convolutional layer’s index, and n describes the depth, i.e. the convolutional layer number. The filter number is also recognized as the width in the i^th layer depicted as m_i. Whereas m_{i − 1} represents the input channels number of the i^th layer. The spatial size, i.e. the filter’s length is p_i. The spatial size of the output feature map is denoted by n_i. This time complexity is applicable at both times, i.e. testing and training, but through a different range. The training time required for every image is approximately 3 times (one for forward and two for backward propagation, respectively) of the time needed for testing per image. The above formula does not account the time cost pooling layers and fc layers; which acquires 5–10% of the computational time.

The overall time complexity of the suggested algorithm can be given as:

where T1, T2, T3, and T4 are the time complexity of PSO, SOM, RF, and DCNN algorithm, respectively. The T1 and T2 cost are incurred at one time in the preprocessing stage, whereas the T3 and T4 costs are incurred at each query stage.

5.3 Performance Results in Terms of Retrieval

The final retrieved images against a given query using the influence of the DCNN are depicted in Tables 2–4, respectively, on the basis of the similarities of color, texture, and shape.

Table 2:

Color Similarity-Based Output Retrieved Images.

Table 3:

Texture Similarity-Based Output Retrieved Images.

Table 4:

Shape Similarity-Based Output Retrieved Images.

Table 6:

Retrieval Results for 10 Images Given as a Query.

S. no.	Query image	Precision	Recall	Accuracy (%)	F1-score
1		0.975	0.922	93.58	0.947
2		0.974	0.948	93.82	0.960
3		0.967	0.918	93.86	0.941
4		0.980	0.940	94.59	0.959
5		0.964	0.946	94.77	0.954
6		0.980	0.942	93.71	0.96
7		0.940	0.917	92.97	0.928
8		0.979	0.932	94.81	0.954
9		0.982	0.948	94.76	0.964
10		0.988	0.979	94.54	0.983

Table 7:

Similarity (%) of Color, Texture, and Shape for the Topmost Nine Relevant Output Images.

S. no.	Relevant output image	Similarity (%) of color	Similarity (%) of texture	Similarity (%) of shape
1		89.91	87.98	92.11
2		92.284	91.033	93.654
3		94.605	93.126	95.12
4		98.769	97.012	98.561
5		95.721	96.117	87.274
6		78.214	85.813	88.891
7		87.661	82.929	85.721
8		78.939	84.862	74.632
9		75.127	84.321	68.519

The metrics used for the evaluation of the framework are accuracy, precision, recall, and F₁-score calculated as below:

1. Accuracy = (TP + TN)/(TP + TN + FP + FN)

2. Precision = TP/(TP + FP)

3. Recall = TP/(TP + FN)

4. F1−Score=2Precision∗RecallPrecision+Recall

Here,

TP = True positive: positive example and prediction positive

TN = True negative: negative example and prediction negative

FP = False positive: negative example but prediction positive

FN = False negative: positive example but prediction negative

The confusion matrix, as shown in Table 5, maintains the examples from every class identified as correct and incorrect.

For example, considering every RF iteration, with output retrieval of 20 images, the TP, TN, FP, and FN calculation gives the count as 8, 8, 2, and 2 correspondingly.

Accuracy calculation is given by:

The retrieval results for 10 example images as queries are reported in Table 6. Table 7 describes the similarity (%) for color, texture, and shape for the topmost output images retrieved. In Tables 8 and 9, the suggested framework is compared with that of Su et al. [26] and the recent deep learning-based approaches proposed by Iandola et al. [12] and Bao et al. [2]. Su et al. [26] proposed the relevance feedback, based upon the mining of the navigation patterns and a query modification scheme. Iandola et al. [12] and Bao et al. [2] experimented on the DCNN using the FlickrLogos-32 database for the retrieval of the logo images. The proposed scheme integrated preprocessing (PSO and SOM) with the query modification and mining-based RF approach and incorporated the DCNN within RF, experimented using the FlickrLogos-27, FlickrLogos-32, and FlickrLogos-32 PLUS databases. The results are promising as shown in the comparison in Tables 8 and 9.

Table 8 depicts the improved retrieval results of the method when the DCNN is integrated with the RF. The optimization (PSO) and clustering techniques (SOM) applied earlier in the preprocessing phase reduced the space for searching and, therefore, the complexity.

The comparison in Table 9 proves the improved retrieval results with the integration of the DCNN in the RF when compared with the other two only deep learning-based models.