CN114821180B - A Weakly Supervised Fine-grained Image Classification Method Based on Soft Threshold Penalty Mechanism - Google Patents

Abstract

The invention provides a weak supervision fine-grained image classification method based on a soft threshold punishment mechanism, which comprises the following steps: step 1: constructing a fine-grained image classification network of a secondary cascade network structure based on a soft threshold punishment mechanism; and 2, step: acquiring an image to be classified; and step 3: preprocessing the image to be classified; and 4, step 4: and classifying the images of the preprocessing result based on the fine-grained image classification network, and outputting the image classification result. The weak supervision fine-grained image classification method based on the soft threshold punishment mechanism optimizes the MMAL-Net network structure, reduces the calculated amount of the whole model by reducing the number of network branch layers, and also reduces the requirement of a training model on hardware; on the basis of a new branch structure, adding a soft threshold punishment mechanism module to resist noise appearing in the image; effectively shielding interference information, thereby improving the overall precision.

Description

A Weakly Supervised Fine-grained Image Classification Method Based on Soft Threshold Penalty Mechanism

technical field

The invention relates to the technical field of image classification and intelligent optimization, in particular to a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism.

Background technique

MMAL-Net is a multi-branch multi-scale learning network, a weakly supervised fine-grained classification method based on global features. Another method used in local feature classification is followed, with a three-level cascade as the overall architecture. The classification accuracy of the MMAL-Net model is high, and it can reach the accuracy rate of SOTA on many data sets, even reaching 94.7% on the aircraft data set, which is currently the highest accuracy rate of this data set. The algorithm flow of MMAL-Net is shown in Figure 2.

MMAL-Net uses the RA-CNN network as the basic structure, and uses a three-level cascaded network. On each level of the network, ResNet is used to extract and classify features. The difference is that at each level Between the network, there are two modules interspersed, which are AOLM (Attention Object Location Module) and APPM (Attention PartProposal Module) modules. These two modules divide the entire three-level network into three branches, which are the original image branch and the main body. Image branch and partial image branch.

ALOM is used to predict the position of objects. The aggregation operation of the feature map is adopted. In the final stage of feature extraction, the feature map is aggregated in the channel dimension, and the corresponding threshold is set to extract the largest connected domain with a higher response, and the receptive field of the corresponding area is mapped back to the original image, which is convenient. The position of the main part in the original image can be found, and then this part is cut out, and then the features are extracted again.

APPM predicts the information of the key area of the object without the need for border boxes or labels. Select some fixed-size sliding windows, and perform pooling operations on the data in the sliding windows, so as to calculate the calculation results of each area, and sort these results, select the area with relatively large results, and perform the calculation on the area. After the non-maximum suppression operation, the image of the part is fed into the network.

The Loss function used in the MMAL-Net three-level network is the basic Cross-entropy loss, and finally the loss value of the three-level network is summed to obtain the final loss.

However, MMAL-Net has the following two disadvantages:

(1) In the entire three-level cascaded network structure, although the number of parameters in the overall network is reduced by sharing a set of parameters, the amount of calculation is significantly increased due to the complex three-level cascaded structure, and because the input of the first two levels of the network are both It is a 448*448 pixel picture, which further increases the calculation amount of the entire network, so the speed of training the network is much slower, and it also takes up a large part of the video memory.

(2) In the setting of the middle branch, the positioning information of the main body can be obtained through the original image, so as to obtain a recognizable area. However, in some fine-grained image classification tasks where there is no obvious subject, the existence of this branch will weaken the entire feature extraction ability.

Contents of the invention

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, optimizes the MMAL-Net network structure, reduces the amount of calculation of the overall model by reducing the number of network branch layers, and also reduces the training model on the hardware. Requirements: On the basis of the new branch structure, add a soft threshold penalty mechanism module to resist the noise in the image; effectively shield the interference information, so that the overall accuracy is improved.

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, including:

Step 1: Based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascaded network structure;

Step 2: Obtain images to be classified;

Step 3: Preprocessing the image to be classified;

Step 4: Based on the fine-grained image classification network, perform image classification on the preprocessing result, and output the image classification result.

Preferably, said step 1: based on a soft threshold penalty mechanism, constructing a fine-grained image classification network with a two-level cascaded network structure, including:

Constructing a first network branch, the first network branch comprising: Input448*448*3, the first ResNet50, Feature14*14*2048, the first GAP, the first FC and the first Softmax connected in sequence;

Construct the second network branch, the second network branch includes: Input224*224*3*mult, the second ResNet50, Feature*7*7*2048*mult, the second GAP, the second FC and the second Softmax connected in sequence ;

Connect Input224*224*3*mult in the second network branch to Input448*448*3 in the first network branch through crop;

Connect Feature14*14*2048 in the first network branch to the crop through APPM;

setting a first loss function RawLoss for the first network branch;

setting a second loss function PartLoss for the second network branch;

Set a soft threshold penalty mechanism in the APPM;

The first network branch, the second network branch, APPM and crop form a fine-grained image classification network with a two-level cascade network structure.

Preferably, the APPM is formed based on SCDA, and the Feature14*14*2048 extracted from the APPM is combined along the channel direction of the pooling layer to obtain a two-dimensional map of 14*14*1, which is used A plurality of preset sliding windows of different sizes perform sliding window calculation on the two-dimensional graph, and the calculation process is shown in formula (2-1):

Among them, H and W are the height and width of the sliding window respectively, A(x, y) is the value corresponding to the coordinate position of the closed two-dimensional image, and a _w is the calculation result of the sliding window.

Preferably, the formula of the first loss function RawLoss is shown in formula (2-2):

Among them, m _i is the i-th sample image, and _ni is the predicted probability of the first convolutional neural network CNN corresponding to the i-th sample image.

Preferably, the formula of the second loss function PartLoss is shown in formula (2-3):

Wherein, q is the number of local feature regions screened out by the second ResNet50, _{miq is the number of qth local feature regions corresponding to the i-th sample image, and n iq is} the second convolutional neural network CNN corresponding to The predicted probability of the number of qth local feature regions corresponding to the i-th sample image.

Preferably, the soft threshold penalty mechanism includes:

Let F(x, y) be the image without noise, N(x, y) be the noise, G(x, y) be the image after noise influence, choose the L _1/2 paradigm to establish the model as formula (2 -5) Shown:

i represents the serial number of the image. When the image is iteratively processed, G( _xi ,y _i )-F( _xi ,y _i ) has a residual error, indicating that noise appears in the image and affects it;

To limit the residual state by soft threshold, first construct the penalty factor ||G(x _i ,y _i )-F(x _i ,y _i )|| _h to limit G(x _i ,y _i )-F(x _i , y _i ) is not greater than 0, thereby reducing the degree of image interference by noise, as shown in formula (2-6):

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value, so the soft threshold can effectively reduce the influence of noise interference.

Preferably, in order to further optimize the soft threshold method, the objective function can be further modified, as shown in formula (2-7):

Among them, V is an auxiliary variable, and λ ₁ and λ ₂ are penalty coefficients. In the calculation process, V is iteratively updated using the soft threshold algorithm.

Preferably, the step 2: obtaining images to be classified includes:

When the user touches and selects a plurality of first photos on the viewing interface of the album to form a circle, and the circle expands along the same expansion direction within a preset first time, acquire and output the preset non-touch circle Select the prompt information, and at the same time, control the circled box to continue to expand along the expansion direction at a preset first expansion speed;

Dynamically acquire the current eye sight of the user;

determining a first fixation point corresponding to the eye line of sight in the viewing interface;

If the first gaze point falls within the circled box in the viewing interface, obtain the distance between the first gaze point and the target frame edge in the expanding direction of the circled box first vertical distance;

Based on the first vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-8):

为预设的第一关系系数；Wherein, v ₁ ′ is the first expansion speed after adjustment, v ₁ is the first expansion speed before adjustment, l ₁ is the first vertical distance,

is the preset first relational coefficient;

If the first gaze point falls within the range to be circled in the expansion direction outside the circled box in the viewing interface and the first gaze point occurs within the preset second time change, obtaining a second vertical distance between the first gaze point and the target frame edge in the expansion direction of the circled box;

Based on the second vertical distance, the first expansion speed is adjusted, and the adjustment formula is as shown in formula (2-9):

为预设的第二关系系数；Wherein, v ₂ ′ is the first expansion speed after adjustment, v ₂ is the first expansion speed before adjustment, l ₂ is the second vertical distance,

is the preset second relational coefficient;

If the first gaze point falls within the range to be circled in the expanding direction outside the circled box in the viewing interface and the first gaze point does not occur within the second time change, obtaining a second photo corresponding to the first gaze point in the viewing interface;

When the second photo just enters the circled box, control the circled box to stop expanding;

Acquiring the movement type of the target frame side in the expansion direction of the circled box;

When the movement type is line feed downward, control the circled box to deselect all the third photos on the right side of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is upward line break, control the circled box to deselect all fourth photos on the left side of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is changing columns to the right, control the circled box to deselect all the fifth photos below the second photo in the column where the second photo is located in the viewing interface;

When the movement type is changing columns to the left, control the circled box to deselect all the sixth photos above the second photo in the column where the second photo is located in the viewing interface;

After the unselection is completed, all the seventh photos circled in the circled frame are used as images to be classified, and the acquisition is completed.

Preferably, the acquiring the second photo corresponding to the first gaze point in the viewing interface includes:

Get the current thumbnail ratio of the display interface;

Acquiring a preset thumbnail ratio threshold corresponding to the size of the display interface;

If the thumbnail ratio is greater than or equal to the thumbnail ratio threshold, determine the eighth photo located at the first gaze point in the viewing interface, and use it as the second photo to complete the acquisition;

Otherwise, determine a plurality of ninth photos located at the first gaze point in the viewing interface;

Based on the ninth photo, a cut-off photo zoom-in confirmation box is generated;

A confirmation box for zooming in on the cut-off photo is suspended in the display interface;

determining a second fixation point corresponding to the user's current eye sight in the viewing interface;

When the second gaze point falls within the cut-off photo zoom-in confirmation frame and the second gaze point has not changed within the preset third time, place the cut-off photo zoom-in confirmation frame at the The ninth photo of the second gaze point is used as the second photo to complete the acquisition.

The present invention provides a weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism, including:

A building block for constructing a fine-grained image classification network with a two-level cascaded network structure based on a soft threshold penalty mechanism;

An acquisition module, configured to acquire images to be classified;

A preprocessing module, configured to preprocess the image to be classified;

The classification module is configured to perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the description, and are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the attached picture:

Fig. 1 is a flow chart of a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism in an embodiment of the present invention;

Fig. 2 is the structural representation of MMAL-Net in the embodiment of the present invention;

3 is a schematic structural diagram of a fine-grained image classification network in an embodiment of the present invention;

FIG. 4 is a schematic diagram of a weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism in an embodiment of the present invention.

detailed description

The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described here are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, as shown in Figure 1, including:

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, including:

Step 1: Based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascaded network structure;

Step 2: Obtain images to be classified;

Step 3: Preprocessing the image to be classified;

Step 4: Based on the fine-grained image classification network, perform image classification on the preprocessing result, and output the image classification result.

The step 1: based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascaded network structure, including:

Constructing a first network branch, the first network branch comprising: Input448*448*3, the first ResNet50, Feature14*14*2048, the first GAP, the first FC and the first Softmax connected in sequence;

Construct the second network branch, the second network branch includes: Input224*224*3*mult, the second ResNet50, Feature*7*7*2048*mult, the second GAP, the second FC and the second Softmax connected in sequence ;

Connect Input224*224*3*mult in the second network branch to Input448*448*3 in the first network branch through crop;

Connect Feature14*14*2048 in the first network branch to the crop through APPM;

setting a first loss function RawLoss for the first network branch;

setting a second loss function PartLoss for the second network branch;

Set a soft threshold penalty mechanism in the APPM;

The first network branch, the second network branch, APPM and crop form a fine-grained image classification network with a two-level cascade network structure.

The APPM is formed based on SCDA, and the Feature14*14*2048 extracted by the APPM is combined along the channel direction of the pooling layer to obtain a two-dimensional map of 14*14*1, using the preset A plurality of sliding windows of different sizes performs sliding window calculation on the two-dimensional graph, and the calculation process is shown in formula (2-1):

Among them, H and W are the height and width of the sliding window respectively, A(x, y) is the value corresponding to the coordinate position of the closed two-dimensional image, and a _w is the calculation result of the sliding window.

The formula of the first loss function RawLoss is shown in formula (2-2):

Among them, m _i is the i-th sample image, and _ni is the predicted probability of the first convolutional neural network CNN corresponding to the i-th sample image.

The formula of the second loss function PartLoss is shown in formula (2-3):

Wherein, q is the number of local feature regions screened out by the second ResNet50, _{miq is the number of qth local feature regions corresponding to the i-th sample image, and n iq is} the second convolutional neural network CNN corresponding to The predicted probability of the number of qth local feature regions corresponding to the i-th sample image.

The soft threshold penalty mechanism includes:

Let F(x, y) be the image without noise, N(x, y) be the noise, G(x, y) be the image after noise influence, choose the L _1/2 paradigm to establish the model as formula (2 -5) Shown:

i represents the serial number of the image. When the image is iteratively processed, G( _xi ,y _i )-F( _xi ,y _i ) has a residual error, indicating that noise appears in the image and affects it;

To limit the residual state by soft threshold, first construct the penalty factor ||G(x _i ,y _i )-F(x _i ,y _i )|| _h to limit G(x _i ,y _i )-F(x _i , y _i ) is not greater than 0, thereby reducing the degree of image interference by noise, as shown in formula (2-6):

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value, so the soft threshold can effectively reduce the influence of noise interference.

In order to further optimize the soft threshold method, the objective function can be further modified, as shown in formula (2-7):

Among them, V is an auxiliary variable, and λ ₁ and λ ₂ are penalty coefficients. In the calculation process, V is iteratively updated using the soft threshold algorithm.

The working principle and beneficial effects of the above-mentioned technical scheme are:

The invention modifies the three-level cascade network of the MMAL-Net model into a two-level cascade mode, and puts the reasoning task of the last level on the first branch. That is to say, during the operation of the entire network model, the effect of classifying fine-grained networks can be obtained only through an ordinary network classification model. Its structure is shown in Figure 3.

Input448*448*3 is the image input (the image size is 448 pixels*448 pixels, three color channels of R (red), G (green), and B (blue));

The first ResNet50 is a network model, a deep residual network, and 50 refers to the number of network layers;

Feature14*14*2048 is 2048 14-pixel*14-pixel feature two-dimensional images;

The first GAP is Global Average Pooling (Global Average Pooling);

The first FC is a fully connected layer (Fully Connection);

The first Softmax is the activation function as the output layer, which is often regarded as a multi-classifier in machine learning. The popular meaning is to input an item and get the probability of the category it may belong to.

Input224*224*3*mult is the fixed-size image input obtained by performing the crop method on Input448*448*3 (partial blocks of the first branch image input, the image size is 224 pixels*224 pixels, R (red), G (green), B (blue) three color channels), mult refers to the fixed number of blocks obtained after image cropping.

The second ResNet50 is the same as the first ResNet50;

Feature*7*7*2048*mult is 2048 7-pixel*7-pixel feature two-dimensional images for mult;

The second GAP, the second FC and the second Softmax are the same as the first GAP, the first FC and the first Softmax respectively;

Crop is cropping, which cuts out a part of the image directly and retains the real size ratio of the original image. Depending on the method of image cropping, the number of cropped images obtained is also different.

After image preprocessing (filtering noise reduction, grayscale and shrinking to 448 pixels*448 pixels), the image shrunk to 448 pixels*448 pixels is used as the first-level input of the network. In the design process of the network model, ResNet50 is used as the backbone network for feature extraction. The residual block is specially designed to obtain the ResNet50 network, where 50 refers to the total number of convolutional layers and fully connected layers in the deep network. The overall accuracy is moderate, and the overall calculation amount is also moderate. The residual block of ResNet50 uses BottleNeck The structure will neither affect the accuracy nor reduce the overall calculation amount. The number of feature channels is 2048, and it is more helpful to select discriminative regions. On the whole, the feature extraction of the two branches of the network adopts the method of parameter sharing, which not only reduces the amount of parameters of the entire network, but also makes the network suitable for images of different sizes and different parts.

The local feature extraction of the network uses the APPM structure, which is formed based on the research of SCDA (Selective Convolutional Descriptor Aggregation, Selective Convolutional Descriptor Aggregation). First, the 14*14*2048 extracted by the module is combined along the channel direction to obtain a 14*14*1 two-dimensional map. The 14*14*1 two-dimensional map is a two-dimensional map of image features with a size of 14 pixels*14 pixels.

Then, use several pre-set sliding windows of different sizes to calculate, and the calculation process inside the sliding window is shown in formula (2-1)

The height and width of the preset sliding window are H and W in the formula respectively, A(x,y) refers to the corresponding value of the coordinate position of the closed two-dimensional graph, and a _w means the value calculated by the current sliding window value. A(x, y) is the value corresponding to the coordinate position of the folded two-dimensional image, specifically the position coordinate value of the 14*14*1 two-dimensional image in the corresponding image to be classified, that is, the original image. Secondly, the _aw values of different positions are sorted, and the area with the larger output value is the area with discriminative features. Finally, in areas where feature areas overlap and cannot be simply sorted, NMS (non maximum suppression, non-maximum suppression) is used to select multiple high-discrimination, low-redundancy candidate areas as local feature areas, which become the second layer's input image.

The loss function adopted in the present invention is the most commonly used Cross-Entropy (cross-entropy) loss in multi-classification problems. The first layer Raw Loss formula is shown in (2-2)

Among them, i represents the sample image, m _i represents the real label of the sample image, which is usually calculated by substituting 1, and _ni refers to the predicted probability of this type of neural network. m _i is the i-th sample image, which is derived from the image intercepted by the sliding window above, a total of 2048 images, that is, the value of i is from 1 to 2048.

The second layer Part Loss formula is shown in (2-3)

Here, q represents the number of selected local feature regions, calculates and obtains the loss value for each region, and finally calculates the average value. m _iq is the qth local feature region corresponding to the i-th sample image, specifically: the sample image is derived from the image after the crop (cropping) of the image preprocessed by the first branch, and its corresponding qth local feature region. The local feature regions selected by the second ResNet50 come from the image to be classified, that is, the preprocessed image of the original image, that is, the image after filtering, noise reduction, and grayscale, with a total of 2048 local feature regions.

The first convolutional neural network CNN comes from the first ResNet50, and ResNet50 is a network model of the convolutional neural network; the second convolutional neural network is the same.

The total loss function formula is shown in (2-4)

Loss _total ＝μLoss _raw +ωLoss _part , (2-4)

Loss _raw and loss _part respectively represent the loss values of the first layer and the second layer, Loss _total represents the overall loss, μ and ω are values between 0-1, representing the influence weight of the two-layer branch on the overall network , to adjust.

In order to reduce the complexity of the network, it can be used better. The lightweight method used in the present invention uses the SqueezeNet (lightweight convolutional neural network) structure. When it comes to dimensionality reduction, the number of input channels is reduced, and then convolution operations are performed with 1*1 and 3*3 convolution kernels in the Expand layer, and finally the operation results are spliced to obtain the final output feature data.

The reduction of the overall network structure level can effectively reduce the total amount of calculation under the condition that the recognition accuracy is still high. However, as far as the original data set is concerned, there is a lot of environmental noise, and the branch of subject monitoring is missing. The present invention introduces a soft threshold penalty mechanism to further improve the local feature extraction capability. The influence of noise on an image can be divided into an additive model and a multiplicative model. Set F(x,y) as an image without noise, N(x,y) as noise, and G(x,y) as the image after noise influence , that is, the original image. In order to better extract features, the present invention uses the L _1/2 paradigm to establish the model, as shown in formula (2-5)

i represents the sample image. When the image is iteratively processed, G( _xi ,y _i )-F( _xi ,y _i ) has a residual error, indicating that noise appears in the image and affects it. In the soft threshold method proposed by the present invention, the residual state is limited by the soft threshold.

为在数学上表示真实值，在式中表示后面式中取得最小值时(x_i,y_i)的值；arg为复数辐角，指的是复数的辐角主值，在此式中argmin即表示后面函数取最小值时，xi和yi的取值；F、N、G仅为函数名。(x _i , y _i ) is the coordinates of the i-th feature image;

In order to express the real value mathematically, the value of ( _xi , y _i ) when the minimum value is obtained in the following formula is expressed in the formula; arg is the argument angle of a complex number, which refers to the main value of the argument angle of a complex number. In this formula, argmin That is to say, when the subsequent function takes the minimum value, the values of xi and yi; F, N, and G are only function names.

First construct the penalty factor ||G( _xi ,y _i )-F( _xi ,y _i )|| _h to limit G( _xi ,y _i )-F( _xi ,y _i ) not greater than 0, Thereby reducing the degree to which the image is disturbed by noise, as shown in formula (2-6)

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value, so the soft threshold can effectively reduce the influence of noise interference. The full name of st is subject to, which means to make...satisfy...In the formula, it means: make the above formula true under the conditions of N( _xi ,y _i )>0,F( _xi ,y _i )>0 .

h is a preset constant, preferably 1/2.

To further optimize the soft threshold method, the objective function can be further modified

Among them, V is an auxiliary variable. In the calculation process, the soft threshold algorithm is used to iteratively update V. Use the soft threshold penalty mechanism to effectively resist the noise in the data.

This application optimizes the network structure of MMAL-Net, reduces the calculation amount of the overall model by reducing the number of network branch layers, and also reduces the hardware requirements of the training model; on the basis of the new branch structure, a soft threshold penalty mechanism module is added, To resist the noise in the image; effectively shield the interference information, so that the overall accuracy is improved.

Fine-grained image classification initially uses strong-supervised deep learning to complete the classification task. The supervision information of the strong-supervised fine-grained image classification method relies on too many labels. In addition to the network for fine-grained classification, a component is required in the entire algorithm framework localized object detection network or semantic segmentation network. This leads to the high cost of data labeling and network structure, which makes the strong supervision method unable to be better applied in the actual production process. Therefore, this application only uses category labels and does not require additional labeling information, which is a weakly supervised method.

In one embodiment, the step 2: obtaining images to be classified includes:

When the user touches and selects a plurality of first photos on the viewing interface of the album to form a circle, and the circle expands along the same expansion direction within a preset first time, acquire and output the preset non-touch circle Select the prompt information, and at the same time, control the circled box to continue to expand along the expansion direction at a preset first expansion speed;

Dynamically acquire the current eye sight of the user;

determining a first fixation point corresponding to the eye line of sight in the viewing interface;

If the first gaze point falls within the circled box in the viewing interface, obtain the distance between the first gaze point and the target frame edge in the expanding direction of the circled box first vertical distance;

Based on the first vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-8):

为预设的第一关系系数；Wherein, v ₁ ′ is the first expansion speed after adjustment, v ₁ is the first expansion speed before adjustment, l ₁ is the first vertical distance,

is the preset first relational coefficient;

If the first gaze point falls within the range to be circled in the expansion direction outside the circled box in the viewing interface and the first gaze point occurs within the preset second time change, obtaining a second vertical distance between the first gaze point and the target frame edge in the expansion direction of the circled box;

Based on the second vertical distance, the first expansion speed is adjusted, and the adjustment formula is as shown in formula (2-9):

为预设的第二关系系数；Wherein, v ₂ ′ is the first expansion speed after adjustment, v ₂ is the first expansion speed before adjustment, l ₂ is the second vertical distance,

is the preset second relational coefficient;

If the first gaze point falls within the range to be circled in the expanding direction outside the circled box in the viewing interface and the first gaze point does not occur within the second time change, obtaining a second photo corresponding to the first gaze point in the viewing interface;

When the second photo just enters the circled box, control the circled box to stop expanding;

Acquiring the movement type of the target frame side in the expansion direction of the circled box;

When the movement type is line feed downward, control the circled box to deselect all the third photos on the right side of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is upward line break, control the circled box to deselect all fourth photos on the left side of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is changing columns to the right, control the circled box to deselect all the fifth photos below the second photo in the column where the second photo is located in the viewing interface;

When the movement type is changing columns to the left, control the circled box to deselect all the sixth photos above the second photo in the column where the second photo is located in the viewing interface;

After the unselection is completed, all the seventh photos circled in the circled frame are used as images to be classified, and the acquisition is completed.

The working principle and beneficial effects of the above-mentioned technical scheme are:

Generally, when a user selects an image that needs to be classified, since there must be a large amount of selection, when performing touch operations on smart terminals (such as mobile phones and tablets, etc.), the user needs to continue to touch the operation, and the experience is poor. If the time is too long, it may cause discomfort to the user's fingers; especially for some users who are engaged in image classification, the experience is not good. For example: when the user opens the mobile phone photo album, the finger touches the upper left corner of the mobile phone screen and moves to the right, so that the photos to be selected in the all-selection interface are selected. Swipe down and select all at the same time. The photos to be selected appear in the new interface, but the finger needs to keep still until the photo is cut off before the finger can be released. Therefore, urgently need to solve.

When the user touches to circle the first photo in the album, the selected first photo forms a marquee frame, and when the marquee frame expands in the same direction within the preset first time (for example: 2 seconds) (for example: When it expands vertically downward), it means that the user needs to continue to select more first photos in the album (for example: the display interface of the album is moving downward). Circle selection prompt information (for example: display on the display interface "You can move your finger! Start automatic circle selection"), and control the circle box to expand along the expansion direction (for example: vertically downward) with the preset first expansion The speed (for example: 1.2cm/s) continues to expand.

At this point, the user will continuously check the first photo that has been circled by the marquee box or the first photo that will be circled by the marquee box to determine whether the deadline is reached. Obtain the user's eye line of sight (obtaining the line of sight belongs to the category of the prior art, and will not be described in detail), and determine the first fixation point corresponding to the eye sight in the viewing interface, and the first fixation point is the position where the user is gazing. If the first fixation point falls within the circled box, it means that the user is viewing the first photo that has been circled by the circled box, and cannot keep up with the first expansion speed of the circled box, based on the first gaze point and the target The first vertical distance of the frame edge slows down the first expansion speed. The larger the first vertical distance, the faster the first expansion speed, and the more the first expansion speed should be decelerated. If the first fixation point falls within the range to be circled in the direction of expansion outside the circled box in the viewing interface and the first fixation point changes within the preset second time (for example: 3 seconds), it means that the user is Check the first photo that is about to be selected by the circle box and the cut-off photo is not yet reached. The larger the second vertical distance between the first gaze point and the edge of the target frame, it means that the first expansion speed is slower, and the first expansion speed needs to be accelerated. . It fully ensures that the expansion speed of the circle selection box can be close to the user, which improves the humanization and the intelligence of the touch-free circle selection. When the first gaze point falls within the range to be circled and the first gaze point does not change in the second time, it means that the cutoff photo has been reached; however, the cutoff photo is not necessarily the last photo in a row or the last photo in a column For a photo, the first photo after the cut-off photo needs to be removed. Further enhance the intelligence and humanization of touch-free circle selection. Get the second photo that is the cutoff photo. When the second photo just enters the frame, useless photos are eliminated based on the movement type of the edge of the target frame. Generally, the movement type is line wrapping downward, and the user views photos in a cycle from left to right. Therefore, the control circle check box deselects all the third photos on the right side of the row where the second photo is located. The same is true when the movement type is line feed upward. When the movement type is changing columns to the right, the user views photos in a cycle from top to bottom. Therefore, the control circle check box deselects all the fifth photos below the second photo. The same is true when the movement type is column change to the left.

In one embodiment, the obtaining the second photo corresponding to the first gaze point in the viewing interface includes:

Get the current thumbnail ratio of the display interface;

Acquiring a preset thumbnail ratio threshold corresponding to the size of the display interface;

If the thumbnail ratio is greater than or equal to the thumbnail ratio threshold, determine the eighth photo located at the first gaze point in the viewing interface, and use it as the second photo to complete the acquisition;

Otherwise, determine a plurality of ninth photos located at the first gaze point in the viewing interface;

Based on the ninth photo, a cut-off photo zoom-in confirmation box is generated;

A confirmation box for zooming in on the cut-off photo is suspended in the display interface;

determining a second fixation point corresponding to the user's current eye sight in the viewing interface;

When the second gaze point falls within the cut-off photo zoom-in confirmation frame and the second gaze point has not changed within the preset third time, place the cut-off photo zoom-in confirmation frame at the The ninth photo of the second gaze point is used as the second photo to complete the acquisition.

The working principle and beneficial effects of the above-mentioned technical scheme are:

When obtaining the second photo as the cut-off photo, generally, it is only necessary to determine the photo at the first gaze point in the viewing interface. However, due to the small size of the mobile phone screen of some smart terminals or the small thumbnail ratio of some photo albums (for example: 1:50, 50 photos are displayed in one interface), the accuracy of line of sight acquisition is limited, so that the first gaze point cannot Cutoff photo for pinpointing the user's gaze.

Therefore, the current thumbnail ratio of the display interface is obtained, and at the same time, the thumbnail ratio threshold of the size of the display interface is obtained; the thumbnail ratio threshold is the size of the display interface, and the first gaze point can be used to accurately locate the cut-off photo of the user's gaze Minimum thumbnail ratio. If the thumbnail ratio is greater than or equal to the thumbnail ratio threshold, it means that the first gaze point can be used to accurately locate the cut-off photo that the user gazes at, and the eighth photo at the first gaze point can be determined as the second photo. Otherwise (the thumbnail ratio is less than the thumbnail ratio threshold), determine the ninth photo of the first gaze point, and generate a cut-off photo zoom-in confirmation box based on the ninth photo for further confirmation by the user. When the user's current second gaze point falls within If there is no change within the preset third time (for example: 2 seconds) within the cut-off photo zoom-in confirmation box, it means that the user is looking at the cut-off photo, and the ninth photo at the second gaze point can be determined as the second photo. . It greatly improves the applicability of different smart terminals, and also improves the accuracy of the cut-off photo determination.

The present invention provides a weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism, as shown in Figure 4, including:

Building block 1 is used to construct a fine-grained image classification network with a two-level cascaded network structure based on a soft threshold penalty mechanism;

Obtaining module 2, for obtaining images to be classified;

A preprocessing module 3, configured to preprocess the image to be classified;

The classification module 4 is configured to perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims (7)

1. A weak supervision fine-grained image classification method based on a soft threshold punishment mechanism is characterized by comprising the following steps:

step 1: constructing a fine-grained image classification network of a secondary cascade network structure based on a soft threshold punishment mechanism;

step 2: acquiring an image to be classified;

and step 3: preprocessing the image to be classified;

and 4, step 4: based on the fine-grained image classification network, carrying out image classification on a preprocessing result and outputting an image classification result;

the step 1: based on a soft threshold punishment mechanism, a fine-grained image classification network of a two-level cascade network structure is constructed, and the method comprises the following steps:

constructing a first network branch, the first network branch comprising: input448 x 3, first ResNet50, feature14 x 2048, first GAP, first FC and first Softmax connected in sequence;

constructing a second network branch, the second network branch comprising: input224 × 3 × mult, second ResNet50, feature × 7 × 2048 × mult, second GAP, second FC, and second Softmax connected in this order;

connecting Input224 x 3 x mult in the second network branch with Input448 x 3 in the first network branch by crop;

connecting Feature14 x 2048 in the first network branch with the crop through APPM;

setting a first loss function RawLoss for the first network branch;

setting a second loss function PartLoss for the second network branch;

setting a soft threshold penalty mechanism in the APPM;

the first network branch, the second network branch, the APPM and the crop form a fine-grained image classification network of a two-level cascade network structure;

the step 2: acquiring an image to be classified, comprising:

when a user touches and selects a plurality of first photos to form a selection frame in a touch control manner on a viewing interface of a photo album and the selection frame is expanded along the same expansion direction within a preset first time, acquiring and outputting preset touch-free selection prompt information, and simultaneously controlling the selection frame to continue to expand along the expansion direction at a preset first expansion speed;

dynamically acquiring the current eye sight of the user;

determining a first point of gaze location within the viewing interface corresponding to the eye gaze;

if the first watching point position is located in the selection box in the viewing interface, acquiring a first vertical distance between the first watching point position and a target frame edge of the selection box in the expansion direction;

adjusting the first expansion speed based on the first vertical distance, wherein the adjustment formula is shown as a formula (2-8):

wherein v is ₁ ' adjusted said first expansion speed, v ₁ For said first expansion speed before adjustment,/ ₁ For the said first vertical distance, the distance is,

is a preset first relation coefficient;

if the first watching point location is located outside the selection circle frame in the viewing interface and within the range to be selected in the expansion direction and the first watching point location changes within a preset second time, acquiring a second vertical distance between the first watching point location and a target frame edge of the selection circle frame in the expansion direction;

adjusting the first expansion speed based on the second vertical distance, wherein the adjustment formula is shown as a formula (2-9):

wherein v is ₂ ' adjusted said first expansion speed, v ₂ For said first expansion speed before adjustment,/ ₂ Is the second vertical distance, and is,

is a preset second relation coefficient;

if the first watching point position is located in the to-be-circled range in the expansion direction outside the circled frame in the viewing interface and the first watching point position is not changed in the second time, acquiring a second photo corresponding to the first watching point position in the viewing interface;

when the second photo just enters the selection frame, controlling the selection frame to stop expanding;

acquiring the movement type of the target frame edge in the expansion direction of the selection frame;

when the moving type is downward line feed, controlling the selection frame to deselect all third photos on the right side of the second photo in the line where the second photo is located in the viewing interface;

when the moving type is upward line feed, controlling the selection frame to deselect all fourth photos on the left side of the second photo in the line where the second photo is located in the viewing interface;

when the moving type is changing columns to the right, controlling the selection circle frame to retreat all fifth photos on the lower side of the second photo in the column of the second photo in the viewing interface;

when the moving type is changing columns to the left, controlling the selection circle frame to retreat all sixth photos on the upper side of the second photo in the column of the second photo in the viewing interface;

after the selection is finished, all seventh pictures selected in the selection frame are taken as images to be classified, and the seventh pictures are obtained;

said obtaining a second photograph within the viewing interface corresponding to the first point of gaze location comprises:

acquiring the current thumbnail proportion of a display interface;

acquiring a preset thumbnail ratio threshold corresponding to the size of the display interface;

if the thumbnail proportion is larger than or equal to the thumbnail proportion threshold value, determining an eighth photo located at the first watching point position in the viewing interface, and taking the eighth photo as a second photo to finish the acquisition;

otherwise, determining a plurality of ninth photos within the viewing interface that are located at the first point of gaze location;

generating a cut-off picture enlargement confirmation frame based on the ninth picture;

displaying the cut-off photo amplification confirmation frame in a suspending manner in the display interface;

determining a second point of gaze location within the viewing interface that corresponds to the user's current eye gaze;

and when the second watching point position is located in the cut-off photo amplification confirmation frame and the second watching point position is not changed within a preset third time, the ninth photo located at the second watching point position in the cut-off photo amplification confirmation frame is taken as the second photo to finish the acquisition.

2. The method for classifying the weakly supervised fine grained image based on the soft threshold penalty mechanism according to claim 1, wherein the APPM is formed based on SCDA, the Feature14 x 2048 extracted from the features by the APPM is folded along the channel direction of the pooling layer to obtain a two-dimensional map of 14 x 1, and the two-dimensional map is subjected to sliding window calculation by using a plurality of preset sliding windows with different sizes, and the calculation process is shown in formula (2-1):

wherein H and W are respectively the height and width of the sliding window, A (x, y) is the numerical value corresponding to the coordinate position of the closed two-dimensional graph, and a _w The result is a sliding window calculation.

3. The method for classifying the weakly supervised fine grained image based on the soft threshold penalty mechanism as recited in claim 1, wherein the formula of the first loss function RawLoss is shown as formula (2-2):

wherein m is _i For the ith sample image, n _i The first convolutional neural network CNN corresponds to the predicted probability of the ith sample image.

4. The method for classifying the weakly supervised fine granular image based on the soft threshold penalty mechanism as recited in claim 1, wherein the formula of the second loss function PartLoss is shown as the formula (2-3):

wherein q is the number of local feature regions screened by the second ResNet50, m _iq The number of the q local feature regions corresponding to the ith sample image, n _iq And the prediction probability of the number of the q local feature areas corresponding to the ith sample image is the second convolutional neural network CNN.

5. The method of claim 1, wherein the soft-threshold penalty mechanism comprises:

let F (x, y) be the image without noise, N (x, y) be the noise, G (x, y) be the image after the influence of noise, select L _1/2 The paradigm is modeled as shown in equation (2-5):

i denotes the number of images, and G (x) when image iteration is performed _i ,y _i )-F(x _i ,y _i ) Residual errors occur, which indicates that noise occurs in the image and causes influence;

the residual error state is limited by a soft threshold, and a penalty factor G (x) is firstly constructed _i ,y _i )-F(x _i ,y _i )|| _h To limit G (x) _i ,y _i )-F(x _i ,y _i ) Not greater than 0, thereby reducing the degree to which the image is disturbed by noise, as shown in equation (2-6):

in the formula, lambda represents a penalty coefficient, and the result can be close to the true value by adjusting the coefficient.

6. The method for classifying the weakly supervised fine grained image based on the soft threshold penalty mechanism as claimed in claim 5, wherein to further optimize the soft threshold method, the objective function is further modified as shown in formula (2-7):

wherein V is an auxiliary variable, λ ₁ And λ ₂ And all the coefficients are penalty coefficients, and in the calculation process, the soft threshold algorithm is used for carrying out iterative update processing on the V.

7. A weakly supervised fine grained image classification system based on a soft threshold penalty mechanism, comprising:

the construction module is used for constructing a fine-grained image classification network of a secondary cascade network structure based on a soft threshold punishment mechanism;

the acquisition module is used for acquiring an image to be classified;

the preprocessing module is used for preprocessing the image to be classified;

the classification module is used for classifying the images of the preprocessing result based on the fine-grained image classification network and outputting the image classification result;

the building module performs the following operations:

constructing a first network branch, the first network branch comprising: input448 x 3, first ResNet50, feature14 x 2048, first GAP, first FC and first Softmax connected in sequence;

constructing a second network branch, the second network branch comprising: input224 × 3 × mult, a second ResNet50, feature × 7 × 2048 × mult, a second GAP, a second FC, and a second Softmax connected in sequence;

connecting Input224 x 3 x mult in the second network branch with Input448 x 3 in the first network branch via crop;

connecting Feature14 x 2048 in the first network branch with the crop through APPM;

setting a first loss function RawLoss for the first network branch;

setting a second loss function PartLoss for the second network branch;

setting a soft threshold penalty mechanism in the APPM;

the first network branch, the second network branch, the APPM and the crop form a fine-grained image classification network of a two-level cascade network structure;

the acquisition module performs the following operations:

when a user selects a plurality of first photos to form a selection frame in a touch control manner on a viewing interface of the photo album and the selection frame is expanded along the same expansion direction within a preset first time, acquiring and outputting preset touch-free selection prompt information, and simultaneously controlling the selection frame to continue to expand along the expansion direction at a preset first expansion speed;

dynamically acquiring the current eye sight of the user;

determining a first point of gaze location within the viewing interface corresponding to the eye gaze;

if the first watching point position is located in the selection frame in the viewing interface, acquiring a first vertical distance between the first watching point position and a target frame edge in the expansion direction of the selection frame;

adjusting the first expansion speed based on the first vertical distance, wherein the adjustment formula is shown as a formula (2-8):

wherein v is ₁ ' adjusted said first expansion speed, v ₁ For said first expansion speed before adjustment,/ ₁ For the said first vertical distance, the distance is,

is a preset first relation coefficient;

if the first watching point location is located outside the selection circle frame in the viewing interface and within the range to be selected in the expansion direction and the first watching point location changes within a preset second time, acquiring a second vertical distance between the first watching point location and a target frame edge of the selection circle frame in the expansion direction;

adjusting the first expansion speed based on the second vertical distance, wherein the adjustment formula is shown as a formula (2-9):

wherein v is ₂ ' adjusted said first expansion speed, v ₂ For said first expansion speed before adjustment,/ ₂ Is the second vertical distance, and is,

is a preset second relation coefficient;

if the first watching point position is located in the to-be-circled range in the expansion direction outside the circled frame in the viewing interface and the first watching point position is not changed in the second time, acquiring a second photo corresponding to the first watching point position in the viewing interface;

when the second picture just enters the selection frame, controlling the selection frame to stop expanding;

acquiring the movement type of the target frame edge in the expansion direction of the selection frame;

when the moving type is line feed, controlling the selection circle frame to retreat all third photos on the right side of the second photo in the line where the second photo is located in the viewing interface;

when the moving type is the line feed, controlling the selection circle frame to retreat all fourth photos on the left side of the second photo in the line where the second photo is located in the viewing interface;

when the moving type is changing columns to the right, controlling the selection circle frame to retreat all fifth photos on the lower side of the second photo in the column of the second photo in the viewing interface;

when the moving type is changing columns to the left, controlling the selection circle frame to retreat all sixth photos on the upper side of the second photo in the column of the second photo in the viewing interface;

after the selection retreating is finished, all seventh photos selected in the circle selection frame are used as images to be classified, and the acquisition is finished;

the obtaining a second photograph within the viewing interface corresponding to the first point of gaze location includes:

acquiring the current thumbnail proportion of a display interface;

acquiring a preset thumbnail proportion threshold corresponding to the size of the display interface;

if the thumbnail proportion is larger than or equal to the thumbnail proportion threshold, determining an eighth photo located at the first watching point position in the viewing interface, and taking the eighth photo as a second photo to finish acquisition;

otherwise, determining a plurality of ninth photos within the viewing interface that are located at the first point of gaze location;

generating a cut-off picture enlargement confirmation frame based on the ninth picture;

displaying the cut-off photo amplification confirmation frame in a suspending manner in the display interface;

determining a second point of gaze location within the viewing interface that corresponds to the user's current eye gaze;

and when the second watching point position is located in the cut-off photo amplification confirmation frame and the second watching point position is not changed within a preset third time, the ninth photo located at the second watching point position in the cut-off photo amplification confirmation frame is taken as the second photo to finish the acquisition.

Images