CN114821180A - A Weakly Supervised Fine-Grained Image Classification Method Based on Soft Threshold Penalty Mechanism - Google Patents

Abstract

The invention provides a weak supervision fine-grained image classification method based on a soft threshold punishment mechanism, which comprises the following steps: step 1: constructing a fine-grained image classification network of a two-level cascade network structure based on a soft threshold punishment mechanism; step 2: acquiring an image to be classified; and step 3: preprocessing the image to be classified; and 4, step 4: and based on the fine-grained image classification network, carrying out image classification on the preprocessing result and outputting an image classification result. The weak supervision fine-grained image classification method based on the soft threshold punishment mechanism optimizes the MMAL-Net network structure, reduces the calculated amount of the whole model by reducing the number of network branch layers, and also reduces the requirement of the training model on hardware; on the basis of a new branch structure, adding a soft threshold punishment mechanism module to resist noise appearing in the image; effectively shielding interference information, thereby improving the overall precision.

Description

A Weakly Supervised Fine-Grained Image Classification Method Based on Soft Threshold Penalty Mechanism

technical field

The invention relates to the technical field of image classification and intelligent optimization, in particular to a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism.

Background technique

MMAL-Net is a multi-branch multi-scale learning network, a weakly supervised fine-grained classification method based on global features. Another method used for local feature classification is followed, and the three-level cascade is used as the overall architecture. The classification accuracy of the MMAL-Net model is high, and it can reach the accuracy of SOTA on many datasets, even reaching 94.7% on the aircraft dataset, which is currently the highest accuracy rate of this dataset. The algorithm flow of MMAL-Net is shown in Figure 2.

MMAL-Net uses the RA-CNN network as the basic structure, and selects a three-level cascade network to carry out. On each level of the network, ResNet is used for feature extraction and classification. The difference is that at each level There are two modules interspersed between the networks, namely AOLM (Attention Object Location Module) and APPM (Attention Part Proposal Module) modules. These two modules divide the entire three-level network into three branches, namely the original image branch and the main body. Image branches and partial image branches.

ALOM is used to predict the position of objects. The aggregation operation of the feature map is adopted. In the final stage of feature extraction, the feature map is aggregated in the channel dimension, the corresponding threshold is set to extract the largest connected domain with high response, and the receptive field of the corresponding region is mapped back to the original image, so that the The position of the main part in the original image can be found, then cut out this part, and then perform feature extraction again.

APPM predicts the information of key regions of objects without the need for bounding boxes or labels. Some fixed-size sliding windows are selected, and the data in the sliding window is pooled to calculate the calculation results of each area, and these results are sorted, and the area with the larger result is selected. After the non-maximum suppression operation, the image of the part is then fed into the network.

The Loss function used in the MMAL-Net three-level network is the basic Cross-entropy loss, and finally the loss values of the three-level network are summed to obtain the final loss.

However, MMAL-Net has the following two disadvantages:

(1) In the entire three-level cascaded network structure, although the number of parameters of the overall network is reduced by sharing a set of parameters, the amount of calculation is significantly increased because of the complex three-level cascaded structure, and because the inputs of the first two levels of network are both It is a 448*448 pixel picture, which further increases the calculation amount of the entire network, so the speed of training the network is much slower, and it also takes up a large part of the video memory.

(2) In the setting of the middle branch, the positioning information of the main body can be obtained through the original image, so as to obtain a distinguishable area. However, in some fine-grained image classification tasks without obvious subjects, the existence of this branch will weaken the entire feature extraction ability.

SUMMARY OF THE INVENTION

The invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, optimizes the MMAL-Net network structure, reduces the calculation amount of the overall model by reducing the number of network branch layers, and also reduces the training model on the hardware. Requirements; On the basis of the new branch structure, a soft threshold penalty mechanism module is added to resist the noise appearing in the image; the interference information is effectively shielded, thereby improving the overall accuracy.

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, including:

Step 1: Based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascade network structure;

Step 2: Obtain the image to be classified;

Step 3: preprocessing the to-be-classified image;

Step 4: Perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

Preferably, the step 1: constructing a fine-grained image classification network with a two-level cascade network structure based on a soft threshold penalty mechanism, including:

Build a first network branch, the first network branch includes: Input448*448*3, the first ResNet50, Feature14*14*2048, the first GAP, the first FC and the first Softmax connected in sequence;

Build a second network branch, the second network branch includes: Input224*224*3*mult, second ResNet50, Feature*7*7*2048*mult, second GAP, second FC and second Softmax connected in sequence ;

Connect Input224*224*3*mult in the second network branch to Input448*448*3 in the first network branch through crop;

Connect Feature14*14*2048 in the first network branch to the crop through APPM;

setting a first loss function RawLoss for the first network branch;

setting a second loss function PartLoss for the second network branch;

A soft threshold penalty mechanism is set in the APPM;

The first network branch, the second network branch, APPM and crop form a fine-grained image classification network with a two-level cascade network structure.

Preferably, the APPM is formed based on SCDA, and the Feature14*14*2048 extracted from the feature by the APPM is closed along the channel direction of the pooling layer to obtain a 14*14*1 two-dimensional image, using A plurality of preset sliding windows of different sizes are used to perform sliding window calculation on the two-dimensional image, and the calculation process is shown in formula (2-1):

Among them, H and W are the height and width of the sliding window respectively, A(x, y) is the value corresponding to the coordinate position of the collapsed two-dimensional image, and a _w is the calculation result of the sliding window.

Preferably, the formula of the first loss function RawLoss is shown in formula (2-2):

Among them, m _i is the ith sample image, and _ni is the prediction probability of the first convolutional neural network CNN corresponding to the ith sample image.

Preferably, the formula of the second loss function PartLoss is shown in formula (2-3):

Wherein, q is the number of local feature regions selected by the second ResNet50, m _iq is the number of the qth local feature regions corresponding to the ith sample image, and n _iq is the second convolutional neural network CNN corresponding to The predicted probability of the number of qth local feature regions corresponding to the ith sample image.

Preferably, the soft threshold penalty mechanism includes:

Let F(x, y) be the image without noise, N(x, y) be the noise, G(x, y) be the image after the noise, and choose the L _1/2 paradigm to build the model as shown in formula (2) -5) shown:

i represents the serial number of the image. When the image is iteratively processed, there is a residual error in G(x _i , y _i )-F(x _i , y _i ), which means that there is noise in the image and affects it;

The residual state is limited by the soft threshold, and the penalty factor ||G(x _i ,y _i )-F(x _i ,y _i )|| _h is constructed first to limit G(x _i ,y _i )-F(x _i , y _i ) is not greater than 0, thereby reducing the degree of image interference by noise, as shown in formula (2-6):

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value. Therefore, the soft threshold can effectively reduce the influence of noise interference.

Preferably, in order to further optimize the soft threshold method, the objective function can be further modified, as shown in formula (2-7):

Among them, V is an auxiliary variable, and λ ₁ and λ ₂ are both penalty coefficients. In the calculation process, a soft threshold algorithm is used to iteratively update V.

Preferably, the step 2: obtaining the images to be classified, including:

When the user touches and selects a plurality of first photos on the viewing interface of the album to form a circled selection box and the circled selection box expands in the same expansion direction within the preset first time, obtain and output the preset touch-free circle selecting prompt information, and at the same time, controlling the encircled box to continue expanding along the expanding direction at a preset first expanding speed;

Dynamically obtain the current eye sight of the user;

determining a first gaze point corresponding to the eye sight in the viewing interface;

If the first gaze point is within the encircled box in the viewing interface, obtain the distance between the first gaze point and the edge of the target frame in the enlarging direction of the encircled box. the first vertical distance;

Based on the first vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-8):

为预设的第一关系系数；Wherein, v ₁ ′ is the first expansion speed after adjustment, v ₁ is the first expansion speed before adjustment, l ₁ is the first vertical distance,

is the preset first relationship coefficient;

If the first gaze point falls within the to-be-selected range outside the encircled box in the viewing interface in the expanding direction and the first gaze point occurs within a preset second time change, and obtain the second vertical distance between the first gaze point and the target frame edge in the enlarging direction of the circle selection frame;

Based on the second vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-9):

为预设的第二关系系数；Wherein, v ₂ ′ is the first expansion speed after adjustment, v ₂ is the first expansion speed before adjustment, l ₂ is the second vertical distance,

is the preset second relationship coefficient;

If the first gaze point is within the to-be-selected range outside the encircled box in the viewing interface in the enlarging direction and the first gaze point does not occur within the second time change, and obtain a second photo corresponding to the first gaze point in the viewing interface;

When the second photo just enters the circled selection box, controlling the circled selection box to stop expanding;

Obtain the movement type of the target frame edge in the expansion direction of the circled selection frame;

When the movement type is line feed downward, controlling the circle selection box to deselect all the third photos to the right of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is line wrap up, controlling the circle selection box to deselect all the fourth photos on the left side of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is to change column to the right, control the circle selection box to deselect all the fifth photos below the second photo in the column where the second photo is located in the viewing interface;

When the movement type is to change columns to the left, controlling the circle selection box to deselect all sixth photos on the upper side of the second photo in the column where the second photo is located in the viewing interface;

After the selection is completed, all the seventh photos circled in the circle selection box are used as images to be classified, and the acquisition is completed.

Preferably, the acquiring the second photo corresponding to the first gaze point in the viewing interface includes:

obtaining the current thumbnail scale of the display interface;

obtaining a preset abbreviated ratio threshold corresponding to the size of the display interface;

If the abbreviated ratio is greater than or equal to the abbreviated ratio threshold, determine the eighth photo located at the first gaze point in the viewing interface, and use it as the second photo to complete the acquisition;

Otherwise, determining a plurality of ninth photos located at the first gaze point in the viewing interface;

Based on the ninth photo, generating a cut-off photo enlargement confirmation box;

suspending and displaying the cut-off photo enlargement confirmation box in the display interface;

determining a second gaze point in the viewing interface that corresponds to the user's current eye sight;

When the second gaze point is located in the cut-off photo enlargement confirmation frame and the second gaze point does not change within the preset third time, the cut-off photo magnification confirmation frame is located in the cut-off photo magnification confirmation frame. The ninth photo of the second gaze point is used as the second photo to complete the acquisition.

The present invention provides a weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism, including:

The building block is used to construct a fine-grained image classification network with a two-level cascade network structure based on the soft threshold penalty mechanism;

The acquisition module is used to acquire the images to be classified;

a preprocessing module for preprocessing the to-be-classified image;

The classification module is configured to perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description, claims, and drawings.

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. In the attached image:

1 is a flowchart of a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism in an embodiment of the present invention;

2 is a schematic structural diagram of MMAL-Net in an embodiment of the present invention;

3 is a schematic structural diagram of a fine-grained image classification network in an embodiment of the present invention;

FIG. 4 is a schematic diagram of a weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism according to an embodiment of the present invention.

Detailed ways

The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, but not to limit the present invention.

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, as shown in Figure 1, including:

The present invention provides a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, including:

Step 1: Based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascade network structure;

Step 2: Obtain the image to be classified;

Step 3: preprocessing the to-be-classified image;

Step 4: Perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

Described step 1: Based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascade network structure, including:

Build a first network branch, the first network branch includes: Input448*448*3, the first ResNet50, Feature14*14*2048, the first GAP, the first FC and the first Softmax connected in sequence;

Build a second network branch, the second network branch includes: Input224*224*3*mult, second ResNet50, Feature*7*7*2048*mult, second GAP, second FC and second Softmax connected in sequence ;

Connect Input224*224*3*mult in the second network branch to Input448*448*3 in the first network branch through crop;

Connect Feature14*14*2048 in the first network branch to the crop through APPM;

setting a first loss function RawLoss for the first network branch;

setting a second loss function PartLoss for the second network branch;

A soft threshold penalty mechanism is set in the APPM;

The first network branch, the second network branch, APPM and crop form a fine-grained image classification network with a two-level cascade network structure.

The APPM is formed based on SCDA, and the Feature14*14*2048 extracted by the APPM is closed along the channel direction of the pooling layer to obtain a 14*14*1 two-dimensional image. Multiple sliding windows of different sizes perform sliding window calculation on the two-dimensional image, and the calculation process is shown in formula (2-1):

Among them, H and W are the height and width of the sliding window respectively, A(x, y) is the value corresponding to the coordinate position of the collapsed two-dimensional image, and a _w is the calculation result of the sliding window.

The formula of the first loss function RawLoss is shown in formula (2-2):

Among them, m _i is the ith sample image, and _ni is the prediction probability of the first convolutional neural network CNN corresponding to the ith sample image.

The formula of the second loss function PartLoss is shown in formula (2-3):

Wherein, q is the number of local feature regions selected by the second ResNet50, m _iq is the number of the qth local feature regions corresponding to the ith sample image, and n _iq is the second convolutional neural network CNN corresponding to The predicted probability of the number of qth local feature regions corresponding to the ith sample image.

The soft threshold penalty mechanism includes:

Let F(x, y) be the image without noise, N(x, y) be the noise, G(x, y) be the image after the noise, and choose the L _1/2 paradigm to build the model as shown in formula (2) -5) shown:

i represents the serial number of the image. When the image is iteratively processed, there is a residual error in G(x _i , y _i )-F(x _i , y _i ), which means that there is noise in the image and affects it;

The residual state is limited by the soft threshold, and the penalty factor ||G(x _i ,y _i )-F(x _i ,y _i )|| _h is constructed first to limit G(x _i ,y _i )-F(x _i , y _i ) is not greater than 0, thereby reducing the degree of image interference by noise, as shown in formula (2-6):

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value. Therefore, the soft threshold can effectively reduce the influence of noise interference.

To further optimize the soft threshold method, the objective function can be further modified, as shown in formula (2-7):

Among them, V is an auxiliary variable, and λ ₁ and λ ₂ are both penalty coefficients. In the calculation process, a soft threshold algorithm is used to iteratively update V.

The working principle and beneficial effects of the above technical solutions are as follows:

The invention modifies the three-level cascade network of the MMAL-Net model into a two-level cascade mode, and puts the reasoning task of the last level on the first branch. That is to say, in the running process of the entire network model, the effect of classifying fine-grained networks can be obtained only through a common network classification model. Its structure is shown in Figure 3.

Input448*448*3 is image input (image size is 448 pixels*448 pixels, R (red), G (green), B (blue) three color channels);

The first ResNet50 is a network model, a deep residual network, and 50 refers to the number of network layers;

Feature14*14*2048 is 2048 14-pixel*14-pixel feature two-dimensional maps;

The first GAP is Global Average Pooling;

The first FC is a fully connected layer (Fully Connection);

The first Softmax is the excitation function as the output layer, which is often regarded as a multi-classifier in machine learning. The popular meaning is to input an item and get the probability of the category it may belong to.

Input224*224*3*mult is a fixed-size image input obtained by performing the crop method on Input448*448*3 (part of the image input block of the first branch, the image size is 224 pixels*224 pixels, R (red), G (green), B (blue) three color channels), mult refers to the number of fixed blocks obtained after the image is cropped.

The second ResNet50 is the same as the first ResNet50;

Feature*7*7*2048*mult is 2048 7-pixel*7-pixel feature two-dimensional maps in mult copies;

The second GAP, the second FC and the second Softmax are respectively the same as the first GAP, the first FC and the first Softmax;

Crop is cropping, which is to directly cut out a part of the image and retain the true size ratio of the original image. Depending on the method of image cropping, the number of resulting cropped images is also different.

After image preprocessing (filtering noise reduction, grayscale, and shrinking to 448 pixels * 448 pixels size), the image reduced to 448 pixels * 448 pixels size is used as the first-level input of the network. In the design process of the network model, ResNet50 is used as the backbone network for feature extraction. The residual block is specially designed to obtain the ResNet50 network, of which 50 refers to the total number of convolutional layers and fully connected layers in the deep network. The overall accuracy is moderate, and the overall calculation amount is also moderate. The ResNet50 residual block uses BottleNeck The structure will neither affect the accuracy nor reduce the overall calculation amount. The number of feature channels is 2048, which is also more helpful for selecting regions with identification. On the whole, the feature extraction of the two branches of the network adopts the method of parameter sharing, which not only reduces the amount of parameters of the entire network, but also makes each network suitable for images of different sizes and different parts.

The local feature extraction of the network adopts the APPM structure, which is formed based on the research of SCDA (Selective Convolutional Descriptor Aggregation). First, the 14*14*2048 extracted from the feature by the module are closed along the channel direction, and a 14*14*1 two-dimensional image is obtained. A 14*14*1 two-dimensional map is a two-dimensional map of image features with a size of 14 pixels*14 pixels.

Then, use several pre-set sliding windows of different sizes to calculate, and the calculation process inside the sliding window is shown in formula (2-1).

The preset height and width of the sliding window are H and W in the formula respectively, A(x,y) refers to the corresponding value of the coordinate position of the collapsed two-dimensional image, and a _w means the value calculated by the current sliding window. numerical value. A(x, y) is the value corresponding to the coordinate position of the collapsed two-dimensional image, specifically the coordinate value of the position of the two-dimensional image of 14*14*1 in the corresponding image to be classified, that is, the original image. Secondly, sort the a _w values of different positions, and the area with the larger output value is the area with discriminative features. Finally, in the area where the feature areas overlap and cannot be easily sorted, NMS (non maximum suppression, non-maximum suppression) is used to select multiple candidate areas with high identification and low redundancy as local feature areas, which become the second The input image for the layer.

The loss function adopted in the present invention is the most commonly used Cross-Entropy loss in the multi-classification problem. The Raw Loss formula of the first layer is shown in (2-2)

Among them, i represents the sample image, m _i represents the real label of the sample image, which is usually calculated by substituting 1, and _ni refers to the prediction probability of this type of neural network. m _i is the ith sample image, which is derived from the images captured by the sliding window above, with a total of 2048 images, that is, the value of i ranges from 1 to 2048.

The second layer Part Loss formula is shown in (2-3)

Here, q represents the number of selected local feature regions, and the loss value is calculated and obtained for each region, and finally the average value is obtained. m _iq is the q th local feature area corresponding to the ith sample image. Specifically, the sample image is derived from the image after cropping (crop) of the preprocessed image of the first branch, and its corresponding q th local feature area. The local feature regions screened by the second ResNet50 come from the image to be classified, that is, the preprocessed image of the original image, that is, the image after filtering, noise reduction, and grayscale. There are a total of 2048 local feature regions.

The first convolutional neural network CNN comes from the first ResNet50, which is a network model of the convolutional neural network; the second convolutional neural network is the same.

The total loss function formula is shown in (2-4)

Loss _total = μLoss _raw +ωLoss _part , (2-4)

Loss _raw , loss _part represent the loss value of the first layer and the second layer respectively, Loss _total represents the overall loss, μ and ω are values between 0-1, which represent the influence weight of the two-layer branch on the overall network , to adjust.

In order to reduce the complexity of the network and enable it to be used better, the lightweight method used in the present invention uses the SqueezeNet (lightweight convolutional neural network) structure. The dimensionality reduction effect is to reduce the number of input channels, and then perform convolution operations with 1*1 and 3*3 convolution kernels in the Expand layer, and finally splicing the running results to obtain the final output feature data.

The reduction of the overall network structure level can effectively reduce the total amount of calculation under the condition that the recognition accuracy is still high. However, as far as the original data set is concerned, there is a lot of environmental noise, and the branch of subject monitoring is missing. The present invention introduces a soft threshold penalty mechanism to further improve the local feature extraction capability. The influence of noise on the image can be divided into an additive model and a multiplicative model. Set F(x, y) as the image without noise, N(x, y) as the noise, and G(x, y) as the image after noise influence. , that is, the original image. In order to better extract features, the present invention selects the L _1/2 normal form to establish the model, as shown in formula (2-5).

i represents the sample image. When the image is iteratively processed, G(x _i , y _i )-F(x _i , y _i ) has a residual error, which means that there is noise in the image and affects it. The soft threshold method proposed by the present invention defines the residual state by the soft threshold.

为在数学上表示真实值，在式中表示后面式中取得最小值时(x_i,y_i)的值；arg为复数辐角，指的是复数的辐角主值，在此式中argmin即表示后面函数取最小值时，xi和yi的取值；F、N、G仅为函数名。(x _i , y _i ) is the coordinate of the i-th feature image;

In order to express the true value mathematically, the value of (x _i , y _i ) when the minimum value is obtained in the following formula is expressed in the formula; arg is the complex argument, which refers to the main value of the complex argument, in this formula argmin That is, when the following function takes the minimum value, the values of xi and yi; F, N, and G are only function names.

First construct the penalty factor ||G(x _i ,y _i )-F(x _i ,y _i )|| _h to restrict G(x _i ,y _i )-F(x _i ,y _i ) not to be greater than 0, Thereby reducing the degree of image interference by noise, as shown in formula (2-6)

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value. Therefore, the soft threshold can effectively reduce the influence of noise interference. The full name of st is subject to, which means to make...satisfy...In the formula, it means: under the condition of N(x _i , y _i )>0, F(x _i , y _i )>0, the above formula is established .

h is a preset constant, preferably 1/2.

To further optimize the soft threshold method, the objective function can be further modified

Among them, V is an auxiliary variable. In the calculation process, a soft threshold algorithm is used to iteratively update V. The soft threshold penalty mechanism is used to effectively resist the noise in the data.

This application optimizes the MMAL-Net network structure, reduces the calculation amount of the overall model by reducing the number of network branch layers, and also reduces the hardware requirements of the training model; on the basis of the new branch structure, a soft threshold penalty mechanism module is added, To resist the noise appearing in the image; effectively shield the interference information, so as to improve the overall accuracy.

Fine-grained image classification originally used strongly supervised deep learning to complete the classification task. The supervision information of the strongly supervised fine-grained image classification method relies on too many annotations. In addition to the network used for fine-grained classification, the entire algorithm framework needs a component Localized object detection network or semantic segmentation network. This leads to the high cost of data annotation and network structure, which makes the strong supervision method unable to be well applied in the actual production process. Therefore, this application only uses category labels, does not require additional labeling information, and belongs to a weakly supervised method.

In one embodiment, the step 2: acquiring the image to be classified includes:

When the user touches and selects multiple first photos on the viewing interface of the album to form a circled selection box, and the circled selection box expands in the same expansion direction within the preset first time, obtain and output the preset touch-free circle selecting prompt information, and at the same time, controlling the encircling box to continue to expand along the expanding direction at a preset first expanding speed;

Dynamically obtain the current eye sight of the user;

determining a first gaze point corresponding to the eye sight in the viewing interface;

If the first gaze point is within the encircled box in the viewing interface, obtain the distance between the first gaze point and the edge of the target frame in the enlarging direction of the encircled box. the first vertical distance;

Based on the first vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-8):

为预设的第一关系系数；Wherein, v ₁ ′ is the first expansion speed after adjustment, v ₁ is the first expansion speed before adjustment, l ₁ is the first vertical distance,

is the preset first relationship coefficient;

If the first gaze point falls within the to-be-selected range outside the encircled box in the viewing interface in the expanding direction and the first gaze point occurs within a preset second time change, and obtain the second vertical distance between the first gaze point and the edge of the target frame in the expansion direction of the circle selection frame;

Based on the second vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-9):

为预设的第二关系系数；Wherein, v ₂ ′ is the first expansion speed after adjustment, v ₂ is the first expansion speed before adjustment, l ₂ is the second vertical distance,

is the preset second relationship coefficient;

If the first gaze point is within the to-be-selected range outside the encircled box in the viewing interface in the enlarging direction and the first gaze point does not occur within the second time change, and obtain a second photo corresponding to the first gaze point in the viewing interface;

When the second photo just enters the circled selection box, controlling the circled selection box to stop expanding;

Obtain the movement type of the target frame edge in the expansion direction of the circled selection frame;

When the movement type is line feed downward, controlling the circle selection box to deselect all the third photos to the right of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is line wrap up, controlling the circle selection box to deselect all the fourth photos on the left side of the second photo in the row where the second photo is located in the viewing interface;

When the movement type is to change column to the right, control the circle selection box to deselect all the fifth photos below the second photo in the column where the second photo is located in the viewing interface;

When the movement type is to change columns to the left, controlling the circle selection box to deselect all sixth photos on the upper side of the second photo in the column where the second photo is located in the viewing interface;

After the selection is completed, all the seventh photos circled in the circle selection box are used as images to be classified, and the acquisition is completed.

The working principle and beneficial effects of the above technical solutions are as follows:

Generally, when a user selects an image that needs to be classified, due to the large amount of selection, when a touch operation is performed on a smart terminal (such as a mobile phone and a tablet, etc.), the user needs to continue to touch the operation, and the experience is poor. If the time is too long, it may cause discomfort to the user's fingers; especially for some users who are engaged in image classification, the experience is even worse. For example, when the user opens the mobile phone album, the finger touches and moves to the right on the upper left corner of the mobile phone screen, so that the photos to be selected in the all-select interface are selected. Swipe down and select all at the same time. The photo to be selected appears in the new interface. However, you need to keep your finger still until the photo is cut off. Therefore, there is an urgent need to solve it.

When the user touches the circle to select the first photo in the album, the selected first picture forms a circle, and the circle expands in the same direction (for example: 2 seconds) within a preset first time (for example: 2 seconds). vertically downward), it indicates that the user needs to continue to select more first photos in the album (for example, the display interface of the album is moving downward). Circle the prompt information (for example: "Finger can be removed! Start automatic circle") on the display interface, and control the circle to expand along the expansion direction (for example: vertically downward) to the preset first expansion The speed (eg: 1.2cm/s) continues to expand.

At this time, the user will continuously check the first photo that has been selected in the circled box or the first picture that is about to be circled in the circled box, to determine whether the photo deadline is reached. Acquire the user's eye sight line (the sight line acquisition belongs to the category of the prior art and will not be described repeatedly), and determine the first gaze point corresponding to the eye sight line in the viewing interface, where the first gaze point is the position the user is looking at. If the first fixation point falls within the circled selection box, it means that the user is not able to keep up with the first expansion speed of the circled selection box when viewing the first photo that has been circled in the circled selection box. Based on the first fixation point and the target The first vertical distance of the frame edge is used to slow down the first expansion speed. The larger the first vertical distance is, the faster the first expansion speed is, and the more the first expansion speed should be slowed down. If the first gaze point falls within the to-be-selected range in the expanding direction outside the circled selection box in the viewing interface and the first gaze point changes within the preset second time (for example: 3 seconds), it means that the user is in the View the first photo that is about to be selected in the circle selection box and the photo has not yet reached the deadline. The larger the second vertical distance between the first gaze point and the edge of the target frame, the slower the first expansion speed, and the more it is necessary to speed up the first expansion speed. . It fully guarantees that the expansion speed of the circle selection box can be close to the user, which improves the humanization and the intelligence of the touch-free circle selection. When the first fixation point falls within the range to be circled and the first fixation point does not change in the second time, it means that the cut-off photo has been reached; however, the cut-off photo is not necessarily the last photo in a row or the last photo in a column For a photo, the first photo after the cut-off photo needs to be eliminated. Further enhance the intelligence and humanization of touch-free circle selection. Acquire a second photo that is the cut-off photo. When the second photo just enters the marquee frame, the useless photo is eliminated based on the movement type of the target frame. Generally, the movement type is line feed downward, and users view photos in a circular view from left to right. Therefore, control the circle selection box to deselect all the third photos on the right side of the row where the second photo is located. When the movement type is newline up, the same is true. When the movement type is right-to-column, the user views the photos in a circular view from top to small. Therefore, control the circle selection box to deselect all the fifth photos below the second photo. When the movement type is left-to-column, the same is true.

In one embodiment, the acquiring the second photo corresponding to the first gaze point in the viewing interface includes:

obtaining the current thumbnail scale of the display interface;

obtaining a preset abbreviated ratio threshold corresponding to the size of the display interface;

If the abbreviated ratio is greater than or equal to the abbreviated ratio threshold, determine the eighth photo located at the first gaze point in the viewing interface, and use it as the second photo to complete the acquisition;

Otherwise, determining a plurality of ninth photos located at the first gaze point in the viewing interface;

Based on the ninth photo, generating a cut-off photo enlargement confirmation box;

suspending and displaying the cut-off photo enlargement confirmation box in the display interface;

determining a second gaze point in the viewing interface that corresponds to the user's current eye sight;

When the second gaze point is located in the cut-off photo enlargement confirmation frame and the second gaze point has not changed within the preset third time, the cut-off photo magnification confirmation frame is located in the cut-off photo magnification confirmation frame. The ninth photo of the second gaze point is used as the second photo to complete the acquisition.

The working principle and beneficial effects of the above technical solutions are as follows:

When acquiring the second photo that is the cut-off photo, generally, it is only necessary to determine the photo located at the first gaze point in the viewing interface. However, due to the small size of the mobile phone screen of some smart terminals or the small abbreviated ratio of some albums (for example: 1:50, 50 photos are displayed in one interface), the accuracy of the line of sight acquisition is limited, making the first gaze point impossible Cutoff photo for pinpointing the user's gaze.

Therefore, the current thumbnail ratio of the display interface is obtained, and at the same time, the thumbnail ratio threshold of the size of the display interface is obtained; the thumbnail ratio threshold is the cut-off photo of the first gaze point that can be used to accurately locate the user's gaze under the size of the display interface. Minimum abbreviated scale. If the abbreviated ratio is greater than or equal to the abbreviated ratio threshold, it means that the first gaze point can be used to accurately locate the cut-off photo of the user's gaze, and the eighth photo at the first gaze point can be determined as the second photo. Otherwise (the abbreviated ratio is less than the abbreviated ratio threshold), determine the ninth photo of the first gaze point, and based on the ninth photo, generate a cut-off photo enlargement confirmation box for the user to further confirm, when the user's current second gaze point falls In the cut-off photo enlargement confirmation box and there is no change within the preset third time (for example: 2 seconds), it is indicated that the user is staring at the cut-off photo, and the ninth photo at the second gaze point can be determined as the second photo. . It greatly improves the applicability of different smart terminals, and also improves the accuracy of cut-off photos.

The present invention provides a weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism, as shown in Figure 4, including:

Building module 1 is used to build a fine-grained image classification network with a two-level cascade network structure based on a soft threshold penalty mechanism;

Obtaining module 2, used to obtain images to be classified;

Preprocessing module 3, for preprocessing the to-be-classified image;

The classification module 4 is configured to perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (10)

1. a weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism, is characterized in that, comprising: Step 1: Based on the soft threshold penalty mechanism, construct a fine-grained image classification network with a two-level cascade network structure; Step 2: Obtain the image to be classified; Step 3: preprocessing the to-be-classified image; Step 4: Perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result. 2. A weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism as claimed in claim 1, wherein the step 1: constructing a fine-grained two-level cascade network structure based on the soft threshold penalty mechanism Image classification networks, including: Build a first network branch, the first network branch includes: Input448*448*3, the first ResNet50, Feature14*14*2048, the first GAP, the first FC and the first Softmax connected in sequence; Build a second network branch, the second network branch includes: Input224*224*3*mult, second ResNet50, Feature*7*7*2048*mult, second GAP, second FC and second Softmax connected in sequence ; Connect Input224*224*3*mult in the second network branch to Input448*448*3 in the first network branch through crop; Connect Feature14*14*2048 in the first network branch to the crop through APPM; setting a first loss function RawLoss for the first network branch; setting a second loss function PartLoss for the second network branch; A soft threshold penalty mechanism is set in the APPM; The first network branch, the second network branch, APPM and crop form a fine-grained image classification network with a two-level cascade network structure. 3. A weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism as claimed in claim 2, wherein the APPM is formed based on SCDA, and the Feature14* extracted from the APPM pair feature 14*2048 is closed along the channel direction of the pooling layer to obtain a two-dimensional image of 14*14*1, and the sliding window calculation is performed on the two-dimensional image with multiple preset sliding windows of different sizes. The calculation process is as follows Formula (2-1) shows:

Among them, H and W are the height and width of the sliding window respectively, A(x, y) is the value corresponding to the coordinate position of the collapsed two-dimensional image, and a _w is the calculation result of the sliding window. 4. a kind of weak supervision fine-grained image classification method based on soft threshold penalty mechanism as claimed in claim 2, is characterized in that, the formula of described first loss function RawLoss is as shown in formula (2-2):

Among them, m _i is the ith sample image, and _ni is the prediction probability of the first convolutional neural network CNN corresponding to the ith sample image. 5. a kind of weakly supervised fine-grained image classification method based on soft threshold penalty mechanism as claimed in claim 2, is characterized in that, the formula of described second loss function PartLoss is as shown in formula (2-3):

Wherein, q is the number of local feature regions selected by the second ResNet50, m _iq is the number of the qth local feature regions corresponding to the ith sample image, and n _iq is the second convolutional neural network CNN corresponding to The predicted probability of the number of qth local feature regions corresponding to the ith sample image. 6. A weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism according to claim 2, wherein the soft threshold penalty mechanism comprises: Let F(x, y) be the image without noise, N(x, y) be the noise, G(x, y) be the image after the noise, and choose the L _1/2 paradigm to build the model as shown in formula (2) -5) shown:

i represents the serial number of the image. When the image is iteratively processed, there is a residual error in G(x _i , y _i )-F(x _i , y _i ), which means that there is noise in the image and affects it; The residual state is limited by the soft threshold, and the penalty factor ||G(x _i ,y _i )-F(x _i ,y _i )|| _h is constructed first to limit G(x _i ,y _i )-F(x _i , y _i ) is not greater than 0, thereby reducing the degree of image interference by noise, as shown in formula (2-6):

In the formula, λ represents the penalty coefficient. Adjusting this coefficient can make the result close to the real value. Due to the limitation of the soft threshold, the approximate value may also appear above the real value. Therefore, the soft threshold can effectively reduce the influence of noise interference. 7. A weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism as claimed in claim 2, characterized in that, in order to further optimize the soft threshold method, the objective function can be further modified, as shown in formula (2-7) Show:

Among them, V is an auxiliary variable, and λ ₁ and λ ₂ are both penalty coefficients. In the calculation process, a soft threshold algorithm is used to iteratively update V. 8. A weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism according to claim 1, wherein the step 2: acquiring an image to be classified, comprises: When the user touches and selects multiple first photos on the viewing interface of the album to form a circled selection box, and the circled selection box expands in the same expansion direction within the preset first time, obtain and output the preset touch-free circle selecting prompt information, and at the same time, controlling the encircled box to continue expanding along the expanding direction at a preset first expanding speed; Dynamically obtain the current eye sight of the user; determining a first gaze point corresponding to the eye sight in the viewing interface; If the first gaze point is within the encircled box in the viewing interface, obtain the distance between the first gaze point and the edge of the target frame in the enlarging direction of the encircled box. the first vertical distance; Based on the first vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-8):

Wherein, v ₁ ′ is the first expansion speed after adjustment, v ₁ is the first expansion speed before adjustment, l ₁ is the first vertical distance,

is the preset first relationship coefficient; If the first gaze point falls within the to-be-selected range outside the encircled box in the viewing interface in the expanding direction and the first gaze point occurs within a preset second time change, and obtain the second vertical distance between the first gaze point and the target frame edge in the enlarging direction of the circle selection frame; Based on the second vertical distance, the first expansion speed is adjusted, and the adjustment formula is shown in formula (2-9):

Wherein, v ₂ ′ is the first expansion speed after adjustment, v ₂ is the first expansion speed before adjustment, l ₂ is the second vertical distance,

is the preset second relationship coefficient; If the first gaze point is within the to-be-selected range outside the encircled box in the viewing interface in the enlarging direction and the first gaze point does not occur within the second time change, and obtain a second photo corresponding to the first gaze point in the viewing interface; When the second photo just enters the circled selection box, controlling the circled selection box to stop expanding; Obtain the movement type of the target frame edge in the expansion direction of the circled selection frame; When the movement type is line feed downward, controlling the circle selection box to deselect all the third photos to the right of the second photo in the row where the second photo is located in the viewing interface; When the movement type is line wrap up, controlling the circle selection box to deselect all the fourth photos on the left side of the second photo in the row where the second photo is located in the viewing interface; When the movement type is to change column to the right, control the circle selection box to deselect all the fifth photos below the second photo in the column where the second photo is located in the viewing interface; When the movement type is to change columns to the left, controlling the circle selection box to deselect all sixth photos on the upper side of the second photo in the column where the second photo is located in the viewing interface; After the selection is completed, all the seventh photos circled in the circle selection box are used as images to be classified, and the acquisition is completed. 9 . The weakly supervised fine-grained image classification method based on a soft threshold penalty mechanism according to claim 8 , wherein the acquiring the second photo corresponding to the first gaze point in the viewing interface. 10 . ,include: obtaining the current thumbnail scale of the display interface; obtaining a preset abbreviated ratio threshold corresponding to the size of the display interface; If the abbreviated ratio is greater than or equal to the abbreviated ratio threshold, determine the eighth photo located at the first gaze point in the viewing interface, and use it as the second photo to complete the acquisition; Otherwise, determining a plurality of ninth photos located at the first gaze point in the viewing interface; Based on the ninth photo, generating a cut-off photo enlargement confirmation box; suspending and displaying the cut-off photo enlargement confirmation box in the display interface; determining a second gaze point in the viewing interface that corresponds to the user's current eye sight; When the second gaze point is located in the cut-off photo enlargement confirmation frame and the second gaze point has not changed within the preset third time, the cut-off photo magnification confirmation frame is located in the cut-off photo magnification confirmation frame. The ninth photo of the second gaze point is used as the second photo to complete the acquisition. 10. A weakly supervised fine-grained image classification system based on a soft threshold penalty mechanism, characterized in that it comprises: The building block is used to construct a fine-grained image classification network with a two-level cascade network structure based on the soft threshold penalty mechanism; The acquisition module is used to acquire the images to be classified; a preprocessing module for preprocessing the to-be-classified image; The classification module is configured to perform image classification on the preprocessing result based on the fine-grained image classification network, and output the image classification result.

Images