TWI769603B - Image processing method and computer readable medium thereof - Google Patents

Abstract

The present invention provides an image processing method comprising: obtaining at least an image of an object to be detected, generating a feature heat map based on the image; inputting both the image and the feature heat map into a deep learning model so as to obtain training feature values; obtaining prediction result from the deep learning model based on the training feature values. The present invention also provides a computer readable medium. The accuracy of object determination can be increased by the application of the present invention.

Description

Image processing method and storage medium

The invention relates to the technical field of image recognition, in particular to an image processing method and a storage medium.

The use of deep learning convolutional networks to discriminate images is the current mainstream, and has excellent performance for image discrimination. It can also be applied to the target detection of images, but the deep learning convolutional network also has certain limitations for the target discrimination. For example, it is difficult for developers to interpret the model generated by the convolutional network, and can only predict whether it is good or bad from the discriminative results of the model. Therefore, it is not easy to control and improve the network in a targeted manner. On the other hand, convolutional networks tend to ignore the correlation between the part and the whole. For example, it cannot extract features such as "position", "size" and "orientation" of the target in the image. However, in some cases, these factors are important basis for target judgment, and the lack of judgment of these factors will affect the accuracy of the model's target judgment.

In view of the above problems, the present invention proposes an image processing method and storage medium to improve the accuracy of target determination in image processing.

A first aspect of the present application provides an image processing method, the method includes: acquiring at least one image of a target to be tested; generating a feature heat map based on the image; Input into the deep learning model to obtain training feature values; obtain prediction results from the deep learning model according to the training feature values.

Preferably, generating a feature heat map based on the image includes: extracting feature values of the image; importing the feature values into a preset matrix; generating the feature heat map based on the imported preset matrix.

Preferably, after extracting the feature value of the image, the method further comprises: normalizing the feature value.

Preferably, the preset matrix includes a plurality of classification blocks, and each classification block includes a plurality of elements, and the importing the eigenvalues into the preset matrix includes: dividing the eigenvalue; import the divided eigenvalues into the corresponding classification block, wherein each eigenvalue corresponds to each element in the classification block.

Preferably, the feature values include feature values describing the size of the object in the image, feature values describing the position of the object in the image, feature values describing the texture of the object, and feature values describing the object Eigenvalues of the orientation in the image.

Preferably, generating the feature heat map based on the imported preset matrix includes: converting all elements in the imported preset matrix into grayscale values; and generating the feature heat map according to the converted preset matrix.

Preferably, the deep learning model includes a first convolutional neural network, a second convolutional neural network and a prediction unit.

Preferably, inputting the image and the feature heatmap into a deep learning model at the same time, and obtaining the training feature value includes: obtaining a first prediction parameter according to the first convolutional neural network and the image; The second convolutional neural network and the feature heat map are used to obtain second prediction parameters; the training feature values are obtained by combining the first prediction parameters and the second prediction parameters.

Preferably, the first convolutional neural network and the second convolutional neural network respectively comprise a plurality of convolutional layers and pooling layers, and the pooling layers are arranged between one or more convolutional layers.

Preferably, at least one of said convolutional layers and at least one of said pooling layers are combined into an intermediate layer of said first convolutional neural network and said second convolutional neural network.

Preferably, the first prediction parameters are extracted from a pooling layer of an intermediate layer of the first convolutional neural network, and at least one first prediction parameter is another first prediction parameter passing through a convolutional layer of the intermediate layer. The parameters output after processing; the second prediction parameters are extracted from the pooling layer of the middle layer of the second convolutional neural network, and at least one second prediction parameter is another second prediction parameter passing through the middle layer The parameters of the output of the convolutional layer after processing.

Preferably, combining the first prediction parameter and the second pre-stored parameter to obtain the training feature value includes: as the first prediction parameter extracted from the last intermediate layer of the first convolutional neural network the third prediction parameter of the image; the second prediction parameter extracted from the last intermediate layer of the second convolutional neural network is used as the fourth prediction parameter of the feature heatmap; combining a plurality of the first predictions parameter and the third prediction parameter, and combining a plurality of the second prediction parameter and the fourth prediction parameter to obtain the training feature value.

Preferably, obtaining the prediction result from the deep learning model according to the training feature value includes: inputting the training feature value to a prediction unit of the deep learning model; and the prediction unit outputting the prediction result.

Preferably, outputting the prediction result by the prediction unit includes: obtaining a calculation result according to a loss function in the prediction unit; The calculation result is exponentially normalized; the normalized calculation result is converted into a corresponding output label, wherein the output label corresponds to the level label of the target.

A second aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned image processing method is implemented.

The image processing method and medium provided by the present invention generate a feature heat map based on the image; input the image and the feature heat map into a deep learning model at the same time to obtain training feature values; according to the training feature value is predicted from the deep learning model. It solves the problem that the convolutional neural network cannot extract the image and learn the feature value of the image measurement data. Using the method provided in this application to discriminate the target in the image can obtain a higher accuracy rate.

S1~S4: Steps

201: The first classification block

202: Second Classification Block

203: The third classification block

204: Fourth Classification Block

100: Image Processing System

101: Get Mods

102: Generating Mods

103: Input module

104: Processing modules

10: Electronics

11: Memory

12: Processor

13: Computer Programs

FIG. 1 is a schematic flowchart of an image processing method provided by an embodiment of the present invention.

2A is a schematic diagram of a classification block of a preset matrix provided by an embodiment of the present invention.

FIG. 2B is a schematic diagram of a preset matrix provided by an embodiment of the present invention.

FIG. 3A is a schematic diagram of a to-be-processed image A according to an embodiment of the present invention.

FIG. 3B is a schematic diagram of a feature heat map corresponding to a to-be-processed image A according to an embodiment of the present invention.

FIG. 3C is a schematic diagram of a to-be-processed image B according to an embodiment of the present invention.

FIG. 3D is a schematic diagram of a feature heat map corresponding to a to-be-processed image B according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a deep learning model provided by an embodiment of the present invention.

FIG. 5 is a schematic diagram of an image processing system provided by an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

In order to understand the objects, features and advantages of the present invention more clearly, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments may be combined with each other in the case of no conflict.

In the following description, many specific details are set forth in order to facilitate a full understanding of the present invention, and the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terms used herein in the description of the present invention are for the purpose of describing specific embodiments only, and are not intended to limit the present invention.

Please refer to FIG. 1 , which is a schematic flowchart of an image processing method according to an embodiment of the present invention. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. For the convenience of description, only the parts related to the embodiments of the present invention are shown.

As shown in FIG. 1 , the image processing method includes the following steps.

Step S1, acquiring at least one image of the target to be measured.

In one embodiment, at least one image can be obtained by photographing the object to be measured by a camera. The camera can be a mobile phone, a monocular camera, a line scan camera, etc., and the target to be tested can be a person, an object, an animal, a mobile phone, a personal computer, and the like. In another embodiment, acquiring at least one image of the object to be measured may be at least one image of the object to be measured transmitted by the receiving server. In other embodiments, at least one image of the object to be tested may be acquired from a local database. In this embodiment, the image may include a complete or partial image of the object to be tested. The image can be of any resolution, and it can also be oversampled or undersampled, depending on actual needs.

Step S2, generating a feature heat map based on the image.

In one embodiment, the method for generating a feature heatmap based on an image includes:

(1) Extract the eigenvalues of the image; In one embodiment, the target to be measured may be one or more targets in the image, and the feature values include a feature value describing the size of the target in the image, a feature value describing the position of the target in the image, and a feature value describing the target's position in the image. The eigenvalues of the texture, and the eigenvalues that describe the orientation of the object in the image. The feature values that describe the size of the target include flaw length, grayscale difference, single-point flaw area, width, flaw brightness, cluster flaw area, and aspect ratio. The feature values describing the position of the target in the image include whether it is in the first area of the hole, whether it is in the second area of the hole, whether it is in the third area of the hole, whether it is in the first area of the vertical partition, whether it is in the second area of the vertical partition , Whether it is in the third area of the vertical partition, whether it is in the fourth area of the vertical partition, and whether it is in the rounded corner area, etc. The feature values that describe the texture of the target include density, cluster density, entropy, contrast, correlation, and uniformity. The feature value describing the direction of the target in the image may be the angle between the target and the X axis or/and the Y axis of the coordinate system (XOY) established according to the image, including angle_0, angle_1, angle_ 2 and so on.

In one embodiment, the goal may be to describe a defect area on the workpiece to be tested. Defects are caused when the workpiece to be tested is scratched, scratched or bruised during the production process, and can also include areas with stains on the workpiece to be tested.

It should be noted that the position of the target in the workpiece to be tested can be confirmed by the position of the target in the image, and the direction of the target in the workpiece to be tested can also be confirmed by the direction of the target in the image.

(2) The eigenvalues are normalized; in one embodiment, the eigenvalues can be normalized to a value in the range of 0 to 1.

(3) Import the eigenvalues into a preset matrix; in this embodiment, the preset matrix may include a plurality of classification blocks, and each classification block includes a plurality of elements respectively. One eigenvalue can be imported per element. The classification block can describe the characteristics of the object, such as size, texture, etc.

Preferably, when the number of feature values is less than the total number of elements in the classification block, 0 is added to the classification block.

For example, as shown in FIG. 2A , the preset matrix includes four sorting blocks, a first sorting block 201 , a second sorting block 202 , a third sorting block 203 and a fourth sorting block 204 . first category Block 201 describes the size of the object. The second classification block 202 describes the texture of the object, the third classification block 203 describes the location of the object, and the fourth classification block 204 describes the orientation of the object. The four classification blocks are size, texture, position and orientation.

As shown in FIG. 2B , the first classification block 201 , the second classification block 202 , the third classification block 203 and the fourth classification block 204 respectively include nine elements. The first classification block 201 imports the feature values describing the target size, such as defect length, grayscale difference, single-point defect area, width, defect brightness, cluster defect area and aspect ratio, and fills in a zero to the first Classification block 201. The second classification block 202 imports the feature values describing the target texture, such as density, cluster density, entropy, contrast, correlation and uniformity, and pads three zeros into the second classification block 202 . The third classification block 203 imports the feature values describing the location of the target, such as whether it is in the first area of the hole, whether it is in the second area of the hole, whether it is in the third area of the hole, whether it is in the first area of the vertical partition, whether it is in the vertical partition The second area, whether it is in the third area of the vertical partition, whether it is in the fourth area of the vertical partition and whether it is in the rounded corner area, and a zero is added to the third classification block 203 . The fourth classification block 204 imports feature values that describe the target orientation, eg, angle_0, angle_1, angle_2, angle_3, angle_4, angle_5, angle_6, angle_7, and angle _8.

(4) Generating a feature heatmap based on the imported preset matrix; in one embodiment, by converting all elements in the preset matrix into grayscale values, and then generating a feature heatmap according to the converted preset matrix.

In one embodiment, each element in the preset matrix is multiplied by 255 and converted into a grayscale value, and then each element in the converted preset matrix is used as a primitive point to obtain a feature heat map. FIG. 3A is the image A to be processed, and FIG. 3B is the feature heat map of the image A to be processed obtained by the above method. FIG. 3C shows the image B to be processed, and FIG. 3D is the feature heat map obtained by the above method for the image B to be processed.

In one embodiment, the feature values of the extracted images are divided according to categories, and then arranged in order of correlation between features, and finally the arranged preset matrix is converted into a feature heatmap, so that the feature heatmap can be compared with the feature heatmap. The images are applied together to a convolutional neural network model.

Preferably, after acquiring the feature heat map of the image, the method for processing the image further includes: adjusting the size of the feature heat map. In order to meet the needs of the deep learning model, the size of the feature heatmap needs to be processed first.

Step S3: Input the image and the feature heatmap into the deep learning model at the same time to obtain the training feature value.

In one embodiment, the image and feature heatmap are simultaneously input into a deep learning model, and the deep learning model includes a first convolutional neural network, a second convolutional neural network and a prediction unit.

In one embodiment, the image 2 is input into the first convolutional neural network of the deep learning model, while the feature heatmap is input into the second convolutional neural network of the deep learning model. The deep learning model is a pre-trained model, which can output the classification level of the target in the image after inputting the image, as shown in Figure 4. The method for training the deep learning model is an existing training method, which will not be repeated here.

It should be noted that the parameters of the deep learning model can be dynamically adjusted, and the deep learning model is not limited to the classification and determination of the target in this application, and is also applicable to the recognition of any other image.

Preferably, the first convolutional neural network and the second convolutional neural network respectively include a plurality of convolutional layers and pooling layers, and the pooling layers are arranged between one or more convolutional layers. At least one convolutional layer and at least one pooling layer are combined into an intermediate layer of the first convolutional neural network and the second convolutional neural network.

Preferably, the structures of the first convolutional neural network and the second convolutional neural network may be the same or different. For example, the middle layer of the first convolutional neural network includes four layers, and the middle layer of the second convolutional neural network also includes four layers. As another example, when the importance of the feature heatmap is low, a simpler neural network architecture can be used, for example, reducing the number of intermediate layers of the second convolutional neural network.

Preferably, inputting the image and the feature heat map into the deep learning model at the same time, and obtaining the training feature value includes: obtaining the first prediction parameter according to the first convolutional neural network and the image; according to the second convolutional neural network and the feature heatmap to obtain the second prediction parameter; combining the first prediction parameter and the second prediction parameter to obtain the training feature value.

In one embodiment, the first prediction parameters may be extracted from the pooling layer of the first convolutional neural network, and at least one first prediction parameter is a parameter output after another first prediction parameter is processed by the convolution layer. The second prediction parameters are extracted from the pooling layer of the second convolutional neural network, and at least one second prediction parameter is a parameter that is output after another second prediction parameter is processed by the convolution layer.

The intermediate layers of the first convolutional neural network may include multiple layers, and each intermediate layer includes a plurality of convolutional layers and pooling layers. The intermediate layers of the second convolutional neural network may also include multiple layers, each intermediate layer including multiple convolutional layers and pooling layers. As shown in FIG. 4 , the middle layer of the first convolutional neural network includes four layers, and four first prediction parameters can be obtained by extracting the prediction parameters of each middle layer in sequence, such as the first prediction parameter A, the first prediction parameter parameter B, first prediction parameter C, and first prediction parameter D. The first prediction parameter A is the parameter extracted from the first intermediate layer, the first prediction parameter B is the parameter extracted from the second intermediate layer, the first prediction parameter C is the parameter extracted from the third intermediate layer, and the first prediction parameter C is the parameter extracted from the third intermediate layer. The prediction parameter D is the parameter extracted from the fourth intermediate layer.

In one embodiment, the attention mechanism may be used to extract the first prediction parameters of each intermediate layer of the first convolutional neural network to obtain a plurality of first prediction parameters of the image. Other methods may also be used to extract the first prediction parameter, which is not limited in this application.

The middle layer of the second convolutional neural network also includes four layers. By extracting the prediction parameters of each middle layer in sequence, four second prediction parameters can be obtained, such as the second prediction parameter A, the second prediction parameter B, the second prediction parameter Prediction parameter C and second prediction parameter D. The second prediction parameter A is the parameter extracted from the first intermediate layer, the second prediction parameter B is the parameter extracted from the second intermediate layer, the second prediction parameter C is the parameter extracted from the third intermediate layer, and the second prediction parameter C is the parameter extracted from the third intermediate layer. The prediction parameter D is the parameter extracted from the fourth intermediate layer. Similarly, the attention mechanism can be used to extract the second prediction parameters of the middle layer of the second convolutional neural network to obtain multiple second prediction parameters of the feature heat map.

It should be noted that, the attention mechanism can also be used to extract the second prediction parameters of each intermediate layer of the second convolutional neural network to obtain multiple second prediction parameters of the image. Other methods may also be used to extract the second prediction parameter, which is not limited in this application.

In this embodiment, the first combined prediction parameter may be generated by combining multiple first prediction parameters, the second combined prediction parameter may be generated by combining multiple second prediction parameters, and then the first combined prediction parameter and the second combined prediction parameter may be connected , which constitute the fully connected layers of the deep learning model.

In one embodiment, combining the first prediction parameter and the second prediction parameter to obtain the training feature value includes:

(1) The first prediction parameter extracted from the last intermediate layer of the first convolutional neural network is used as the third prediction parameter. In this embodiment, the parameters of the last layer of the first convolutional neural network can be extracted to obtain the third prediction parameters of the image.

(2) The second prediction parameter extracted from the last intermediate layer of the second convolutional neural network is used as the fourth prediction parameter of the feature heatmap. In this embodiment, the parameters of the last layer of the second convolutional neural network can be extracted to obtain the fourth prediction parameter of the feature heat map.

(3) Combining multiple first prediction parameters and third prediction parameters, and combining multiple second prediction parameters and fourth prediction parameters to obtain training feature values. In this embodiment, the first combined prediction parameter may be generated by combining multiple first prediction parameters and the third prediction parameter, the second combined prediction parameter may be generated by combining multiple second prediction parameters and the fourth prediction parameter, and then the A combined prediction parameter and a second combined prediction parameter constitute a fully connected layer of the deep learning model.

It can be understood that the training feature value includes at least the first combined prediction parameter and the second combined prediction parameter.

Step S4, obtaining a prediction result from the deep learning model according to the training feature value.

In this embodiment, the prediction unit outputs the prediction result by inputting the training feature value to the prediction unit of the deep learning model. In one embodiment, the fully connected layer can be input into the prediction unit, the calculation result can be obtained according to the loss function in the prediction unit, the calculation result can be exponentially normalized, and the normalized calculation result can be converted into the corresponding output label, wherein , the output label corresponds to the level label of the target.

In one embodiment, the prediction unit may be a classifier, the classifier may be a Softmax loss function, and the result of the loss function is equivalent to the probability that the input image is classified into each level label. rate distribution. The fully connected layer is input to the Softmax loss function to obtain the calculation result, and then the calculation result is exponentially normalized, and finally converted into the corresponding output label. The output label corresponds to the level label of the target.

This application uses image processing technology to extract the bullseye image in the image to measure the feature value of the data, and generates a feature heatmap of the image based on the feature value, and uses the feature heatmap and the image together to learn and match the image with the convolutional neural network. Prediction to increase the features of image measurement data that cannot be extracted from the original image by a variety of convolutional neural networks.

The convolutional neural network of the present application can not only learn the feature values of the image itself, but also learn the associated image (such as feature heatmap) features specified by the user at the same time, which solves the problem of the convolutional neural network. For example, it is impossible to extract and learn the feature values of image measurement data such as "position", "size", and "direction". Using the present application to detect and discriminate images can obtain higher accuracy.

FIG. 1 to FIG. 4 describe the image processing method of the present invention in detail, and through the method, the accuracy of target determination in image processing can be improved. The functional modules of the software system and the hardware device architecture for implementing the image processing method will be introduced below with reference to FIG. 5 and FIG. 6 .

FIG. 5 is a structural diagram of an image processing system according to an embodiment of the present invention.

In some embodiments, the image processing system 100 may include a plurality of functional modules composed of program code segments. The program codes of each program segment in the image processing system 100 can be stored in the memory of the computer device and executed by at least one processor in the computer device to realize the function of image detection.

Referring to FIG. 5 , in this embodiment, the image processing system 100 can be divided into a plurality of functional modules according to the functions performed by the image processing system 100 , and each functional module is used to execute each step in the corresponding embodiment of FIG. processing function. In this embodiment, the functional modules of the image processing system 100 include: an acquisition module 101 , a generation module 102 , an input module 103 , and a processing module 104 . The functions of each functional module will be described in detail in the following embodiments.

The acquiring module 101 is used for acquiring at least one image of the object to be measured. In one embodiment, at least one image can be obtained by photographing the object to be measured by a camera. The camera can be a mobile phone, a monocular camera, Line scan cameras, etc., the target to be tested can be people, objects, animals, mobile phones, personal computers, etc. In another embodiment, acquiring at least one image of the object to be measured may be at least one image of the object to be measured transmitted by the receiving server. In other embodiments, the image of the object to be tested may be obtained from a local database. In this embodiment, the image may include a complete or partial image of the object to be tested. The image can be of any resolution, and it can also be oversampled or undersampled, depending on actual needs.

The generation module 102 is used to generate a feature heatmap based on the image.

In one embodiment, the generation module 102 may be used to:

(1) Extract the feature values of the image. In one embodiment, the feature values include feature values that describe the size of the object, feature values that describe the location of the object in the image, feature values that describe the texture of the object, and feature values that describe the orientation of the object in the image. The feature values that describe the target size include length, grayscale difference, area, width, brightness, color, saturation, and aspect ratio. The feature values describing the position of the target in the image include whether it is in the first area, whether it is in the second area, whether it is in the third area, whether it is in the first area of the vertical partition, whether it is in the second area of the vertical partition, whether it is in the first area of the vertical partition. The third area, whether it is in the fourth area of the vertical partition and whether it is in the rounded area, etc. The feature values that describe the target texture include density, cluster density, entropy, contrast, correlation, and uniformity. The feature value describing the direction of the target in the image can be the angle between the target and the X-axis or/and the Y-axis of the coordinate system (XOY) established according to the image, including angle_0, angle_1, angle_2, etc. .

(2) Normalize the eigenvalues. In one embodiment, the eigenvalues may be normalized to a value in the range of 0-1.

(3) Import the eigenvalues into the preset matrix; in one embodiment, the preset matrix may include a plurality of classification blocks, each classification block includes a plurality of elements, and each element can import one eigenvalue. Classification blocks can describe characteristics of objects such as size, texture, color, etc.

Preferably, when the number of feature values is less than the total number of elements in the classification block, 0 is added to the classification block.

For example, as shown in FIG. 2A , the preset matrix includes four sorting blocks, a first sorting block 201 , a second sorting block 202 , a third sorting block 203 and a fourth sorting block 204 . The first classification block 201 describes the size of the object. The second classification block 202 describes the texture of the object, the third classification block 203 describes the location of the object, and the fourth classification block 204 describes the orientation of the object. The four classification blocks are size, texture, position and orientation.

As shown in FIG. 2B , the first sorting block 201 , the second sorting block 202 , the third sorting block 203 and the fourth sorting block 204 respectively include nine elements. The first classification block 201 imports the feature values describing the target size, such as length, grayscale difference, color, width, brightness, area, and aspect ratio, and adds a zero to the first classification block 201 . The second classification block 202 imports the feature values of the target texture, such as density, cluster density, entropy, contrast, correlation and uniformity, and pads three zeros into the second classification block 202 . The third classification block 203 imports the feature values describing the location of the target, such as whether it is in the first area, whether it is in the second area, whether it is in the third area, whether it is in the first area of the vertical partition, whether it is in the second area of the vertical partition , whether it is in the third area of the vertical partition, whether it is in the fourth area of the vertical partition and whether it is in the rounded corner area, and add a zero to the third classification block 203 . The fourth classification block 204 imports feature values describing the target orientation, eg, angle_0, angle_1, angle_2, angle_3, angle_4, angle_5, angle_6, angle_7, and angle _8.

(4) Generate a feature heatmap based on the imported preset matrix. In one embodiment, all elements in the preset matrix are converted into grayscale values, and then a feature heat map is generated according to the converted preset matrix.

In one embodiment, each element in the preset matrix is multiplied by 255 to convert to a grayscale value. Then use each element in the converted preset matrix as a primitive point to obtain a feature heat map. FIG. 3A shows the image A to be processed, and FIG. 3B is the feature heat map of the image A to be processed. FIG. 3C shows the image B to be processed, and FIG. 3D is the feature heat map of the image B to be processed.

In one embodiment, the generating module 102 can be used to divide the feature values of the extracted images according to categories, and then arrange them in order of correlation between the features, and finally convert the arranged preset matrix into a feature heat map, Enables feature heatmaps to be applied to convolutional neural network models along with images.

Preferably, the generating module 102 can adjust the size of the feature heatmap after acquiring the feature heatmap of the image to meet the needs of the deep learning model.

The input module 103 is used to simultaneously input the image and the feature heatmap into the deep learning model to obtain training feature values.

In one embodiment, the input module 103 may simultaneously input the image and the feature heatmap into a deep learning model, where the deep learning model includes a first convolutional neural network, a second convolutional neural network and a prediction unit.

In one embodiment, the image is fed into the first convolutional neural network of the deep learning model, while the feature heatmap is fed into the second convolutional neural network of the deep learning model. The deep learning model is a pre-trained model, which can output the detection result of the target after inputting the image, as shown in Figure 4. The method for training the deep learning model is an existing training method, which will not be repeated here.

It should be noted that the parameters of the deep learning model can be dynamically adjusted, and the deep learning model is not limited to the determination of the target in this application, and is also applicable to the recognition of any other image.

The processing module 104 is used to obtain the prediction result from the deep learning model according to the training feature value.

In one embodiment, the processing module 104 is a deep learning model, and can output prediction results from the prediction unit of the deep learning model.

In one embodiment, the prediction unit may be a classifier, the classifier may be a Softmax loss function, and the result of the loss function is equivalent to the probability distribution of the input image being assigned to each level label. The training feature value is input to the Softmax loss function to obtain the calculation result, and then the calculation result is exponentially normalized, and finally converted into the corresponding output label. The output label corresponds to the detection result label of the target.

FIG. 6 is a schematic diagram of a functional module of an electronic device according to an embodiment of the present invention. The electronic device 10 includes a memory 11 , a processor 12 , and a computer program 13 , such as an image processing program, stored in the memory 11 and executable on the processor 12 .

In this embodiment, the electronic device 10 may be, but not limited to, a smart phone, a tablet computer, a computer device, and the like.

Exemplarily, the computer program 13 may be divided into one or more modules/units, and one or more modules/units are stored in the memory 11 and executed by the processor 12 . One or more modules/units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 13 in the electronic device 10 . For example, the computer program 13 may be divided into modules 101-104 in FIG. 5 .

Those skilled in the art can understand that the schematic diagram 6 is only an example of the electronic device 10, and does not constitute a limitation on the electronic device 10. The electronic device 10 may include more or less components than those shown in the figure, or combine certain components, or Various components such as the electronic device 10 may also include input and output devices and the like.

The so-called processor 12 may be a central processing unit (Central Processing Unit, CPU), and may also include other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC) , Off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor 12 is the control center of the electronic device 10 , and uses various interfaces and lines to connect various parts of the entire electronic device 10 .

The memory 11 can be used to store computer programs 13 and/or modules/units. The processor 12 executes or executes the computer programs and/or modules/units stored in the memory 11 and calls the data to realize various functions of the electronic device 10 . The memory 11 may include an external storage medium or a memory. In addition, the memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, Smart Media Card (SMC), Secure Digital (Secure Digital, SD) card, flash memory card (Flash Card), at least one disk memory device, flash memory device, or other volatile solid state memory device.

If the modules/units integrated in the electronic device 10 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the method embodiments of the present invention can also be implemented by computer programs. The computer program can be stored in a computer-readable storage medium, and when executed by the processor, the computer program can implement the steps of each method embodiment.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present invention.

S1~S4: Steps

Claims (13)

An image processing method, which is improved in that the method includes: acquiring at least one image of a target to be tested; generating a feature heat map based on the image; inputting the image and the feature heat map into deep learning at the same time In the model, the training feature value is obtained, and the deep learning model includes a first convolutional neural network, a second convolutional neural network and a prediction unit; according to the first convolutional neural network and the image, the first convolutional neural network is obtained. a prediction parameter; obtain a second prediction parameter according to the second convolutional neural network and the feature heatmap; obtain the training feature value by combining the first prediction parameter and the second prediction parameter; The training feature values obtain prediction results from the deep learning model. The image processing method according to claim 1, wherein generating a feature heatmap based on the image comprises: extracting feature values of the image; importing the feature values into a preset matrix; based on the imported preset The matrix generates the feature heatmap. The image processing method according to claim 2, wherein after extracting the feature value of the image, the method further comprises: normalizing the feature value. The image processing method according to claim 3, wherein the preset matrix includes a plurality of classification blocks, and each classification block includes a plurality of elements, and the importing the eigenvalues into the preset matrix includes: according to The type of the classification block divides the feature value; the divided feature value is imported into the corresponding classification block, wherein each feature value corresponds to each element in the classification block. The image processing method according to any one of claims 2 to 4, wherein the feature value includes a feature value describing the size of the object in the image, and the feature value describing the size of the object in the image eigenvalues of the position in the image, eigenvalues describing the texture of the object, and eigenvalues describing the orientation of the object in the image. The image processing method according to claim 2, wherein generating the feature heat map based on the imported preset matrix comprises: converting all elements in the imported preset matrix into grayscale values; and A preset matrix generates the feature heatmap. The image processing method according to claim 1, wherein the first convolutional neural network and the second convolutional neural network respectively comprise a plurality of convolutional layers and pooling layers, and the pooling layers are set in between one or more convolutional layers. The image processing method according to claim 7, wherein at least one of the convolutional layers and at least one of the pooling layers is combined into a middle of the first convolutional neural network and the second convolutional neural network Floor. The image processing method according to claim 8, further comprising: extracting the first prediction parameters from the pooling layer of the middle layer of the first convolutional neural network, and at least one first prediction parameter is another A first prediction parameter is a parameter output after being processed by the convolutional layer of the middle layer; the second prediction parameter is extracted from the pooling layer of the middle layer of the second convolutional neural network, at least one second prediction parameter The parameter is another second prediction parameter that is output after being processed by the convolutional layer of the intermediate layer. The image processing method according to claim 8, wherein combining the first prediction parameter and the second pre-stored parameter to obtain the training feature value includes: The first prediction parameter extracted from the layer is used as the third prediction parameter of the image; the second prediction parameter extracted from the last intermediate layer of the second convolutional neural network is used as the fourth prediction of the feature heatmap. parameter; A plurality of the first prediction parameters and the third prediction parameters are combined, and a plurality of the second prediction parameters and the fourth prediction parameters are combined to obtain the training feature value. The image processing method according to claim 1, wherein obtaining a prediction result from the deep learning model according to the training feature value comprises: inputting the training feature value into a prediction unit of the deep learning model; the predicting The unit outputs the predicted result. The image processing method according to claim 11, wherein the prediction unit outputting the prediction result comprises: obtaining a calculation result according to a loss function in the prediction unit; exponentially normalizing the calculation result; The calculated result is converted into a corresponding output label, wherein the output label corresponds to the level label of the target. A computer-readable storage medium is improved in that a computer program is stored thereon, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

Images