CN112581483B - Self-learning-based plant leaf vein segmentation method and device - Google Patents

Abstract

The application provides a self-learning-based plant leaf vein segmentation method and device, and relates to the technical field of data processing, wherein the method comprises the following steps: training the marked plant leaf sample through a deep neural network model, and processing the unmarked plant leaf picture through an acquisition feature extraction module, a rough vein extraction module and a fine vein extraction module to acquire a rough vein picture and a fine vein picture; and fusing the rough vein image and the fine vein image to obtain a vein segmentation image as the labeling information of the plant leaf image without labeling, and training the deep neural network model according to a preset loss function so that the trained deep neural network model processes the plant leaf image to be processed to obtain a plant leaf segmentation result. Therefore, the model can automatically learn information in a large number of unmarked pictures by using a small number of marked pictures, so that the generalization performance is improved, and the efficiency and the accuracy of the plant leaf vein segmentation are improved.

Description

Method and device for segmentation of plant leaf veins based on self-learning

technical field

The present application relates to the technical field of data processing, and in particular, to a method and device for segmenting plant leaf veins based on self-learning.

Background technique

The leaves of plants are important organs of plants, and the outline and veins of leaves are important components of leaf morphological characteristics. They contain the intrinsic attributes and important genetic information of plants, and leaf veins are regarded as the "fingerprints" of leaves, not only as important parameters for measuring plant growth and development, growth status, genetic characteristics and other biochemical reaction processes, but also as a classification of plants. The important basis for identification and identification is widely used in agricultural production and scientific research services. Plant species on the earth are huge, and the extraction of plant leaf veins is of great significance in botany, agricultural production and horticulture. Traditional plant vein segmentation methods are done manually, such as using chemical reagents, high-resolution scanners and X-rays, which all require specialized technicians and complex equipment and are inefficient. With the development of artificial intelligence and computer vision, deep learning methods are promising in solving such problems, however, existing methods usually require a large number of finely annotated images for training, and the annotation process is tedious and labor-intensive.

SUMMARY OF THE INVENTION

The present application aims to solve one of the technical problems in the related art at least to a certain extent.

To this end, the first purpose of this application is to propose a method for segmenting plant leaf veins based on self-learning, using a deep learning algorithm framework to clearly and accurately extract the leaf outline and leaf veins in a picture containing plant leaves, which can be directly To process photos taken in a simple background, no special image preprocessing is required, and the neural network is trained by using as few labeled picture samples (such as 10 pictures) and a large number of unlabeled picture samples (such as hundreds of pictures) as possible. Iterative self-learning training is carried out, and through the continuous iterative process, more and more clear and complete contours and leaf vein segmentation maps are obtained. The algorithm greatly reduces the need for a large amount of labeled data in the training process, and makes full use of the information of the unlabeled pictures themselves, so that the model can learn the features related to the contour and leaf veins in repeated iterations, making the extraction of the contour and leaf veins clearer and more accurate. Finally, the outline and veins of the input image can be segmented end-to-end, which not only saves the complex process of previous image preprocessing and improves the generalization performance, but also makes full use of the information of unlabeled data, which is more conducive to the model to extract leaves. characteristic information.

The second objective of the present application is to propose a self-learning-based device for segmenting plant leaf veins.

In order to achieve the above purpose, the first aspect of the present application proposes a method for segmenting plant leaf veins based on self-learning, including:

Obtain the labeled plant leaf picture samples, and train the labeled plant leaf samples through a deep neural network model to obtain a feature extraction module, a rough leaf vein extraction module, and a fine leaf vein extraction module;

Obtain the unlabeled plant leaf picture and input the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and obtain the rough leaf vein map and the fine leaf vein map of the unlabeled plant leaf picture;

The rough leaf vein map and the fine leaf vein map are fused to obtain the leaf vein segmentation map of the unlabeled plant leaf picture, and the leaf vein segmentation map is used as the label information of the unlabeled plant leaf picture and according to A preset loss function is used to train the deep neural network model, so that the trained deep neural network model can process the picture of the plant leaf to be processed, and obtain the plant leaf segmentation result.

The self-learning-based plant leaf vein segmentation method in the embodiment of the present application trains the labeled plant leaf samples through a deep neural network model, and obtains the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for unlabeled plant leaves. The image is processed to obtain the rough leaf vein map and the fine leaf vein map; the rough leaf vein map and the fine leaf vein map are fused to obtain the leaf vein segmentation map as the label information of the unlabeled plant leaf image, and the deep neural network model is performed according to the preset loss function. training, so that the trained deep neural network model can process the plant leaf pictures to be processed to obtain plant leaf segmentation results. Therefore, using a very small number of labeled pictures allows the model to automatically learn the information in a large number of unlabeled pictures, thereby improving generalization and improving the efficiency and accuracy of plant leaf vein segmentation.

In an embodiment of the present application, an unlabeled plant leaf picture is obtained and input into the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and the unlabeled plant leaf picture is obtained. Coarse and fine vein maps, including:

The feature extraction module performs feature extraction on the unlabeled plant leaf picture to obtain a feature map;

The rough leaf vein extraction module processes the feature map to obtain the intermediate layer feature map and the rough leaf vein map;

The fine vein extraction module processes the feature map and the intermediate layer feature map to obtain the fine vein map.

In an embodiment of the present application, using the leaf vein segmentation map as the label information of the unlabeled plant leaf picture and training the deep neural network model according to a preset loss function, including:

Input the leaf vein segmentation map as the label information of the unlabeled plant leaf picture into the deep neural network model for training, calculate the difference between the training result and the label information according to the loss function, and adjust the deep neural network model parameters until the training conditions are met.

In an embodiment of the present application, a confidence map is generated according to the rough leaf vein segmentation map, and the loss of the region corresponding to the high-confidence region in the confidence map and the labeled plant leaf picture sample is calculated, and the loss function is as follows:

l(x,y)=L={l ₁ ,l ₂ ,...,l _N } ^T (1)

l _n = -w _n [y _n ·logx _n ] (2)

Among them, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, y _n is the label value of the corresponding position of the marked image, w _n ∈ {0,1} is the confidence weight, when the corresponding output position When the confidence is high, w _n is 1, otherwise it is 0.

In an embodiment of the present application, the loss of the area corresponding to the fine vein map and the labeled plant leaf picture sample is calculated, and the loss function is as follows:

l(x,y)=L={l ₁ ,l ₂ ,...,l _N } ^T (3)

l _n = -y _n ·logx _n (4)

Wherein, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, and y _n is the label value of the corresponding position of the marked picture.

In order to achieve the above purpose, a second aspect of the present application provides a device for segmenting plant leaf veins based on self-learning, including:

obtaining a training module for obtaining the labeled plant leaf picture samples, and training the labeled plant leaf samples through a deep neural network model to obtain a feature extraction module, a rough leaf vein extraction module and a fine leaf vein extraction module;

The obtaining module is used to obtain the unlabeled plant leaf picture and input the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and obtain the rough leaf vein map and the fine vein picture of the unlabeled plant leaf picture. leaf vein diagram

A processing module, configured to fuse the rough leaf vein map and the fine leaf vein map, obtain a leaf vein segmentation map of the unlabeled plant leaf picture, and use the leaf vein segmentation map as the unlabeled plant leaf picture and the deep neural network model is trained according to the preset loss function, so that the trained deep neural network model can process the plant leaf picture to be processed and obtain the plant leaf segmentation result.

The self-learning-based plant leaf vein segmentation device of the embodiment of the present application trains the labeled plant leaf samples through a deep neural network model, and obtains the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for the unlabeled plant leaves. The image is processed to obtain the rough leaf vein map and the fine leaf vein map; the rough leaf vein map and the fine leaf vein map are fused to obtain the leaf vein segmentation map as the label information of the unlabeled plant leaf image, and the deep neural network model is performed according to the preset loss function. training, so that the trained deep neural network model can process the plant leaf pictures to be processed to obtain plant leaf segmentation results. Therefore, using a very small number of labeled pictures allows the model to automatically learn the information in a large number of unlabeled pictures, thereby improving generalization and improving the efficiency and accuracy of plant leaf vein segmentation.

In an embodiment of the present application, the acquisition module is specifically used for:

The feature extraction module performs feature extraction on the unlabeled plant leaf picture to obtain a feature map;

The rough leaf vein extraction module processes the feature map to obtain the intermediate layer feature map and the rough leaf vein map;

The fine vein extraction module processes the feature map and the intermediate layer feature map to obtain the fine vein map.

In an embodiment of the present application, the processing module is specifically used for:

Input the leaf vein segmentation map as the label information of the unlabeled plant leaf picture into the deep neural network model for training, calculate the difference between the training result and the label information according to the loss function, and adjust the deep neural network model parameters until the training conditions are met.

In an embodiment of the present application, a confidence map is generated according to the rough leaf vein segmentation map, and the loss of the region corresponding to the high-confidence region in the confidence map and the labeled plant leaf picture sample is calculated, and the loss function is as follows:

l(x,y)=L={l ₁ ,l ₂ ,...,l _N } ^T (1)

l _n = -w _n [y _n ·logx _n ] (2)

Among them, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, y _n is the label value of the corresponding position of the marked image, w _n ∈ {0,1} is the confidence weight, when the corresponding output position When the confidence is high, w _n is 1, otherwise it is 0.

In an embodiment of the present application, the loss of the area corresponding to the fine vein map and the labeled plant leaf picture sample is calculated, and the loss function is as follows:

l(x,y)=L={l ₁ ,l ₂ ,...,l _N } ^T (3)

l _n = -y _n ·logx _n (4)

Wherein, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, and y _n is the label value of the corresponding position of the marked picture.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a self-learning-based plant leaf vein segmentation method provided in Embodiment 1 of the present application;

Fig. 2 is an example diagram of a self-learning-based plant leaf vein segmentation method according to an embodiment of the application;

3 is an example diagram of picture data according to an embodiment of the present application;

4 is a flowchart of extracting pseudo-labels for leaf vein segmentation from unlabeled pictures using a pre-trained model according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a plant leaf vein segmentation device based on self-learning provided by an embodiment of the present application.

Detailed ways

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

The following describes the method and device for segmenting plant leaf veins based on self-learning according to the embodiments of the present application with reference to the accompanying drawings.

The present application is based on the self-learning method for plant leaf vein segmentation, and realizes the self-learning-based contour and leaf vein segmentation tasks under a very small number of labeled samples. The segmentation effect from coarse to fine is achieved, and the training is not limited to several labeled data, which also makes the method have better generalization.

FIG. 1 is a schematic flowchart of a method for segmenting plant leaf veins based on self-learning according to Embodiment 1 of the present application.

As shown in Figure 1, the self-learning-based method for segmenting plant leaf veins includes the following steps:

Step 101: Obtain the labeled plant leaf picture samples, and train the labeled plant leaf samples through a deep neural network model to obtain a feature extraction module, a rough leaf vein extraction module, and a fine leaf vein extraction module.

Step 102 , obtaining an unlabeled plant leaf image and inputting the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and obtaining the rough leaf vein map and the fine leaf vein map of the unlabeled plant leaf image.

Step 103, fuse the rough leaf vein map and the fine leaf vein map to obtain the leaf vein segmentation map of the unlabeled plant leaf picture, and use the leaf vein segmentation map as the labeling information of the unlabeled plant leaf picture, and perform a deep neural network analysis according to a preset loss function. The network model is trained, so that the trained deep neural network model can process the plant leaf picture to be processed, and obtain the plant leaf segmentation result.

Specifically, the present application uses a deep learning model to extract leaf information and uses these intermediate features to infer leaf vein segmentation, and uses a self-learning learning mode to use a very small number of labeled pictures to let the model automatically learn information from a large number of unlabeled pictures , thereby improving generalization. This makes the application more robust against local noise. When using the algorithm, users do not need to understand the principle behind the algorithm, and the trained model can complete the task of plant classification.

This application uses a deep learning algorithm framework to clearly and accurately extract the leaf outline and leaf veins in pictures containing plant leaves, and use as few labeled picture samples (such as 10 pictures) and a large number of unlabeled picture samples ( Such as hundreds of pictures) iterative self-learning training of neural network to improve network performance and generalization.

Specifically, as shown in Figure 2, one is to use a small number of labeled pictures for pre-training, and three modules are obtained through the training of the deep neural network model - a feature extraction module, a rough leaf vein extraction module, and a fine leaf vein extraction module; The above feature extraction module is used to extract features from the newly arrived unlabeled images, and then based on this feature, the coarse vein extraction module and the fine vein extraction module are used to obtain high-confidence pseudo-labels with different finenesses through inference, respectively. Label fusion; thirdly, based on the self-learning iterative process, the obtained pseudo-label is used as the annotation of the input image to retrain the neural network, and through continuous iterative training, a clearer and more complete outline and leaf vein segmentation map are obtained.

Specifically, a small number of labeled image samples are shown in Figure 3, in which the white part of the labeled image is the contour and leaf vein to be segmented. The one is that there is only the input image on the left, and there is no sample of the manually annotated image on the right.

In the embodiment of the present application, the feature extraction module performs feature extraction on unlabeled plant leaf pictures to obtain feature maps; the rough leaf vein extraction module processes the feature maps to obtain intermediate layer feature maps and the rough leaf vein maps; fine leaf vein extraction The module processes the feature map and the middle layer feature map to obtain a fine vein map.

In the embodiment of the present application, a confidence map is generated according to the rough leaf vein segmentation map, and the loss of the high-confidence region in the confidence map and the area corresponding to the labeled plant leaf picture sample is calculated, and the loss function is as follows:

l(x,y)=L={l ₁ ,l ₂ ,...,l _N } ^T (1)

l _n = -w _n [y _n ·logx _n ] (2)

Among them, x _n is the predicted value of the nth pixel output by the rough vein extraction module, y _n is the label value of the corresponding position of the marked image, and w _n ∈ {0,1} is the confidence weight. When the corresponding output position is high When the confidence is used, w _n is 1, otherwise it is 0, that is, no loss is passed.

In the embodiment of the present application, the loss of the area corresponding to the fine vein map and the labeled plant leaf picture sample is calculated, and the loss function is as follows:

l(x,y)=L={l ₁ ,l ₂ ,...,l _N } ^T (3)

l _n = -y _n ·logx _n (4)

Among them, x _n is the predicted value of the nth pixel output by the rough leaf vein extraction module, and y _n is the label value of the corresponding position of the marked image. It is considered that the inference result is of high confidence, so there is no corresponding weight w _n .

Thus, by minimizing the above two loss functions, the relevant model is trained through the back-propagation algorithm.

Specifically, in order to obtain features containing rich image and semantic information, a feature extraction module is designed, which converts a fixed-size input image into a fixed-size feature map. In order to obtain clear and accurate leaf veins, the mesophyll and leaf veins are separated, and the leaves and the background are distinguished at the same time, a rough leaf vein extraction module and a fine leaf vein extraction module are designed. The input of the rough leaf vein extraction module is the feature map output by the feature extraction module, and the output is The rough segmentation map of the same size as the original input leaf image, that is, the probability value of each point as the extraction target, and the confidence level of each pixel point can be obtained. Infer again, and use multi-scale feature map inference to obtain a local fine vein map. In order to improve the generalization ability of each module of the present application, the rough vein extraction module and the fine vein extraction module share the image feature extraction module. In addition, the fine vein extraction module also uses the multi-scale feature maps of each intermediate layer in the rough vein extraction module.

The first step uses the convolutional neural network (CNN) method, which includes three modules: feature extraction module, rough leaf vein extraction module, and fine leaf vein extraction module. These three modules describe the shape features of plant leaf images on the one hand, and use the These features, inferring global and local leaf vein segmentation maps from different granularities, can be combined to achieve leaf contour and leaf vein segmentation. In addition, the basic performance of each model obtained through pre-training in this step is the basis for further learning of the entire algorithm. In order to improve the effectiveness of the method, the usual deep learning method will increase the amount of sample data to improve the model performance, and this application is aimed at It is designed for a small amount of labeled data. When the amount of data is kept at such a small amount, a more efficient network model - Residual Network (ResNet) is selected. The advantage of using this network is that there is no need to manually design a feature extraction method. Rich spatial structure and semantic features are extracted for contour and leaf vein segmentation. The residual network structure parameters specifically implemented by the feature extraction module in this application are shown in Table 1. From convolution 2 to convolution 5, they are composed of 6, 8, 12, and 6 convolution kernels of the same size, except for each layer. The first convolution kernel has a stride of 2, and the remaining convolution kernels have a stride of 1 (except for convolution 2, which has all strides of 1, because pooling 1 has a stride of 2, which is already in this step. reduced feature map size). The rough vein extraction module uses deconvolution to gradually increase the resolution of the feature map, and finally obtains the rough vein map of the same size as the input image. The specific network parameters are shown in Table 2. The fine vein extraction module is implemented by a convolution kernel with a size of 1*1, and a simple and effective network structure is used to achieve the purpose of secondary inference. The specific network parameters are shown in Table 3.

Table 1 Feature extraction module structure parameter table

Table 2 Structure parameters of rough leaf vein extraction module

layer name category size step size input size output size Deconvolution 1 Deconvolution 3*3 2*2 516*14*14 256*28*28 Deconvolution 2 Deconvolution 3*3 2*2 256*28*28 128*56*56 Deconvolution 3 Deconvolution 3*3 2*2 128*56*56 64*112*112 Deconvolution 4 Deconvolution 3*3 2*2 64*112*112 64*224*224 Deconvolution 5 Deconvolution 3*3 2*2 64*224*224 1*448*448

Table 3 Structure parameter table of fine vein extraction module

layer name category size step size input size output size Convolution 1 convolution 1*1 1*1 1029*1*1 512*1*1 Convolution 2 convolution 1*1 1*1 513*1*1 256*1*1 Convolution 3 convolution 1*1 1*1 257*1*1 256*1*1 Convolution 4 convolution 1*1 1*1 257*1*1 64*1*1 Convolution 5 convolution 1*1 1*1 65*1*1 1*1*1

It should be noted that, when the feature extraction module performs feature extraction and segmentation on the original image, different deep learning network models can be used to achieve this, and the size of the feature map can also be implemented in different schemes.

The second step is to extract the feature map from the new unlabeled image by using the above feature extraction module, and then further use the rough vein extraction module and the fine vein extraction module to obtain high-confidence pseudo-graphs with different finenesses after inference based on this feature. label, and fuse the two pseudo-labels, the entire flow chart is shown in Figure 4. The feature extraction module operates on the input image to obtain a feature map with rich semantic information; the rough leaf vein extraction module gradually increases the resolution while using the information in these feature maps for segmentation, and the segmented image has the same size as the original input image. Since only a part of the result has high confidence, it is called a rough vein segmentation map; for the part with low confidence, it is sampled, and the fine vein extraction module is used for secondary inference. The input includes the feature map obtained by the feature extraction module, the intermediate layer feature map of the rough vein module, and the points sampled in the rough vein confidence map. These information are synthesized to generate the fine vein segmentation results corresponding to the sampling points. Finally, the rough leaf veins and the fine leaf veins are fused to obtain the final leaf vein segmentation map, which is the pseudo-label of the unlabeled leaf image.

It should be noted that the rough leaf vein extraction module can also use different deep learning network models to segment the features, and the sampling method of the confidence map can adopt various methods such as uniform sampling and non-uniform sampling. In the process of secondary inference of sampling point features, different deep learning models and fusion methods of pseudo-labels can also be used, such as additive, multiplicative, maximum, minimum and other methods.

即为分割结果，不需要再进行迭代训练。In the third step, the leaf vein segmentation map obtained in the second step, that is, the pseudo-label, is regarded as the label of the unlabeled image, and is input into the model obtained in the first step. Repeated retraining, as the following algorithm flow. As mentioned above, the feature extraction module uses Residual Network (ResNet) as the skeleton, and the rough vein extraction module includes multiple deconvolution layers. After the training converges, for any test image, the leaf vein segmentation process is similar to the above process, and the output is directly

That is, the segmentation result, no need for iterative training.

以及置信度图D_c；根据置信度图采样N个最不确定的点；计算这些点所对应的各个特征图的位置；结合特征图ε，粗糙叶脉图

以及粗糙叶脉提取模块的中间层特征，用精细叶脉提取模块做二次推断，得到精细叶脉

将粗糙叶脉图

和精细叶脉

融合，得到伪标签

将

作为I的标签计算损失函数l，作梯度反向传播更新模型参数。返回第一步迭代，直到收敛。Specifically, in the self-learning iterative process, input: unlabeled leaf image I, feature extraction module me obtained by pre-training, rough leaf vein extraction module m _c and fine leaf vein extraction module _{m f ,} and the secondary inference scale N; _Unlabeled leaf image: use the feature extraction module to extract the feature map of the image ε=me (I); use the rough vein extraction module to calculate the rough vein map of the leaf

And the confidence map D _c ; sample N most uncertain points according to the confidence map; calculate the position of each feature map corresponding to these points; combine the feature map ε, the rough vein map

And the middle layer features of the rough vein extraction module, use the fine vein extraction module to do secondary inference, and get the fine veins

The rough vein diagram

and fine veins

Fusion to get pseudo labels

Will

The loss function l is calculated as the label of I, and the model parameters are updated by gradient back-propagation. Return to the first iteration until convergence.

Therefore, for the plant leaves whose leaf veins are to be extracted, it is only necessary that the entire leaf is a simple background and is completely included in the photographed photos. Apply for the use of a deep learning model to automatically learn and extract leaf features, and then use these intermediate features to automate Through two-stage inference, a high-confidence leaf vein segmentation map is obtained. The secondary inference process can achieve the effect of allowing the model to infer the local information again and correct the errors in the first inference, which makes the results extracted by this application more accurate and robust.

In addition, the present application uses a self-learning learning mode to use a very small number of labeled pictures to allow the model to automatically learn the information in a large number of unlabeled pictures, which can improve the generalization ability of the present application. At the same time, the dependence on a large number of labeled pictures during model training is reduced, and the unlabeled data is fully utilized, which also makes the application scenarios of the present application wider.

Based on the description of the above embodiments, the present application uses the deep learning method to extract the multi-scale feature information of the leaves; for the leaves of the plants, the outline and the leaf veins of the leaves are extracted by the deep learning network - rough vein extraction module. For the leaves of plants, the contours and veins of the leaves are extracted through the deep learning network-fine vein extraction module. For the leaves of plants, the obtained contours and leaf veins are further inferred and segmented by means of confidence map sampling and confidence inference. For the pre-training process of each module, only a very small number of labeled leaf pictures are required. For the self-learning process, only the unlabeled leaf pictures can make the iterative self-learning algorithm converge without using additional labeled data. Combining confidence inference with self-learning, confidence inference provides a guarantee for self-learning pseudo-labels, and the model obtained from self-learning provides a more convincing confidence map for the inference process. For the generation of pseudo-labels, pseudo-labels are obtained by fusing the coarse vein map and the fine vein map.

For related technologies, most of them have extremely high requirements on the quality of the collected images, such as hyperspectral images, projected scanned images, and some require manual and complex processing to obtain the data formats required by the algorithm, such as manual cropping, point cloud data, The disadvantage of denoising and binarization is that it has high requirements for acquisition equipment. In real life, the most convenient way is to use a simple handheld device to shoot directly. The inability to directly input such a simple shooting original image makes the method inconvenient to apply. The processed image will lose the fine features such as the color texture of the image itself. Moreover, these methods use image processing algorithms that require manual parameter adjustment, which often makes it impossible to completely remove noise, and there are many burrs in the segmented leaf veins, which in turn makes the final effect particularly dependent on the quality of the input image. The process cannot be automated. More importantly, the leaves in the real scene often have finer veins, and these methods either require special equipment to obtain high-definition pictures to obtain fine results, or only simple and rough pictures can be obtained for simple pictures. Leaf veins, which makes these methods inconvenient and generalizable. However, a deep learning segmentation algorithm similar to the present invention often requires a large amount of carefully labeled training data to improve the effect, and a method that can automate processing and reduce the dependence on manually labeled data is urgently needed at this stage.

For the plant leaves whose veins are to be extracted in this application, only leaves with a clean background photographed by ordinary handheld devices (mobile phones, cameras, etc.) are required, and the veins can be automatically extracted without any preprocessing of the pictures. A very small amount of labeled data is required to train the model. Compared with other solutions, this has great advantages and convenience, saving time and manpower.

This application uses the deep learning method model to extract the leaf features, and only needs to use the pictures obtained by the portable handheld camera. It does not require special sensors and complex preprocessing work. The test pictures can be directly input into the trained model. The disadvantage of high image quality requirements, the effect and generalization are better.

The present application adopts a deep learning method model to automatically learn parameters without manual parameter adjustment, and the denoising effect is better. The application does not require special sensors to obtain pictures, which greatly reduces the difficulty of application, broadens the application scenarios, and captures richer feature information in pictures through neural networks, avoiding the problem of information loss in related technologies.

This application avoids tedious manual preprocessing operations, is closer to the actual scene, and overcomes the limitations of manually designing parameters in the related art, and the deep learning neural network algorithm used in this application has small errors and strong robustness. wide range.

The deep learning network used in this application is much better than Canny operator filtering in terms of denoising effect, has strong robustness, avoids the problem of burrs forming on the edges of leaf veins, and reduces the generation of broken areas of leaf veins, and finally only needs to test The automatic leaf vein segmentation can be completed by inputting the image into the trained model, without the need for human intervention in the middle processing process.

The deep learning network used in the present application can be processed automatically, avoiding so many thresholds that need to be manually set, and the algorithm of the present application has small errors, no burr, strong robustness, and high flexibility.

The training and test pictures of this application are all pictures taken directly by ordinary cameras, and point clouds are obtained without preprocessing, which avoids the loss of information caused by the preprocessing process. In the related art, only one main leaf vein in the middle can be obtained, while the present application can obtain a finer multi-level leaf vein, and the effect is better.

This application does not require such high-quality pictures, but only needs pictures taken by ordinary cameras, which greatly reduces the difficulty of application of the technology. In addition, the error of the deep learning algorithm in the present application is smaller than that of the skeletonization algorithm in the related art, the obtained leaf vein segmentation map is more refined, and at the same time, the problem of too thin or too thick leaf veins caused by continuous refinement in the skeletonization algorithm is avoided. The pixel width of the leaf veins in the original image is closer.

The self-learning-based plant leaf vein segmentation method in the embodiment of the present application trains the labeled plant leaf samples through a deep neural network model, and obtains the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for unlabeled plant leaves. The image is processed to obtain the rough leaf vein map and the fine leaf vein map; the rough leaf vein map and the fine leaf vein map are fused to obtain the leaf vein segmentation map as the label information of the unlabeled plant leaf image, and the deep neural network model is performed according to the preset loss function. training, so that the trained deep neural network model can process the plant leaf pictures to be processed to obtain plant leaf segmentation results. Therefore, using a very small number of labeled pictures allows the model to automatically learn the information in a large number of unlabeled pictures, thereby improving generalization and improving the efficiency and accuracy of plant leaf vein segmentation.

In order to realize the above embodiments, the present application also proposes a plant leaf vein segmentation device based on self-learning.

FIG. 5 is a schematic structural diagram of a plant leaf vein segmentation device based on self-learning according to an embodiment of the present application.

As shown in FIG. 5 , the plant leaf vein segmentation device based on self-learning includes an acquisition training module 510 , an acquisition module 520 and a processing module 530 .

The acquisition training module 510 is used to acquire the labeled plant leaf picture samples, and train the labeled plant leaf samples through the deep neural network model, and obtain a feature extraction module, a rough leaf vein extraction module and a fine leaf vein extraction module.

The obtaining module 520 is configured to obtain the unlabeled plant leaf picture and input the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and obtain the rough leaf vein picture and the fine vein extraction module of the unlabeled plant leaf picture. Detailed leaf vein diagram.

The processing module 530 is configured to fuse the rough leaf vein map and the fine leaf vein map, obtain the leaf vein segmentation map of the unlabeled plant leaf picture, and use the leaf vein segmentation map as the unlabeled plant leaf The labeling information of the picture and the deep neural network model are trained according to the preset loss function, so that the trained deep neural network model processes the plant leaf picture to be processed, and obtains the plant leaf segmentation result.

The self-learning-based plant leaf vein segmentation device of the embodiment of the present application trains the labeled plant leaf samples through a deep neural network model, and obtains the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for the unlabeled plant leaves. The image is processed to obtain the rough leaf vein map and the fine leaf vein map; the rough leaf vein map and the fine leaf vein map are fused to obtain the leaf vein segmentation map as the label information of the unlabeled plant leaf image, and the deep neural network model is performed according to the preset loss function. training, so that the trained deep neural network model can process the plant leaf pictures to be processed to obtain plant leaf segmentation results. Therefore, using a very small number of labeled pictures allows the model to automatically learn the information in a large number of unlabeled pictures, thereby improving generalization and improving the efficiency and accuracy of plant leaf vein segmentation.

It should be noted that, the foregoing explanations of the embodiment of the method for segmenting plant leaf veins based on self-learning are also applicable to the device for segmenting veins of plant leaves based on self-learning in this embodiment, which will not be repeated here.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless expressly and specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those of ordinary skill in the field of the present application can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program is in When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (6)

1. a plant leaf vein segmentation method based on self-learning, is characterized in that, comprises: Obtain the labeled plant leaf picture samples, and train the labeled plant leaf samples through a deep neural network model to obtain a feature extraction module, a rough leaf vein extraction module, and a fine leaf vein extraction module; Obtain the unlabeled plant leaf picture and input the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and obtain the rough leaf vein map and the fine leaf vein map of the unlabeled plant leaf picture; The rough leaf vein map and the fine leaf vein map are fused to obtain the leaf vein segmentation map of the unlabeled plant leaf picture, and the leaf vein segmentation map is used as the label information of the unlabeled plant leaf picture and according to The preset loss function is used to train the deep neural network model, so that the trained deep neural network model processes the plant leaf picture to be processed, and obtains the plant leaf segmentation result; Wherein, the obtained unlabeled plant leaf picture is input to the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and the rough leaf vein map and the fine leaf vein of the unlabeled plant leaf picture are obtained. Figures, including: The feature extraction module performs feature extraction on the unlabeled plant leaf picture to obtain a feature map; The rough leaf vein extraction module processes the feature map, and obtains the intermediate layer feature map, the rough leaf vein map, and a confidence map; Sampling N most uncertain points according to the confidence map; calculating the position of each feature map corresponding to the point; Combining the feature map, the rough leaf vein map and the intermediate layer features of the rough leaf vein extraction module, the fine leaf vein extraction module is used for secondary inference to obtain a fine leaf vein map; The method also includes: Generate a confidence map according to the rough leaf vein segmentation map, and calculate the loss of the high-confidence region in the confidence map and the area corresponding to the labeled plant leaf image sample. The loss function is as follows: l(x, y)=L={l ₁ , l ₂ , ..., l _N } ^T (1) l _n = -w _n [y _n ·logx _n ] (2) Among them, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, y _n is the label value of the corresponding position of the marked image, w _n ∈ {0, 1} is the confidence weight, when the corresponding output position When the confidence is high, w _n is 1, otherwise it is 0. 2 . The method according to claim 1 , wherein the deep neural network model is trained according to a preset loss function by using the leaf vein segmentation map as the label information of the unlabeled plant leaf picture. 3 . ,include: Input the leaf vein segmentation map as the label information of the unlabeled plant leaf picture into the deep neural network model for training, calculate the difference between the training result and the label information according to the loss function, and adjust the deep neural network model parameters until the training conditions are met. 3. The method of claim 1, wherein Calculate the loss of the area corresponding to the fine leaf vein map and the labeled plant leaf image sample, and the loss function is as follows: l(x, y)=L={l ₁ , l ₂ , ..., l _N } ^T (3) l _n = -y _n ·logx _n (4) Wherein, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, and y _n is the label value of the corresponding position of the marked picture. 4. a plant leaf vein segmentation device based on self-learning, is characterized in that, comprises: obtaining a training module for obtaining the labeled plant leaf picture samples, and training the labeled plant leaf samples through a deep neural network model to obtain a feature extraction module, a rough leaf vein extraction module and a fine leaf vein extraction module; The obtaining module is used to obtain the unlabeled plant leaf picture and input the feature extraction module, the rough leaf vein extraction module and the fine leaf vein extraction module for processing, and obtain the rough leaf vein map and the fine vein picture of the unlabeled plant leaf picture. leaf vein diagram A processing module, configured to fuse the rough leaf vein map and the fine leaf vein map, obtain a leaf vein segmentation map of the unlabeled plant leaf picture, and use the leaf vein segmentation map as the unlabeled plant leaf picture and training the deep neural network model according to the preset loss function, so that the trained deep neural network model processes the plant leaf pictures to be processed, and obtains the plant leaf segmentation results; Wherein, the acquisition module is specifically used for: The feature extraction module performs feature extraction on the unlabeled plant leaf picture to obtain a feature map; The rough leaf vein extraction module processes the feature map, and obtains the intermediate layer feature map, the rough leaf vein map, and a confidence map; Sampling N most uncertain points according to the confidence map; calculating the position of each feature map corresponding to the point; Combining the feature map, the rough leaf vein map and the intermediate layer features of the rough leaf vein extraction module, the fine leaf vein extraction module is used for secondary inference to obtain a fine leaf vein map; Wherein, the device is also used for: Generate a confidence map according to the rough leaf vein segmentation map, and calculate the loss of the high-confidence region in the confidence map and the area corresponding to the labeled plant leaf image sample. The loss function is as follows: l(x, y)=L={l ₁ , l ₂ , ..., l _N } ^T (1) l _n = -w _n [y _n ·logx _n ] (2) Among them, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, y _n is the label value of the corresponding position of the marked image, w _n ∈ {0, 1} is the confidence weight, when the corresponding output position When the confidence is high, w _n is 1, otherwise it is 0. 5. The apparatus according to claim 4, wherein the processing module is specifically used for: The leaf vein segmentation map is input into the deep neural network model as the label information of the unlabeled plant leaf picture for training, and the difference between the training result and the label information is calculated according to the loss function, and the value of the deep neural network model is adjusted. parameters until the training conditions are met. 6. The apparatus of claim 4, wherein Calculate the loss of the area corresponding to the fine leaf vein map and the labeled plant leaf image sample, and the loss function is as follows: l(x, y)=L={l ₁ , l ₂ , ..., l _N } ^T (3) l _n = -y _n ·logx _n (4) Wherein, x _n is the predicted value of the nth pixel point output by the rough leaf vein extraction module, and y _n is the label value of the corresponding position of the marked picture.

Images