EP4064817A1

EP4064817A1 - Method for estimating a course of plant rows

Info

Publication number: EP4064817A1
Application number: EP20811569.1A
Authority: EP
Inventors: Maurice Gohlke; Sandra Amend; Daniel DI MARCO; Markus Hoeferlin
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2019-11-25
Filing date: 2020-11-20
Publication date: 2022-10-05
Also published as: US20230403964A1; DE102019218177A1; WO2021105006A1

Abstract

The invention relates to a method (100) for estimating a course of a plant row in a field while the field is being crossed in a direction of travel (36) substantially parallel to the plant row, comprising the following steps: capturing (S102) a plurality of images of the field; substantially in sync with obtaining (S104) position information relating to the position in which the individual images are captured on the field; classifying (S106) pixels or regions in individual images as crop plants; arranging (S108) the classified images in a global context using the position information; and estimating (S110) the course of the plant rows by determining a probability distribution (38) of the pixels or regions classified as crop plants in the global context along a direction perpendicular to the direction of travel.

Description

Method for estimating a course of rows of plants

description

The present invention relates to a method for estimating a course of rows of plants in a field.

The estimation of a course of rows of plants for the agricultural cultivation of a field is predominantly based on the processing of camera images, one of the two methods described below being used in most cases.

The first method involves segmenting the vegetation from the ground either by distinguishing between green (plants) and brown (ground) in the visible color range or taking into account the NDVI index, which is calculated using image information in the near-infrared range is recognized. The plant row is then estimated by means of straight line recognition between the individual plants that were previously segmented as vegetation. The line recognition is carried out using the Hough transformation.

In the second method, the plants are first segmented. A center point estimate is then carried out and the row of plants is estimated by means of a straight line through the plant center points. The straight lines are estimated using the RANSAC algorithm, the Least Square method, etc.

It is therefore the object of the present invention to provide a method for a more robust and more accurate estimation of a row of plants than is possible with the methods known up to now in the prior art for estimating the course of plant rows. The object is achieved by the method according to claim 1. Advantageous refinements are given in the dependent claims.

Embodiments of the present invention are described below with reference to the accompanying figures. Show it:

1 shows a flow chart of the method according to the invention;

Figure 2 is an image of a field that is semantically segmented; 3 shows another image of a field for which a probability distribution of pixels classified as useful plants is determined.

Detailed description of embodiments

In agriculture, seeds are sown in a field from which crops grow. A field can be understood to be a delimited area of land for the cultivation of useful plants or a part of such a field. A useful plant is understood to be an agriculturally used plant which itself or its fruit is used, e.g. as food, feed or as an energy plant. The seeds and consequently the plants are primarily arranged or sown in rows, with a predetermined distance between the individual useful plants in which objects can be present between the rows and between the individual plants within a row. However, the objects are undesirable because they reduce the yield of the plants or represent a disruptive influence during cultivation and / or harvesting. An object can be understood to mean any plant that is different from the useful plant, or any object. Objects can in particular weeds,

To be woods and stones.

In order to reduce the mentioned negative influence of weeds, they are either removed mechanically, e.g. by a milling machine, or sprayed with a pesticide by a sprayer. For this reason, a vehicle, on which a device for processing the plants is attached, travels the field along a predetermined route, i.e. in a lane between two adjacent rows of plants in the field, and the individual plants are processed in the meantime.

The vehicle is a vehicle specifically provided for working the field, such as an agricultural robot. However, the vehicle can also be an agricultural vehicle such as a tractor, a trailer, etc., or an aircraft such as a drone. The vehicle drives or flies across the field in a direction of travel that is essentially parallel to a row direction in which the plants are planted at the predetermined distance from one another. The vehicle drives through the field autonomously, but can also drive through the field based on control by an operator. The individual steps S102 to S110 of the method 100 according to the invention shown in FIG. 1 are described below. It should be noted that the method 100 is carried out continuously during a run, ie a continuous estimation of the row of plants takes place during the run.

In a first method step S102, a plurality of images of a surface of the field are acquired by an image acquisition means. The image acquisition means is a camera, such as a CCD camera, a CMOS camera, etc., which acquires an image in the visible area and provides it as RGB values or as values in another color space. The image acquisition means can, however, also be a camera that acquires an image in the infrared range. An image in the infrared range is particularly suitable for capturing plants, as the reflection of the plants is significantly increased in this frequency range. The image acquisition means can also be, for example, a mono, RGB, multispectral, hyperspectral camera. In addition, other data can be acquired using sensors such as 3D sensors, etc. The image acquisition means can also provide a depth measurement, e.g. by a stereo camera, a time-of-flight camera, etc. It is possible for several image acquisition means to be present on the vehicle and for several images to be acquired essentially synchronously by the different image acquisition means and data from different sensors.

The field on which the plants and objects are present is captured by the image capturing means while driving the vehicle on which the image capturing means is attached. The image capturing means is attached to the vehicle in such a way that an image sensor of the image capturing means is essentially parallel to a surface of the field. The image sensor of the image acquisition means can, however, also be inclined towards the surface of the field, for example in a direction of travel of the vehicle, in order to cover a larger area of the field.

The vehicle to which the image capturing means is attached drives or flies the field and the image capturing means captures the images at a predetermined time interval. The images are preferably captured in such a way that they overlap. For this reason, the image capturing means captures several images per second during the movement, as a result of which the images strongly overlap at a low movement speed. However, it is also possible for the images to be recorded in such a way that they do not overlap. The multitude of images can also be recorded as video. The images are then stored in a memory and are then available for further processing. The following step S104 is carried out essentially synchronously with step S102. In step S104, position information is acquired using position detection means. When driving through the rows of plants by an agricultural vehicle, GNSS systems, such as RTK-GPS, are used, which enable the vehicle to be localized with high precision in the field. The position detection means can also obtain the position information using high-precision GPS, odometry, visual odometry, encoder wheels or the use of sensors that estimate the speed based on optical features (e.g. cameras, speed-over-ground sensors, etc.). The position information is given as world coordinates, but can also be given as field coordinates, longitude + latitude, etc., for example. The position information is then correlated with the image recorded in step S102, so that the position in the field at which the image is recorded can be precisely determined.

For this purpose, the position information obtained is assigned to a point in the center of the image. The position information can also be assigned to another point in the image, e.g. a corner point. It should be noted that distances between the mounting position of the image capturing means and the mounting position of the position capturing means on the vehicle must be taken into account when correlating the position information with the captured image. The spatial extent of the image on the field in an X and a Y direction can then be determined using an image angle of the image capturing means and the distance of the image capturing means to the floor surface. If the image sensor of the image acquisition means is inclined to the surface of the field, this inclination must also be taken into account when calculating the spatial extent of the image on the field. In this way it is possible to determine the section of the field shown by the picture. Taking into account the resolution of the image, position information and consequently a position on the field can also be assigned to each of the pixels of the image. This procedure can also be applied to images that are acquired by another image acquisition means and to data that are acquired by different sensors.

In step S106, the pixels in the images are classified so that it is determined which of the pixels in the image represent a useful plant. In addition, it is also determined which of the pixels represent a certain type of weed or generally a weed and which of the pixels represent the bottom of the field. The images captured in step S102 become this individually semantically segmented, ie a classification of each individual pixel in the images is carried out and the individual pixels of the images are classified as useful plants, weeds or weeds or soil. It is also conceivable that areas that are composed of several pixels are semantically segmented.

Methods and architectures for the semantic segmentation of images are known from the prior art. In the present embodiment, a fully convolutional densenet is used as described in Jegou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017). "The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 11-19). is revealed. However, a fully convolutional neural network can also be used, as described in Long, J., Shelhamer, E., & Darreil,

T. (2015). "Fully convolutional networks for semantic segmentation". In Proceedings of the IEEE Conference on Computer Vision and pattern recognition (pp. 3431-3440). is revealed. However, it is also possible to use another known method for semantic segmentation of images.

Figure 2 shows an image of a field that is semantically segmented. A class is assigned to each pixel in the image and the pixel is colored according to the assigned class. In FIG. 2, black pixels represent the class of useful plants and bordered but not filled areas represent a class of weeds. The soil is not colored for the sake of simplicity. The semantic segmentation may have incorrect classifications. In the image shown in FIG. 2, pixels (dashed lines) in outer regions 22, 24 of the useful plant 20, in this case a sugar beet, are recognized as weeds. However, since the useful plant 20, with the exception of these small areas 22, 24, is correctly recognized as a useful plant, the method described below is robust against these unavoidable small errors.

In the following step S108, the semantically segmented images, which are captured during the same driving process and for which the position information was obtained in S104, are arranged using the position information in a global context which, compared to the pixel coordinate system on the image level, is a more global coordinate system, such as a above-mentioned world coordinate system. A position in the field, which is recorded due to the movement of the vehicle and the rapid repetition rate when recording the images in different images from different perspectives, has the same position information in all images and is therefore global Context placed in the same place. In the context of the present invention, the expression “global context” can be understood to mean an environment with its own coordinate system which includes the captured field areas and in or relative to which the images can be arranged.

In step S110 the course of a row of plants, that is to say consequently the coordinates of the row of plants in the global context, is estimated. The starting point for this are the images classified pixel by pixel, which are arranged in the global context. FIG. 3 shows an excerpt from the global context in which three useful plants 30, 32, 34 are recognized and shown in black. In addition, a large number of weeds are recognized and displayed with a frame. As already mentioned above, it is assumed that the movement of the vehicle takes place parallel to the row 36 of plants. An estimate of the plant row can thus be carried out in that the parameters of a probability distribution 38 of the pixels representing the useful plant (in the case of a pixel-by-pixel classification) or areas (in the case of a classification for larger areas or superpixels) in the global context, as in Fig 3, can be determined along a direction perpendicular to the direction of travel 36. The probability distribution 38 is preferably a symmetrical probability distribution and in particular a normal or Gaussian distribution.

A center of the row of plants corresponds to a calculated expected value of the probability distribution 38 and a width of the row of plants can be derived from a variance of the probability distribution 38. In this way, not only the course of the row of plants, but also a width of the row of plants can be specified. Due to the consideration of the pixels of all useful plants 30, 32, 34 in the previously acquired images for the estimation of the plant row, the method according to the invention is extremely robust against incorrect classification of individual pixels in the image. The method is also robust against inaccuracies in the row which are often generated during sowing due to rolled seeds or seeds that have been spread twice.

On the basis of this estimate of the row of plants in the captured images, a course of the row of plants in front of the vehicle can be estimated. The row of plants in front of the vehicle is estimated as a continuing straight line. The estimated course of the row of plants in front of the vehicle can in turn be integrated into the global context, so that the result of the pixel-by-pixel classification of useful plants is improved as the vehicle continues to travel. For this purpose, a function is implemented in the area of the plant row, the pixels in the area of the estimated plant row in front of the field have a higher probability assigns to classify it as a crop. The function can be a trapezoidal function, a rectangular function or another suitable function. In this way, a classification of the pixels as useful plants in the area of the estimated row can be weighted higher without the existing row estimation influencing the future row estimation.

After the estimated course of the plant row is known in the global context, this is converted to a coordinate system of the vehicle. This conversion makes it possible for a processing tool to be guided in a simple manner along a row of plants or for the vehicle or its wheels to be automatically controlled between two rows. In addition, a message can be displayed to a driver when driving manually when he is steering the vehicle into the area of a row of plants. Since the method according to the invention is able to determine the width of the row of plants, the plant row can also be reliably prevented from being traversed in the edge areas of the row of plants, so that fewer useful plants are destroyed due to imprecise traversing.

The method is also able to provide the user with further information. During a run, a distance between two crops in a row is determined so that conclusions can be drawn about regular seed application and the emergence of the individual seeds. In addition, by determining the width of individual rows, taking into account the variance of the probability distribution, it is possible to draw conclusions about the scattering of the seeds in the width direction.

Claims

Expectations

1. A method (100) for estimating a course of a row of plants in a field while driving the field along a direction of travel (36) essentially parallel to the row of plants, comprising the following steps:

Capturing (S102) a plurality of images from the field; essentially in sync with one

Obtaining (S104) position information about the position at which the individual images are captured on the field;

Classifying (S106) pixels or areas in individual images as useful plants;

Arranging (S108) the classified images, in particular in a global context, using the position information; and

Estimating (S110) the course of the row of plants by determining a probability distribution (38) of the pixels or areas classified as useful plants, in particular in a global context along a direction perpendicular to the direction of travel.

2. The method according to claim 1, wherein the pixels or areas in the image are classified as useful plants, weeds or soil by semantic segmentation.

3. The method according to any one of the preceding claims, wherein an expected value of the probability distribution corresponds to a center of the row of plants and a variance of the probability distribution corresponds to a width of the row of plants.

4. The method according to any one of the preceding claims, wherein the probability distribution is a normal distribution.

5. The method according to any one of the preceding claims, wherein a course of the estimated row of plants is converted into a coordinate system of a vehicle driving the field.

6. The method according to any one of the preceding claims, wherein the further course of the estimated row of plants in front of a vehicle driving through the field is estimated as a straight line.

7. The method according to claim 6, wherein the vehicle is automatically controlled in such a way that it drives in a lane between two estimated rows of plants adjacent in front of the vehicle.

8. The method of claim 6 or 7, wherein the row of plants estimated in front of the vehicle is used to improve the classification of pixels or areas in the plurality of images.

9. The method according to any one of the preceding claims, wherein a distance between two useful plants in an estimated row is determined in order to determine a quality of seed application.

10. The method according to any one of the preceding claims, wherein a variance of the probability distribution is used to determine a quality of seed application in the direction perpendicular to the direction of travel.

11. Computing unit for estimating a course of a row of plants in a field while driving the field along a direction of travel (36) essentially parallel to the row of plants, the computing unit being set up to carry out the following steps:

Receiving a plurality of captured images of the field; essentially in sync with one

Receiving an acquired position information about the position at which the individual images are recorded on the field;

Classifying (S106) pixels or areas in individual images as useful plants;

12. Agricultural working machine with a computing unit according to claim 11.