JP2021111273A - Learning model generation method, program and information processor - Google Patents

Abstract

To provide a learning model generation method, a program, and an information processor that accurately output information related to an object from a photographed image by using combination data based on the photographed image including the object.SOLUTION: A learning model generation method includes the steps of: acquiring a photographed image including an object photographed by an imaging apparatus mounted on a moving object; acquiring a plurality of object extraction images obtained by extracting the object in association with the photographed image; and generating, based on the acquired photographed image, the plurality of object extraction images and training data including information related to the object, a learning model that outputs information related to the object when the photographed image and the plurality of object extraction images are input.SELECTED DRAWING: Figure 9

Description

The present technology relates to a learning model generation method, a program, and an information processing device.

Conventionally, a detection method has been proposed in which various information is detected from an image captured in front of a moving body such as a vehicle. Patent Document 1 discloses a vehicle white line detection device that is mounted on a vehicle and can more accurately detect a white line on the road on which the vehicle travels.

Japanese Unexamined Patent Publication No. 2007-220013

However, the method of Patent Document 1 has a problem that the ability to detect a white line to be detected from a captured image is not sufficient.

An object of the present disclosure is to provide a learning model generation method, a program, and an information processing device that accurately output information about an object from a captured image.

In the method of generating a learning model in one aspect of the present disclosure, an image taken by an image pickup device mounted on a moving body including an object is acquired, and a plurality of object extraction images obtained by extracting the object are obtained. Information about an object when a captured image and a plurality of object extracted images are input based on training data that is acquired in association with the captured image and includes the acquired captured image and a plurality of object extracted images and information about the object. Generate a training model that outputs.

According to the present disclosure, it is possible to accurately output information about an object from a captured image.

It is a schematic diagram of the learning model generation system in 1st Embodiment. It is a block diagram which shows the configuration example of an information processing apparatus. It is a block diagram which shows the configuration example of a control unit. It is a conceptual diagram which shows the generation method of a unit group value. It is explanatory drawing explaining the content of a data unit. It is explanatory drawing explaining an example of the combination method of the white line extracted image. It is explanatory drawing explaining the structure of the learning model. It is a figure which shows the example of the table which stored the group ID corresponding to n and m in a matrix form. It is a flowchart which shows an example of the generation processing procedure of a learning model. It is a flowchart which shows an example of the detailed procedure of unit group value acquisition. It is a flowchart which shows an example of the generation processing procedure of the learning model in 2nd Embodiment. It is a flowchart which shows an example of the detailed procedure of the unit group value acquisition in 2nd Embodiment. It is a block diagram which shows the structural example of the estimation system in 3rd Embodiment. It is a conceptual diagram which shows the estimation processing method. It is a flowchart which shows an example of the estimation processing procedure using a learning model. It is a flowchart which shows an example of the detailed procedure of the unit group value acquisition in 3rd Embodiment. It is a figure which shows the screen example displayed on the display device. It is a flowchart which shows an example of the estimation processing procedure using the learning model in 4th Embodiment. It is a flowchart which shows an example of the detailed procedure of the unit group value acquisition in 4th Embodiment.

The present invention will be specifically described with reference to the drawings showing the embodiments thereof.

(First Embodiment)
FIG. 1 is a schematic diagram of the learning model generation system 100 according to the first embodiment. The learning model generation system 100 includes an information processing device 1, a control unit 200 of the mobile body 2, and an image pickup device 31. The information processing device 1 and the control unit 200 are communicably connected via a network N1 such as the Internet or a public switched telephone network.

The moving body 2 is provided with a moving mechanism such as a vehicle, a motorcycle, a helicopter, a ship, or a drone, and an image pickup device 31 mounted on the moving body 2 photographs the outside of the moving moving body 2. Hereinafter, the moving body 2 will be described as being a vehicle. The image pickup device 31 is, for example, a drive recorder, which is a device for photographing the outside of the moving body 2 with a camera and recording the photographed video data on a recording medium such as an SD card. The imaging device 31 may include, for example, a distance measuring sensor such as a radar or a lidar (LIDAR: Laser Imaging Detection and Ranging) in addition to an image sensor such as a camera. The ranging sensor outputs the transmitted wave and receives the reflected wave from the object, so that the position and speed of the object can be calculated from the reception state of the reflected wave. The image pickup device 31 is connected to the control unit 200. The control unit 200 transmits the moving image captured by the imaging device 31 to the information processing device 1.

The information processing device 1 is, for example, a server computer. The information processing device 1 generates a learning model that outputs information about an object included in the captured image based on the information acquired from the control unit 200. In the first embodiment, the information processing device 1 is described as one server computer, but the functions or processes may be distributed among a plurality of server computers, or a plurality of virtually generated information processing devices 1 may be generated in one large computer. It may be one of the server computers (instances) of. The information processing device 1 may be installed inside the mobile body 2.

The configuration and detailed processing contents of such a learning model generation system 100 will be described below.

FIG. 2 is a block diagram showing a configuration example of the information processing device 1. The information processing device 1 includes a control unit 10, a storage unit 11, a communication unit 12, and an operation unit 13. The control unit 10 is a processor using one or a plurality of CPUs (Central Processing Units), GPUs (Graphics Processing Units), etc., and uses a built-in memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory). , Control each component to execute processing. The control unit 10 performs various information processing, control processing, and the like by reading and executing the program 1P stored in the storage unit 11.

The storage unit 11 includes, for example, a hard disk or a non-volatile memory such as an SSD (Solid State Drive). The storage unit 11 stores programs and data referred to by the control unit 10 including the program 1P. The program 1P stored in the storage unit 11 may be recorded on a recording medium so that it can be read by a computer. The storage unit 11 stores the program 1P read from the recording medium 1A by a reading device (not shown). Further, the program 1P may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 11. The storage unit 11 may be composed of a plurality of storage devices, or may be an external storage device connected to the information processing device 1.

A plurality of learning models 1M are further stored in the storage unit 11. The learning model 1M is a discriminator that identifies information about an object included in a captured image, and is a learning model generated by machine learning. The learning model 1M is defined by its definition information. The definition information of the learning model 1M includes, for example, structural information and layer information of the learning model 1M, channel information included in each layer, and learned parameters. Definition information about the learning model 1M is stored in the storage unit 11. The details of the learning model 1M will be described later.

The communication unit 12 is a communication interface that realizes communication via the network N1. The control unit 10 can communicate with the control unit 200 via the network N1 by the communication unit 12.

The operation unit 13 is an interface that accepts user operations, and includes a physical button, a mouse, and a touch panel device with a built-in display. The operation unit 13 receives an operation input from the user and sends a control signal according to the operation content to the control unit 10.

FIG. 3 is a block diagram showing a configuration example of the control unit 200. The control unit 200 is, for example, an ECU (Electronic Control Unit) for controlling the equipment of the mobile body 2, and includes a control unit 20, a storage unit 21, a first communication unit 22, a second communication unit 23, and the like.

The control unit 20 is a processor using one or a plurality of CPUs, GPUs, etc., and controls each component unit using a built-in memory such as a ROM and a RAM to execute processing. The control unit 20 can sequentially acquire time information by the built-in timer. The control unit 20 executes information processing based on the program stored in the storage unit 21.

The storage unit 21 includes a non-volatile memory such as an EEPROM (Electronically Erasable Programmable Read Only Memory). The storage unit 21 stores a program executed by the control unit 20, data necessary for executing the program, and the like. The storage unit 21 may store a log of the speed of the moving body 2 during movement by associating the time information obtained by the timer built in the control unit 20 with the time information.

The first communication unit 22 is a communication interface using a communication protocol such as CAN (Control Area Network) or Ethernet (registered trademark), and the control unit 20 connects to the mobile communication line N2 via the first communication unit 22. It communicates with various connected devices, other ECUs, etc. The device connected to the first communication unit 22 via the mobile communication line N2 includes an image pickup device 31.

The second communication unit 23 is a communication interface for wireless communication using mobile communication protocols such as 3G, LTE, 4G, 5G, and WiFi, and information is provided via an antenna connected to the second communication unit. Sends and receives data to and from the processing device 1. Communication between the second communication unit 23 and the information processing device 1 is performed via, for example, an external network N1 such as a public line network or the Internet.

In the learning model generation system 100 configured as described above, the information processing device 1 generates a unit group value X composed of a combination of a plurality of data including the captured image data based on the acquired captured image. The information processing device 1 generates the learning model 1M described later using the training data set including the generated unit group value X.

The learning model determines various information by extracting the feature amount of the captured image including the object. When an captured image capable of detecting an object with high accuracy is used, it is highly possible that a highly accurate estimation result can be obtained. On the other hand, when a captured image is used, for example, the object is unclear or a part of the object is missing, the learning model is based on the information about the object detected with low accuracy. The estimation accuracy may be low. The learning model generation system 100 improves the estimation accuracy of the learning model for the object by generating training data using the unit group value X composed of a combination of a plurality of data whose detection accuracy of the object is improved.

FIG. 4 is a conceptual diagram showing a method of generating the unit group value X. The unit group value X is acquired through the processing of data unit generation and unit group generation based on the captured image. The generation of the unit group value X (t) based on the captured image at time t will be specifically described with reference to FIG.

The information processing device 1 first acquires a grayscale captured image and an object extracted image in association with the captured image. The captured image is captured by the image pickup device 31 and acquired by the information processing device 1 via the control unit 200. The captured image is obtained as a moving image, and is composed of a still image of a plurality of frames acquired based on a predetermined frame rate such as 60 frames per second. The captured image may be a set of a plurality of still images acquired at predetermined intervals.

The information processing device 1 generates a grayscale captured image converted into a grayscale image from the captured image including RGB (Red Green Blue) values. The grayscale captured image may include pixel values from 0 to 255, or may include values normalized to continuous values from 0 to 1. The captured image itself acquired from the imaging device 31 may be a grayscale image.

Further, an object extraction image is generated based on the captured image of either the RGB image or the grayscale image. In the first embodiment, the object extraction image is image data obtained by extracting an object from a captured image, and is a binarized image. The objects extracted from the captured image are, for example, white lines on the road, guardrails, medians, traffic lights, road signs, vehicles, surrounding advertisements, people, and the like. In the first embodiment, the captured image is an image obtained by capturing a white line indicating the traveling path of the moving body 2, and the object extraction image is a binary white line extraction image obtained by extracting the white line as the object. do.

A known method may be used as a method for generating a white line extracted image from the captured image. For example, the information processing device 1 may generate an image in which an object is extracted by using an algorithm such as LaneNet or U-Net by a machine learning model. The LaneNet model is a learning model that performs image segmentation, and generates a binarized image in which a white line, which is an object, is extracted based on an captured image. The information processing device 1 performs the above processing for each frame of the captured image to generate a white line extracted image corresponding to each captured image.

The white line extracted image may be generated by using a technique such as pattern matching. In this case, the information processing apparatus 1 first detects the local feature amount from the captured image, and pattern-matches the detected local feature amount with the feature amount of the object held in advance to include the object. The area is specified and a white line extraction image is generated. Further, the information processing apparatus 1 may execute a process of binarizing the generated white line extracted image based on the brightness value of the captured image to generate the binarized image. A known method may be used as the method of binarizing the image. For example, for each pixel in the white line extracted image converted to grayscale, the brightness value (pixel value) of the pixel is compared with the threshold value calculated by a predetermined method. When the pixel value of a pixel is larger than the threshold value, the information processing device 1 determines the pixel as a black pixel and sets the pixel value to 1. On the other hand, when the pixel value is equal to or less than the threshold value, the pixel is determined to be a white pixel and the pixel value is set to 0. The information processing device 1 performs the above processing for each pixel to generate a binary white line extraction image.

Although the example of generating the object extracted image from the captured image has been described above, the object extracted image is not limited to the one generated from the captured image. When the sensor data obtained by extracting the object by a rider or the like is acquired, the information processing device 1 may convert the acquired sensor data into image data to generate the object extracted image.

Next, the information processing device 1 generates a data unit composed of a grayscale captured image and a plurality of object extraction images (white line extraction images). FIG. 5 is an explanatory diagram illustrating the contents of the data unit. The time t data unit at time t includes a grayscale captured image at time t and an object extracted image at n times before and after time t and time t, respectively. That is, the time t data unit contains a grayscale image of time t, time t, time t-1, time t + 1, time t-2, time t + 2, ..., Time t-n, time t. Each + n white line extracted image (2n + 1 in total) is included. n is a positive natural number. In the example of FIG. 5, n = 1, and the time t data unit includes a grayscale captured image at time t and a white line extracted image at time t, time t + 1, and time t-1. Similarly, the time t-1 data unit includes a grayscale captured image at time t-1 and a white line extracted image at time t-2, time t-1 and time t. The time t + 1 data unit includes a grayscale image taken at time t + 1 and a white line extracted image at time t, time t + 1 and time t + 2. In this embodiment, n is described as a positive natural number, but n may be 0. The combined white line extraction image is not limited to the one in which n white line extraction images of the same number before and after are combined around the time t. The number of the white line extracted images to be combined before and after the time t, that is, the number on the past side and the number on the future side may be different.

One data unit value is generated from each data unit at each time. One data unit value is stored and processed as one data obtained by converting the pixel values of each of the grayscale captured image and the combined white line extracted image into a one-dimensional or two-dimensional array format and concatenating them. In the first embodiment, the data unit value is one matrix data composed of a combination of two channel data of the image data of the gray scale captured image and the image data of the combined white line extracted image obtained by combining 2n + 1 white line extracted images. Is.

FIG. 6 is an explanatory diagram illustrating an example of a method of combining white line extracted images. The information processing device 1 combines the white line extracted image at time t and the white line extracted images at n times before and after time t to generate a combined white line extracted image. In the lower part of FIG. 6, as an example, an example of generating a combined white line extraction image in which three white line extraction images of time t, time t + 1 and time t-1 are combined in the case of n = 1 will be described. Each white line extracted image is, for example, image data having the same number of pixels, and has a pixel value of each pixel. The information processing device 1 combines the corresponding pixels of each white line extracted image to generate one image data. For example, when all the pixels of the same array (pixel number) in each white line extracted image are 0, the pixel value of the array in the combined white line extracted image is set to 0. When the pixels of the same array in each white line extracted image are 0, 0, 1, the pixel value of the array in the combined white line extracted image is set to 1. In this way, by combining the information of the plurality of data before and after, the data supplemented with the information not detected in one object extraction image is generated. The method of combining a plurality of images is an example and is not limited to the above example.

The explanation will be continued by returning to FIG. The information processing device 1 further generates one unit group composed of one or a plurality of combinations of the above-mentioned data units. The unit group at time t is composed of data units at time t and m times before and after time t. That is, the unit groups at time t include time t data unit, time t-1 data unit, time t + 1 data unit, ..., Time t-m data unit, and time t + m data unit (total 2 m + 1). Is included. m is 0 or a positive natural number. The unit group is not limited to a combination of m data units of the same number before and after the time t. The number of data units to be combined before and after the time t, that is, the number on the past side and the number on the future side may be different.

One unit group value X is generated from the unit group at the time t generated as described above. One unit group value X is a value obtained by combining 2m + 1 data unit values. For example, pixel values of 2m + 1 grayscale captured images and combined white line extracted images are set in a one-dimensional or two-dimensional array format. It is stored and processed as one data obtained by converting to and concatenating. In the first embodiment, the unit group value X is one matrix data composed of a combination of 4m + 2 channels of the image data of 2m + 1 grayscale captured images and the image data of 2m + 1 combined white line extracted images. ..

In the above description, the data unit value and the unit group value X are each a combination of the grayscale captured image data and the white line extracted image data, but they are included in the data unit value and the unit group value X. The data is not limited to the above example. The data unit value and the unit group value X may be a combination of the captured image data including the RGB values and the white line extracted image data, respectively.

Using the unit group value X described above, the information processing device 1 generates a learning model 1M. FIG. 7 is an explanatory diagram illustrating the configuration of the learning model 1M. The learning model 1M is generated and trained by deep learning using a neural network. The learning model 1M is, for example, a CNN (Convolution Neural Network). In the example shown in FIG. 7, the learning model 1M has an input layer for inputting a unit group value X based on a captured image, an output layer for outputting information about an object, and an intermediate layer (hidden) for extracting feature amounts of image data. Layer) and.

The control unit 10 generates a grayscale captured image and an object extracted image with respect to the captured image as described above, and executes preprocessing to generate a unit group value X based on a data unit and a unit group formed by combining these. do. The learning model 1M in the first embodiment includes a learning model 1M in a narrow sense and a learning model module in a broad sense including the above-mentioned preprocessing.

The input data input to the input layer of the learning model 1M is the unit group value X. In the first embodiment, the unit group value X is two-channel data of grayscale captured image data obtained by photographing a white line indicating a traveling path of a moving body 2 and white line extracted image data obtained by extracting a white line which is an object. include. The white line extracted image has a function as a mask image to be multiplied by the captured image.

The intermediate layer is composed of, for example, a convolution layer, a pooling layer and a fully connected layer. A plurality of convolution layers and pooling layers may be provided alternately. The convolution layer and the pooling layer extract the features of the captured image data and the white line extracted image data input through the input layer by the calculation using the channels of each layer. The fully connected layer combines the data whose feature portions are extracted by the convolution layer and the pooling layer into one node, and outputs the feature amount converted by the activation function. The feature quantity is output to the output layer through the fully connected layer.

The output data output from the output layer of the learning model 1M is information about the object. In the first embodiment, as information about the object, the location lane of the moving body 2 indicating the position on the traveling path where the moving body 2 is located is output. The output layer includes channels corresponding to the set location lanes, and outputs the accuracy for each location lane as a score. The information processing device 1 can use the location lane having the highest score or the location lane having a score equal to or higher than the threshold value as the output value of the output layer. The output layer may have one output channel that outputs the most accurate location lane instead of having a plurality of output channels that output the accuracy of each location lane. In this way, the learning model 1M outputs information about the object when the unit group value X is input.

As the learning model 1M described above, different learning models 1M are prepared according to the values of n and m described above. That is, different learning models 1M are generated according to the number of white line extracted images included in the combined white line extracted image and the number of data unit values to be combined to generate the unit group value X to be input to the learning model 1M. The configuration of each learning model 1M may have the same configuration or may have a configuration in which the number of layers of the intermediate layer is different. The information processing device 1 determines the values of n and m, for example, by accepting a user's input. The information processing device 1 separates captured images for each group specified by the determined combination of n and m, and generates a learning model 1M for each group. FIG. 8 is a diagram showing an example of a table in which group IDs corresponding to n and m are stored in a matrix. For example, group 1 is stored when n = 1 and m = 0, group 2, ... Is stored when n = 1 and m = 1. The information processing device 1 generates a learning model 1M using different training data for each group ID.

Although the learning model 1M has been described above as being a CNN, the learning model 1M is not limited to the CNN. When time series data is acquired, a neural network other than CNN, for example, a recurrent neural network (RNN: Recurrent Neural Network) or an LSTM (Long Short Term Memory) network may be used. Further, the model may be a model learned by another algorithm such as a support vector machine that does not use a neural network or a regression tree.

Needless to say, the content example of the learning model 1M is not limited to the example shown in FIG. The learning model 1M may be trained to output output information corresponding to the input information as appropriate according to the object included in the captured image. For example, the learning model 1M may output the classification of advertisements or the number of advertisements according to the advertisements that are the objects. The learning model 1M may output the number of people according to the person who is the object. The learning model 1M may output the number of vehicles according to the vehicle that is the object.

FIG. 9 is a flowchart showing an example of the generation processing procedure of the learning model 1M. The following processing is executed by the control unit 10 according to the program 1P stored in the storage unit 11 of the information processing device 1. The execution timing of the process may be, for example, a periodic timing, or may be a timing at which a new captured image is recorded by the imaging device 31 and transmitted from the control unit 200.

The control unit 10 of the information processing device 1 acquires an captured image from the control unit 200 (step S11), and temporarily stores the acquired image in the storage unit 11. The captured image is an image of the outside of the moving body 2 photographed and recorded by the imaging device 31, and includes a white line indicating a traveling path of the moving body 2.

The control unit 10 accesses the storage unit 11 and acquires an captured image. The control unit 10 converts the captured image into grayscale and acquires the grayscale captured image (step S12). The control unit 10 may acquire a grayscale captured image from the control unit 200. The control unit 10 extracts an object from the captured image by using a machine learning model such as LaneNet, and acquires a white line extracted image which is an object extracted image (step S13). The white line extracted image is a binarized image. The control unit 10 stores the grayscale captured image and the white line extracted image in association with each other. The control unit 10 executes the above processing for all frames included in the captured image.

When the control unit 10 acquires sensor data from a distance measuring sensor such as a rider, the control unit 10 may generate a white line extracted image from the sensor data instead of generating a white line extracted image based on the captured image. In this case, the captured image and the sensor data are each accompanied by information regarding the acquisition time point, and the control unit 10 may acquire the captured image in association with the sensor data at the same time point.

The control unit 10 acquires the unit group value X based on the acquired grayscale image and the white line extracted image (step S14). FIG. 10 is a flowchart showing an example of a detailed procedure for acquiring the unit group value X. The processing procedure shown in the flowchart of FIG. 10 corresponds to the details of step S14 in the flowchart of FIG.

The control unit 10 determines the values of n and m for generating the unit group value X, for example, by receiving the input of the user by the operation unit 13 (step S141). Based on the determined value of n, the control unit 10 has a gray scale image taken at time t and n times before and after time t and time t, that is, white lines at each time from time t-n to time t + n. A time t data unit composed of extracted images (2n + 1 in total) is generated (step S142). The control unit 10 acquires a data unit value that combines the grayscale captured image data in the generated time t data unit and the combined white line extracted image data based on the white line extracted image at each time (step S143).

The control unit 10 is composed of time t and m time before and after time t, that is, data units (total 2 m + 1) at each time from time tm to time t + m based on the determined value of m. Unit group is generated (step S144). The control unit 10 acquires a unit group value X that combines the data unit values at each time of the generated unit group (step S145), and returns the process to step S15 in the flowchart of FIG.

The explanation will be continued by returning to FIG. The control unit 10 generates a training data set in which the acquired unit group value X is labeled with information about the object (for example, the location lane of the moving body 2) (step S15). The control unit 10 performs the above-mentioned processing for each frame of the captured image, and generates a plurality of training data sets in which the location lanes are labeled with the unit group value X. The control unit 10 collects the unit group value X based on a large amount of captured images and the label data, and stores the collected data as a training data set in a database (not shown). In this case, the unit group value X is stored with the values of n and m attached.

The control unit 10 refers to the table shown in FIG. 8 based on the values of n and m associated with the unit group value X, acquires the group ID specified by the combination of the values of n and m, and acquires the unit group. The value X is sorted into groups (step S16).

Using the generated training data set, the control unit 10 generates a learning model 1M that outputs the location lane at time t when the unit group value X at time t is input (step S17). Specifically, the control unit 10 accesses the database and acquires a set of training data sets used for generating the learning model 1M. The training dataset includes the unit group value X and the location lane. The control unit 10 inputs the unit group value X to the input layer of the learning model 1M. In this case, since the control unit 10 generates different learning models 1M, 1M, 1M, etc. for each group ID according to the configuration of the unit group, the training data is stored in the learning model 1M corresponding to each group of the separated unit group value X. Enter.

The control unit 10 acquires the predicted value of the location lane from the output layer. At the stage before the start of learning, it is assumed that the definition information describing the learning model 1M is given an initial setting value. The control unit 10 compares the predicted value of the location lane with the location lane which is the correct answer value by using, for example, the back-propagation method, and learns the parameters and weights in the intermediate layer so that the difference becomes small. When the learning is completed when the magnitude of the difference and the number of learnings satisfy the predetermined criteria, the optimized parameters are obtained. The control unit 10 stores the learning models 1M, 1M, 1M ... Generated for each group in the storage unit 11, and ends a series of processes.

According to this embodiment, the learning model 1M is generated using the training data set including the unit group value X that combines a plurality of data generated from the captured image. The learning model 1M has higher accuracy than the case where one frame image is used as it is by using the input data including the object extraction image in which the extraction accuracy of the object to be the mask image is improved and the captured image. Estimation processing becomes possible.

As the learning model 1M, a plurality of learning models 1M are prepared according to the combination contents of data. Each learning model 1M can be trained by training data including a unit group value X based on individually fixed values of n and m, and it is possible to improve the detection accuracy.

(Second Embodiment)
In the second embodiment, the values of n and m are determined based on various information according to the captured image. Hereinafter, the differences between the second embodiment and the first embodiment will be described. Since the other configurations other than the configurations described later are the same as those of the learning model generation system 100 of the first embodiment, the common configurations are designated by the same reference numerals and detailed description thereof will be omitted. The information processing device 1 in the second embodiment receives additional information including the speed of the moving body and the frame rate of the captured image from the control unit 200, and the moving image captured by the imaging device 31.

The value of n in the second embodiment is determined based on the detection accuracy of the object in the object extracted image. The detection accuracy may be derived from, for example, the output accuracy obtained from LaneNet, the correlation value of pattern matching in the object extraction image, or the like. The detection accuracy is classified into a plurality of stages such as high / medium / low in descending order of accuracy, for example. The information processing device 1 stores in advance a table (not shown) in which each stage of detection accuracy and a value of n are associated with each other. When the detection accuracy of the object is low, the value of n is set to be large. That is, when the detection accuracy of the object is low, the detection accuracy of the learning model is improved by using more front and rear object extraction images. On the other hand, when the detection accuracy of the object is high, the value of n is set to be small. This is because when the detection accuracy of the object is high, the high detection accuracy can be obtained from the learning model even if the number of objects extracted is small. The method for deriving the detection accuracy is not limited to the above example. The detection accuracy may be estimated by using a machine learning model or the like that has learned the object extraction image and the detection accuracy of the object in the object extraction image.

The value of m in the second embodiment is determined based on the moving distance of one frame of the captured image. The moving distance of one frame is obtained by dividing the speed of the moving body 2 by the frame rate of the captured image, and is classified into a plurality of stages such as large / medium / small. The information processing device 1 stores a table (not shown) in which the moving distance and the value of m are associated with each other in advance. The larger the moving distance, the larger the value of m is set. Since the moving distance increases as the speed increases, the detection accuracy of the learning model 1M is improved by setting the value of m to be large, that is, to combine more data units. Since the moving distance increases as the frame rate decreases, the value of m is set to increase. If the moving distance exceeds a predetermined threshold value, the continuous frames may be images that are significantly different. Therefore, the value of m is small, for example, it may be set to 0.

FIG. 11 is a flowchart showing an example of the generation processing procedure of the learning model 1M in the second embodiment. The following processing is executed by the control unit 10 according to the program 1P stored in the storage unit 11 of the information processing device 1. The same step numbers are assigned to the processes common to those in FIG. 9 in the first embodiment, and detailed description thereof will be omitted.

The control unit 10 of the information processing device 1 acquires an captured image and additional information from the control unit 200 (step S21), and temporarily stores the acquired captured image and additional information in the storage unit 11. The captured image includes a white line indicating the traveling path of the moving body 2. The additional information includes, for example, historical data of the speed of the moving body 2, the frame rate of the captured image, and the like. Time information is associated with the additional information. The speed data may be acquired from the speedometer of the moving body 2 or may be acquired based on position information such as GPS (Global Positioning System) data of the moving body 2.

The control unit 10 acquires a grayscale captured image from the captured image (step S12), and acquires a binary object extracted image (white line extracted image) from which the white line that is the object is extracted (step S13). The control unit 10 acquires the unit group value X based on the acquired grayscale image and the white line extracted image (step S14). FIG. 12 is a flowchart showing an example of a detailed procedure for acquiring the unit group value X in the second embodiment. The processing procedure shown in the flowchart of FIG. 12 corresponds to the details of step S14 in the flowchart of FIG. The same step numbers are assigned to the processes common to those in FIG. 10 in the first embodiment, and detailed description thereof will be omitted.

The control unit 10 acquires the detection accuracy of the object corresponding to the white line extracted image at time t based on, for example, a score indicating the detection accuracy of LaneNet (step S241). Next, the control unit 10 acquires the moving distance at time t based on the speed of the moving body 2 acquired as additional information and the frame rate of the captured image (step S242). The control unit 10 refers to each of the tables (not shown) and determines the values of n and m for generating the time t data unit based on the acquired detection accuracy and the movement distance (step S141).

The control unit 10 generates a time t data unit composed of a grayscale captured image and a white line extracted image (total 2n + 1) at time t based on the specified value of n (step S142). The control unit 10 acquires a data unit value in which the combined white line extraction image data of the generated time t data unit and the grayscale captured image data are combined (step S143).

Based on the specified value of m, the control unit 10 generates a unit group composed of data units (total 2m + 1) at time t and m times before and after time t (step S144). The control unit 10 acquires the unit group value X in which the generated unit group data unit values are combined (step S145), and returns the process to step S15 in the flowchart of FIG.

The explanation will be continued by returning to FIG. The control unit 10 generates a training data set in which the acquired unit group value X is labeled with information about the object (step S15). The control unit 10 refers to the table shown in FIG. 8 based on the values of n and m associated with the unit group value X, acquires the group ID specified by the combination of the values of n and m, and acquires the unit group. The value X is sorted into groups (step S16).

Using the generated training data set, the control unit 10 generates a learning model 1M that outputs the location lane at time t when the unit group value X at time t is input (step S17). The control unit 10 stores the learning model 1M generated for each group in the storage unit 11, and ends a series of processes.

In each of the above embodiments, an example in which the unit group value X is classified into groups based on n and m has been described, but the group classification is not limited to those based on n and m. The unit group value X may be divided into groups based on either n or m, that is, for example, detection accuracy or movement speed.

According to the present embodiment, the combination content of the data constituting the unit group is determined according to the detection accuracy according to the captured image and the moving distance. By generating the learning model 1M according to the state of the object included in the captured image, more accurate estimation processing becomes possible.

(Third Embodiment)
In the third embodiment, information about the object estimated by using the learning model is provided. FIG. 13 is a block diagram showing a configuration example of the estimation system 110 according to the third embodiment. Hereinafter, the differences between the third embodiment and the first embodiment will be described. Since the other configurations other than the configurations described later are the same as those of the learning model generation system 100 of the first embodiment, the common configurations are designated by the same reference numerals and detailed description thereof will be omitted. The estimation system 110 includes a control unit 200 of the mobile body 2 and an image pickup device 31.

The control unit 200 of the third embodiment stores the program 2P and the program and data referred to by the control unit 20 including the plurality of learning models 1M in the storage unit 21. The program 2P stored in the storage unit 21 may be recorded on a recording medium so that it can be read by a computer. The storage unit 21 stores the program 2P read from the recording medium 2A by a reading device (not shown). Further, the program 2P may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 21. The control unit 20 functions as an information processing device that estimates information about the object based on the captured image by reading and executing the program 2P.

The display device 32 is further connected to the first communication unit 22 via the mobile communication line N2. The display device 32 is, for example, a liquid crystal display or the like, and displays information output from the control unit 200. Further, the display device 32 has an operation unit 33 such as a touch panel, receives the user's operation on the operation unit 33, and notifies the control unit 200 of the received operation content. The display device 32 may be shared with, for example, a car navigation device.

In the estimation system 110 configured as described above, the estimation process using the learning model 1M is executed. In the third embodiment, the learning model 1M is used to output the estimation result of the position on the traveling path according to the captured image including the white line indicating the traveling path of the moving body 2. The position on the traveling path where the moving body 2 is located is indicated by, for example, the location lane of the moving body 2. FIG. 14 is a conceptual diagram showing an estimation processing method. In FIG. 14, for example, a process of outputting an estimation result of the location lane between two points from the point A to the point B will be described.

The control unit 200 of the moving body 2 first acquires an captured image between two points recorded by the imaging device 31. The control unit 200 determines the values of n and m with respect to the learning model 1M by accepting the input of the user or the like. The control unit 200 refers to the table shown in FIG. 8 and identifies the group ID according to the combination of n and m at each determined time point. Each frame of the captured image is divided into groups according to the specified group ID.

Each frame of the separated captured image is input to the learning model 1M corresponding to the group ID after being preprocessed to generate a unit group value X from the frame based on the values of n and m corresponding to each group. NS. From each learning model 1M, the estimation result of the location lane corresponding to the frame at each time point is output. The estimation results at each time point are combined in a time series and output as a series of estimation result data via the display device 32 or the like.

FIG. 15 is a flowchart showing an example of an estimation processing procedure using the learning model 1M. The following processing is executed by the control unit 20 according to the program 2P stored in the storage unit 21 of the control unit 200. The execution timing of the process is, for example, the timing at which a new moving image is recorded by the image pickup apparatus 31.

The control unit 20 acquires an captured image captured and recorded by the imaging device 31 (step S31). The captured image is, for example, a moving image obtained by photographing the outside of the moving body 2 between two points from the point A to the point B, and includes a white line indicating a traveling path of the moving body 2. The captured image is accompanied by information about the time of shooting.

The control unit 20 converts the captured image into grayscale and acquires the grayscale captured image (step S32). The control unit 20 may acquire a captured image captured in grayscale from the imaging device 31. The control unit 20 extracts an object from the captured image by using a machine learning model such as LaneNet, and acquires the object extracted image (step S33). In the third embodiment, the object to be extracted is a white line, and the object extraction image is a binary white line extraction image. The control unit 20 stores the grayscale captured image and the white line extracted image in association with each other.

The control unit 20 acquires the unit group value X based on the acquired grayscale image and the white line extracted image (step S34). FIG. 16 is a flowchart showing an example of a detailed procedure for acquiring the unit group value X in the third embodiment. The processing procedure shown in the flowchart of FIG. 16 corresponds to the details of step S34 in the flowchart of FIG.

The control unit 20 determines the values of n and m for generating the unit group value X by receiving the input of the user by the operation unit 33 (step S341). n and m are values for determining the number of white line extracted images to be combined and the number of data unit values to be combined, respectively.

Based on the determined value of n, the control unit 20 is time t data composed of a grayscale image taken at time t and white line extracted images (total 2n + 1) at time t and n times before and after time t. Generate a unit (step S342). The control unit 10 acquires a data unit value that combines the grayscale captured image data in the generated time t data unit and the combined white line extracted image data based on the white line extracted image at each time (step S343).

The control unit 10 is composed of time t and m time before and after time t, that is, data units (total 2 m + 1) at each time from time tm to time t + m based on the determined value of m. Unit group is generated (step S344). The control unit 10 acquires a unit group value X that combines the data unit values at each time of the generated unit group (step S345), and returns the process to step S35 in the flowchart of FIG. The unit group value X is acquired with the values of n and m attached.

Returning to FIG. 15, the description will be continued. The control unit 20 refers to the table in which the values of n and m and the group ID described in FIG. 8 are associated with each other based on the values of n and m associated with the unit group value X, and by combining the values of n and m. Acquire the specified group ID. The control unit 20 classifies each frame of the captured image into groups based on the acquired group ID (step S35).

The control unit 20 selects the learning model 1M corresponding to the acquired group ID from the plurality of learning models 1M to be stored (step S36). The control unit 20 inputs the unit group value X obtained by performing the above-mentioned preprocessing on the captured image into the selected learning model 1M as input information (step S37). The control unit 20 acquires information about the object output from the learning model 1M (step S38). The output information is, for example, the location lane corresponding to each frame. The control unit 20 generates a series of estimation result data in which the acquired estimation results at each time point are combined in a time series (step S39). The control unit 20 stores the generated estimation result data in association with the captured image in the storage unit 11, outputs the estimation result data via the display device 32 or the like (step S40), and ends a series of processes.

In the above, an example of executing the estimation process by the learning model 1M after acquiring a series of moving images has been described, but the timing of executing the estimation process is not limited. The control unit 20 may execute the above-mentioned processing at the timing when the imaging device 31 starts photographing, and output the estimation result data based on the captured image acquired in real time. In this case, the estimation result data is not generated as a series of data but may be output at any time.

Further, the control unit 20 may output information according to the estimation result data. For example, when it is determined that the vehicle deviates from the traveling lane based on the transition of the estimation result of the location lane, the control unit 20 uses the display device 32, a speaker (not shown), or the like to display an image, an alarm, a sound, or the like. It may output support information due to vibration or the like. The control unit 20 may output a control signal to the equipment of the moving body 2.

FIG. 17 is a diagram showing an example of a screen displayed by the display device 32. FIG. 17 is a diagram showing an example of the estimation result screen 320 including the estimation result data. The estimation result screen 320 includes the captured image 321 at one time, the estimation result 322 and the estimation information 323 associated with the captured image 321.

The captured image 321 is a one-frame still image cut out from the recorded moving image. The estimation result 322 is information about the object output by the learning model 1M, such as the location lane, the number of vehicles, the number of advertisements, and the like. In the example of FIG. 17, the estimation result 322 is the location lane, and is shown using lane numbers such as 1, 2, and 3 in the order of proximity to the sidewalk. The estimation information 323 is information related to the estimation process based on the captured image. In the example of FIG. 17, the estimation information 323 includes text data indicating a file name of the captured image, a shooting time, values of n and m used for estimation, and numerical values of estimation accuracy, respectively. The estimated information 323 is not limited to text data, but may be illustration, voice, or the like. The estimated accuracy may be highlighted by changing, for example, the color, size, blinking / lighting, and display state of the numerical value according to the accuracy. The control unit 20 acquires the estimation result 322 and the estimation information 323 at the same point in association with the captured image 321. The control unit 20 generates screen data of the estimation result screen 320 including the acquired captured image 321 and the text data of the estimation result 322 and the estimation information 323, and outputs the screen data via the display device 32. The user can recognize the estimation result of the learning model 1M by the display device 32.

In the above, the learning model 1M has been described as being used for processing in the control unit 200. However, not limited to this, the learning model 1M is stored in another information processing device communicably connected to the control unit 200, and is stored in the other information processing device based on the captured image obtained from the control unit 200. It may be used in the process of outputting the estimation result.

Further, the learning model 1M may be used in an analysis device such as a server that is not directly connected to the control unit 200 by communication. The analysis device acquires an image captured by the control unit 200 via another information processing device communicably connected to the control unit 200, and uses the learning model 1M based on the acquired image to obtain an object. You may execute the analysis process of the information about. Further, the learning model 1M may perform re-learning based on new data acquired by the analysis process or the like. The analysis device further creates training data using the new correction information, and retrains the learning model 1M using the training data. By performing re-learning, the accuracy of estimation of the learning model 1M can be further improved.

According to this embodiment, the estimation process is executed using a learning model 1M that is different for each frame of the captured image. By using the learning model 1M corresponding to each frame, it is possible to acquire output information with high estimation accuracy.

(Fourth Embodiment)
In the fourth embodiment, the detection accuracy and the moving speed according to the captured image are acquired, and the values of n and m are determined. Hereinafter, the differences between the fourth embodiment and the third embodiment will be described. Since the other configurations other than the configurations described later are the same as those of the estimation system 110 of the third embodiment, the common configurations are designated by the same reference numerals and detailed description thereof will be omitted.

FIG. 18 is a flowchart showing an example of an estimation processing procedure using the learning model 1M in the fourth embodiment. The following processing is executed by the control unit 20 according to the program 2P stored in the storage unit 21 of the control unit 200. The same step numbers are assigned to the processes common to those in FIG. 15 of the third embodiment, and detailed description thereof will be omitted.

The control unit 20 acquires the captured image and additional information captured and recorded by the imaging device 31 (step S51). The captured image includes a white line indicating the traveling path of the moving body 2. The additional information includes, for example, historical data of the speed of the moving body 2, the frame rate of the captured image, and the like. Information about the time point is associated with the additional information.

The control unit 20 converts the captured image into grayscale and acquires the grayscale captured image (step S32). The control unit 20 extracts an object from the captured image by using a machine learning model such as LaneNet, and acquires the object extracted image (step S33). In the fourth embodiment, the object to be extracted is a white line, and the object extraction image is a binary white line extraction image. The control unit 20 stores the grayscale captured image and the white line extracted image in association with each other.

The control unit 20 acquires the unit group value X based on the acquired grayscale image and the white line extracted image (step S34). FIG. 19 is a flowchart showing an example of a detailed procedure for acquiring the unit group value X in the fourth embodiment. The processing procedure shown in the flowchart of FIG. 19 corresponds to the details of step S34 in the flowchart of FIG. The same step numbers are assigned to the processes common to those in FIG. 16 of the third embodiment, and detailed description thereof will be omitted.

The control unit 20 acquires the detection accuracy required for the learning model 1M by accepting the user's input by the operation unit 33 (step S441). The control unit 20 may acquire the detection accuracy determined by using another machine learning model or the like.

The control unit 20 acquires the moving distance at time t based on the speed of the moving body 2 acquired as additional information and the frame rate of the captured image (step S442). The control unit 20 refers to each of the tables (not shown) and determines the values of n and m for generating the unit group value X based on the detection accuracy and the moving distance (step S341).

Based on the determined value of n, the control unit 20 is time t data composed of a grayscale image taken at time t and white line extracted images (total 2n + 1) at time t and n times before and after time t. Generate a unit (step S342). The control unit 10 acquires a data unit value that combines the grayscale captured image data in the generated time t data unit and the combined white line extracted image data based on the white line extracted image at each time (step S343).

Based on the determined value of m, the control unit 10 generates a unit group composed of data units (total 2m + 1) at time t and m times before and after time t (step S344). The control unit 10 acquires a unit group value X that combines the data unit values at each time of the generated unit group (step S345), and returns the process to step S35 in the flowchart of FIG.

Returning to FIG. 18, the description will be continued. The control unit 20 refers to the table in which the values of n and m and the group ID described in FIG. 8 are associated with each other based on the values of n and m associated with the unit group value X, and by combining the values of n and m. Acquire the specified group ID. The control unit 20 classifies each frame of the captured image into groups based on the acquired group ID (step S35).

The control unit 20 selects the learning model 1M corresponding to the acquired group ID from the plurality of learning models 1M to be stored (step S36). The control unit 20 inputs the unit group value X obtained by performing the above-mentioned preprocessing on the captured image into the selected learning model 1M as input information (step S37). The control unit 20 acquires information about the object output from the learning model 1M (step S38). The control unit 20 generates a series of estimation result data in which the acquired estimation results at each time point are combined in a time series (step S39). The control unit 20 stores the generated estimation result data in association with the captured image in the storage unit 11, outputs the estimation result data via the display device 32 or the like (step S40), and ends a series of processes.

According to the present embodiment, the estimation process is executed using the learning model 1M that is different for each frame of the captured image according to the required detection accuracy and the moving distance. Since the learning model 1M to be used is selected according to the state of the captured image, it is possible to acquire output information with higher estimation accuracy.

It should be noted that each processing sequence described in each of the above-described embodiments is not limited, and changes in the procedure can be tolerated as long as the properties are not violated. For the above processing sequence, for example, the execution order of each processing step may be changed, or a plurality of processing steps may be executed at the same time. Each time a series of processing sequences are executed, the order of each processing step is changed. It may be different.

It should be noted that the embodiments disclosed as described above are exemplary in all respects and should not be considered restrictive. The technical features described in each example can be combined with each other and the scope of the invention is intended to include all modifications within the claims and scope equivalent to the claims. Will be done.

1 Information processing device 2 Mobile 200 Control unit 31 Imaging device 32 Display device 10, 20 Control unit 11,21 Storage unit 1P, 2P Program 1M Learning model

Claims (16)

Acquires an image captured including an object imaged by an image pickup device mounted on a moving body, and obtains an image.
A plurality of object extraction images from which the objects have been extracted are acquired in association with the captured images, and the images are acquired.
Based on the acquired captured image and training data including multiple object extracted images and information about the object, a learning model that outputs information about the object when the captured image and a plurality of object extracted images are input is generated. How to generate a training model. The method for generating a learning model according to claim 1, wherein a plurality of binarized images obtained by extracting the object from each of the captured image and the captured image adjacent to the captured image in time series are acquired. Acquire an captured image including a white line showing the traveling path of a moving body,
A plurality of white line extracted images from which the white lines have been extracted are acquired in association with the captured image, and the images are acquired.
Based on the training data including the acquired captured image and the plurality of white line extracted images and the position on the traveling path where the moving body is located, the captured image including the white line indicating the traveling path of the moving body and the plurality of extracted white lines. The method for generating a training model according to claim 1 or 2, wherein when a white line extracted image is input, the training model that outputs the position on the traveling path where the moving body is located is generated. The captured image at the first time and a plurality of object extraction images at a plurality of times before and after the first time and the first time are acquired.
The learning model is generated based on the acquired training data including a plurality of object extraction images at the first time and a plurality of times before and after the first time, an image captured at the first time, and information about the object. The method for generating a learning model according to any one of claims 1 to 3. The method for generating a learning model according to claim 4, wherein the number of the object extracted images is determined based on the detection accuracy of the object included in the object extracted image at the first time. The first time and the first time including a data unit composed of a captured image at the first time and a plurality of object extraction images at the first time and a plurality of times before and after the first time. Acquire captured images and multiple object extraction images in multiple data units at multiple times before and after,
When the captured images in a plurality of data units and the extracted images of a plurality of objects are input based on the training data including the acquired images captured in the plurality of data units and the extracted images of the plurality of objects and the information about the objects, the objects The method for generating a learning model according to claim 4 or 5, wherein the learning model for which information is output is generated. The method for generating a learning model according to claim 6, wherein the number of data units is determined based on the moving speed of the moving body and the frame rate of the captured image at the first time. Acquires an image captured including an object imaged by an image pickup device mounted on a moving body, and obtains an image.
A plurality of object extraction images from which the objects have been extracted are acquired in association with the captured images, and the images are acquired.
Based on training data including captured images and multiple object extracted images and information about the objects, a learning model trained to output information about the objects when the captured images and multiple object extracted images are input. , A program for causing a computer to perform a process of inputting an acquired captured image and a plurality of object extraction images and outputting information on the object. The program according to claim 8, wherein a computer is made to execute a process of acquiring a plurality of binarized images obtained by extracting the object from each of the captured image and the captured images adjacent to the captured image in time series. Acquire an captured image including a white line showing the traveling path of a moving body,
A plurality of white line extracted images from which the white lines have been extracted are acquired in association with the captured image, and the images are acquired.
Based on the training data including the captured image and the plurality of white line extraction images and the position on the traveling path where the moving body is located, the captured image including the white line indicating the traveling path of the moving body and the plurality of white line extractions from which the white lines are extracted are extracted. When an image is input, a plurality of captured images including a white line indicating the acquired traveling path of the moving body and a plurality of extracted white lines are extracted from the learning model trained to output the position on the traveling path where the moving body is located. The program according to claim 8 or 9, for causing a computer to execute a process of inputting a white line extracted image and outputting a position on a traveling path where the moving body is located. The learning model is learned based on training data including a plurality of object extraction images at a first time and a plurality of times before and after the first time, an image captured at the first time, and information about the object. The program according to any one of claims 8 to 10. Obtain the detection accuracy of the object included in the object extraction image at the first time,
The program according to claim 11, wherein a computer is made to execute a process of selecting a learning model corresponding to the acquired detection accuracy from a plurality of types of the learning models prepared according to the detection accuracy. The first time and the first time including a data unit composed of a captured image at the first time and a plurality of object extraction images at the first time and a plurality of times before and after the first time. Acquire captured images and multiple object extraction images in multiple data units at multiple times before and after,
Information about an object when an image captured by a plurality of data units and an image extracted from a plurality of objects are input based on training data including an image captured by a plurality of data units and an image extracted from a plurality of objects and information about the object. A request for causing a computer to perform a process of inputting an image captured in a plurality of acquired data units and a plurality of object extraction images into the training model trained to output the data and outputting information on the object. The program according to any one of claim 11 and claim 12. The moving speed of the moving body and the frame rate of the captured image at the first time are acquired.
The program according to claim 13, wherein a computer is made to execute a process of selecting a learning model corresponding to the acquired movement speed and frame rate from a plurality of types of the learning models prepared according to the movement speed and the frame rate. A first acquisition unit that acquires an captured image including an object imaged by an imaging device mounted on a moving body, and a first acquisition unit.
A second acquisition unit that acquires a plurality of object extraction images from which the objects have been extracted in association with the captured images, and a second acquisition unit.
The captured image and the plurality of object extraction images are input based on the training data including the captured image acquired by the first acquisition unit, the plurality of object extraction images acquired by the second acquisition unit, and information about the object. An information processing device equipped with a generator that generates a learning model that outputs information about an object when the information is processed. A first acquisition unit that acquires an captured image including an object imaged by an imaging device mounted on a moving body, and a first acquisition unit.
A second acquisition unit that acquires a plurality of object extraction images from which the objects have been extracted in association with the captured images, and a second acquisition unit.
A learning model learned to output information about an object when a captured image and a plurality of object extracted images are input based on training data including a captured image and a plurality of object extracted images and information about the object. ,
Information processing including an image captured by the first acquisition unit and an output unit that inputs a plurality of object extraction images acquired by the second acquisition unit and outputs information about the object to the learning model. Device.

Images