JP7070157B2 - Image processing program, image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing program, an image processing apparatus, and an image processing method.

Conventionally, there has been known a background subtraction technique for extracting a pixel region from a captured image with a predetermined subject as an extraction target. The background subtraction technique is a technique for extracting a pixel region to be extracted by taking a difference between an input image which is a captured image including an extraction target and a background image which is a captured image not including an extraction target. When the background subtraction technique is applied, the pixel region to be extracted can be extracted, while the pixel region other than the extraction target associated with the extraction target (for example, the shadow of the subject) is also extracted.

On the other hand, for example, in the following patent documents, the shadow of a subject is obtained by applying the background subtraction technique focusing on the difference between the subject to be extracted and the shadow of the subject and the texture component (alternating current component). An extraction method for extracting the excluded pixel area has been proposed.

Japanese Patent Application Laid-Open No. 2003-141546

However, in the case of the above extraction method, for example, a region having a small texture component among the subjects to be extracted is excluded in the same manner as the shadow of the subject, and the pixel region of the subject to be extracted is appropriately (excess or deficient). There is a problem that it cannot be extracted.

One aspect is to improve the extraction accuracy of extracting the pixel region to be extracted from the image.

According to one aspect, the image processing program
The first image including the extraction target and the second image not including the extraction target are acquired, and the image is obtained.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From, the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image are acquired.
The pixel region to be extracted is extracted based on the difference between the first intermediate image and the second intermediate image.
Let the computer perform the process.

It is possible to improve the extraction accuracy of extracting the pixel area to be extracted from the image.

It is a figure which shows an example of the system configuration of an image processing system and the functional configuration of an image processing apparatus. It is a figure which shows an example of the hardware composition of an image processing apparatus. It is a figure which shows the specific example of the image information for learning. It is a figure which shows the specific example of the 1st image acquired by the 1st image acquisition part. It is a figure which shows the specific example of the 2nd image acquired by the 2nd image acquisition part. It is the first figure which shows the specific example of the processing by an object identification part. It is a figure which shows the relationship between the identification target and the non-discrimination target, and the extraction target and the non-extraction target. It is the 2nd figure which shows the specific example of the processing by an object identification part. FIG. 3 is a third diagram showing a specific example of processing by the object identification unit. It is a figure which shows the specific example of the processing by a region extraction part. It is a flowchart which shows the flow of image processing by an image processing unit. It is the first figure which shows the application example of an image processing system. It is the 2nd figure which shows the application example of an image processing system.

Hereinafter, each embodiment will be described with reference to the attached drawings. In the present specification and the drawings, the components having substantially the same functional configuration are designated by the same reference numerals, and duplicate description thereof will be omitted.

[First Embodiment]
<System configuration of image processing system and functional configuration of image processing device>
First, the system configuration of the image processing system and the functional configuration of the image processing device will be described. FIG. 1 is a diagram showing an example of a system configuration of an image processing system and a functional configuration of an image processing device.

As shown in FIG. 1, the image processing system 100 includes an image pickup device 110, an image processing device 120, and a learning image information storage unit 130. The image pickup apparatus 110 and the image processing apparatus 120 are communicably connected to each other. Similarly, the image processing device 120 and the learning image information storage unit 130 are communicably connected to each other.

The image pickup device 110 is installed at a predetermined position, shoots in a fixed shooting direction, and transmits the shot image (shooting image) to the image processing device 120. The captured image transmitted by the image pickup apparatus 110 includes
A "first image" (so-called input image) in which the image processing device 120 determines whether or not an extraction target is included, and
-A "second image" (so-called background image) that does not include the extraction target and was shot at different timings under the same shooting conditions as when the first image was shot.
Is included.

The extraction target refers to a subject (generally an animal body) that is a subject designated by the administrator of the image processing system 100 and changes during shooting among the subjects photographed by the image pickup apparatus 110. It shall be.

An image processing program is installed in the image processing device 120, and when the program is executed, the image processing device 120 has a first image acquisition unit 121 and a second image acquisition unit 122, which are examples of acquisition units. Functions as. Further, the image processing device 120 functions as an object identification unit 123 and an intermediate image acquisition unit 124, which are examples of the identification unit. Further, the image processing device 120 functions as a region extraction unit 125, which is an example of the extraction unit.

The first image acquisition unit 121 acquires the first image transmitted from the image pickup apparatus 110. The first image acquisition unit 121 notifies the object identification unit 123 and the area extraction unit 125 of the acquired first image.

The second image acquisition unit 122 acquires the second image transmitted from the image pickup apparatus 110. The second image acquisition unit 122 notifies the object identification unit 123 of the acquired second image.

The object identification unit 123 has a function of reading the learning image information stored in the learning image information storage unit 130 and learning the neural network, inputting the first image and the second image, and executing the learned neural network. It has a function to make it.

The intermediate image acquisition unit 124 acquires an intermediate image output from a predetermined layer by inputting the first image and the second image and causing the object identification unit 123 to execute the trained neural network.

When the object identification unit 123 executes the trained neural network by inputting the first image, the intermediate image acquisition unit 124 acquires the first intermediate image output from a predetermined layer.

Further, when the object identification unit 123 executes the trained neural network by inputting the second image, the intermediate image acquisition unit 124 acquires the second intermediate image output from a predetermined layer.

The intermediate image acquisition unit 124 notifies the region extraction unit 125 of the acquired first intermediate image and the second intermediate image.

The area extraction unit 125 extracts the pixel region to be extracted in the first image notified from the first image acquisition unit 121 by using the first intermediate image and the second intermediate image notified from the intermediate image acquisition unit 124. do. In the first embodiment, "extracting the pixel area to be extracted" means that the pixel area to be extracted is cut out from the first image and the coordinates of each pixel of the pixel area to be extracted are defined. It shall also include specifying in the first image.

The learning image information storage unit 130 stores learning image information used by the object identification unit 123 when learning a neural network.

In the above description, it is assumed that the object identification unit 123 has both a function of learning the neural network and a function of executing the trained neural network. However, it is sufficient that the object identification unit 123 has at least a function of executing a trained neural network, and another device may have a function of training the neural network. In this case, the image processing device 120 does not need to be communicably connected to the learning image information storage unit 130, and a trained neural network or neural network trained by another device using the learning image information. It suffices to have a function to acquire the weight of the network.

<Hardware configuration of image processing device>
Next, the hardware configuration of the image processing apparatus 120 will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the image processing device. As shown in FIG. 2, the image processing device 120 includes a CPU (Central Processing Unit) 201 and a ROM (Read Only).
It has a Memory) 202 and a RAM (Random Access Memory) 203. The CPU 201, ROM 202, and RAM 203 form a so-called computer.

Further, the image processing device 120 includes an auxiliary storage device 204, a display device 205, an operation device 206, an I / F (Interface) device 207, and a drive device 208. The hardware of the image processing device 120 is connected to each other via the bus 209.

The CPU 201 is an arithmetic device that executes various programs (for example, an image processing program, etc.) installed in the auxiliary storage device 204.

ROM 202 is a non-volatile memory. The ROM 202 functions as a main storage device for storing various programs, data, and the like necessary for the CPU 201 to execute various programs installed in the auxiliary storage device 204. Specifically, ROM 202 has BIOS (Basic Input / Output System) and EFI (Extensible).
It functions as a main memory device that stores boot programs such as Firmware Interface).

RAM 203 includes DRAM (Dynamic Random Access Memory) and SRAM (Static).
Random Access Memory) and other volatile memories. The RAM 203 functions as a main storage device that provides a work area that is expanded when various programs installed in the auxiliary storage device 204 are executed by the CPU 201.

The auxiliary storage device 204 is an auxiliary storage device that stores various programs and information used when various programs are executed.

The display device 205 is a display device that displays the internal state of the image processing device 120. The operation device 206 is an input device for the administrator of the image processing device 120 to input various instructions to the image processing device 120.

The I / F device 207 is connected to the image pickup device 110, and transmits and receives captured images between the image pickup device 110 and the image processing device 120. Further, the I / F device 207 is connected to the learning image information storage unit 130, and transmits / receives learning image information between the image processing device 120 and the learning image information storage unit 130.

The drive device 208 is a device for setting the recording medium 210. The recording medium 210 referred to here includes a medium such as a CD-ROM, a flexible disk, a magneto-optical disk, or the like, which records information optically, electrically, or magnetically. Further, the recording medium 210 may include a semiconductor memory or the like for electrically recording information such as a ROM or a flash memory.

The various programs installed in the auxiliary storage device 204 are installed, for example, by setting the distributed recording medium 210 in the drive device 208 and reading the various programs recorded in the recording medium 210 by the drive device 208. Will be done. Alternatively, various programs installed in the auxiliary storage device 204 may be installed by being downloaded from the network.

<Specific example of image information for learning>
Next, a specific example of the learning image information stored in the learning image information storage unit 130 will be described. FIG. 3 is a diagram showing a specific example of learning image information. As shown in FIG. 3, the learning image information 300 includes "image data ID", "file name", "identification target", and "storage destination" as information items.

An identifier for identifying image data is stored in the "image data ID". The example of FIG. 1 shows that N image data are stored in the learning image information storage unit 130.

The file name of the image data is stored in the "file name". The "identification target" is a general object (hereinafter referred to as "general object") included in each image data, which is a correct answer when the object identification unit 123 learns the neural network. The information to be shown is stored. For example, the object identification unit 123 trains the neural network so that it can be identified as "object A" when the image data of the image data ID = "ID001" is input to the neural network. In the first embodiment, in order to simplify the explanation, the object identification unit 123 identifies four types of general objects (“object A”, “object B”, “object C”, and “object D”). As such, it is assumed that the neural network is trained.

In the "storage destination", a folder name indicating a storage destination in which the image data is stored in the learning image information storage unit 130 is stored.

In the case of the learning image information 300 shown in FIG. 3, each image data is shown to include one of the identification targets, but each image data of the learning image information 300 includes an identification target. It may contain no image data. Further, each image data of the learning image information 300 may include image data including a non-identification target (general object other than the identification target).

Further, each image data of the learning image information 300 shown in FIG. 3 needs to include image data including an extraction target, but may include image data not including an extraction target. Further, each image data of the learning image information 300 may include image data including a non-extraction target (a subject other than the extraction target among the subjects photographed by the image pickup apparatus 110).

<Specific examples of the first image and the second image>
Next, specific examples of the first image acquired by the first image acquisition unit 121 and the second image acquired by the second image acquisition unit 122 will be described.

(1) Specific Example of First Image FIG. 4 is a diagram showing a specific example of the first image acquired by the first image acquisition unit. As shown in FIG. 4, the first image 400, which is a specific example of the input image, includes a plurality of general objects. The example of FIG. 4 shows that the first image 400 includes "object A", "object F", "object G", and "object H" as general objects.

As described above, among the general objects included in the first image 400, the "object A" is a general object to be identified when the object identification unit 123 executes the trained neural network. Further, in the first embodiment, the "object A" is also a general object to be extracted by the region extraction unit 125. On the other hand, among the general objects included in the first image 400, the "object F", "object G", and "object H" are general objects that are not identified when the object identification unit 123 executes the trained neural network. It is an object, and is also a non-extractable general object that is not extracted by the area extraction unit 125.

(2) Specific Example of Second Image FIG. 5 is a diagram showing a specific example of the second image acquired by the second image acquisition unit. As shown in FIG. 5, the second image 500, which is a specific example of the background image, includes a plurality of general objects. The example of FIG. 5 shows that the second image 500 includes "object F" and "object I" as general objects.

The general objects (“object F” and “object I”) included in the second image 500 are all general objects that are not identified when the object identification unit 123 executes the trained neural network, and are regions. It is also a non-extractable general object that is not extracted by the extraction unit 125. As described above, the second image 500 does not include a general object to be extracted.

When the first image 400 is a general object to be identified and includes a general object to be non-extracted (“target B”, “target C”, “target D”), the second image The 500 also needs to include the general object (“object B”, “object C”, “object D”).

<Specific example of processing by the object identification unit>
Next, as a specific example of the process by the object identification unit 123, a process of reading image information for learning and learning a neural network to identify a general object, and a process of inputting a first image and a second image and learning have been completed. The process of executing the neural network will be described.

(1) Specific Example of Processing for Learning Neural Network First, a specific example of processing for learning an neural network in order to read image information for learning and identify a general object will be described. FIG. 6 is a first diagram showing a specific example of processing by the object identification unit.

As shown in FIG. 6A, the object identification unit 123 has a convolutional neural network (CNN) having each layer from the first layer to the third layer, and each image of the learning image information 300. Data is entered.

For reference, FIG. 6B shows a diagram showing the processing of a general convolutional neural network. In the first embodiment, four types of identification targets are illustrated for the sake of simplification of the description, but in reality, as shown in FIG. 6B, about 1000 types of identification targets are handled. become.

Returning to the description of FIG. 6 (a). As described above, each image data of the learning image information 300 is known image data including any of the identification targets. The example of FIG. 6A shows how N pieces of image data of image data ID = "ID001" to "ID00N" are input as each image data of the image information 300 for learning.

Further, the object identification unit 123 learns the filter coefficient of the convolutional neural network so that the identification target included in the input N image data can be appropriately identified. The example of FIG. 6A shows how to learn the filter coefficient so that four types of general objects of “object A”, “object B”, “object C”, and “object D” can be identified. There is.

In the convolutional neural network, the local features extracted by the convolution are converted into higher-order features each time the layers are passed, and finally the identification result is obtained. Therefore, an image (feature map) including a feature amount that contributes to the identification of the identification target is output from each layer of the learned convolutional neural network.

Specifically, each feature map output from each layer of the convolutional neural network has a feature amount that contributes to the identification of the identification target (“target A”, “target B”, “target C”, “target D”). It is included. On the other hand, from each feature map output from each layer of the convolutional neural network, the features that contribute to the identification of the non-discrimination target (“target F”, “target G”, “target H”, “target I”) are It has been excluded. In the first embodiment, a set of feature maps output from each layer of the convolutional neural network is referred to as an intermediate image.

In this way, from each layer of the convolutional neural network, an intermediate image which is an image including the feature amount necessary for identifying the identification target is output. Therefore, when the area extraction unit 125 uses the intermediate image when extracting the pixel area to be extracted from the first image, the convolutional neural network trained by the object identification unit 123 is
-The extraction target is included in the identification target (the extraction target is not a non-identification target).
-The identification target does not include the non-extraction target that changes during shooting, such as the non-extraction target that accompanies the extraction target.
Is a condition.

The non-extraction target attached to the extraction target refers to a subject that changes with the change of the subject during shooting, such as a shadow of the subject to be extracted.

FIG. 7 is a diagram showing the relationship between the identification target and the non-discrimination target and the extraction target and the non-extraction target. Of these, FIG. 7A shows a set of identification targets of the convolutional neural network. In the case of the convolutional neural network whose identification target is the general object shown in FIG. 7A, the intermediate image contains a feature amount that contributes to the identification of "person", "automobile", "dog" and the like.

Further, in the case of the convolutional neural network whose identification target is the general object shown in FIG. 7A, the intermediate image is used to identify other than "people", "automobiles", "dogs", etc. (non-identification targets). Contributed features are not included. For example, the intermediate image does not contain features that contribute to identifying "shadows".

Therefore, as shown in FIG. 7B, when the first image includes the “person” that is the extraction target and the “shadow” that is the non-extraction target associated with the extraction target, the intermediate image is an intermediate image. Remains a feature quantity that contributes to distinguishing a "person". On the other hand, in the intermediate image, the feature amount that contributes to distinguishing the "shadow" is excluded.

(2) Specific example of processing for executing a neural network 1
Next, a specific example 1 of the process of executing the trained neural network will be described. FIG. 8 is a second diagram showing a specific example of processing by the object identification unit. In FIG. 8, the object identification unit 123 has a trained convolutional neural network, and the first image acquired by the first image acquisition unit 121 is input. The example of FIG. 8 shows how the first image 400 including the “object A”, the “object F”, the “object G”, and the “object H” is input.

Here, the convolutional neural network included in the object identification unit 123 is learned so that "object A", "object B", "object C", and "object D" are appropriately identified as identification targets. Therefore, when the first image 400 is input, each feature map output in the first layer or the second layer has "target A", "target B", "target C", and "target D". The feature amount of any one of the identification targets is included. In other words, in each feature map output in the first layer or the second layer, the feature amount of the non-identification target that does not correspond to any of "target A", "target B", "target C", and "target D". Is excluded.

The first image 400 includes "target A" among "target A", "target B", "target C", and "target D", which are identification targets of the trained convolutional neural network. Therefore, each feature map output in the first layer or the second layer includes the feature amount of "target A", and is a non-identification target other than "target A", "target F" and "target G". ", The feature amount of" target H "is excluded.

The example of FIG. 8 shows how the object identification unit 123 outputs the first intermediate image 800, which is a set of feature maps, from the first layer or the second layer by executing the trained convolutional neural network. ing. The first intermediate image 800 output from the first layer or the second layer is acquired by the intermediate image acquisition unit 124 and notified to the region extraction unit 125.

(3) Specific example 2 of processing for executing a neural network
Next, a specific example 2 of the process of executing the trained neural network will be described. FIG. 9 is a third diagram showing a specific example of processing by the object identification unit. The difference from FIG. 8 is that, in the case of FIG. 9, the object identification unit 123 inputs the second image acquired by the second image acquisition unit 122. The example of FIG. 9 shows how the object identification unit 123 inputs the second image 500 including the “object F” and the “object I”.

As described above, the convolutional neural network included in the object identification unit 123 is learned so that "object A", "object B", "object C", and "object D" are appropriately identified as identification targets. Therefore, when the second image 500 is input, each feature map output in the first layer or the second layer has "target A", "target B", "target C", and "target D". The feature amount of any one of the identification targets is included. In other words, in each feature map output in the first layer or the second layer, the feature amount of the non-identification target that does not correspond to any of "target A", "target B", "target C", and "target D". Is excluded.

The second image 500 does not include any of the “target A”, “target B”, “target C”, and “target D” that are the identification targets of the trained convolutional neural network. Therefore, the feature amount of any one of "target A", "target B", "target C", and "target D" is not included in each feature map output in the first layer or the second layer. .. Further, the feature quantities of "object F" and "object I" which are non-identification targets are excluded.

The example of FIG. 9 shows how the object identification unit 123 outputs the second intermediate image 900, which is a set of feature maps, from the first layer or the second layer by executing the trained convolutional neural network. ing. The second intermediate image 900 output from the first layer or the second layer is acquired by the intermediate image acquisition unit 124 and notified to the region extraction unit 125. As described above, when the intermediate image is taken out from the i-th layer, the object identification unit 123 does not execute the processing after the i + 1 layer.

<Specific example of processing by the area extraction unit>
Next, a specific example of the processing by the region extraction unit 125 will be described. FIG. 10 is a diagram showing a specific example of processing by the region extraction unit.

As shown in FIG. 10A, the region extraction unit 125 generates the difference image 1010 using the first intermediate image 800 and the second intermediate image 900 notified by the intermediate image acquisition unit 124. The difference between the first intermediate image 800 and the second intermediate image 900 is that the first intermediate image 800 includes a feature amount to be identified.

In the case of the example of FIG. 10A, since the first intermediate image 800 contains the feature amount of the "object A" to be identified, the difference between the first intermediate image 800 and the second intermediate image 900. The difference image 1010 obtained by calculating the absolute value includes a pixel region of “target A”.

Although not explicitly shown in the example of FIG. 10A, among the feature quantities of the identification target included in the first intermediate image 800 and the second intermediate image 900, the features of the non-extracted target that do not change during shooting. The quantity is excluded by calculating the absolute value of the difference.

Assuming that the pixel value of each pixel of the first intermediate image 800 is F ₁ (c, x, y) and the pixel value of each pixel of the second intermediate image 900 is F ₂ (c, x, y), the difference image 1010 is , | F ₁ (c, x, y) -F ₂ (c, x, y) |. c indicates one of a plurality of feature maps, and x and y indicate the coordinates of each pixel.

If the first intermediate image 800 and the second intermediate image 900 include, for example, n feature maps, n difference images are generated. Therefore, the region extraction unit 125 generates the difference image 1010 by adding the difference images thereof. That is, the pixel value of each pixel of the difference image 1010 is

と表すことができる。

It can be expressed as.

The area extraction unit 125 sets the pixel value of each pixel in the pixel region of "target A" to "1" in the difference image 1010, and sets the pixel value of each pixel other than the pixel region of "target A" in the difference image 1010. The value is set to "0" (that is, the coordinates of each pixel in the pixel area to be extracted are specified).

Further, as shown in FIG. 10B, the region extraction unit 125 generates the extracted image 1020 by multiplying the first image 400 notified by the first image acquisition unit 121 by the difference image 1010. As described above, in the difference image 1010, the pixel value of each pixel in the pixel area of "target A" is "1", and the pixel value of each pixel other than the pixel area of "target A" is "0". be. Therefore, by multiplying the first image 400 by the difference image 1010, the region extraction unit 125 can generate the extraction image 1020 including the pixel region of the “target A” of the first image 400 (that is,). The pixel area to be extracted can be cut out).

<Flow of image processing>
Next, the flow of image processing (processing for executing a neural network) by the image processing device 120 will be described. FIG. 11 is a flowchart showing the flow of image processing by the image processing apparatus.

In step S1101, the first image acquisition unit 121 acquires the first image transmitted from the image pickup apparatus 110.

In step S1102, the first image acquisition unit 121 inputs the acquired first image to the object identification unit 123.

In step S1103, the object identification unit 123 outputs the first intermediate image from the first layer or the second layer by executing the trained convolutional neural network, and the intermediate image acquisition unit 124 outputs the output first. Get an intermediate image.

In step S1104, the second image acquisition unit 122 acquires the second image transmitted from the image pickup apparatus 110.

In step S1105, the second image acquisition unit 122 inputs the acquired second image to the object identification unit 123.

In step S1106, the object identification unit 123 outputs the second intermediate image from the first layer or the second layer by executing the learned convolutional neural network, and the intermediate image acquisition unit 124 outputs the output second. Get an intermediate image.

In step S1107, the area extraction unit 125 generates a difference image by calculating the difference between the first intermediate image and the second intermediate image.

In step S1108, the area extraction unit 125 generates an extracted image by multiplying the first image by the difference image.

<Differences from general image processing>
Next, the difference between the image processing (FIG. 11) executed by the image processing apparatus 120 and general image processing will be described.

(1) Differences from general image processing No. 1 (image processing that applies background subtraction technology to image texture components) In the case of image processing that applies background subtraction technology to image texture components, the subject to be extracted Areas with a small texture component are excluded. As a result, only the contour line to be extracted is extracted, and the pixel area inside the contour line cannot be extracted.

On the other hand, according to the image processing by the image processing apparatus 120, the pixel region can be extracted even in the region where the texture component is small among the subjects to be extracted. In addition, the pixel region of the shadow of the subject accompanying the extraction target is not extracted.

(2) Difference from general image processing 2 (discrimination process using neural network) In the case of discrimination process using neural network, the first image is input and the trained neural network is executed. It is possible to determine whether or not the first image includes an extraction target. However, in the case of the identification process using the neural network, it is not possible to extract the pixel region of the subject to be extracted from the first image.

On the other hand, according to the image processing by the image processing apparatus 120, the pixel region of the subject to be extracted can be extracted from the first image without excess or deficiency.

<Application example of image processing system>
Next, an application example of the image processing system 100 will be described.

(1) Application example 1
FIG. 12 is a first diagram showing an application example of an image processing system. As shown in FIG. 12, by connecting the image processing system 100 to the free viewpoint image generation device 1210, the free viewpoint image generation system 1200 can be formed.

In forming the free viewpoint image generation system 1200, a plurality of image pickup devices (in addition to the image pickup device 110, image pickup devices 1220_1 to 1220_m) are connected to the image processing device 120, and the subject is one of the identification targets. It is assumed that the 1230 is photographed from different directions. Further, the same image processing is applied to the image data output from each of the plurality of image pickup devices, and the area extraction unit 125 has a pixel region of the subject 1230 (a number of pixel regions corresponding to the number of image pickup devices). Therefore, it is assumed that the pixel area (pixel area that does not include shadows) is extracted.

As shown in FIG. 12, the free viewpoint image generation device 1210 has a Visual Hull unit 1211, a rendering unit 1212, and an output unit 1213.

The Visual Hull unit 1211 restores the three-dimensional structure of the subject 1230 by using the pixel regions of the subject 1230 (the number of pixel regions corresponding to the number of image pickup devices) extracted by the region extraction unit 125.

The rendering unit 1212 renders an image from an arbitrary viewpoint by using the three-dimensional structure restored in the Visual Hull unit 1211.

The output unit 1213 outputs the image from the designated viewpoint among the images from an arbitrary viewpoint rendered by the rendering unit 1212.

In this way, by connecting the image processing system 100 to the free viewpoint image generation device 1210 to form the free viewpoint image generation system 1200, it is possible to appropriately (just enough) output the image of the arbitrary viewpoint of the subject 1230. Is possible.

(2) Application example 2
FIG. 13 is a second diagram showing an application example of the image processing system. As shown in FIG. 13, by connecting the image processing system 100 to the video monitoring device 1310, the video monitoring system 1300 can be formed.

Further, as shown in FIG. 13, the video monitoring device 1310 has a determination unit 1311. The determination unit 1311 counts the number of pixels included in the pixel area to be extracted extracted by the area extraction unit 125, and if the number of pixels is equal to or greater than a predetermined threshold value, it is determined to be a suspicious person or a suspicious object. And output a message.

In this way, by connecting the image processing system 100 to the video monitoring device 1310 to form the video monitoring system 1300, it is possible to appropriately determine the pixel area of a suspicious person or a suspicious object (without excess or deficiency). Become.

As is clear from the above description, the image processing apparatus 120 according to the first embodiment acquires a first image including an extraction target and a second image not including an extraction target, and a plurality of images including the extraction target. Input to a neural network trained to identify the identification target.

Further, the image processing apparatus 120 according to the first embodiment acquires a first intermediate image corresponding to the first image and a second intermediate image corresponding to the second image from a predetermined layer of the neural network. Further, the image processing apparatus 120 according to the first embodiment multiplies the difference image obtained from the difference between the first intermediate image and the second intermediate image with the first image, so that the image processing device 120 can be extracted from the first image. Extract the pixel area.

As a result, according to the image processing apparatus 120 according to the first embodiment, it is possible to appropriately (excess or deficiency) extract the pixel region to be extracted from the image. That is, it is possible to improve the extraction accuracy of extracting the pixel region to be extracted.

[Second Embodiment]
In the first embodiment, the timing of acquiring the first image and the second image is not particularly mentioned, but for example, it is desirable that the second image is acquired immediately before or immediately after the acquisition of the first image. This is because the extraction accuracy of extracting the pixel region to be extracted can be further improved by generating the difference image under the same conditions such as the weather and the time zone in the surrounding environment. That is, the "shooting conditions" described in the first embodiment include the conditions of the surrounding environment of the image pickup device 110 in addition to the conditions of the image pickup device 110 side.

Further, in the first embodiment, the first image acquisition unit 121 and the second image acquisition unit 122 have been described as acquiring the first image and the second image directly from the image pickup apparatus 110. However, the first image acquisition unit 121 and the second image acquisition unit 122 may acquire the first image and the second image from the storage destination in which the image data transmitted from the image pickup apparatus 110 is stored. Further, the first image acquisition unit 121 and the second image acquisition unit 122 may perform various preprocessing such as noise removal processing and color correction processing when acquiring the first image and the second image.

Further, in the first embodiment, the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image have been described as being acquired from the same layer, but they are acquired from different layers. You may. However, when acquiring from different layers, it is assumed that the intermediate image is enlarged by performing various interpolation processes.

Further, in the first embodiment, when the difference image is generated, the absolute value of the difference between the first intermediate image and the second intermediate image is added, but the first intermediate image and the second intermediate image are described as being added. The square of the difference may be added.

Further, in the first embodiment, in generating the difference image, the plurality of feature maps included in the first intermediate image and the plurality of feature maps included in the second intermediate image are all used. The difference image may be generated by using a part of the feature map of.

It should be noted that the disclosed technique may have the form described in the appendix below.
(Appendix 1)
The first image including the extraction target and the second image not including the extraction target are acquired, and the image is obtained.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From, the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image are acquired.
The pixel region to be extracted is extracted based on the difference between the first intermediate image and the second intermediate image.
An image processing program that allows a computer to perform processing.
(Appendix 2)
A difference image is generated by calculating the difference between the first intermediate image and the second intermediate image, and the difference image is multiplied by the first image to obtain the first image. The image processing program according to Appendix 1, wherein the pixel region to be extracted is extracted.
(Appendix 3)
The feature is that the difference image is generated by adding the absolute values of the differences between the plurality of feature maps included in the first intermediate image and the plurality of feature maps included in the second intermediate image. The image processing program according to Appendix 2.
(Appendix 4)
The first image and the second image are any of the appendices 1 to 3 characterized in that the image pickup devices installed at the same position are images taken at different timings under the same shooting conditions. The image processing program described in the appendix of.
(Appendix 5)
The image processing program according to any one of Supplementary Provisions 1 to 4, wherein the plurality of identification targets do not include a non-extraction target associated with the extraction target.
(Appendix 6)
When the first image is an identification target of any of the plurality of identification targets and includes an object that is not an extraction target, the second image including the object is acquired. The image processing program according to Appendix 5, which is a feature.
(Appendix 7)
An acquisition unit that acquires a first image that includes an extraction target and a second image that does not include an extraction target.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From the identification unit for acquiring the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image.
An image processing apparatus comprising: an extraction unit for extracting a pixel region to be extracted based on a difference between the first intermediate image and the second intermediate image.
(Appendix 8)
The first image including the extraction target and the second image not including the extraction target are acquired, and the image is obtained.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From, the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image are acquired.
The pixel region to be extracted is extracted based on the difference between the first intermediate image and the second intermediate image.
An image processing method in which a computer performs processing.

The present invention is not limited to the configurations shown here, such as combinations with other elements in the configurations and the like described in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form thereof.

100: Image processing system 110: Image pickup device 120: Image processing device 121: First image acquisition unit 122: Second image acquisition unit 123: Object identification unit 124: Intermediate image acquisition unit 125: Area extraction unit 300: Learning image information 400: First image 500: Second image 800: First intermediate image 900: Second intermediate image 1010: Difference image 1020: Extracted image 1200: Free viewpoint image generation system 1210: Free viewpoint image generation device 1220_1 to 1220_m: Image pickup device 1230: Subject 1300: Image monitoring system 1310: Image monitoring device 1311: Judgment unit

Claims (5)

The first image including the extraction target and the second image not including the extraction target are acquired, and the image is obtained.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From, the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image are acquired.
The pixel region to be extracted is extracted based on the difference between the first intermediate image and the second intermediate image.
An image processing program that allows a computer to perform processing. A difference image is generated by calculating the difference between the first intermediate image and the second intermediate image, and the first image and the difference image are used to obtain the difference image from the first image. The image processing program according to claim 1, wherein a pixel region to be extracted is extracted. The feature is that the difference image is generated by adding the absolute values of the differences between the plurality of feature maps included in the first intermediate image and the plurality of feature maps included in the second intermediate image. The image processing program according to claim 2. An acquisition unit that acquires a first image that includes an extraction target and a second image that does not include an extraction target.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From the identification unit for acquiring the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image.
An image processing apparatus comprising: an extraction unit for extracting a pixel region to be extracted based on a difference between the first intermediate image and the second intermediate image. The first image including the extraction target and the second image not including the extraction target are acquired, and the image is obtained.
The first image and the second image are input to a neural network trained to identify a plurality of identification targets including an extraction target, and a predetermined layer among the plurality of layers of the neural network is input. From, the first intermediate image corresponding to the first image and the second intermediate image corresponding to the second image are acquired.
The pixel region to be extracted is extracted based on the difference between the first intermediate image and the second intermediate image.
An image processing method in which a computer performs processing.

Images