JP7177280B2 - Image recognition device, image recognition method, and image recognition program - Google Patents

Description

The present invention relates to an image recognition device, an image recognition method, and an image recognition program.

There are techniques for recognizing an object represented in an image. For example, techniques for recognizing a person in a photographed image, and for each pixel of an image, recognizing the type of object such as a person, road, sky, etc. represented by that pixel (so-called semantic segmentation). segmentation)) technology is known. In recent years, in order to perform recognition processing with higher accuracy, recognition processing may be performed using a machine learning model that has been learned based on a teacher image.

When performing recognition processing using a machine learning model on which learning has been performed, it is necessary to prepare a teacher image used for the learning. The angle of view of a captured image and the vertical orientation of the captured image differ depending on the posture of the imaging device and the position where the imaging device is placed. Therefore, when a teacher image is captured under random conditions, the teacher image includes captured images with various angles of view and orientations. A machine learning model that has been trained using captured images with various angles of view and orientations as teacher images can perform recognition processing on input images captured with various angles of view and orientations. The number of images increases, making it difficult to improve recognition accuracy. In addition, in order to ensure the necessary recognition accuracy, it is necessary to increase the number of parameters, and there is a problem that the scale of calculation becomes large.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems. , an image recognition device, an image recognition method, and an image recognition program with low computational load.

In order to solve the above problems, an image recognition apparatus according to the present invention is a machine learning model in which learning is performed based on a teacher image captured by an imaging device in a first state represented by first state information. a recognizing unit; an input image obtaining unit that obtains an input image captured by an imaging device in a second state; and second state information representing the second state; an image conversion unit that generates a converted image by applying a conversion process to the image to approximate the image captured in the first state, wherein the recognition unit performs recognition processing on the converted image. characterized in that

An aspect of the present invention is characterized in that the conversion processing is rotation of the input image.

An aspect of the present invention is characterized in that the conversion processing is cutting out a portion of the input image.

An aspect of the invention is characterized in that the portion of the input image that is cropped includes a center of the input image and a horizontal axis represented in the input image.

An aspect of the present invention is characterized in that the transformation processing is affine transformation of the input image.

In one aspect of the present invention, the recognition unit is a machine learning model that has already undergone learning based on a teacher image captured by the imaging device in a third state, and the image conversion unit includes the third It is characterized in that the conversion process is performed so as to bring the image closer to the image captured in the first state or the third state, which is closer to the second state.

An image recognition method according to the present invention includes an input image obtaining step of obtaining an input image captured by an imaging device and state information, and applying a conversion process to the input image to generate a converted image. and a recognition unit, which is a machine learning model trained based on a teacher image captured by an imaging device in the first state represented by the first state information, performs recognition processing on the converted image. a recognition step to be executed, wherein the input image is an image captured by an imaging device in a second state, and the state information obtained in the input image obtaining step is second state information representing the second state; and the conversion processing is a conversion processing for bringing the input image closer to the image captured in the first state based on the second state information.

An image recognition program according to the present invention generates a transformed image by applying an input image acquisition procedure for acquiring an input image captured by an imaging device and state information, and applying transformation processing to the input image. and a recognition unit, which is a machine learning model trained based on a teacher image captured by an imaging device in the first state represented by the first state information, performs recognition processing on the converted image. an image recognition program for causing a computer to execute a recognition procedure to be executed, wherein the input image is an image captured by an imaging device in a second state, and the state information acquired in the input image acquiring step is the second state information representing a second state, and the conversion process is a conversion process that approximates the input image to an image captured in the first state based on the second state information. characterized by

1 is a configuration diagram of an image recognition device according to an embodiment of the present invention; FIG. 1 is a functional block diagram showing an example of functions implemented in an image recognition device according to one embodiment of the present invention; FIG. FIG. 4 is a flow diagram showing an example of the flow of learning processing performed by the image processing apparatus according to one embodiment of the present invention; FIG. 3 is a flow diagram showing an example of the flow of image recognition performed by the image processing apparatus according to one embodiment of the present invention; It is a figure showing an example of image recognition concerning one embodiment of the present invention. It is a figure showing an example of image recognition concerning one embodiment of the present invention.

An embodiment of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a configuration diagram of an image recognition device 10 according to this embodiment.

The image recognition device 10 according to this embodiment is, for example, a computer such as a game console or a personal computer. Also, the image recognition device 10 may be an imaging device such as a digital camera. As shown in FIG. 1, the image recognition device 10 according to this embodiment includes a processor 12, a storage unit 14, an operation unit 16, and a display unit 18, for example.

The processor 12 is a program-controlled device such as a CPU that operates according to a program installed in the image recognition apparatus 10, for example.

The storage unit 14 is a storage element such as ROM or RAM, a hard disk drive, or the like. The storage unit 14 stores programs and the like executed by the processor 12 .

The operation unit 16 is a user interface such as a keyboard, a mouse, and a controller of a game console, and receives user's operation input and outputs a signal indicating the content of the input to the processor 12 .

The display unit 18 is a display device such as a liquid crystal display, and displays various images according to instructions from the processor 12 .

The image recognition device 10 may include a communication interface such as a network board, an optical disk drive for reading optical disks such as DVD-ROMs and Blu-ray (registered trademark) disks, a USB (Universal Serial Bus) port, and the like.

FIG. 2 is a functional block diagram showing an example of functions implemented in the image recognition device 10 according to one embodiment of the present invention. As shown in FIG. 2 , the image recognition device 10 includes a learning section 202 including a recognition section 204 and a parameter storage section 206 , an input image acquisition section 208 and an image conversion section 210 .

The recognition unit 204 is a machine learning model that has been learned based on a teacher image captured by the imaging device in the first state represented by the first state information. Specifically, for example, the recognition unit 204 is a machine learning model implemented by, for example, a convolutional neural network (CNN). A teacher image is a plurality of images captured in a fixed state by an imaging device such as a camera. Depending on this state, the horizontal and vertical angles of view (or focal lengths) determined by the lens of the imaging device, the angle of the imaging device with respect to the vertical direction, and the like are determined. Depending on the state, the area in the real space to be imaged is determined.

In other words, the imaging device converts incident light from a certain area in the real space into an image. It is determined by the angle and the like. Information representing imaging conditions such as the positional relationship of the lens and the attitude of the imaging device (angle of the imaging element), which are elements that determine the area, is hereinafter referred to as state information. For example, the teacher image has a lens focal length of 35 mm, and the angle formed by the bottom surface of the imaging device (or the lower end of the imaging device arranged inside the imaging device) and the vertical direction is 90 degrees. (that is, a state in which the imaging device is horizontally fixed).

Further, the recognition unit 204 executes recognition processing on the converted image. Specifically, for example, the recognition unit 204 recognizes a person, a building, or the like as a subject included in a converted image, which will be described later. The recognition unit 204 may also perform semantic segmentation for recognizing the type of object represented by the pixel, such as a person, road, sky, etc., for each pixel of the transformed image.

The recognition unit 204 is a machine-learning model that has been pre-learned based on the teacher image captured by the imaging device in the first state as described above. The learning is characterized in that the teacher image used for learning is an image captured by an imaging device in a constant state, but other than that, conventionally known methods may be used. The learning will be described with reference to FIG. 3 showing the mode of learning.

First, a plurality of teacher images are captured while the imaging device is fixed in the first state. Then, the recognition unit 204 acquires the plurality of teacher images (S302). Also, the learning unit 202 acquires the result of recognition processing performed on the teacher image acquired in S302 (S304). The recognition process is not a process performed by the recognition unit 204 included in the learning unit 202, but a process for obtaining a correct recognition result by any conventionally known method. Therefore, the learning unit 202 acquires a correct recognition result for the teacher image acquired in S302.

Then, the learning unit 202 performs learning of the recognition unit 204 using the teacher image and the correct recognition result of the teacher image (S306). In the learning, for example, a comparison result (hereinafter referred to as an error) between the correct recognition result and the result of recognition processing performed by the recognition unit 204 on the input teacher image is specified. The error may be data having a value of 0 or more and 1 or less. In this case, the error takes a value of 0 when the correct recognition result and the result of the recognition processing performed by the recognition unit 204 on the input teacher image match, and takes a value of 0 when they do not match. Data that takes 1 may be used. Based on the error, the learning unit 202 updates the parameter value of the recognition unit 204 by, for example, the error back propagation method.

Then, the learning unit 202 confirms whether or not a predetermined end condition is satisfied (S308). If the predetermined end condition is not satisfied (S308: N), the learning unit 202 returns to the processing shown in S306. If the predetermined termination condition is satisfied (S308: Y), the processing shown in this processing example is terminated. As a result, updating of the parameter values of the recognition unit 204 is repeatedly executed. As described above, the recognition unit 204 implemented in the image recognition device 10 is a machine learning model that has been executed.

A parameter storage unit 206 stores parameters of the recognition unit 204, which is a machine learning model. Specifically, for example, the parameter storage unit 206 stores parameters determined by performing learning according to the flow shown in FIG.

The input image acquisition unit 208 acquires an input image captured by the imaging device in the second state and second state information representing the second state. Specifically, for example, the input image acquisition unit 208 acquires an image captured by an imaging device in an arbitrary state (for example, the focal length is 28 mm and the angle between the bottom surface of the imaging device and the vertical direction is 120 degrees). An image and second state information representing the state are obtained.

Here, the control unit that controls the operation of the lens arranged inside the imaging device can acquire information representing the focal length. Also, when the imaging device has a gyro sensor, the gyro sensor can acquire information representing the angle between the bottom surface of the imaging device and the vertical direction. The input image acquisition unit 208 can acquire second state information representing the second state from a control unit or a gyro sensor included in the imaging device.

Note that the input image acquisition unit 208 may acquire combinations of a plurality of input images and second state information according to the number of images to be subjected to recognition processing. The first state information is constant, but the second state may be arbitrary, so the second state information may be different for each acquired input image.

Based on the second state information, the image conversion unit 210 generates a converted image by applying conversion processing to bring the input image closer to the image captured in the first state. Specifically, for example, the transform processing is rotation of the input image. The image conversion unit 210 converts the input image captured by the imaging device in the second state (the focal length is 28 mm and the angle between the bottom surface of the imaging device and the vertical direction is 120 degrees) to the first state (lens The focal length is 28 mm, and the angle between the bottom surface of the imaging device and the vertical direction is 90 degrees). Since the focal lengths in the first state and the second state are the same, the angles of view are the same. On the other hand, the angle between the first state and the second state differs by 30 degrees. Therefore, the image conversion unit 210 rotates the input image counterclockwise by 30 degrees.

It should be noted that the conversion processing may be cutting out a part of the input image. Specifically, for example, the image conversion unit 210 converts the input image captured by the imaging device in the second state (the focal length is 28 mm and the angle between the bottom surface of the imaging device and the vertical direction is 90 degrees). , and the image captured in the first state (the focal length of the lens is 35 mm, and the angle between the bottom surface of the imaging device and the vertical direction is 90 degrees). In this case, the angles in the first state and the second state are the same. On the other hand, since the focal lengths are different between the first state and the second state, the angles of view are different. Therefore, the image conversion section 210 may cut out a region corresponding to the angle of view captured in the first state from the input image captured in the second state.

Also, the transformation processing may be affine transformation of the input image. Specifically, the image conversion unit 210 may scale, shear, rotate, and translate the input image so as to bring the input image closer to the image captured in the first state.

As described above, by using a machine learning model trained using a teacher image captured in the first state, which is a constant state, and the second state information representing the state of the imaging device at the time of shooting, Conditions regarding the state of the imaging device when the teacher image and the input image are captured can be matched. Therefore, highly accurate recognition processing can be executed with a low computational load.

In the above, an embodiment has been described in which learning is performed based on the teacher image captured by the imaging device in the first state. Alternatively, it may be a machine learning model that has already been trained. In this case, based on the second state information, the image conversion unit 210 performs conversion processing to bring the image closer to the image captured in the first state or the third state, which is closer to the second state.

Specifically, for example, the recognition unit 204 recognizes a plurality of teacher images captured with the imaging device fixed in the first state and a plurality of teacher images captured with the imaging device fixed in the third state. It may be a machine learning model in which learning is executed according to the flow shown in FIG. 3 based on a plurality of mixed teacher images. In this case, parameters are determined that are different from parameters determined using only a plurality of teacher images captured with the imaging device fixed in the first state. The recognition unit 204 executes recognition processing by using the parameters.

Further, for example, the recognition unit 204 performs learning based on a plurality of teacher images captured with the imaging device fixed in the first state, and learning performed based on a plurality of teacher images captured with the imaging device fixed in the third state. It may be a machine learning model in which learning is performed based on a plurality of teacher images that have been obtained, and are performed individually. In this case, the parameter storage unit 206 stores the first parameter determined when learning is performed based on the teacher image captured by the imaging device in the first state and the teacher image captured by the imaging device in the third state. and a third parameter determined when image-based learning is performed. Then, after it is determined whether the state represented by the second state information is closer to the first state or the third state, parameters associated with the state determined to be closer are selected. The recognition unit 204 executes recognition processing by using the selected parameters.

In the above case as well, conditions regarding the state of the imaging device when the teacher image and the input image are captured can be matched, so highly accurate recognition processing can be executed with a low computational load.

Next, an example of recognition processing performed by the image recognition device 10 according to the present embodiment will be described with reference to flowcharts illustrated in FIGS. 4 to 6. FIG. It is assumed that the recognition unit 204 is a machine learning model that has already been learned based on teacher images captured by the imaging device in the third state as well as in the first state. Specifically, for example, in the first state, the focal length is 35 mm, and the angle between the bottom surface of the imaging device and the vertical direction is 90 degrees. In the third state, the focal length is 28 mm, Assume that the angle between the bottom surface of the imaging device and the vertical direction represents 90 degrees.

First, the input image acquisition unit 208 acquires an input image captured by the imaging device in the second state (S402). Also, the input image acquisition unit 208 acquires second state information representing the second state (S404). Specifically, the input image shown in the example of FIG. 5 is an image captured by the imaging device in the second state in which the focal length is 35 mm and the angle between the bottom surface of the imaging device and the vertical direction is 120 degrees. is. The input image shown in the example of FIG. 6 is an image captured by the imaging device in the second state in which the focal length is 20 mm and the angle between the bottom surface of the imaging device and the vertical direction is 90 degrees. The input image acquisition unit 208 acquires second state information representing the state when the input image shown in FIG. 5 or 6 was captured.

Next, it is determined whether the state represented by the second state information is closer to the first state or the third state, which is the state when the teacher image was captured (S406). The determination is made based on the focal length in each state and the difference in the angle between the bottom surface of the imaging device and the vertical direction. For example, if only one of the focal length or the angle differs in each state, it is determined which of the first state and the third state is closer to the second state based on the magnitude of the absolute value of the difference. When both the focal length and the angle are different in each state, it is determined which of the first state and the third state is closer to the second state by appropriately weighting or the like according to the purpose of use. If it is determined that the second state is close to the first state, the process proceeds to S408.

When it is determined that the second state is close to the first state, the image conversion unit 210 applies conversion processing to bring the input image closer to the image captured in the first state based on the second state information. (S408). In this case, as shown in FIG. 5, the image conversion unit 210 rotates the input image counterclockwise by 30 degrees. Although FIG. 5 shows the image after only rotation, the image conversion unit 210 further cuts the image so that the top, bottom, left, and right edges of the image are parallel to the vertical and horizontal directions. may be performed.

On the other hand, if it is determined in S406 that the second state is close to the third state, the process proceeds to S410. Based on the second state information, the image conversion unit 210 applies conversion processing to bring the input image closer to the image captured in the third state (S410). In this case, as shown in FIG. 6, the image conversion unit 210 cuts out a region corresponding to the angle of view captured in the third state from the input image captured in the second state.

Here, when the image conversion unit 210 cuts out the input image, the region to be cut out may be moved so as to include an important region of the input image. The important area is determined according to the central area of the input image, the area where the user's line of sight is concentrated when the image is captured, the face recognition area which is separately performed, and the like. The size of the movement should be appropriately weighted according to the purpose of use, etc., so as to maximize the total value of the degree of matching of the state information and the ratio of the important area occupied by the cut out image. It is determined. Based on the second state information, the image conversion unit 210 applies conversion processing to the input image so as to approximate the image captured in the first state. be. By moving the region to be cut out in this manner, image recognition can be executed for a region that is highly likely to be the region intended by the user.

For example, when cropping to include the central region of the input image, the image transform unit 210 crops the input image to include the horizontal axis 602 and the center 604 as shown in FIG. Information representing the horizontal axis 602 is determined based on the second state information acquired from the gyro sensor or the like of the imaging device. When the horizontal axis 602 of the input image exists at the top of the image, as in the upper image in FIG. 6, the image conversion unit 210 cuts out an area corresponding to the angle of view with a focal length of 28 mm from the upper central portion of the input image. . When the horizontal axis 602 of the input image exists at the bottom of the image as in the lower image of FIG. cut out

Then, the recognition unit 204 performs image recognition on the converted image created in S408 or S410 (S412). Here, the parameter storage unit 206 stores the first parameter determined when learning is performed based on the teacher image captured by the imaging device in the first state, and the teacher image captured by the imaging device in the third state. and a third parameter that was determined when learning was performed based on the images. The converted image created in S408 is subjected to recognition processing by the recognition unit 204 using the first parameter. On the other hand, the converted image created in S410 is subjected to recognition processing by the recognition unit 204 using the third parameter.

The functions described above may be implemented by causing the processor 12 to execute a program containing instructions corresponding to the functions described above, which is installed in the image recognition apparatus 10, which is a computer. This program may be supplied to the image recognition apparatus 10 via a computer-readable information storage medium such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, flash memory, etc., or via the Internet or the like. . Also, the states in which the teacher image is captured are not limited to the two types of the first state and the third state, and may be three or more types as long as they are limited.

It should be noted that the present invention is not limited to the above-described embodiments. In addition, the specific character strings and numerical values described above and the specific character strings and numerical values in the drawings are examples, and are not limited to these character strings and numerical values.

Claims (4)

a recognition unit, which is a machine learning model for which learning is performed based on a teacher image captured by the imaging device in the first state represented by the first state information;
an input image acquisition unit that acquires an input image captured by the imaging device in the second state and second state information representing the second state;
an image conversion unit that generates a converted image by applying a conversion process that makes the input image closer to the image captured in the first state based on the second state information;
including
The recognition unit performs recognition processing on the converted image ,
the conversion processing is cutting out a part of the input image;
An image recognition apparatus , wherein the portion of the input image to be cropped includes a center of the input image and a horizontal axis represented in the input image . The recognition unit is further a machine learning model that has been trained based on the teacher image captured by the imaging device in the third state,
The image conversion unit performs the conversion process to approximate an image captured in a state closer to the second state than the first state or the third state,
2. The image recognition device according to claim 1 , wherein: an input image acquisition step of acquiring an input image captured by an imaging device and state information;
an image conversion step of generating a converted image by applying a conversion process to the input image;
a recognition step of performing recognition processing on the converted image by a recognition unit, which is a machine learning model trained based on a teacher image captured by an imaging device in the first state represented by the first state information;
including
the input image is an image captured by the imaging device in the second state;
the state information obtained in the input image obtaining step is second state information representing the second state;
The conversion process is a conversion process that approximates the input image to an image captured in the first state based on the second state information ,
the conversion processing is cutting out a part of the input image;
the portion of the input image that is cropped includes the center of the input image and a horizontal axis represented in the input image;
An image recognition method characterized by: an input image acquisition procedure for acquiring an input image captured by an imaging device and state information;
an image conversion procedure for generating a converted image by applying a conversion process to the input image;
a recognition procedure for executing recognition processing on the converted image by a recognition unit, which is a machine learning model trained based on the teacher image captured by the imaging device in the first state represented by the first state information;
An image recognition program that causes a computer to execute
the input image is an image captured by the imaging device in the second state;
the state information acquired in the input image acquisition step is second state information representing the second state;
The conversion process is a conversion process that approximates the input image to an image captured in the first state based on the second state information ,
the conversion processing is cutting out a part of the input image;
the portion of the input image that is cropped includes the center of the input image and a horizontal axis represented in the input image;
An image recognition program characterized by:

Images