JP2016167164A - Image recognition device, image recognition method and image recognition program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform highly accurate face authentication on a face image including a face with a small captured-size, blur or the like.SOLUTION: An image recognition device comprises: setting means which compares the whole facial area of an input image with the whole facial area of each of registered images, calculates the face similarity between the input image and the registered image, retrieves the registered image corresponding to the input image on the basis of the face similarity, and sets a face organ position stored in association with the corresponding registered image as the face organ position of the input image; feature extraction means which extracts a feature quantity of each face organ of the input image on the basis of the face organ position set by the setting means and extracts a feature quantity of each face organ of each registered image on the basis of the stored face organ position; and face identification means which calculates the face organ similarity between the feature quantity of each face organ of the input image and the feature quantity of each face organ of each registered image and identifies whether the face captured in the input image and the face captured in each registered image are the face of the same person on the basis of the calculated face organ similarity.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image recognition apparatus, an image recognition method, and an image recognition program.

  An alignment technique for identifying the position of an object existing in an image and aligning the size of the object is very important in general image recognition. For example, in order to make the position and size of the face uniform, as in Non-Patent Document 6, finer feature points of the face surface such as the corners of the eyes and the eyes are extracted, and the size and position of the face are based on the extracted feature points There is a technology that keeps a constant.

  Among image recognition techniques based on such alignment techniques, there is a face authentication technique that identifies who the face in the video is. For example, as in Non-Patent Document 1, a feature value called Local Binary Pattern (hereinafter referred to as LBP feature) is extracted from an input luminance face image in which the position and size of the face are made uniform. Then, by comparing the feature amount extracted from the input luminance face image with the feature amount extracted from the previously registered luminance face image, the input face is identified.

  It should be noted that the region from which the feature amount is extracted does not divide the image evenly, and the direction set with reference to finer feature points on the face surface such as the corners of the eyes and the eyes becomes more robust in the direction of the face. Such face authentication technology has been used in comparatively good shooting conditions such as a short distance from an imaging device to a subject as seen in auto shutter and entry / exit management of a digital camera.

  In recent years, research on face authentication for low-resolution face images with small distant faces or blurs that are obtained by surveillance cameras has been conducted, and high resolution of images is one means to deal with this. There is imaging. That is, face authentication is performed after the high resolution of the low-resolution face image is pre-processed and the amount of information is restored. As a high-resolution image technique, for example, as in Non-Patent Document 2, a technique called hallucination that approximates the face of one person with the linear sum of the faces of another person has been proposed.

T. Pajdla, and J. Matas, “Face Recognition with Local Binary Patterns”, ECCV, pp. 469 − 481, 2004 K. Huang, R. Hu, “Face hallucination via K−selection mean constrained sparse representation”, ICIP, pp. 882 − 885, 2012 M. Turk, A. Pentland, Eigenfaces for Recognition, Journal of Cognitive Neurosicence, Vol. 3, No. 1, 1991, pp. 71-86 B. Li, H. Chang, “Hallucinating Facial Images and Features”, ICPR, pp. 1-4, 2008 P. Viola, M. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features”, in Proc. Of CVPR, vol.1, pp.511−518, December, 2001 T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, “Active Shape Models −Their Training and Application”, Computer Vision and Image Understanding, Vol. 61, No. 1, January, pp. 38 − 59, 1995 I. Kemelmacher−Shlizerman, “3D Face Reconstruction from a Single Image Using a Single Reference Face Shape”, PAMI, pp. 394 − 405, 2011

  Image resolution techniques such as the hallucination technique described above divide an image into multiple blocks and perform high resolution for each divided block, so the face position and size are kept constant. It is necessary to align. However, since the amount of information of a low resolution image (low resolution image) before high resolution is greatly lost, there is a problem that it is not possible to accurately detect the position of the face or eyes. It was. The present invention has been made in view of the above-described problems, and can accurately align a small face or blur that is far away, and even a face image that includes noise, and a highly accurate face. It aims at providing the technique which can implement | achieve authentication.

In order to achieve the above object, the image recognition apparatus of the present invention provides:
Registration means for storing a plurality of registered images in association with the facial organ positions of each facial organ shown in the registered image;
Comparing the entire face area of the input image and the entire face area of each registered image of the plurality of registered images to calculate a face similarity between the input image and the registered image, and based on the face similarity Searching for the registered image corresponding to the input image, and setting the face organ position stored in association with the corresponding registered image as the face organ position of the input image;
A feature amount of each facial organ of the input image is extracted based on the facial organ position set by the setting means, and each registered image based on the facial organ position stored by the registration means Feature extraction means for extracting feature quantities of facial organs;
A facial organ similarity between the facial organ feature quantity of the input image extracted by the feature extraction unit and the facial organ feature quantity of each registered image is calculated, and based on the calculated facial organ similarity degree Face identifying means for identifying whether the face shown in the input image and the face shown in each of the registered images are the faces of the same person;
It is characterized by providing.

  According to the present invention, it is possible to accurately align a small face or blur that is far away, and even a face image that includes noise, and to realize highly accurate face authentication.

It is a block diagram which shows the structure of the image recognition apparatus of 1st Embodiment. It is a flowchart which shows the process in the whole process and face image registration mode of 1st Embodiment. It is a flowchart which shows the process in the face identification mode of 1st Embodiment. It is a flowchart which shows the detail of the face organ position setting process in step S1401. It is a figure explaining the conversion image generation process in step S1411. It is a figure explaining the process which searches the pair of the conversion image and registration image in step S1412. It is a figure explaining the process which sets the facial organ position of the registration image in step S1413 to the facial organ position of a conversion image. It is a figure explaining the process which calculates the similarity degree of the feature-value of the facial organ of the conversion image in step S1701, and the feature-value of the facial organ of a registration image. It is a block diagram which shows the structure of the image recognition apparatus of 2nd Embodiment. It is a flowchart which shows the process in the face identification mode of 2nd Embodiment. It is a figure explaining the relationship between the registration image in 2nd Embodiment, a conversion image, the high-resolution conversion image, and a registration image. It is a figure which shows the failure of a high-resolution face image when a high-resolution face image is produced | generated from a low-resolution face image by hallucination technique. It is a block diagram which shows the structure of the image recognition apparatus of 3rd Embodiment. It is a flowchart which shows the process in the face identification mode of 3rd Embodiment. It is a flowchart which shows the process from step S2401 of 3rd Embodiment to step S2801 in detail. It is a figure which shows the characteristic of 3rd Embodiment. It is a figure explaining the relationship between the registration image in 3rd Embodiment, a conversion image, and a twice conversion image. It is a figure explaining the relationship between the twice-converted image, the twice-converted high-resolution image, and the registered image in the third embodiment. It is a figure explaining the relationship between the twice conversion high resolution image in a 3rd embodiment, and a registration image. It is a figure which shows an example of the hardware constitutions of the image recognition apparatus which concerns on the 1st-3rd embodiment of this invention.

Focusing on the processing of the entire face authentication technique, there is a registration process for registering a face image of a person to be specified in advance in face authentication. Face images registered in the registration process are often acquired on the spot or face images that have already been shot are acquired via an interface such as a flash memory. There are often.
In the present invention, the face feature point detected from the face image registered in the registration process is not applied to the low-resolution face image input in the conventional manner by applying the face feature point detection technique to perform positioning. Is applied to the input low-resolution face image.
In addition to this, by incorporating the technique of the present invention that is robust against positional deviation with respect to the high-resolution technique, the face authentication accuracy for the low-resolution face image can be greatly improved.

[First Embodiment]
In the first embodiment, the face organ position detected from the face image registered in the registration process is set as the face organ position of the input low-resolution face image, and face recognition is executed. The details will be described below.

<Hardware configuration>
FIG. 14 shows an example of the hardware configuration of the image recognition apparatus in the present embodiment. The image recognition apparatus 1400 includes a CPU (Central Processing Unit) 1401, a ROM (Read Only Memory) 1402, and a RAM (Random Access Memory) 1403. Further, a secondary storage device 1404, a display unit 1405, an operation unit 1406, a network communication unit 1407, a network connection unit 1408, a USB communication unit 1409, a USB connection unit 1410, and a connection bus 1411 are provided.
The CPU 1401 controls the entire apparatus by executing a control program stored in the ROM 1402 or the RAM 1403.

The ROM 1402 is a nonvolatile memory, and stores a control program and various parameter data. The control program is executed by the CPU 1401 and causes the apparatus to function as means for executing each process described below.
A RAM 1403 is a volatile memory, and temporarily stores image data, a control program, and an execution result thereof.

The secondary storage device 1404 is a rewritable secondary storage device such as a hard disk or a flash memory, and stores an OS (Operating System), application programs, image data, and the like.
The CPU 1401 reads out the program and OS stored in the secondary storage device 1404 to the memory 1403. Various functions of the image recognition apparatus can be realized by executing the program on the memory 1403.

Note that the execution of the program may be performed by a single processor, or may be executed by the cooperation of a plurality of processors. Further, a dedicated circuit (ASIC) for executing a specific process may be provided, and the specific process may be executed by the dedicated circuit.
Further, software (program) describing processing to be described later may be acquired and executed via a network or various storage media.

  The display unit 1405 is composed of a display device such as an LCD. The operation unit 1406 includes an input device such as a keyboard and a mouse. A network communication unit 1407 connects the image recognition apparatus to a network and performs various communications. A network connection unit 1408 connects the network communication unit 1407 to a network medium.

The network communication unit 1407 and the network connection unit 1408 correspond to at least one of a wired LAN and a wireless LAN. These specific forms take necessary functions and forms according to the corresponding LAN. A USB communication unit 1409 communicates with various peripheral devices via a USB interface. The USB connection unit 1410 includes a USB connector.
The connection bus 1411 connects the CPU 1401, the ROM 1402, the RAM 1403, the secondary storage device 1404, and the like to input / output data mutually.

<Functional configuration>
FIG. 1 is a block diagram illustrating an example of a functional configuration of the image recognition apparatus according to the first embodiment. As shown in FIG. 1, the functional configuration of the present embodiment includes an image acquisition unit 110, a face position detection unit 120, a face organ position detection unit 130, a face organ position setting unit 140, a feature extraction unit 150, and a face image registration unit 160. , And a face identification unit 170.

The image acquisition unit 110 acquires a plurality of registration images. An image for registration or a registered image is also referred to as a registered image.
In addition, the image acquisition unit 110 acquires an image (hereinafter, also referred to as “input image”) that is a target for which it is determined whether the image matches the registered image.
The face position detection unit 120 detects the position of the face shown in the input image and the registered image.

As for the registered image, the facial organ position detector 130 detects the position of each facial organ (right eye, left eye, mouth, etc.) shown in the registered image.
The facial organ position detection unit 130 does not detect the position of each facial organ in the input image with respect to the input image.

  The facial organ position setting unit 140 compares the entire face area of the input image with the entire face area of each registered image of the plurality of registered images, calculates the face similarity between the input image and the registered image, Search for a registered image that becomes a corresponding pair. Here, a registered image having the highest face similarity with the input image is searched. Then, the facial organ position stored in association with the registered image having the highest facial similarity is set as the facial organ position of the input image. Note that this search is for setting a facial organ position as a reference for extracting features, and not a registered image with the highest face similarity but a registered image with a face similarity higher than a predetermined threshold. You may make it do. The same applies to the following search.

The facial organ position setting unit 140 also
A plurality of converted images having at least one of a face size, a face position, and a face orientation are generated from the input image, and the degree of face similarity between each of the generated converted images and each of the plurality of registered images To calculate
Search for the pair of the converted image and registered image with the highest facial similarity,
It is preferable to set the face organ position stored in association with the registered image with the highest face similarity as the face organ position of the converted image with the highest face similarity.

With respect to the registered image, the feature extraction unit 150 extracts a feature amount of each facial organ of the registered image based on the facial organ position detected by the facial organ position detection unit 130.
With respect to the input image, the feature extraction unit 150 extracts feature amounts of each facial organ from the converted image of the input image based on the facial organ position set by the facial organ position setting unit 140.

The face image registration unit 160 includes a plurality of registered images, the position of each facial organ shown in each registered image, and the feature amount of each facial organ (the feature amount of the right eye, the feature amount of the left eye, the feature amount of the mouth, etc.) And store them in association with each other.
For example,
The first registered image
The position of the right eye, the position of the left eye, the position of the mouth in the first registered image;
The right eye feature amount, left eye feature amount, mouth feature amount, and the first registered image,
Store it in association.

  The face identification unit 170 extracts the feature amount of each facial organ from the input image extracted by the feature extraction unit 150 and the facial organ of each registered image extracted by the feature extraction unit 150 and stored in the facial image registration unit 160. The facial organ similarity with the feature amount is calculated. Then, based on the calculated facial organ similarity, it is discriminated whether the face shown in the input image and the face shown in each registered image are the faces of the same person.

2 and 3 show the overall flow of the first embodiment of the present invention, and in the following, the first embodiment will be described in detail using this overall flow.
In step S1001 of FIG. 2A, first, it is determined whether or not the registration mode is selected. If the registration mode is selected, the process proceeds to the face image registration mode.

<Face image registration mode>
In step S1101 in FIG. 2B, the image acquisition unit 110 acquires a registered image. The image acquisition unit 110 passes through a condensing element such as a lens, an imaging element such as a CMOS or a CCD that converts light into an electrical signal, and an AD converter that converts an analog signal into a digital signal. Get the data. Further, by performing thinning processing or the like, it is also possible to acquire a face image converted into, for example, VGA (640 × 480 [pixel]) or QVGA (320 × 240 [pixel]). In addition to shooting, a registered image can be acquired through a flash memory or the like. Therefore, a relatively high-resolution face image is registered as the registered image.

In step S <b> 1201, the face position detection unit 120 detects the position of the center of gravity of the face, the left and right eyes, the mouth, and the like using a technique such as Non-Patent Document 5.
In step S1202, the first regularity that the face size is a predetermined size and the face orientation is upright using affine transformation or the like from the center of gravity of the face, left and right eyes or mouth detected in step S1201. Generate a digitized image. There is a method of defining the size of the face as the Euclidean distance between the left and right eyes.

In step S1301, the facial organ position detection unit 130 uses the technique as described in Non-Patent Document 6 for the first normalized image generated in step S1201, and more fine facial organ positions such as the corners of the eyes and the eyes. Is detected.
In step S1302, a second normalized image in which the face size is a predetermined size and the face orientation is upright using the position of the center of gravity of finer feature points such as the corners of the eyes and the eyes detected in step S1301. Is generated.

In step S1501, a feature extraction region is set for the second normalized image generated in step S1302 based on the facial organ position detected in step S1301, and an LBP feature as in Non-Patent Document 1 is set from the region. To extract.
In step S1601, the face image registration unit 160 stores the personal ID, the face organ position detected in step S1301, the second normalized image generated in step S1302, and the feature amount generated in step S1501 in a memory or the like. To do.
In conventional face authentication, in step S1601, the face organ position detected in step S1301 and the second normalized image generated in step S1302 are not stored, and only the personal ID and the feature amount are stored. It is common.

  The above processing is registration processing performed in the face image registration mode. In step S1501, feature extraction is performed on the normalized image, and the feature amount extracted in step S1601 is also stored in the memory. However, the personal ID, the face organ position detected in step S1301, and the normalized image generated in step S1302 are stored, and the feature extraction is not performed in the registration process. In the face identification mode, feature extraction may be performed from the input image and feature extraction may be performed from the registered image.

<Face identification mode>
If the face identification mode is selected in step S1001 of FIG. 2A, the process proceeds to the face identification mode.
Since the processing from step S1102 to step S1204 in FIG. 3 is the same as the processing from step S1101 to step S1202 in the face image registration mode, description thereof will be omitted. However, in the first embodiment of the present invention, the face image acquired in the face identification mode is a low-resolution face image such as a small face in the distance or a blurred face.

In step S1401, the facial organ position setting process in the facial organ position setting unit 140, which is a point of the present embodiment, will be described.
FIG. 4 shows a detailed processing flow of the facial organ position setting processing in step S1401.
FIG. 5 is a diagram illustrating the converted image generation processing in step S1411.
In step S1411, a plurality of converted images having different scales (sizes), shift amounts, and rotation amounts are generated from the first normalized image generated in step S1204 as shown in FIG. A plurality of converted images having different scales, shift amounts, and rotation amounts are generated using, for example, affine transformation.

FIG. 6A is a diagram illustrating processing for searching for a pair of a converted image and a registered image in step S1412.
In step S1412, as shown in FIG. 6A, a plurality of converted images having different scales, shift amounts, and rotation amounts generated in step S1411 are registered with a converted image having the highest similarity by matching the registered image. Search for a pair with an image. For the matching, for example, normalized cross-correlation is used.

  A plurality of converted images are generated by scale conversion. A plurality of converted images are generated by both shift conversion and rotation conversion. Further, scale conversion, shift conversion, and rotation conversion may be appropriately combined. Scale conversion and shift conversion may be performed. Further, rotation conversion, scale conversion, and shift conversion may be performed.

FIG. 6B is a diagram illustrating processing for setting the facial organ position of the registered image to the facial organ position of the converted image in step S1413.
In step S1413, as shown in FIG. 6B, the facial organ position of the registered image (first registered image in the example of FIG. 6B) searched in step S1412 is changed to the variation image (first converted image in the example of FIG. 6B). Set as the facial organ position. Note that the facial organ position of the registered image is a result of the facial organ position detection in step S1301 as described above.

  In conventional face authentication, face organ position detection is performed on the first normalized image generated in step S1204 as in step S1301. However, if the first normalized image generated in step S1204 is a low-resolution image, a correct position cannot be obtained even if face organ position detection is performed. However, only relatively low-frequency component information such as face contour information remains even in a low-resolution image.

  Accordingly, in the present embodiment, face organ position detection is not performed on the first normalized image generated in step S1204. Instead, a registered image having the highest similarity with the converted image is searched by matching the appearance of the entire face with the first normalized image generated in step S1204, and the registered image having the highest similarity. The face organ position detection result is used as the face organ position of the converted image.

In step S1502, a feature extraction region is set using the result of face organ position detection of the registered image set in step S1413, and feature extraction is performed on the region.
In step S1701, the degree of similarity (facial organ similarity) between the feature amount extracted in step S1502 and the feature amount stored in step S1601 is calculated to identify an individual.
For example, as shown in FIG. 6C, the degree of similarity between each facial organ of the first converted image and each facial organ of the first registered image, and each facial organ of the first converted image and each face of the second registered image Calculate organ similarity. Then, based on the calculated facial organ similarity, the person shown in the first input image and the person shown in the first registered image are the same person or the person shown in the first input image And whether the person shown in the second registered image is the same person.

[Second Embodiment]
In the second embodiment, after setting the facial organ position of the input image, the converted image of the input image is subjected to high resolution. Therefore, the similarity between each facial organ of the converted image with high resolution and each facial organ of the registered image is calculated, and is reflected in the input image based on the calculated similarity (facial organ similarity). Identify whether the person and the person in the registered image are the same person.

<Functional configuration>
FIG. 7A is a block diagram illustrating an example of a functional configuration of the image recognition apparatus according to the second embodiment. As shown in FIG. 7A, the functional configuration 200 of the present embodiment includes an image acquisition unit 110, a face position detection unit 120, a face organ position detection unit 130, a face organ position setting unit 140, a feature extraction unit 151, and a face image registration unit. 160, a face identification unit 171, and a high-resolution face image generation unit 210.

The difference between the image recognition apparatus 200 in the second embodiment shown in FIG. 7A and the image recognition apparatus 100 in the first embodiment shown in FIG. 1 is as follows.
The image recognition apparatus 200 includes a high-resolution face image generation unit 210, but the image recognition apparatus 100 does not include a functional unit corresponding to the high-resolution face image generation unit 210.
The high-resolution face image generation unit 210 generates a high-resolution face image from the converted image having the highest face similarity.

With respect to the input image, the feature extraction unit 151 uses a high-resolution image (high-resolution converted image) based on the converted image of the input image based on the facial organ position set by the facial organ position setting unit 140. Extract features of each facial organ.
The face identification unit 171 calculates the facial organ similarity between the feature amount of each facial organ of the high-resolution face image and the feature amount of each facial organ of each registered image, and based on the calculated facial organ similarity. The face shown in the input image and the face shown in each registered image are identified as the faces of the same person.

FIG. 7B is a flowchart showing a flow of processing in the face identification mode in the second embodiment. The processing from step S1102 to step S1401 in FIG. 7B is the same as the processing from step S1102 to step S1401 in FIG.
FIG. 7C is a flowchart showing a flow of a series of processes in the second embodiment.

In step S1450, the high-resolution face image generation unit 210 increases the resolution of the converted image. As a result of the matching described above, the conversion image to be high-resolution is the conversion of the pair that the facial organ position setting unit 140 determines to have the highest similarity (face similarity) among the pair of the conversion image and the registered image. It is an image.
In step S1502, the feature extraction unit 151 extracts a feature amount from the converted image having a high resolution.
In step S <b> 1701, the similarity between the feature amount extracted from the high-resolution converted image by the feature extraction unit 151 and the feature amount extracted from the registered image by the feature extraction unit 151 and stored in the face image registration unit 160. (Facial organ similarity) is calculated. Then, based on the calculated facial organ similarity, the face identifying unit 170 identifies whether the person shown in the input image and the person shown in the registered image are the same person.

[Third Embodiment]
First, the hallucination technique, which is a high-resolution face image generation technique, will be described.
・ Description of hallucination technology
The hallucination technique is a technique for generating a high-resolution face image from a low-resolution face image. The outline of the principle is that an input low-resolution face image is approximated by another person's high-resolution face image. Details will be described.

First, a high resolution dictionary (Equation 1) in which high resolution and low resolution are paired using face images of various persons in advance is prepared by learning.
A plurality of pairs of high resolution and low resolution are stored as shown in Equation 2.
In addition, when each of the first pair constituting the high resolution dictionary, the low resolution dictionary constituting the first pair, and the high resolution dictionary constituting the first pair is described as Equation 3, The relationship among the resolution dictionary, the low resolution dictionary, and the high resolution dictionary is as shown in Equation 4.

  When a luminance image is used as the high resolution dictionary D, a high resolution luminance image and a low resolution luminance image are stored as a pair. For example, a high-resolution luminance image is an image that is clear enough to identify who the face in the video is. On the other hand, a low-resolution luminance image is an image in which it is difficult to specify who the face in the video is because the face is too small or the face is blurred.

Next, cut a low-resolution partial images from the input low-resolution face image I _L, the pair dictionary high resolution and low resolution stored the low-resolution partial images to the high resolution of the dictionary D Of these, approximation is performed using a linear sum of low-resolution dictionaries. Equation 5 is an approximation result.
As a result, a low resolution dictionary and a coupling coefficient α (α1, α2, α3,...) Approximating the low resolution partial image are obtained, and a high resolution dictionary and a coupling coefficient α corresponding to the low resolution dictionary are obtained. A high-resolution partial image is generated using (α1, α2, α3,...). Expression 6 is an expression representing the generated high-resolution partial image.

  Note that the high-resolution dictionary D may use a base image common to face images such as edges instead of a luminance image. As an example of the base image, there is an eigenface by principal component analysis as in Non-Patent Document 3.

Next, the problem of hallucination technology is explained.
・ Problems of hallucination technology
In the hallucination technology, the low-resolution face image is divided into a plurality of blocks, and the high-resolution dictionary D is used for each block to achieve high resolution. A highly accurate alignment is required. If high-accuracy alignment has not been achieved, the generated high-resolution face image may partially fail, and sufficient face authentication accuracy may not be realized.

  For example, when a high-resolution image (high-resolution image) 18003 is generated from the low-resolution face image 18001 as shown in FIG. 8, the human eye (left eye) included in the block 18002 is a block 18004 corresponding to the block 18002. Can be recognized as human eyes. However, the human eye (right eye) straddling blocks 801 and 802 fails in blocks 803 and 804 corresponding to blocks 801 and 802 and cannot be recognized as human eyes.

On the other hand, when the high-resolution image 18007 is generated from the low-resolution face image 18005 obtained by shifting the face position of the low-resolution face image 18001 to the left, the human eye (right eye) included in the block 18006 corresponds to the block. In block 18008 to be recognized as human eyes. However, the phenomenon that the human eye (left eye) straddling blocks 805 and 806 fails in blocks 807 and 808 corresponding to blocks 805 and 806 and cannot be recognized as human eyes. To do.
As described above, in the hallucination technology, it is very important to align the position of the eyes and mouth with a predetermined position. If this is not done, a phenomenon may occur in which part of the image breaks down. . However, it is difficult to align the low-resolution face image with high accuracy. Therefore, in the third embodiment, a plurality of high-resolution face images are generated by the high-resolution processing, and the region with the highest similarity is used among them, so that only the region that has not failed is used. Face recognition. The details will be described below.

  FIG. 9 is a block diagram illustrating an example of a functional configuration of the image recognition apparatus according to the third embodiment. As shown in FIG. 9, the functional configuration of the present embodiment includes an image acquisition unit 110, a face position detection unit 120, a face organ position detection unit 130, a face organ position setting unit 140, a feature extraction unit 152, and a face image registration unit. 160, and a high-resolution image generation unit 310 and a face identification unit 340. The face identification unit 340 includes a similarity calculation unit 320 and a similarity integration unit 330.

  The image acquisition unit 110, the face position detection unit 120, the face organ position detection unit 130, the face organ position setting unit 140, and the face image registration unit 160 are the image acquisition unit 110 or the like in the first embodiment or the second embodiment. Since it is the same as that of FIG.

  The high-resolution face image generation unit 310 generates a plurality of twice-converted images having at least one of a face size, a face position, and a face orientation from the converted image having the highest face similarity. Then, a twice-converted high-resolution face image (hereinafter, also simply referred to as a high-resolution face image) is generated from each twice-converted image of the plurality of twice-converted images.

The feature extraction unit 152
Based on the facial organ position set by the facial organ position setting unit 140, the feature amount of each facial organ of the twice converted high-resolution facial image is extracted,
Based on the facial organ position stored in the facial image registration unit 160, the feature amount of each facial organ of each registered image is extracted.

  The similarity calculation unit 320 calculates the facial organ similarity between the feature amount of each facial organ of the twice-converted high-resolution face image and the feature amount of each facial organ of each registered image.

The similarity integration unit 330 includes at least the highest facial organ similarity for the first facial organ calculated using the first two-time converted high-resolution facial image,
The integrated facial organ similarity is obtained by integrating the highest value of the facial organ similarity for the second facial organ calculated using the second two-time converted high-resolution facial image.
The first facial organ is, for example, the right eye, and the second facial organ is, for example, the left eye.

Here, “at least” means the maximum value of the facial organ similarity for the third facial organ (for example, mouth) calculated using the third two-time transformed high-resolution facial image,
The integrated face obtained by integrating the maximum value of the facial organ similarity for the fourth facial organ (a facial organ other than the right eye, the left eye, and the mouth) calculated using the fourth two-time conversion high resolution facial image This means that organ similarity may be obtained.

The “first two-time conversion high-resolution face image” means “a combination of the first facial organ of the image and the first facial organ of the registered image” to “a facial organ similarity for the first facial organ” This means a twice-converted high-resolution image from which the “highest value” is obtained.
The “second two-time conversion high-resolution face image” means “the combination of the second facial organ of the image and the second facial organ of the registered image” to “the facial organ of the second facial organ” This means a twice-converted high-resolution image from which the “maximum similarity” is obtained.

A “maximum value of facial organ similarity for the first facial organ” is obtained from a certain “two-time transformed high-resolution facial image”, and “second-transformed high-resolution facial image” is obtained from the same “second-transformed high-resolution facial image”. It is possible that the “maximum value of facial organ similarity for the facial organs” is obtained. In this case, the “first two-time conversion high-resolution face image” and the “second two-time conversion high-resolution face image” are the same “two-time conversion high-resolution face image”.

  Based on the integrated facial organ similarity (integrated similarity) obtained by the similarity integrating unit 330, the face identifying unit 340 determines that the face shown in the input image and the face shown in each registered image are the same person. Identify face.

10 and 11 show the flow of processing in the face identification mode in the third embodiment of the present invention.
Since the face registration mode is the same as the face registration mode of the first embodiment, description thereof is omitted.

<Face identification mode>
The processing from step S2102 to step S2302 in FIG. 10 is the same as the processing from step S1101 to step S1202 in FIG.
Before describing step S2303 and subsequent steps, points in the third embodiment of the present invention will be described.

<Points of Third Embodiment>
As described above, when the hallucination technique is used, a phenomenon such that the position of eyes or mouth cannot be adjusted to a predetermined position and a part of the image breaks down occurs. Therefore, in the third embodiment, in step S2303, a plurality of converted images having different shift amounts, scales, and rotation amounts are generated, the resolution conversion processing is performed on the generated converted images, and the registered images are compared. Face recognition is performed using only regions with high similarity. As a result, face authentication is performed using only the unbroken area.

FIG. 12 shows a process flow in which a high-resolution face image is generated from a plurality of converted images, and face authentication is performed using an unbroken area (block) in the generated high-resolution face image. It is a figure which shows an outline.
The converted image 18001 and the converted image 18005 are examples of converted images having different shift amounts. The block 18004 of the high-resolution face image 18003 generated from the converted image 18001 and the block 18008 of the high-resolution face image 18007 generated from the converted image 18005 have high facial organ similarity with the registered image (that is, it has failed). This is an example of a region that is not.

The position of the block 18004 that does not fail corresponds to the position of the block 18011 of the registered image 18809, and the position of the block 18008 that does not fail corresponds to the position of the block 18010. Then, the face is identified based on the similarity between the feature extracted from the block 18004 and the feature extracted from the block 18011, and the similarity between the feature extracted from the block 18008 and the feature extracted from the block 18010.
The above is the point of the third embodiment.
Subsequently, processing in the third embodiment will be sequentially described.

<Processing Flow in Third Embodiment>
In step S2303, a plurality of converted images having different shift amounts, scales, and rotation amounts are generated from the normalized image generated in step S2302. For example, affine transformation is used.
As illustrated in FIG. 13A, first to third converted images are generated from an input image by scale conversion, shift conversion, rotation conversion, and the like. As in the first and second embodiments, a plurality of converted images are generated by scale conversion, shift conversion, and rotation conversion, and the conversions may be appropriately combined.

<Search for pairs>
In step S2304, as in step S1402 of FIG. 4, a search is made for a pair of a converted image and a registered image having the highest similarity.

<Generate twice converted image>
In step S2305, a plurality of converted images having different shift amounts, scales, and rotation amounts (hereinafter also referred to as “twice-converted images”) are generated from the pair of converted images determined to have the highest similarity in step S2304. .
In the example shown in FIG. 13A, the first converted image is the converted image having the highest similarity, and the first two-time converted image and the second two-time converted image are generated from the first converted image. doing.
The shift amount, scale, and rotation amount in step S2305 in FIG. 10 are smaller than the shift amount, scale, and rotation amount in step S2303. This is because step S2303 is a pre-process for comparing the similarity of the whole face, whereas step S2305 is a pre-process for comparing the similarity of a facial organ that is smaller than the entire face. For example, in step S2303, the shift is performed in units of 1 cm, and in step S2305, the shift is performed in units of 1 mm.

In step S2401, high resolution is performed by applying the hallucination technique as described above to the plurality of converted images generated in step S2305.
In the example shown in FIG. 13B, the first two-time conversion image is increased in resolution to form a first two-time conversion high-resolution image, and the second two-time conversion image is increased in resolution to obtain the second two-time conversion. It was set as the conversion high resolution image.

<Feature extraction>
In step S2502, LBP features as described in Non-Patent Document 1 are extracted from all the high-resolution face images generated in step S2401.
<Calculation of similarity>
In step S2701, the similarity between the LBP feature extracted in step S2502 from a block having a high-resolution face image and the LBP feature extracted from the same block in the face image stored in step S1601 (FIG. 2) is calculated.

In the example shown in FIG. 13B,
Calculating the similarity between the LBP feature extracted from the block 18022 of the first twice-converted image and the LBP feature extracted from the block 18010 of the first registered image;
The similarity between the LBP feature extracted from the block 18008 of the second twice-converted image and the LBP feature extracted from the block 18010 of the first registered image is calculated.

In the example shown in FIG. 13C,
Calculating the similarity between the LBP feature extracted from the block 18004 of the first twice-converted image and the LBP feature extracted from the block 18011 of the first registered image;
The degree of similarity between the LBP feature extracted from the block 18012 of the second twice-converted image and the LBP feature extracted from the block 18011 of the first registered image is calculated.

<Similarity integration>
In step S2702, the similarity of the pair having the highest similarity is selected from the similarities of the plurality of pairs having the same block position, and the selected similarities are integrated.
For example, the similarity (facial organ similarity) between the block 18022 of the high-resolution face image 18003 in FIG. 13B and the block 18010 of the registered image 18809,
Similarity between the block 18008 of the high-resolution face image 18007 and the block 18010 of the registered image 18009 (face organ similarity), and the similarity of the pair having the highest similarity (pair of the block 18008 and block 18010) Select the degree.

Similarly, the similarity (facial organ similarity) between the block 18004 of the high-resolution face image 18003 in FIG. 13C and the block 18011 of the registered image 18009,
Similarity between the block 18012 of the high-resolution face image 18007 and the block 18011 of the registered image 18009 (face organ similarity), and the similarity between the pair having the highest similarity (pair of the block 18004 and the block 18011) Select the degree.
Then, the selected similarity (similarity of a pair of block 18008 and block 18010, similarity of a pair of block 18004 and block 18011) is integrated.

The pair similarity between the block 18008 of the high-resolution face image 18007 and the block 18010 of the registered image 18809 is selected, and similarity integration described later is performed on the selected similarity.
In step S2801, face identification is performed based on the integrated similarity.

<From feature extraction to similarity integration>
Steps S2502 to S2702 will be described in detail with reference to FIG.
First, in step S3010, facial organ features are extracted.
In step S3011, the similarity between facial organs is calculated.
In step S3012, if the facial organ similarity is not calculated for all the twice-converted high-resolution images, the process returns to step S3010, and the processes after step S3010 are repeated for the new twice-converted high-resolution images. If the facial organ similarity is calculated for all the twice-converted high-resolution images, the process proceeds to step S3013.

For example, it is assumed that from the first two-time conversion high-resolution image to the n-th two-time conversion high-resolution image are generated as the two-time conversion high-resolution image (n is an arbitrary natural number).
in this case,
Calculating a first similarity between the right eye of the first two-time converted high-resolution image and the right eye of the first registered image;
Calculating a second similarity between the right eye of the second two-time converted high-resolution image and the right eye of the first registered image;
:
Calculating the (n−1) th similarity between the right eye of the (n−1) th two-time conversion high resolution image and the right eye of the first registered image;
If the n-th similarity between the right eye of the n-th two-time conversion high-resolution image and the right eye of the first registered image is calculated, the process proceeds to step S3013.

In step S3013, the highest facial organ similarity is stored.
For example, as described above, it is assumed that from the first two-time conversion high-resolution image to the n-th two-time conversion high-resolution image are generated as the two-time conversion high resolution image (n is an arbitrary natural number).
in this case,
The highest similarity among the first similarity, the second similarity,..., The (n−1) th similarity, and the nth similarity for the right eye is stored.

In step S3014, if the maximum value of the facial organ similarity is not stored for all the facial organs, the process returns to step S3010, and the processing after step S3010 is repeated for the new facial organ.
For example, when the facial organ similarity is calculated for the right eye, left eye, and mouth, and the highest value is stored and integrated to identify the face, the highest value is stored for the right eye, but the highest value is still stored for the left eye. Assume that no value is stored.

in this case,
Calculating a first similarity between the left eye of the first two-time converted high-resolution image and the left eye of the first registered image;
Calculating a second similarity between the left eye of the second two-time converted high-resolution image and the left eye of the first registered image;
:
When the n-th similarity between the left eye of the n-th two-time conversion high-resolution image and the left eye of the first registered image is calculated, the first similarity, the second similarity,. The highest value among the (n−1) -th similarity and the n-th similarity is stored.

  Then, it returns to step S3010 and repeats the process after step S3010 regarding a mouth. The first similarity between the mouth of the first two-time converted high-resolution image and the mouth of the first registered image, ... the mouth of the n-th two-time converted high-resolution image and the first registered image The n-th similarity with the mouth is calculated, and the highest value among the first to n-th similarities is stored.

If the maximum value of the facial organ similarity is stored for all the facial organs, the process proceeds to step S3015.
In step S3015, the facial organ similarity is integrated.
For example, the highest value of the facial organ similarity for the right eye, the highest value of the facial organ similarity for the left eye, and the highest value of the facial organ similarity for the mouth are summed.

Which of the first to n-th two-time converted high-resolution images is combined with the highest value of each facial organ similarity may differ depending on the facial organ.
For example, for the right eye, the similarity between the first two-time converted high resolution image and the first registered image is the highest value,
For the left eye, the similarity between the first two-time converted high resolution image and the first registered image is the highest value,
For the mouth, the similarity between the third twice-converted high-resolution image and the first registered image can be the highest value.
Or, for the right eye, the similarity between the first two-time converted high resolution image and the first registered image is the highest value,
For the left eye, the similarity between the third two-time conversion high resolution image and the first registered image is the highest value,
For the mouth, the similarity between the fifth twice-converted high-resolution image and the first registered image can be the highest value.

In step S3016, if the facial organ similarity is not integrated for all the registered images, the process returns to step S3010, and the processes after step S3010 are repeated for the new registered image.
When the facial organ similarities are integrated for all the facial organs for all the registered images and the combinations with all the converted images, the process proceeds to step S2801, and face identification is executed.

  As described above, in the third embodiment, a plurality of high-resolution face images are generated by high-resolution processing, and by using an area having the highest similarity among them, only an unbroken area is obtained. Use face authentication. As a result, since accurate alignment is difficult, even when a phenomenon in which the hallucination technique is applied and a part of the image breaks down occurs, highly accurate face authentication can be performed.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 110 Image acquisition part 120 Face position detection part 130 Face organ position detection part 140 Face organ position setting part 150 Feature extraction part 160 Face image registration part 170 Face identification part

Claims (7)

Registration means for storing a plurality of registered images in association with the facial organ positions of each facial organ shown in the registered image;
Comparing the entire face area of the input image and the entire face area of each registered image of the plurality of registered images to calculate a face similarity between the input image and the registered image, and based on the face similarity Searching for the registered image corresponding to the input image, and setting the face organ position stored in association with the corresponding registered image as the face organ position of the input image;
A feature amount of each facial organ of the input image is extracted based on the facial organ position set by the setting means, and each registered image based on the facial organ position stored by the registration means Feature extraction means for extracting feature quantities of facial organs;
A facial organ similarity between the facial organ feature quantity of the input image extracted by the feature extraction unit and the facial organ feature quantity of each registered image is calculated, and based on the calculated facial organ similarity degree Face identifying means for identifying whether the face shown in the input image and the face shown in each of the registered images are the faces of the same person;
An image recognition apparatus comprising: The setting means includes
A plurality of converted images having at least one of a face size, a face position, and a face orientation are generated from the input image, and each of the generated converted images and each of the plurality of registered images Calculate facial similarity,
Search for a corresponding pair of the converted image and the registered image based on the face similarity,
The face organ position stored in association with the paired registered image is set as the face organ position of the paired converted image,
The feature extraction means includes
Based on the facial organ position set by the setting means, extract feature quantities of each facial organ of the paired conversion image,
Based on the facial organ position stored by the registration means, the feature amount of each facial organ of each registered image is extracted,
The face identification means includes
Calculating a facial organ similarity between the feature amount of each facial organ of the converted image extracted by the feature extraction unit and the feature amount of each facial organ of the registered image extracted by the feature extraction unit;
2. The image according to claim 1, wherein the face in the input image and the face in the registered images are identified as the face of the same person based on the calculated facial organ similarity. Recognition device.   The image recognition apparatus according to claim 1, wherein the input image has a low resolution, and the registered image has a high resolution. Further comprising a generating means for generating a high-resolution face image from the paired converted images,
The feature extraction means includes
Based on the facial organ position set by the setting means, extract feature quantities of each facial organ of the high-resolution facial image,
Based on the facial organ position stored by the registration means, the feature amount of each facial organ of each registered image is extracted,
The face identification means includes
Calculating a facial organ similarity between the facial feature of each high resolution facial image and the facial feature of each registered image;
4. The method according to claim 2, wherein the face in the input image and the face in each of the registered images are identified based on the calculated face organ similarity. Image recognition device. The generating means includes
Generating a plurality of twice-converted images in which at least one of a face size, a face position, and a face orientation is different from the converted image having the highest face similarity,
Generating a high-resolution face image from each twice-converted image of the plurality of twice-converted images;
The feature extraction means includes
Based on the facial organ position set by the setting means, extract feature quantities of each facial organ of the high-resolution facial image,
Based on the facial organ position stored by the registration means, the feature amount of each facial organ of each registered image is extracted,
The face identification means includes
Calculating a facial organ similarity between the facial feature of each high resolution facial image and the facial feature of each registered image;
A facial organ similarity for the first facial organ calculated using at least the first high-resolution facial image;
Based on the integrated similarity obtained by integrating the facial organ similarity for the second facial organ calculated using the second high-resolution facial image, the face reflected in the input image and the face The image recognition apparatus according to claim 4, wherein a face in each registered image is identified as a face of the same person. A registration step of storing a plurality of registered images in association with the facial organ positions of each facial organ shown in the registered image;
Comparing the entire face area of the input image and the entire face area of each registered image of the plurality of registered images to calculate a face similarity between the input image and the registered image, and based on the face similarity Searching for the registered image corresponding to the input image, and setting the facial organ position stored in association with the corresponding registered image as the facial organ position of the input image;
A feature amount of each facial organ of the input image is extracted based on the facial organ position set in the setting step, and each registered image based on the facial organ position stored in the registration step A feature extraction step for extracting feature quantities of facial organs;
Calculating a facial organ similarity between the facial organ feature quantity of the input image extracted in the feature extraction step and the facial organ feature quantity of each registered image, and based on the calculated facial organ similarity degree And a face identification step of identifying whether the face shown in the input image is the same person's face as the face shown in each of the registered images. The program for functioning a computer as each means of the image recognition apparatus of any one of Claims 1 thru | or 5.