JP5375988B2 - Image signal reproducing apparatus, image signal reproducing method, imaging apparatus, and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image signal playback device and an image signal playback method, capable of easily identify a prescribed person out of a plurality of persons displayed on a monitor. <P>SOLUTION: An image signal playback device 600 comprises: a feature amount storage part 160 to memorize a feature amount of a face of a specific person; an image acquisition part 604 to acquire an image signal; a face extraction part 180 to extract one or a plurality of faces within an image frame of the image signal; a similarity calculation part 182 to calculate a feature amount of extracted one or a plurality of faces and to calculate a similarity to the feature amount of the face of the specific person; a similarity comparator 184 to compare the similarity with a prescribed threshold; an image superimposition part 190 to superimpose an index in a mode to specify a face at a face or its periphery having a similarity more than or equal to the prescribed threshold within an image frames; a monitor 622; and an image output unit 130 to output onto the monitor 622 an image signal superimposed with the index. <P>COPYRIGHT: (C)2012,JPO&amp;INPIT

Description

  The present invention relates to an image signal reproducing apparatus and an image signal reproducing method for reproducing a moving image and a still image including a human image.

In an imaging device such as a video camera, autofocus (A
F) A function, an auto exposure (AE) function that adjusts the exposure according to the shutter speed, and the size of the aperture are installed, and the difference in the imaging environment can be determined by using such imaging control (focus adjustment, exposure adjustment, etc.). It is possible to correct and image the subject beautifully. In addition, since focus adjustment and exposure adjustment are performed only on a subject located at an arbitrary distance, a subject desired by the photographer, for example, a person, is usually selected as the target. There are various techniques for specifying a subject to be subjected to such focus adjustment or exposure adjustment.

As a method for identifying a person on an image, face information is extracted from a captured image signal and compared with registered face information registered in advance to determine whether the extracted face matches the registered face information. A technique is known (for example, Patent Document 1). Using such technology, entrance authentication to facilities has been implemented.

In addition, the above face authentication technology is applied to an electronic camera, face recognition is performed based on the face information of a person registered in advance, the person who is the main subject is designated, and information such as the facial expression of the main subject is obtained. A technique for utilizing and performing various processes has also been devised (for example, Patent Document 2).

JP 2007-334623 A JP 2007-282119 A

For example, when imaging a children's athletic meet, it may not be easy to find their children among a plurality of children who are competing at relatively distant places. In particular, in imaging devices that require reduction in size and weight, liquid crystal monitors and viewfinders are also downsized, and very rough images are output compared to actual imaging pixels. It is difficult to find your child from multiple children.

Although the zoom function may be used to recognize a child's face, zooming during imaging requires a certain level of skill and often captures unintended subjects. In some cases, I missed the imaging opportunity, and sometimes felt uncomfortable feelings such as observing an irrelevant person with zoom during playback.

SUMMARY OF THE INVENTION In view of the above problems, the present invention has an object to provide an image signal reproduction device and an image signal reproduction method capable of easily specifying a predetermined person from a plurality of persons displayed on a monitor. Yes.

In order to solve the above-mentioned problems , the present invention comprises means described in 1) to 4) below.
That is,
1) a feature amount storage unit for storing a feature amount of a face of a specific person;
A face extraction unit for extracting one or more faces in the screen in the acquired image signal;
A degree-of-similarity calculating unit that calculates one or a plurality of extracted facial feature quantities and calculates a similarity degree to the facial feature quantity of the specific person;
A similarity comparison unit for comparing the similarity with a predetermined threshold;
An image superimposing unit that superimposes an index in a manner that identifies the face on or around the face having a similarity equal to or greater than the predetermined threshold in the screen;
An image output unit that outputs an image signal on which the index is superimposed ;
An identity determination unit that determines the identity of one or more faces on a predetermined number of previous screens and one or more faces on the current screen;
The similarity calculation unit calculates the similarity of one or more faces on the current screen using also the similarity of one or more faces on a predetermined number of previous screens determined to be identical. ,
The image signal reproducing apparatus, wherein the index changes a color, a pattern, or a shape according to the similarity of the one or more faces.
2) Extract one or more faces in the screen from the acquired image signal,
Calculating the extracted feature quantity of one or more faces, calculating the degree of similarity with the acquired facial feature quantity of the specific person,
Determining the identity of one or more faces on the previous predetermined number of screens and one or more faces on the current screen,
One or more of the current screen using the similarity between the feature quantity of one or more faces on a predetermined number of previous screens determined to be identical and the acquired face feature quantity of a specific person The degree of similarity between the facial feature quantity of the particular person and the facial feature quantity of the specific person is calculated, and the similarity degree between the one or more facial feature quantities of the current screen and the facial feature quantity of the specific person And a predetermined threshold value,
In the screen, an index is displayed by changing the color, pattern, or shape in accordance with the similarity of the one or more faces in a manner that identifies the face on or around the face having the similarity equal to or greater than the predetermined threshold. An image signal reproduction method comprising: superposing and outputting an image signal on which the index is superimposed.
3) a feature amount storage unit for storing a feature amount of a face of a specific person;
An image sensor that photoelectrically converts a subject image to generate an image signal;
A face extraction unit for extracting one or more faces in the screen in the image signal;
A degree-of-similarity calculating unit that calculates one or a plurality of extracted facial feature quantities and calculates a similarity degree to the facial feature quantity of the specific person;
A similarity comparison unit for comparing the similarity with a predetermined threshold;
An image superimposing unit that superimposes an index in a manner that identifies the face on or around the face having a similarity equal to or greater than the predetermined threshold in the screen;
A monitor,
An image output unit that outputs an image signal superimposed with the index to the monitor;
An identity determination unit that determines the identity of one or more faces on a predetermined number of previous screens and one or more faces on the current screen;
The similarity calculation unit calculates the similarity of one or more faces on the current screen using also the similarity of one or more faces on a predetermined number of previous screens determined to be identical. ,
The imaging apparatus is characterized in that a color, a pattern, or a shape is changed according to the similarity of the one or more faces.
4) Acquire facial features of a specific person,
Photoelectrically convert the subject image to generate an image signal,
Extracting one or more faces in the screen in the image signal;
Calculating the extracted feature quantity of one or more faces,
Determining the identity of one or more faces on the previous predetermined number of screens and one or more faces on the current screen,
The similarity between the feature quantity of one or more faces and the feature quantity of the face of the specific person on the predetermined number of screens that have been determined to be identical is also used to determine one or more of the current screen. Calculating the degree of similarity between the feature amount of the face and the feature amount of the face of the specific person,
Comparing the similarity between the feature quantity of one or more faces of the current screen and the feature quantity of the face of the specific person and a predetermined threshold;
In the screen, an index is displayed by changing the color, pattern, or shape in accordance with the similarity of the one or more faces in a manner that identifies the face on or around the face having the similarity equal to or greater than the predetermined threshold. An imaging method characterized by superimposing.

  In the present invention, a predetermined person can be easily identified from a plurality of persons displayed on the monitor, and the user can capture or reproduce a truly desired image based on at least the identification information. Become.

1 is an external view illustrating an example of an imaging apparatus according to a first embodiment. It is a block diagram which shows the structure of the imaging device in 1st Embodiment. It is explanatory drawing for demonstrating the parameter | index table in 1st Embodiment. It is explanatory drawing explaining the superimposition of the parameter | index for person discrimination | determination in the liquid crystal monitor in 1st Embodiment. It is a flowchart explaining the flow of a process of the imaging method in 1st Embodiment. It is a block diagram which shows the structure of the imaging device in 2nd Embodiment. It is explanatory drawing explaining the superimposition of the parameter | index in 2nd Embodiment. It is the flowchart explaining the flow of the process of the imaging method in 2nd Embodiment. It is the flowchart which showed the flow of the specific process of the maximum similarity calculation step in FIG. 8 in 2nd Embodiment. It is a block diagram which shows the structure of the image signal reproducing | regenerating apparatus in 3rd Embodiment. 12 is a flowchart illustrating a flow of an image signal reproduction method according to the third embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials, and other specific numerical values shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same function and configuration are denoted by the same reference numerals, and redundant description is omitted, and elements not directly related to the present invention are not illustrated. To do.

In recent imaging apparatuses that are desired to be small and light, liquid crystal monitors and viewfinders are also miniaturized, and it is difficult to specify a person in the image. In such a monitor, it is difficult to identify and track a person to be imaged such as his child from a plurality of distant subjects. In addition, in the existing image signal reproduction device, it may take time to find a desired person each time the screen is switched, or it may be difficult to specify the target when using the partial zoom function in the screen. there were.

In the following embodiments, an imaging apparatus, an imaging method, an image signal reproduction apparatus, and an image signal reproduction that can easily identify a specific person even if a plurality of persons are displayed small on the monitor as described above. It aims to provide a method. Hereinafter, the configuration of the imaging apparatus of the embodiment and the imaging method using the imaging apparatus will be described, and then the configuration of the image signal playback apparatus and the image signal playback method using the image signal playback apparatus will be described.

(First embodiment: imaging apparatus 100)
FIG. 1 is an external view illustrating an example of an imaging apparatus 100 according to the first embodiment. The imaging apparatus 100 has portability and includes a main body 102, an imaging lens 104, an operation unit 106, and a liquid crystal monitor 108 as a monitor.

The main body 102 records the image data captured through the imaging lens 104 so that it can be viewed again, and the recording timing and angle of view are adjusted in accordance with user input to the operation unit 106. In addition, it accepts an input of switching between imaging modes such as outdoors, indoors, and night views from the user.
Further, the user can view the image data actually recorded with reference to the image displayed on the liquid crystal monitor 108, and can capture the subject at a desired position and occupied area. In the present embodiment, a liquid crystal monitor has been exemplified as a monitor (display).
Not only LCD monitors, but also organic EL (Electro Luminescence), LED (Light Emitting Dio)
de) It may be composed of a display or the like.

FIG. 2 is a block diagram illustrating a configuration of the imaging apparatus 100 according to the first embodiment. The imaging device 100 includes an imaging unit 120, a signal processing unit 122, an image storage unit 124, an image processing unit 126, a recording I / F unit 128, an image output unit 130, a memory device 132, and an imaging control unit. 134. The image storage unit 124, the image processing unit 126, the recording I / O
The F unit 128, the image output unit 130, and the imaging control unit 134 are connected via a system bus 136.

The imaging unit 120 captures a subject through the imaging lens 104 and generates image data. Specifically, the imaging unit 120 photoelectrically irradiates light such as an object image incident through the IR cut filter 140 that blocks near-infrared light, a focus lens 142 used for focus adjustment, a stop 144 used for exposure adjustment, and the imaging lens 104. CCD (Charge Coupled) that converts and generates image signal
Devices) 146, an amplifier 148 that amplifies an image signal from the image sensor 146, and an A / D that converts the amplified image signal into digital image data
A drive control unit 1 that controls the drive of the converter 150, the focus lens 142, and the diaphragm 144.
52 and its drive circuit 154.

The signal processing unit 122 performs signal processing such as forming a luminance signal and a color signal on the input signal to form a color video signal, and transmits the color video signal to the image storage unit 124. In addition, the control signal is output to the drive control unit 152 by obtaining the average luminance of the screen.

The image storage unit 124 includes a buffer memory such as an SDRAM (Synchronous-DRAM), temporarily stores image data, and allows the image processing unit 126 and the like to refer to the image data.

The image processing unit 126 converts the image data from the image storage unit 124 into MPEG-2, MPEG-
4. Generate data for recording by compressing in a format such as MPEG-4 / AVC. Further, the image processing unit 126 reduces the image data according to an instruction from the imaging control unit 134 and generates a display image (view image). On the other hand, when reproducing the recording data, the image processing unit 126 executes a process of decompressing and restoring the compressed storage data.

The recording I / F unit 128 encodes the image data through an encoding process to generate a recording signal (data stream), and records the storage signal on an arbitrary recording medium 156. As the arbitrary recording medium 156, a medium that does not require a power source such as a DVD or a BD, or a medium that requires a power source such as a RAM, an EEPROM, a nonvolatile RAM, a flash memory, or an HDD can be applied.
Also, a separate recording medium that can be connected from the outside can be used.

The memory device 132 stores a program processed by the imaging control unit 134. The memory device 132 also functions as the feature amount storage unit 160 and has an index table 162. The feature amount storage unit 160 stores a feature amount related to the face of a specific person imaged in advance by the user. The index table 162 will be described in detail later. Hereinafter, “face” simply refers to the entire face that can be cut out from the image signal, “face position” indicates the relative position of any point on the screen, and “face occupation area” indicates the face Indicates the area that occupies the screen.

The imaging control unit 134 manages and controls the entire imaging apparatus 100 using a semiconductor integrated circuit,
Various calculations necessary for imaging and the like are executed. The imaging control unit 134 also includes a face extraction unit 180.
Also, it functions as a similarity calculation unit 182, a similarity comparison unit 184, a coordinate specification unit 186, and a timer unit 188.

The face extraction unit 180 extracts a face from the captured image data acquired by the imaging unit 120. Then, the face extraction unit 180 derives coordinates, occupied area, and feature points in the screen of the face from the extracted face, and stores information on the coordinates and occupied area in the face screen in the memory device 132, The points are transmitted to the similarity calculation unit 182. As an extraction method, for example, JP-A-2001-16.
The occupied area of the face is extracted by the feature point extraction process described in Japanese Patent No. 573. Such feature point extraction processing is not limited to being performed by the face extraction unit 180, and may be performed by a similarity calculation unit 182 described later.

The similarity calculation unit 182 first calculates the face feature amount from the face and the face feature points extracted by the face extraction unit 180. The feature amount is information that characterizes the face, and features of the face (eyes, mouth, nose,
The face is identified using the relative position of the feature portion such as the ear), the distance between the feature points, the size of the feature portion, the outline of the face, the skin color, the hair color, the amount of hair, and the like. Next, the similarity calculation unit 182 reads the feature amount of the face of a specific person designated by the user from the feature amount storage unit 160 and compares it with one or more faces for which the feature amount has been calculated, The similarity between the person's face and the imaged person's face is obtained. The similarity indicates an analogy between two face images, and is represented by a value of 0 to 100, for example.
If it is, it can be judged that it is another person, and if it is 100, it is the same person.

FIG. 3 is an explanatory diagram for explaining the index table 162. The indicator table 162 is
The similarity is associated with the color of an index (here, an arrow) indicating a specific person. For example, when the similarity calculated by the similarity calculation unit 182 is 70 to 80, it is shown in white, when it is 81 to 90, it is shown in blue, and when it is 91 to 100, it is shown in red. Here, a predetermined threshold described later is set to 7
Since 0 is set, when the similarity is less than 70, it is assumed that no other color is associated with the other person. In the present embodiment, an arrow is used as an index, but various specific displays such as a mark other than an arrow such as a pointing mark and a line along the contour of the face can be used.

The similarity comparison unit 184 compares the similarity calculated by the similarity calculation unit 182 with a predetermined threshold set in advance, and if the similarity is larger than the predetermined threshold, the similarity determination unit 186 determines the similarity of the face.
To communicate. In the present embodiment, the predetermined threshold value is set to 70, but the present invention is not limited to this value, and the user can arbitrarily set a numerical value.

The coordinate specifying unit 186 acquires color data corresponding to the degree of similarity larger than the predetermined threshold received from the similarity comparison unit 184 with reference to the index table 162, and the face extraction unit 180 stores the color data in the memory device 132. With reference to the data of the position and occupied area of each face, the coordinate data of the area excluding all the faces of the imaged person in the vicinity of the face of the imaged person is transmitted to the image superimposing unit 190.

The image output unit 130 includes an image superimposing unit 190 and a D / A converter 192. The image superimposing unit 190 superimposes, for example, an arrow (index) colored in the color data acquired from the similarity calculating unit 182 on the coordinates acquired from the coordinate specifying unit 186 of the image signal. With this configuration, the user can intuitively and surely select an appropriate subject from a plurality of face candidates. In addition, even if there are multiple faces that are very similar to a specific person, and the discrimination cannot be made, the user can process all of the colored faces on the screen, etc.
Imaging can be performed more reliably without missing a desired person.

In the present embodiment, the color of the arrow indicates the degree of similarity, but the pattern or shape of the arrow may be changed. A plurality of indicators may be prepared and an indicator corresponding to the similarity may be displayed, or the similarity may be displayed as a number as it is. Also, the degree of similarity may be indicated by changing the color or brightness of a person or the face itself.

Further, since the coordinate data acquired from the similarity calculation unit 182 is a value calculated so that the arrow does not overlap the face of the person on the screen, the image superimposing unit 190 sets the arrow to all the persons on the screen. Superimpose in a position that does not overlap the face. With this configuration, it is possible to avoid a situation in which the arrow overlaps the face of another person and the face of the other person cannot be visually recognized, and stable imaging can be performed in a state where all faces can be recognized. .

The D / A converter 192 processes the digital image data acquired from the image storage unit 124 into a viewable image signal and outputs the processed image signal to the liquid crystal monitor 108. The image data includes an image signal such as an arrow superimposed by the image superimposing unit 190. An imager (user) can identify an imaging target while visually recognizing the video on the liquid crystal monitor 108. Here, the output destination of the image signal is the liquid crystal monitor 108, but the image output unit 130 has a video terminal for outputting an image to the outside, so that it can be used in various image display devices such as a separate monitor. It is also possible to connect.

  Hereinafter, a specific processing operation of the imaging apparatus 100 will be described.

FIG. 4 is an explanatory diagram for explaining superimposition of indices for person discrimination in the liquid crystal monitor 108. Here, a soccer game is assumed as an imaging target, and four soccer players 202A, 202B, and 202 are displayed on the liquid crystal monitor 108 that displays the captured image.
C and 202D are projected. Although the actually captured image signal has a high resolution, since the liquid crystal monitor 108 is small here, the person 202A, 202B, 202C, 20
Assume that 2D identification is difficult.

The user registers in advance an image of a specific person 202C to be imaged in the imaging apparatus 100, and searches for the person 202C through the operation unit 106 before imaging.
Give instructions to. The imaging apparatus 100 converts an image captured by the imaging unit 120 into a signal processing unit 122.
Is stored in the buffer memory of the image storage unit 124 and transmitted to the image output unit 130 and the face extraction unit 180.

The face extraction unit 180 performs the person 202A, 202B, 202C, 202 by the feature point extraction process.
The four faces of D are extracted, the face position, occupied area, and feature point are derived, information on the face position and occupied area is stored in the memory device 132, and the similarity calculation unit 182 calculates the face and feature point. To communicate.

The similarity calculation unit 182 displays the feature amount of a specific person designated by the user as the feature amount storage unit 160.
And the feature amount is calculated based on the face and feature points transmitted from the face extraction unit 180, and the feature amount of each person in the calculation result is compared with the feature amount of the specific person, so that the specific person The similarity indicating the analogy with the face image is calculated and stored in the memory device 132 for each face. Here, the similarity of each person is 85 for the person 202A, 65 for the person 202B, 95 for the person 202C, and 20 for the person 20
Suppose 2D is 45.

Next, the similarity comparison unit 184 compares the predetermined threshold value set in advance with the similarity for each face. The predetermined threshold value may be changed according to a user input, or may be a fixed value. Here, it is assumed that the predetermined threshold is 70.

First, for the face of the person 202A, the calculated similarity 85 is compared with the predetermined threshold 70, and it is determined that the similarity is higher. Then, the coordinate specifying unit 186 refers to the index table 162, the blue color data corresponding to the similarity 85, and the person 202A, 20 near the face of the person 202A.
2B, 202C, and 202D, the coordinate data of the area excluding the occupied area of the face is used as the image superimposing unit 190.
Send to. In the image superimposing unit 190, a blue arrow graphic is superimposed on the image data from the transmitted color data and coordinate data and displayed on the liquid crystal monitor 108 (arrow 300A).

Similarly, the similarity comparison unit 184 compares the face similarity 65 of the person 202B with the predetermined threshold 70 and determines that the similarity is smaller, and does not process the person 202B and proceeds to the next face processing. To do.

Similarly, the similarity degree 95 of the face of the person 202C is compared with the predetermined threshold 70, and it is determined that the similarity degree is larger. The index table 162 is referred to, and the red color data corresponding to the similarity degree 95 and the C face The coordinate data of the area excluding the occupied areas of the faces of the persons 202A, 202B, 202C, and 202D in the vicinity of is transmitted to the image superimposing unit 190. The image superimposing unit 190 superimposes a red arrow graphic on the image data from the transmitted color data and coordinate data and displays the image on the liquid crystal monitor 108 (
Arrow 300B).

Similarly, the similarity 45 of the face of the person 202D is compared with a predetermined threshold 70, and it is determined that the similarity is smaller, and no special processing is performed on the person 202D.

The above processing may be performed every frame during imaging. The imaging control unit 134 may perform processing using a pulse signal synchronized with a frame as a trigger, or may perform processing based on a count value of a timer unit 188 (see FIG. 2) that counts the frame period. .

(Imaging method)
FIG. 5 is a flowchart illustrating the processing flow of the imaging method according to the first embodiment.

When one or more specific persons 202 are designated from images of the person 202 registered in advance by the user (S300) and imaging is started (S302), the image signal acquired by the imaging unit 120 is the signal processing unit 122, The information is transmitted to the face extraction unit 180 of the imaging control unit 134 via the image storage unit 124, and the face extraction unit 180 extracts a face from the image data (S304).

If no face is extracted (NO in S306), the process ends. When at least one face is extracted (YES in S306), the following processing is repeatedly executed for all the extracted faces.

First, the similarity calculation unit 182 calculates a feature amount from the extracted face (S308). Less than,
In the case where there are a plurality of specific persons 202 designated by the user, if only one person is designated, the feature quantity is stored in the feature quantity storage unit 16 for that particular person 202.
The similarity is obtained from 0, and the similarity is calculated from the feature amounts of both (S310).

If the similarity is larger than a predetermined threshold value set in advance (YES in S312), the index table 1
62, the color data corresponding to the similarity is acquired (S314), and the coordinates where the other faces do not overlap are calculated in the vicinity of the coordinates of the face (S316). The image superimposing unit 190 is connected to the liquid crystal monitor 10.
The arrow of the color designated by the color data is superimposed on the calculated coordinate position in the screen displayed in 8 (S318).

If the calculated similarity is equal to or less than the predetermined threshold (NO in S312), no special process is performed on the face.

There are a plurality of specific persons 202 designated by the user, and the above-described similarity calculation is still performed (S310).
If the specific person 202 that has not performed the process from the drawing to the arrow drawing (S318) remains (YES in S320), the steps after the similarity calculation (S310) for one of the remaining specific persons 202 are performed. Execute. This process is performed by a specific person 20 specified by the user.
Repeat until 2 is over.

When the number of unprocessed specific persons is zero (NO in S320), it is determined whether or not there is a face that has not been subjected to the processing below the feature amount calculation (S308) (S322). Repeat the process. When there is no unprocessed face (NO in S322), the imaging method ends. The above processing is executed for each frame.

According to the present embodiment, when the user takes an image, the liquid crystal monitor 10 of the image pickup apparatus displays a predetermined person.
8 can be easily found and confirmed, and an appropriate subject can be imaged without feeling uncomfortable, such as zooming and imaging a person who is not supposed to be mistaken. Further, even if a plurality of persons with similar faces are displayed on the liquid crystal monitor 108, the degree of similarity can be determined by color, so that a specific person can be determined more reliably.

(Second Embodiment: Imaging Device 400)
In the first embodiment, the similarity is calculated for each frame, and the index is superimposed on the image of the liquid crystal monitor 108 based on the result. In this method, the degree of similarity may change from frame to frame according to changes in face orientation and facial expression, and the index (arrow, color, number, etc.) corresponding to the degree of change changes with the change, and the user is troublesome. There is a possibility of thinking.

In the second embodiment, not only the current frame but also a plurality of immediately preceding predetermined frames are referred to calculate a stable similarity, suppress changes in the index to be displayed, and annoying the user. Can be reduced.

FIG. 6 is a block diagram illustrating a configuration of the imaging apparatus 400 according to the second embodiment. The imaging device 100 includes an imaging unit 120, a signal processing unit 122, an image storage unit 124, an image processing unit 126, a recording I / F unit 128, an image output unit 130, a memory device 132, and an imaging control unit. 402. The image storage unit 124, the image processing unit 126, the recording I / O
The F unit 128, the image output unit 130, and the imaging control unit 402 are connected via a system bus 136.

The imaging unit 120, signal processing unit 122, image storage unit 124, image processing unit 126, recording I
The / F unit 128, the image output unit 130, and the memory device 132 have substantially the same functions as the constituent elements described in the first embodiment, and thus redundant description is omitted. Here, imaging control with different configurations is performed. The unit 402 will be mainly described.

Unlike the imaging apparatus 100, the imaging apparatus 400 is provided with a distance calculation unit 404 and an identity determination unit 406 in the imaging control unit 402. The distance calculation unit 404 reads information on the positions (specifically expressed in coordinates) of the faces of the previous frame and the current frame in the screen and the occupied area information from the memory device 132, and the face position and the previous frame of the current frame. The distances from all face positions are calculated. For example, with the lower left corner of the liquid crystal monitor 108 as the origin, the face is at the coordinates of X1 pixels in the horizontal direction from the origin, and the coordinates of Y1 pixels in the vertical direction, and is also at the coordinates of X2 pixels in the horizontal direction and Y2 pixels from the origin The face distance D is D = √ ((X1−X2) ² + (Y1−Y2) ² ).
Is calculated by

The identity determination unit 406 determines the identity of all faces included in the frame A and all faces included in the frame B with respect to, for example, the frame A and the frame B (in order of the imaging time) in which the imaging time changes. . The identity indicates whether the faces included in the two frames with different imaging times can be regarded as the same person.

In the present embodiment, the identity is also determined based on the distance between one or more faces between faces in the screen and the occupied area in the screen between the frames of one or more faces. Here, the former is the identity of the distance, and the latter is the identity of the area.

The identity of the distance is determined from the distance in the screen of the face between the frames calculated by the distance calculation unit 404, and the distance between a certain face included in the frame B and all the faces in the frame A is
When it is larger than the first predetermined value set in advance, it is determined that the distances are not identical, and the faces are not the same or do not exist in the frame A. The existing similarity regarding the face is not referred to when calculating the similarity of the person in the frame B.

When the distance in the screen of the face between the frames is equal to or less than the first predetermined value, the identity determination unit 406 determines the identity of the areas of the two faces. The identity of the area is determined from the absolute value of the difference in the occupied area of each person's face in the frame A and frame B screens derived by the face extraction unit 180. If the absolute value of the difference in the area occupied by the face is greater than a preset second predetermined value, it is determined that there is no area identity, and the existing similarity for that face is calculated by calculating the similarity of the person in frame B. It is not referred to when. If the absolute value of the difference in the occupied area of the face is smaller than a second predetermined value set in advance, it is determined that there is an area identity, the two faces are the faces of the same person, and the existing similarities related to that face The degree is referred to when the similarity of the person in frame B is calculated.

By using the same area, for example, the subject in the frame A can be captured by the imaging device 400.
The image pickup device 400 is located far away from the camera and is the same position in the frame B.
Even when another subject appears at a position close to, since the occupied area of the face in the screen differs according to the perspective method, the identity determination unit 406 determines that there is no identity and can correctly determine that the person is another person.

Thus, when the distance between one or more faces in the screen between frames is smaller than a predetermined value and the absolute value of the difference in occupied area in the screen is smaller than a predetermined value, the faces between frames are It can be determined that they are the same person. With this configuration, faces that can be regarded as the same between frames can be reliably extracted, and as a result, stable imaging can be performed.

Further, the identity determination unit 406 may determine the identity taking into consideration not only the distance and the occupied area of the face in the screen but also the amount of face light and the feature amount between frames. With this configuration, it is possible to improve the performance of tracking a person.

FIG. 7 is an explanatory diagram for explaining the superimposition of indices in the second embodiment. Fig. 7 (a
) Is an image taken at a certain time. FIG. 7B is an image captured at a time one frame after FIG. 7A. Here, the procedure when calculating the similarity for each person 202 imaged in the frame B with the image in FIG. 7A as the frame A and the image in FIG. 7B as the frame B will be described. . However, in order to clearly indicate the distance, the time of one frame is long.

The distance calculation unit 404 includes the persons 202E, 202F, 202G, and 20 included in the frame B.
For each of 2H and 202I, the persons 202A, 202B, and 20 included in the frame A
The distance in the screen with each of 2C and 202D is calculated.

Subsequently, the identity determination unit 406 compares each distance in the screen calculated by the distance calculation unit 404 with the first predetermined value to determine the degree of distance identity. In the case of the face of the person 202E, the person 202A
, 202B, 202C, and 202D, and the distance is greater than the first predetermined value, so it is determined that the distances are not identical. Therefore, it is determined that there is no person who is the same as the person 202E in the frame A, and the similarity of the frame A is not referred to when calculating the similarity of the person 202E.

Similarly, the position of the face of the person 202F in frame B and the persons 202A, 202B in frame A,
The distances in the screen with the face positions of 202C and 202D are calculated, and each is compared with a predetermined distance. The distance in the screen between the position of the face of the person 202F and the position of the face of the person 202A in frame A is smaller than the first predetermined value. Accordingly, the face hardly moves between the frames A and B, and it is determined that the distance is the same.

Next, the absolute value of the difference between the occupied area of the face in the screen of the person 202F and the occupied area of the face in the screen of the person 202A is calculated, and the identity of the area is determined. Since the absolute value of the area difference is smaller than the second predetermined value, it is determined that there is an area identity, and the face of the person 202F and the person 202A
Can be determined to be the faces of the same person.

Here, since both the conditions of the identity of the distance and the identity of the area are satisfied, the identity determination unit 406 determines that the person 202F of the frame B and the person 202A of the frame A are the same person, The degree of similarity of the person 202A is stored in the memory device 132 in association with 202F.

Similarly, the identity determination unit 406 includes the face of the person 202G, the face of the person 202B, and the person 202H.
And the face of the person 202C and the face of the person 202I and the face of the person 202D are determined to be the faces of the same person, and the respective similarities are associated and stored in the memory device 132.

The identity determination unit 406 compares two consecutive frames from the current frame to, for example, 29 frames before, and repeats the same processing sequentially from the newest imaging time to the oldest. Here, in order to simplify the description, processing prior to two frames is omitted.

For example, when there is a frame C in which there is no person 202 that is determined to be the same as the person 202 included in the frame B (current frame) during the process, the person 202 does not exist before the frame C. Therefore, the distance calculation process by the distance calculation unit 404 and the identity determination process by the identity determination unit 406 are not performed for the person 202.

Accordingly, the similarity calculation unit 182 calculates the similarity of the person 202E in the frame B when 2
Refers to the similarity up to 9 frames before, but there was no person judged to be the same person,
The similarity is a value calculated from the feature amount of only the person 202E in the frame B.

On the other hand, the similarity of the person 202F is the same in the similarity of the person 202A registered in the memory device 132 and in the previous frame because there is the person 202A who is determined to be the same person in the frame A and the previous frame. It is calculated with reference to all similarities of the person 202 determined to be a person. The similarity calculation unit 182 calculates the maximum similarity (maximum similarity) from all the similarities referred to, and sets the maximum similarity as the similarity of the person 202F in frame B.

The similarity comparison unit 184 refers to the maximum similarity calculated by the similarity calculation unit 182 and compares it with a predetermined threshold. In the case of the face of the person 202F, the maximum similarity is larger than a predetermined threshold value, so the coordinate specifying unit 186 uses the color data (blue) corresponding to the maximum similarity and the persons 202E, 202F, 202F near the face of the person 202F. The coordinate data of the areas excluding the occupied areas of the faces 202G, 202H, and 202I are transmitted to the image superimposing unit 190. The image superimposing unit 190 superimposes a blue arrow figure on the image data from the received color data and coordinate data and displays the image on the liquid crystal monitor (
Arrow 300A).

Similarly, the person 202G refers to the similarity of the face of the person 202B of the frame A and the person 202 determined to be the same person in the previous frame, and uses the maximum value as the similarity of the frame B. In the case of a face, since the maximum similarity is smaller than a predetermined threshold value, no special processing is performed. Since the maximum similarity is 95 for the face of the person 202H, a red arrow graphic is superimposed on the image data and displayed on the liquid crystal monitor (arrow 300B). Person 202I is person 2
Similar to 02G, since the degree of similarity is low even when referring to a past frame, no special processing is performed for the face of the person 202I.

In this way, not only the current frame but also a plurality of frames from the previous 29 frames are referred to calculate the similarity of the face in the screen, and the accuracy of calculating the similarity of the subject is improved. With such a configuration, it is possible to avoid a phenomenon in which the original similarity suddenly falls down due to a fine operation such as when the subject changes the direction of the face, and the index changes complicatedly. It is possible to capture an image of a desired subject appropriately and reliably with a stable index.

Further, when calculating the similarity, it is determined that there is the sameness of the distance and the sameness of the area.
By calculating the maximum value of the similarity of one or a plurality of faces in the nine frames and the similarity of the face in the current frame as the similarity, the similarity may be instantaneously reduced due to the movement of the subject or external factors. Even in such a case, it is possible to maintain the maximum similarity within a frame from a predetermined number before, and to eliminate the influence of the similarity variation.

In this embodiment, the identity of the distance and the identity of the area are used to determine the identity of the person between the frames. In addition to this, the object tracking method described in the non-patent document “Journal of the Institute of Image Information and Television Engineers Vol. 62, No. 6, pp 849-855” (for example, Particl Filt
The object tracking may be determined as the same person using tracking of the object by er).

(Imaging method)
FIG. 8 is a flowchart for explaining the processing flow of the imaging method according to the second embodiment. The processes already described with reference to FIG. 5 in the first embodiment are denoted by the same reference numerals and the description thereof is omitted.

In FIG. 5, the similarity calculation step (S310) and the step of comparing the similarity with a predetermined threshold (S312) are respectively compared with the maximum similarity calculation step (S500) and the maximum similarity with a predetermined threshold in FIG. This is replaced with the step (S518).

Unlike the imaging method according to the first embodiment, the imaging method according to the second embodiment compares the maximum similarity between a plurality of immediately preceding frames and the current frame with a predetermined threshold (S518).
), And determines the face to be drawn by the arrow. In this way, it is possible to avoid a phenomenon in which the original similarity suddenly falls and the index changes complicatedly, and it is possible to capture a desired subject appropriately and reliably with a stable index through a stable similarity. Become.

FIG. 9 is a flowchart showing a specific processing flow of the maximum similarity calculation step (S500) in FIG.

Similar to the first embodiment, the feature amount storage unit 160 stores the feature amount of a specific person designated by the user.
And the similarity with the feature quantity obtained from one of the faces of the current frame is calculated (S5).
02).

Furthermore, information such as the coordinates and occupied area in the screen of each face of the previous frame is acquired from the memory device 132 (S504). The distance calculation unit 404 calculates the relative distance between the face and each face from the coordinates (S506), and determines the identity of the distance. If there is a face having the same identity (YES in S508), the identity of the area is determined from the absolute value of the difference in occupied area for all the faces having the same identity. If there is area identity (S510 YES), all faces with area identity are determined to be the same person, and the similarity of the frame immediately before the corresponding face is stored in association with the similarity of the latest frame. (S512).

There is no identity of the distance with all faces included in the previous frame (S508 NO)
), If the area occupied by all faces is not identical (NO in S510), the person does not exist in the previous frame, and it is determined that the person has appeared from the latest frame. Do not refer.

After storing the similarity of the previous frame (S512), 29
It is determined whether or not processing has been performed up to the previous frame (S514). If 29 frames have not been reached (YES in S514), information on the previous frame is acquired (S50).
4) Compared with the oldest frame out of the two frames processed up to immediately before, the degree of similarity for a maximum of 29 frames is derived.

When processing has been completed from the current frame to 29 frames before (NO in S514)
), Or when the same person does not exist in the frame and the processing is completed up to 29 frames, all the similarities registered in the memory device 132 are referred to in S514, and the maximum value of the similarities is calculated.
(Third Embodiment: Image Signal Reproducing Device 600)
FIG. 10 is a block diagram showing a configuration of an image signal reproduction device 600 according to the present embodiment. The image signal reproduction device 600 includes an operation unit 602, a memory device 132, and an image storage unit 124.
An image output unit 130, an image acquisition unit 604, an image processing unit 606, and a reproduction control unit 608.
It is comprised including. The image storage unit 124, the image acquisition unit 604, and the image processing unit 606
The reproduction control unit 608 and the image output unit 130 are connected via a system bus 136. The image storage unit 124, the image output unit 130, and the memory device 132 have substantially the same functions as those of the memory device 132 already described in the first embodiment, and thus redundant description is omitted. Here, the image signal reproduction device 600 is used. The difference in configuration will be mainly described.

  The above-described imaging apparatuses 100 and 400 display images through the liquid crystal monitor 108 that is configured integrally with the image capturing apparatus 100 and 400. However, the image signal reproduction apparatus 600 displays images through an external monitor 622 that is configured separately. To do. The operation unit 602 accepts operations such as playback, stop, fast forward, and rewind according to user input. Further, in the present embodiment, as with the imaging apparatus 100, it is also used when the user designates a specific person that the user wants to find on the screen.

The image acquisition unit 604 receives MPEG-2, MPEG from a recording medium 156 such as DVD or HD.
-4, and image data compressed in a format such as MPEG-4 / AVC. Image processing unit 6
In step 06, compressed image data is acquired from the image acquisition unit 604 and decompressed / restored.

The reproduction control unit 608 manages and controls the entire image signal reproduction apparatus 600 using a semiconductor integrated circuit, and executes various operations necessary for reproduction and the like. Further, similarly to the imaging control unit 402 of the imaging apparatus 400 in the second embodiment, the face extraction unit 180, the similarity calculation unit 182, the similarity comparison unit 184, the coordinate specification unit 186, the timer unit 188, and the distance calculation unit 402 , Functions as the identity determination unit 404.
(Image signal playback method)
FIG. 11 is a flowchart showing the flow of the image signal reproducing method in the present embodiment. One or more specific persons 202 are designated from images of the person 202 registered in advance by the user (S300), and reproduction of the image signal is started (S700). The subsequent processing flow is shown in FIG.
Is substantially the same as the flowchart showing the flow of the imaging method in the second embodiment shown in FIG.

Further, in this embodiment, as in the first and second embodiments, the identity between a plurality of immediately preceding frames is determined from the distance in the frame, the occupied area, etc., and it is determined that there is identity. In this case, using the similarities in a plurality of immediately preceding frames, for example, the similarity having the maximum value is set as the similarity of the current frame. Furthermore, according to the similarity of one or more faces,
The color, pattern, or shape is changed and superimposed on a position that does not overlap one or more faces of the person on the screen.

With existing image signal playback devices, it may take time to find the desired person each time the screen is switched, or the location where the partial zoom function in the screen is used cannot be specified, such as when the person image is small. there were. However, in the present embodiment, the image signal reproduction device 600 calculates the similarity between the facial feature quantity in the screen and the facial feature quantity of the specific person stored in advance, and uses the index when it is higher than a predetermined threshold value. Point to the face.

With such a configuration, for example, when a drama video or the like is reproduced, if the desired actor's image is registered in advance, the relative position of the eyes and mouth even when the scene is suddenly switched to a video in a dark place. Since the face is recognized based on the facial feature points such as, even in a video with poor visibility that cannot be confirmed by human eyes, the user can quickly find the actor based on the index.

In particular, when a famous director or actor appears as an unexpected extra, the director or actor can be reliably captured.

Also, even when using the partial zoom function in the screen, an index is displayed on the faces of a plurality of people in the screen without narrowing down the candidate for the target person to be seen, and which person image is actually It is left to the user to decide whether or not to reproduce. Here, by delegating the decision to the user, even if the machine determines that the wrong person image is the most probable person, the final selection of the person is executed by the user, so that the user's will is surely confirmed. It can be reflected.

Furthermore, after confirming that the face image with the highest degree of similarity reproduced in advance is the person to be specifically viewed, the person with the highest degree of similarity is used by combining the zoom function and the function of this embodiment. It is also possible to continuously zoom the image.

As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

Note that each step in the imaging method and the image signal reproduction method of the present specification does not necessarily have to be processed in time series in the order described in the flowchart, and may include processing in parallel or by a subroutine.

  The present invention can be used in a moving image imaging apparatus and an imaging method.

100: Imaging device 108 ... Liquid crystal monitor (monitor)
DESCRIPTION OF SYMBOLS 130 ... Image output part 132 ... Memory device 146 ... Image sensor 160 ... Feature-value memory | storage part 162 ... Index table 180 ... Face extraction part 182 ... Similarity calculation part 184 ... Similarity comparison part 190 ... Image superimposition part 400 ... Imaging device 406 ... Identity determination unit 600 ... Image signal reproduction device 604 ... Image acquisition unit

Claims (4)

A feature amount storage unit for storing a feature amount of a face of a specific person;
A face extraction unit for extracting one or more faces in the screen in the acquired image signal;
A degree-of-similarity calculating unit that calculates one or a plurality of extracted facial feature quantities and calculates a similarity degree to the facial feature quantity of the specific person;
A similarity comparison unit for comparing the similarity with a predetermined threshold;
An image superimposing unit that superimposes an index in a manner that identifies the face on or around the face having a similarity equal to or greater than the predetermined threshold in the screen;
An image output unit that outputs an image signal on which the index is superimposed ;
An identity determination unit that determines the identity of one or more faces on a predetermined number of previous screens and one or more faces on the current screen;
The similarity calculation unit calculates the similarity of one or more faces on the current screen using also the similarity of one or more faces on a predetermined number of previous screens determined to be identical. ,
The image signal reproducing apparatus, wherein the index changes a color, a pattern, or a shape according to the similarity of the one or more faces. Extract one or more faces in the screen from the acquired image signal,
Calculating the extracted feature quantity of one or more faces, calculating the degree of similarity with the acquired facial feature quantity of the specific person,
Determining the identity of one or more faces on the previous predetermined number of screens and one or more faces on the current screen,
One or more of the current screen using the similarity between the feature quantity of one or more faces on a predetermined number of previous screens determined to be identical and the acquired face feature quantity of a specific person The degree of similarity between the facial feature quantity of the particular person and the facial feature quantity of the specific person is calculated, and the similarity degree between the one or more facial feature quantities of the current screen and the facial feature quantity of the specific person And a predetermined threshold value,
In the screen, an index is displayed by changing the color, pattern, or shape in accordance with the similarity of the one or more faces in a manner that identifies the face on or around the face having the similarity equal to or greater than the predetermined threshold. An image signal reproduction method comprising: superposing and outputting an image signal on which the index is superimposed. A feature amount storage unit for storing a feature amount of a face of a specific person;
An image sensor that photoelectrically converts a subject image to generate an image signal;
A face extraction unit for extracting one or more faces in the screen in the image signal;
A degree-of-similarity calculating unit that calculates one or a plurality of extracted facial feature quantities and calculates a similarity degree to the facial feature quantity of the specific person;
A similarity comparison unit for comparing the similarity with a predetermined threshold;
An image superimposing unit that superimposes an index in a manner that identifies the face on or around the face having a similarity equal to or greater than the predetermined threshold in the screen;
A monitor,
An image output unit that outputs an image signal superimposed with the index to the monitor;
An identity determination unit that determines the identity of one or more faces on a predetermined number of previous screens and one or more faces on the current screen;
The similarity calculation unit calculates the similarity of one or more faces on the current screen using also the similarity of one or more faces on a predetermined number of previous screens determined to be identical. ,
The imaging apparatus is characterized in that a color, a pattern, or a shape is changed according to the similarity of the one or more faces. Get the facial features of a specific person ,
Photoelectrically convert the subject image to generate an image signal,
Extracting one or more faces in the screen in the image signal;
Calculating the extracted feature quantity of one or more faces,
Determining the identity of one or more faces on the previous predetermined number of screens and one or more faces on the current screen,
Using the degree of similarity between the feature quantity of one or more faces on the predetermined number of previous screens determined to be identical and the feature quantity of the face of the specific person, one or more of the current screen calculating a similarity between the feature quantity and the feature quantity of the face of a specific person's face,
Comparing the similarity between the feature quantity of one or more faces of the current screen and the feature quantity of the face of the specific person and a predetermined threshold;
In the screen, an index is displayed by changing the color, pattern, or shape in accordance with the similarity of the one or more faces in a manner that identifies the face on or around the face having the similarity equal to or greater than the predetermined threshold. An imaging method characterized by superimposing.

Images