JP5711296B2 - Imaging method and imaging apparatus - Google Patents

Description

  The present invention relates to an imaging method and an imaging apparatus for capturing images of favorite expressions of both a photographer and a subject.

  In recent years, various systems have been proposed in which digital photographic images obtained by users themselves can be widely disclosed via a network. In a system that can publish photos on such a network, the published photo image may be seen by many people, so if the user wants to take a photo image that looks good and publish it as much as possible Seem.

  Here, as a proposal regarding a technique for photographing a subject with a facial expression according to a photographer's preference, for example, in Patent Document 1, the feature amount of facial images with a facial expression desired by the photographer (such as eye and nose distribution) ) Is stored in a predetermined storage unit, and at the time of shooting, a plurality of images are acquired by one shooting instruction, and a plurality of images are acquired, and stored in advance in the face image An image most similar to the feature amount is selected and recorded.

JP 2006-115406 A

  Here, in Patent Document 1, since it is not possible to confirm whether the subject has a favorite expression before shooting, a plurality of images are shot with one shooting instruction. For this reason, the photographing time tends to be long, and it is necessary to hold a plurality of images at a time, so that the memory capacity tends to increase.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an imaging method and an imaging apparatus capable of obtaining images of favorite expressions of both a photographer and a subject by one shooting. To do.

In order to achieve the above object, the imaging method according to the first aspect of the present invention acquires through image data by continuously operating an image sensor, sequentially displays the acquired through image data, and acquires the through image acquired above. When the eyes, mouth, and head of the face are detected from the data, and the score is specified by the subject for a plurality of comparison facial expression images stored in advance, the detected eyes The number of points associated with the facial expression image matched with the facial expression and the direction of the neck tilt or the face orientation by matching the detection results of the head, mouth and head with the plurality of facial expression images When the score is not specified, the facial expression and the neck tilt or face by matching the detection results of the detected eyes, mouth and head with the plurality of facial expression images Number according to the direction of Was calculated as the number that represents the smile associated with the matched expression image, the computed the face portion of the expressions and neck tilted way or the through image data expression level corresponding to the numerical value corresponding to the orientation of the face It is characterized by being displayed in a superimposed manner.

In order to achieve the above object, an imaging device according to a second aspect of the present invention includes an imaging element that acquires through image data, a display unit that sequentially displays the acquired through image data,
When the eyes, mouth, and head of the face are detected from the acquired through image data , and points are specified by the subject for a plurality of facial expression images for comparison stored in advance. , By matching the detected results of the detected eyes, mouth and head with the plurality of facial expression images, the facial expression and the numerical value corresponding to the manner of tilting the neck or the direction of the face are matched to the facial expression image When the score is calculated and the score is not designated, the facial expression and neck of the face are matched by matching the detected eye, mouth and head detection results with the plurality of facial expression images. A facial expression level calculation circuit that calculates a numerical value according to how to tilt or the direction of the face as a score representing a smile associated with the matched facial expression image, and the calculated facial expression and the method of tilting the neck or Face orientation The expression level according to numerical values Flip characterized by comprising a control unit for superimposing displayed on the through image data.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging method and imaging device which can obtain the image of the favorite expression of both a photographer and a to-be-photographed object by one imaging | photography can be provided.

It is a block diagram which shows the structure of the camera which concerns on one Embodiment of this invention. It is a figure shown about the outline | summary of operation | movement in the camera shown in FIG. It is a flowchart shown about the flow of a process at the time of calculating R value. FIG. 4A is a diagram showing an example of a facial expression with a high R value, and FIG. 4B is a diagram showing an example of a facial expression with a low R value. It is a figure which shows the distribution curve which showed the relationship between R value and a scoring result. It is a flowchart shown about the main operation control of the camera which concerns on one Embodiment of this invention. It is a flowchart shown about the scoring process of a facial expression image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a camera 100 according to an embodiment of the present invention. A camera 100 shown in FIG. 1 includes a microprocessing unit (MPU) 101, an operation unit 102, a photographing lens 103, an autofocus (AF) circuit 104, an aperture mechanism 105, an aperture control circuit 106, and an image sensor 107. An analog front end (AFE) circuit 108, an image processing circuit 109, a display control circuit 110, a display panel 111, a focus determination circuit 112, a face detection circuit 113, an expression level calculation circuit 114, a compression / The decompression unit 115, the recording control unit 116, the facial expression database 116 a, the recording medium 117, and the auxiliary light emitting circuit 118 are included.

  The MPU 101 sequentially controls each block constituting the camera 100 according to the operation of the operation unit 102 from the user. The operation unit 102 includes a release switch for causing the camera 100 to perform shooting, a changeover switch for switching the operation mode of the camera 100 between a shooting mode, a playback mode, and the like. Although details will be described later, the operation unit 102 includes an operation unit having a function of a designation unit for causing a person who is a subject to designate a favorite facial expression.

  The MPU 101 also has a timer for measuring the time when the user took a picture. Thereby, the photographing time is associated with the image data obtained by photographing.

  The taking lens 103 forms an image from the subject 200 on the image sensor 107. The photographing lens 103 includes a lens for autofocus (focusing). During autofocus, the MPU 101 controls the AF circuit 104 while switching the lens position of the focusing lens while monitoring the lens position of the focusing lens, and performs focusing of the photographing lens 103. The photographing lens 103 also includes a zoom lens, and the angle of view at the time of photographing can be switched by switching the position of the zoom lens. The aperture mechanism 105 is a mechanism that is provided in the vicinity of the photographing lens 103 or has an effect of a shutter or an aperture. The aperture mechanism 105 is opened to a predetermined aperture by the aperture control circuit 106 at the start of shooting and closed at the end of shooting. At the time of shooting, by driving the zoom lens of the shooting lens 103, the aperture mechanism 105, etc., the photographer can take a photograph with his / her favorite shooting effect (composition, background processing, motion description, etc.). it can.

  The image sensor 107 has a light receiving surface configured by arranging a large number of pixels, and converts an image from the subject 200 received through the photographing lens 103 into an electrical signal. In the present embodiment, the image sensor 107 may use either a CCD system or a CMOS system.

  The AFE circuit 108 includes a CDS circuit, a gain adjustment circuit, an AD conversion circuit, and the like, and is obtained by performing analog processing such as correlated double sampling processing and gain adjustment processing on the analog electric signal obtained in the image sensor 107. The obtained image signal is converted into digital data and input to the image processing circuit 109. The AFE circuit 108 is also provided with a function of reading signals from a plurality of pixels constituting the image sensor 107 at a time. For example, the SFE can be improved by collectively reading out signals of 4 pixels (2 × 2 pixels) and 9 pixels (3 × 3 pixels) by the AFE circuit 108 and adding the read electrical signals. . By such processing, the apparent sensitivity of the image sensor 107 can be increased.

  Further, the AFE circuit 108 also has a function of selecting an output signal from the image sensor 107, and can extract only signals from a limited range of pixels among all effective pixels of the image sensor 107. In general, the image sensor 107 can read signals at high speed by performing thinning. By using this function, if a through image for composition confirmation obtained by continuously operating the image sensor 107 is read out at high speed and processed by the image processing circuit 109 and then displayed on the display panel 111, the user can display the display panel 111. It is possible to perform framing using the through image displayed above.

  The image processing circuit 109 corrects the color, gradation, sharpness, etc. in the image data input from the AFE circuit 108, or amplifies the level of the image data to a predetermined level to adjust it to the correct shading and the correct gray level. Various image processing is performed.

  Further, in order to display various images such as the framing image as described above, the image processing circuit 109 resizes (reduces) the image data input from the AFE circuit 108 so that the image data can be displayed on the display panel 111. It also has a function to

  The display control circuit 110 converts the image data obtained by the image processing circuit 109 and the image data recorded on the recording medium 117 into a video signal, and causes the display panel 111 to display an image based on the converted video signal. The display panel 111 is a display panel such as a liquid crystal display panel or an organic EL display panel. The display control circuit 110 also has a multi-screen display function for displaying a plurality of images superimposed on the display panel 111. With this multi-screen display function, it is also possible to superimpose a character display of mode setting and various warning displays on the captured image or the captured image.

  The focus determination circuit 112 detects the contrast of the image obtained by the image processing circuit 109. The contrast detected by the focus determination circuit 112 is used during autofocus. That is, while the AF circuit 104 changes the lens position of the focusing lens of the photographing lens 103, the focus determination circuit 112 sequentially detects the contrast of the image, and stops the focusing lens at the lens position where the detected contrast is maximized. By doing so, the photographing lens 103 can be focused.

  The face detection circuit 113 corresponds to a particular face of the subject in the image from the distribution of feature points in the image obtained by the image processing circuit 109 (for example, if the subject is a person, eyes, nose, mouth, etc.). Detect part. The expression level calculation circuit 114 calculates an R value (details will be described later) that is a value for obtaining the expression level of the subject from the face image detected by the face detection circuit 113.

  The compression / decompression unit 115 compresses the image data processed by the image processing circuit 109 at the time of shooting. The compression / decompression unit 115 decompresses image data compressed and recorded on the recording medium 117. The recording control unit 116 records the image data compressed by the compression / expansion unit 115 on the recording medium 117 with accompanying data.

  The facial expression database 116a as a storage unit stores an R value corresponding to an image of a favorite facial expression of the person designated by the person who is the subject 200, and a score designated by the subject 200 corresponding to the image. .

  The auxiliary light emitting circuit 118 irradiates the subject 200 with auxiliary light according to the situation at the time of shooting. This prevents insufficient brightness and unevenness during shooting.

  FIG. 2 is a diagram showing an outline of the operation of the camera shown in FIG. Before shooting, first, various facial expressions are presented on the subject 200, and each facial expression is scored. The images of various facial expressions to be presented at this time may be pre-photographed, for example, as a collection of facial expression samples, and stored in the facial expression database 116a. You may make it obtain by image | photographing in the state made to do. Here, in the present embodiment, the expression value presented to the subject 200 is associated with R values (Ra to Rd in FIG. 2), which will be described in detail later. Then, when scoring is performed by the subject 200, the scoring result is associated with the R value and stored in the facial expression database 116a. In the example of FIG. 2, an example is shown in which the subject 200 performs scoring of 10-level evaluation on the four facial expression images of the image A, the image B, the image C, and the image D. The number of images to be scored is not limited to four, and the scoring is not limited to 10-level evaluation.

  During framing before shooting, the camera 100 sequentially captures the subject 200 to acquire a through image, and sequentially displays the through image on the display panel 111. In the present embodiment, the score corresponding to the R value of the human face image in the through image sequentially acquired during framing is extracted from the facial expression database 116a, and based on this score, the current facial expression is the extent of the subject 200. Judge whether the facial expression you like. Then, this determination result is provided to the photographer in the form of a level display (bar display) as indicated by reference numeral 111a, for example.

  Next, the R value will be described. FIG. 3 is a flowchart showing the flow of processing when calculating the R value. In FIG. 3, the expression level calculation circuit 114 detects the eyes and mouth of the face in the image from the shadow distribution in the image of the face detected by the face detection circuit 113 (step S101). More specifically, the face detection circuit 113 detects a face portion that is substantially circular in the image while enhancing the contrast of the image. Then, the expression level calculation circuit 114 detects the eyes and nose from the distribution of shadows in the face.

  After detecting the eyes and nose in step S101, the facial expression level calculation circuit 114 obtains the area EA (see FIG. 4A) of the white-eye portion above the straight line A connecting the eyes of both eyes (step S102). Next, the facial expression level calculation circuit 114 obtains the area EB (see FIG. 4A) of the white-eye portion below the straight line A connecting the eyes of both eyes (step S103). After obtaining EA and EB, the facial expression level calculation circuit 114 obtains a value RE obtained by normalizing the difference between EA and EB by the sum of EA and EB (step S104). Here, when the subject 200 is a smile that he wants to show to others, the EB becomes close to 0 as shown in FIG. On the contrary, when the subject 200 has a troubled face that he / she does not want to show to others, the EA becomes close to 0 as shown in FIG. Therefore, it is possible to determine the facial expression from the magnitude of RE.

  After obtaining RE, the facial expression level calculation circuit 114 obtains the area LA (see FIG. 4B) of the upper lip portion of the straight line B connecting both ends of the mouth (step S105). Next, the facial expression level calculation circuit 114 obtains the area LB (see FIG. 4B) of the lower lip portion of the straight line B connecting both ends of the mouth (step S106). After obtaining LA and LB, the facial expression level calculation circuit 114 obtains a value RL obtained by normalizing the difference between LB and LA by the sum of LA and LB (step S107). Here, in the case of a smile, since LA is close to 0 as shown in FIG. 4A, the result RL increases. On the contrary, in the case of a troubled face, LB becomes small as shown in FIG. 4B, and therefore the result RL becomes small. Therefore, it is possible to determine the facial expression from the size of RL.

  After obtaining RE and RL, the facial expression level calculation circuit 114 obtains the sum R of RE and RL (step S108). The larger the R value, the closer the expression at that time is to a smile (good expression). Furthermore, as shown by the arrow C in FIG. 4A, if there is a shadow on the edge of the mouth or teeth are visible, the probability of smile is high. Therefore, the facial expression level calculation circuit 114 determines whether teeth (white portions) are detected in the mouth portion and whether there is a shadow on the mouth end portion (step S109). If it is determined in step S109 that teeth are detected at the mouth and there is a shadow at the mouth edge, the facial expression level calculation circuit 114 adds 1 to R (step S110).

  On the other hand, if it is determined in step S109 that teeth are detected in the mouth portion and there is no shadow at the mouth end portion, or after step S110, the facial expression level calculation circuit 114 uses the arrow D in the eyebrows portion (between eyes). It is determined whether there is a wrinkle as shown (step S111). If it is determined in step S111 that there is a wrinkle in the portion between the eyebrows, the facial expression level calculation circuit 114 subtracts 1 from R (step S112). That is, a facial expression in which a wrinkle is detected between the eyebrows cannot be said to be a smile, so the determination in step S111 is performed.

  The R value obtained as described above is higher as the smile is closer. In general, it is considered that the higher the R value, the better the expression. However, since the person who becomes the subject 200 does not always like the expression with the higher R value, images of various expressions are given to the person who becomes the subject 200. The scoring result is stored, and the scoring result is stored in the facial expression database 116a as a facial expression preference for each subject. FIG. 5 is a diagram showing a distribution curve showing the relationship between the R value and the scoring result. It should be noted that the score corresponding to the R value that has not been scored is preferably calculated by, for example, interpolation calculation.

  For example, in the example of FIG. 5, it can be seen that the subject A thinks that the facial expression image of R1 having a moderate R value among the facial expression images presented by the camera 100 is the best. On the other hand, it can be seen that the subject B thinks that the image of the expression of R2, which has an R value higher than R1, is the highest. By presenting such a facial expression preference of the subject 200 so as to be understood by the photographer before shooting, the photographer takes a picture while taking into account the preference of the facial expression for each subject and taking into account his own preference. Can be done.

  Hereinafter, specific operation of the camera will be described. FIG. 6 is a flowchart showing the main operation control of the camera according to the present embodiment.

  Before photographing, the MPU 101 performs a facial expression image scoring process (step S201). Details of the scoring process will be described later. After the scoring process of the facial expression image in step S201, the MPU 101 determines whether or not a photographing instruction is given by the operation of the operation unit 102 by the photographer (step S202). If it is determined in step S202 that a shooting instruction has not been issued, the MPU 101 executes various controls for displaying a through image for framing on the display panel 111 (step S203). That is, the MPU 101 operates the image sensor 107 continuously to sequentially acquire through image data, and the acquired through image data is processed by the image processing circuit 109, and then a through image is displayed on the display panel 111 based on the through image data. Let After displaying the through image, the MPU 101 detects the face image in the through image by the face detection circuit 113. After the face image is detected, the MPU 101 uses the expression level calculation circuit 114 to calculate an R value for the expression of the face in the through image as shown in FIG. 3 (step S204). Then, the MPU 101 extracts the score corresponding to the R value calculated by the facial expression level calculation circuit 114 with reference to the facial expression preference distribution curve corresponding to the subject 200 stored in the facial expression database 116a (step S205). ). After extracting the score corresponding to the facial expression (R value) during the through image display, the MPU 101 displays the score on the display panel 111 in a form such as a level display (bar display) 111a as shown in FIG. The displayed through image is superimposed and displayed (step S206). Thereafter, the process returns to step S202.

  In the determination in step S202, when a shooting instruction is given, the MPU 101 performs a shooting operation (step S207). That is, after the MPU 101 performs focus adjustment of the photographing lens 103, the image pickup device 107 is operated with the diaphragm mechanism 105 being squeezed to a predetermined opening, and the photographed image data obtained thereby is processed in the image processing circuit 109. Let Thereafter, the captured image data processed by the image processing circuit 109 is compressed by the compression / decompression unit 115, and the compressed image data is recorded on the recording medium 117. Thereafter, the processing in FIG. 6 is terminated.

  FIG. 7 is a flowchart showing the scoring process of the facial expression image in step S201 of FIG. In FIG. 7, the MPU 101 displays various facial expression images on the display panel 111 via the display control circuit 110 in order to give the subject 200 a desired facial expression score (step S <b> 301). The facial expression images to be displayed here may be displayed one by one on the display panel 111, or a plurality of images may be displayed simultaneously. Thereafter, the MPU 101 determines whether or not a point designation operation has been performed by the operation of the operation unit 102 by the subject 200 (step S302).

  When the score is specified for the facial expression image in the determination in step S302, the MPU 101 associates the R value of the facial expression image with the score (step S303). Next, the MPU 101 sequentially calculates the number of points corresponding to each R value by interpolation calculation (step S304). On the other hand, in the determination in step S302, when the score is not specified for the facial expression image (for example, after a predetermined time has elapsed since the facial expression image is displayed), the R value of each facial expression image is high. A high score is assigned from the one (step S305). This is because, as described above, an expression with a high R value is close to a smile and everyone is likely to feel a good expression.

  After the facial expression image is scored in step S304 or step S305, the MPU 101 stores the R value and the score of each facial expression image in association with each facial expression in the facial expression database 116a (step S306). Thereafter, the process of FIG. 7 is exited.

  As described above, according to the present embodiment, the photographer can take a picture at his / her favorite timing while confirming the expression level of the subject 200 when displaying a through image before photographing. Also, for example, when the level is low, the photographer gives advice (“laugh more”, “make a serious face”, “close your mouth”) or stimulus (say a joke or gag) to the subject. You can also create a facial expression that the subject likes. As described above, in this embodiment, the photographer takes a picture at one time by photographing at his / her favorite timing while looking at the level of the facial expression that the photographer wants to photograph before actually photographing. You can take a photo with a facial expression that reflects the taste of both the subject and the subject.

  In general, a facial expression with a high R value is considered to be a smile, but depending on the subject, this is not necessarily the case, so by actually scoring images of various facial expressions, It is possible to reflect the preference.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. For example, in the above-described embodiment, the R value and the score are associated with each other and stored in the facial expression database 116a, but the facial expression image and the score may be directly associated with each other. In this case, it is possible to extract the corresponding score by matching the face part of the through image with the face part of the expression image. In the case of such matching, the score may be determined including not only the expression but also how the neck is tilted and the direction of the face.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 100 ... Camera, 101 ... Micro processing unit (MPU), 102 ... Operation part, 103 ... Shooting lens, 104 ... Auto-focus (AF) circuit, 105 ... Aperture mechanism, 106 ... Aperture control circuit, 107 ... Imaging element, 108 ... Analog front end (AFE) circuit 109 ... Image processing circuit 110 ... Display control circuit 111 ... Display panel 112 ... Focus determination circuit 113 ... Face detection circuit 114 ... Expression level calculation circuit 115 ... Compression / decompression unit 116 ... Recording control unit, 116a ... Facial expression database, 117 ... Recording medium

Claims (3)

Through image data is obtained by continuously operating the image sensor,
Display the acquired through image data sequentially,
Detecting the eyes, mouth and head of the face from the acquired through image data,
When a score is specified by a subject for a plurality of facial expression images for comparison stored in advance, the detection result of the detected eyes, mouth and head and the plurality of facial expression images matching the calculated as the number associated with the numeric matched expression image corresponding to the direction of the tilted way or facial expressions and neck of the face portion, when the number is not specified, and the detected eye A smile associated with a facial expression image in which numerical values according to facial expression and neck tilt or face orientation are matched by matching the detection results of the head, mouth and head with the plurality of facial expression images Is calculated as a point representing
The expression level of the face and the expression level corresponding to the numerical value corresponding to the way of tilting the head or the face are superimposed and displayed on the through image data .
An imaging method characterized by the above.   The imaging method according to claim 1, wherein the expression level is displayed as a bar in a plurality of stages. An image sensor for obtaining through image data;
A display unit for sequentially displaying the acquired through image data;
When the eyes, mouth, and head of the face are detected from the acquired through image data , and points are specified by the subject for a plurality of facial expression images for comparison stored in advance. , By matching the detected results of the detected eyes, mouth and head with the plurality of facial expression images, the facial expression and the numerical value corresponding to the manner of tilting the neck or the direction of the face are matched to the facial expression image When the score is calculated and the score is not designated, the facial expression and neck of the face are matched by matching the detected eye, mouth and head detection results with the plurality of facial expression images. A facial expression level computation circuit that computes a numerical value according to the tilting direction or face orientation as a score representing a smile associated with the matched facial expression image ;
A control unit that superimposes and displays on the through image data a facial expression level corresponding to a numerical value corresponding to the calculated facial expression and neck tilt or face orientation;
An imaging apparatus comprising:

Images