JP2008225886A - Image display device, image display method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image display device, an image display method, and a program for separating and analyzing a principal object and a background, and surely retrieving and displaying a photo that evokes a sense of nostalgia. <P>SOLUTION: When image data is input to a server 400, a face part of a person and a background in the image data are separated, and the degree of nostalgia is determined from a facial expression in the face part and a kind of an object in the background. When the image that evokes a sense of nostalgia is reproduced, the image data is selected according to the degree of nostalgia for each piece of the image data, and the selected images are reproduced in PC500. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image display device, an image display method, and a program for displaying a plurality of images including face images in a predetermined priority order.

  In recent years, with the increase in capacity of recording media, a larger number of photographs have been taken than in the film photography era. Moreover, since such a photograph is obtained as a digital image, it can be easily sent to various places via a network or the like.

  In addition, past film photographs are digitized using a scanner or the like, so that it is possible to view images of chaotic chaos anywhere.

  As described above, in recent years, it has become possible to acquire a large amount of images, but on the other hand, it has become a great effort to search for images to be viewed from among them. As before, in the era when photos were taken only on occasions such as commemorative photos, it is relatively easy to organize photos, and because they are organized, search is also easy, but as in recent years It's hard to find where you need the images when you get a lot of photos on a daily basis.

There is a technique proposed in Patent Document 1 regarding such a technique related to image retrieval. The proposal of Patent Literature 1 reads an image obtained by photographing the vicinity of a railway line station and the like from a database, specifies the age, and displays the specified images in chronological order.
JP 2005-184785 A

  Here, as in Patent Document 1, it is not always possible to search for a nostalgic image even if the age is simply determined. In fact, there are cases where the search cannot be performed unless the main subject portion and the background portion are considered separately. This is because nostalgia belongs to emotions, and the facial expressions of people at the time of shooting and the people who were together at the time of shooting are important factors. In addition, since the age cannot be specified from the facial expression of a person, it is difficult to determine whether or not the photograph of the person is an old photograph. The specific accuracy of this age is better when using the background.

  The present invention has been made in view of the above circumstances, an image display device capable of separating and analyzing a main subject and a background portion, and reliably searching for and displaying a photo reminiscent of nostalgia. An object is to provide an image display method and a program.

  In order to achieve the above object, an image display device according to a first aspect of the present invention is an image display device for displaying a plurality of images including a face image in a predetermined priority order. A face image detection unit that detects a background image other than the face, an image evaluation unit that evaluates display priority of the image from the facial expression and the background image, And a display control unit that displays the images in a priority order corresponding to the display priority.

  In order to achieve the above object, an image display method according to the second aspect of the present invention is an image display method for displaying a plurality of images including a face image in a predetermined priority order. Detecting a portion, detecting an image of a background portion other than the face portion, evaluating the display priority of the image based on the image of the face portion and the background portion, and according to the priority order according to the display priority. The image is displayed.

  In order to achieve the above object, an image display program according to a third aspect of the present invention is an image display program for displaying a plurality of images including face images in a predetermined priority order. A step of detecting a portion, a step of detecting an image of a background portion other than the face portion, a step of evaluating display priority of the image based on the images of the face portion and the background portion, and the display priority And causing the computer to execute the step of displaying the images in the priority order according to the above.

  According to the present invention, it is possible to provide an image display device, an image display method, and a program capable of separating and analyzing a main subject and a background portion and reliably searching for and displaying a photo reminiscent of nostalgia. Can do.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a digital camera 100 as an example of a photographing apparatus including an image display apparatus according to an embodiment of the present invention. A digital camera 100 shown in FIG. 1 includes a microprocessing unit (MPU) 101, an operation unit 102, a photographing lens 103, an autofocus (AF) circuit 104, an aperture mechanism 105, an aperture control circuit 106, and an image sensor. 107, analog front end (AFE) circuit 108, image processing circuit 109, display control circuit 110, display panel 111, shape detection circuit 112, distance detection circuit 113, face detection circuit 114, compression / The expansion unit 115 includes a recording control circuit 116, a recording medium 117, a communication unit 118, and an auxiliary light emitting circuit 119. The digital camera 100 is connected to a server 400 and a PC 500 via a network 300 so as to be able to communicate.

  The MPU 101 sequentially controls each block constituting the digital camera 100 according to the operation of the operation unit 102 from the user. The operation unit 102 is obtained from the server 400, a release switch for causing the digital camera 100 to perform shooting, a changeover switch for switching the operation mode of the digital camera 100 to any one of the shooting mode, the playback mode, and the digest playback mode. The switch etc. which switch on / off of information display are included.

  The MPU 101 also has a timer for measuring the time when the user took a picture. Thereby, the photographing time is associated with the image data obtained by photographing.

  The taking lens 103 forms an image from the subject 200 on the image sensor 107. The photographing lens 103 includes a lens for autofocus (focusing). During autofocus, the MPU 101 controls the AF circuit 104 while switching the lens position of the focusing lens while monitoring the lens position of the focusing lens, and performs focusing of the photographing lens 103. The diaphragm mechanism 105 is a mechanism that is provided in or near the photographing lens 103 and produces the effects of a shutter and a diaphragm. The aperture mechanism 105 is opened to a predetermined aperture by the aperture control circuit 106 at the start of shooting and closed at the end of shooting. By changing the aperture of the aperture mechanism 105 at the time of shooting, the depth of field of the shooting lens 103 changes. Thereby, it is possible to switch the expression such as adjusting the degree of blurring of the background subject to make the main subject appear at the time of shooting or to clearly depict the background subject.

  The image sensor 107 has a light receiving surface configured by arranging a large number of pixels, and converts an image from the subject 200 received through the photographing lens 103 into an electrical signal. In the present embodiment, the image sensor 107 may use either a CCD system or a CMOS system.

  The AFE circuit 108 includes a CDS circuit, a gain adjustment circuit, an AD conversion circuit, and the like, and is obtained by performing analog processing such as correlated double sampling processing and gain adjustment processing on the analog electric signal obtained in the image sensor 107. The obtained image signal is converted into digital data and input to the image processing circuit 109. The AFE circuit 108 is also provided with a function of reading signals from a plurality of pixels constituting the image sensor 107 at a time. For example, the AFE circuit 108 can read out signals of 4 pixels (2 × 2 pixels) and 9 pixels (3 × 3 pixels) in a lump and add these read-out electric signals to improve the S / N. . By such processing, the apparent sensitivity of the image sensor 107 can be increased.

  Further, the AFE circuit 108 also has a function of selecting an output signal from the image sensor 107, and can extract only signals from pixels in a limited range among all effective pixels of the image sensor 107. In general, the image sensor 107 can read signals at high speed by performing thinning. Using this function, if an image obtained for composition confirmation by the image sensor 107 is read at high speed and processed by the image processing circuit 109 and then displayed on the display panel 111, the user is displayed on the display panel 111. It is possible to perform framing using an image.

  The image processing circuit 109 corrects the color, gradation, sharpness, etc. in the image data input from the AFE circuit 108, or amplifies the level of the image data to a predetermined level to adjust it to the correct shading and the correct gray level. Various image processing is performed.

  Further, in order to display various images such as the framing image as described above, the image processing circuit 109 resizes (reduces) the image data input from the AFE circuit 108 so that the image data can be displayed on the display panel 111. It also has a function to

  The display control circuit 110 converts the image data obtained by the image processing circuit 109 and the image data recorded on the recording medium 117 into a video signal, and displays it on the display panel 111. The display panel 111 is a display panel such as a liquid crystal display panel or an organic EL display panel. The display control circuit 110 also has a multi-screen display function for displaying a plurality of images superimposed on the display panel 111. With this multi-screen display function, it is also possible to superimpose a character display of mode setting and various warning displays on the captured image or the captured image.

  The shape detection circuit 112 detects contour information in the image data obtained by the image processing circuit 109.

  The distance detection circuit 113 detects the contrast of the image obtained by the image processing circuit 109. The contrast detected by the distance detection circuit 113 is used during autofocus. That is, while the AF circuit 104 changes the lens position of the focusing lens of the photographing lens 103, the distance detection circuit 113 sequentially detects the contrast of the image, and stops the focusing lens at the lens position where the detected contrast is maximized. By doing so, the photographing lens 103 can be focused. It is also possible to detect the distance of the main subject, the distance of the background subject, and the like based on the information on the lens position at the time of focusing.

  The face detection circuit 114 obtains an image of a particular face portion of the main subject from the distribution of feature points in the image obtained by the image processing circuit 109 (for example, if the main subject is a person, eyes, nose, mouth, etc.). To detect.

  The shape detection circuit 112, the distance detection circuit 113, and the face detection circuit 114, in the sense of improving the quality of a photograph, determine what kind of photograph the user is going to take at the time of photographing and optimize the photographing control. It is used for the purpose (such as setting to portrait mode). Further, when the image is reproduced, the main subject (the face portion) and the background portion of the image can be separated by the face detection function by the face detection circuit 114. Thereby, the shape of the eyes and mouth can be detected from the face portion, and compared with a database 116a in the recording control circuit 116 described later, it can be determined whether the face is an expressionless face or an expressional face. Further, by determining how many faces are detected in the image by the face detection circuit 114, it is possible to determine whether the image is a group photo.

  The compression / decompression unit 115 compresses the image data processed by the image processing circuit 109 at the time of shooting, and expands the image data read from the recording medium via the recording control circuit 116 at the time of shooting.

  The recording control circuit 116 records the image data compressed by the compression / expansion unit 115 on the recording medium 117 with accompanying data such as a shooting time. The recording control circuit 116 has a database 116a for comparing images such as face portions. The contents of the database 116a may be recorded when the camera is manufactured, but may be rewritable via a network from the server 400 or the like outside the digital camera 100.

  The communication unit 118 transmits image data or the like recorded on the recording medium 117 to a personal computer (hereinafter referred to as a PC) 500 or a server 400 outside the digital camera 100, or receives various information from the server 400 or the like. Communication control when The communication unit 118 can transfer an image captured by the digital camera 100 to another person, or can add information using a service on a network (for example, a service linked with a network-connected site). Can be received.

  The auxiliary light emitting circuit 119 irradiates the subject 200 with auxiliary light according to the situation at the time of shooting. This prevents insufficient brightness and unevenness during shooting.

  In the digital camera 100 having such a configuration, in this embodiment, it is possible to perform digest playback by selecting and displaying an image that easily recalls a memory from among many images. That is, when the user operates the operation unit 102 to set the operation mode of the digital camera 100 to the digest playback mode, an image that is easy to identify the age and easily appeal to emotion is selected from the images recorded on the recording medium 117. Then, the image is preferentially displayed on the display panel 111.

  Hereinafter, playback in the digest playback mode will be further described. In the following description, an example in which the digest playback mode function is realized on a server outside the digital camera 100 will be described. However, by providing the MPU 101 of the digital camera 100 with a server function to be described later, the digital camera 100 can also perform playback in the digest playback mode.

  FIG. 2 is a diagram showing a configuration of a communication system including an image display device according to an embodiment of the present invention. In the communication system shown in FIG. 2, a server 400, a PC 500, and a PC 600 are connected via a network 300 so that data communication is possible. Further, image data is input to the PC 600 from the digital camera 100 and the scanner 700, and this image data can be transmitted to the server 400 via the network 300. Here, in the example of FIG. 2, a single server 400 has a database and image determination function. These functions can be distributed to a plurality of servers to provide a lighter system.

  Note that the communication system described below can take measures against situations where the amount of image data is too large to examine images one by one, such as when organizing images such as family chronology. It is a system that can.

  Photo image data taken by the digital camera 100 and photo image data (image data 800) captured by the scanner 700 are transmitted via the PC 600 and the network 300. Here, the image data obtained by the digital camera 100 is attached with accompanying information such as a shooting date and time and a title. However, face position data in an image obtained at the time of shooting may be added. By assigning face position data and the like, the server 400 does not need face detection described later, and the processing amount of the server 400 can be reduced.

  In the network 300, there is a server 400 that performs various services by making full use of information that goes there. In the present embodiment, when the digest playback mode is selected by the user, the server 400 selects the image data 900 that can be remembered from the past from the image data 800 sent from the PC 600. Display control is performed based on the image data 900 so that the image can be viewed on the PC 500. As a result, other users can share a photographic image captured by the digital camera 100 or an image captured by the scanner 700.

  Such a server 400 includes a communication unit 401, a recording unit 402, an access reaction unit 403, a face detection unit 404, a facial expression detection unit 405, a background detection unit 406, an image determination unit 407, a database classified by age. 408, an image selection unit 409, and a display control unit 410. Here, each function of the server 400 may be configured by hardware or may be configured as software.

  The communication unit 401 performs communication control when the server 400 performs communication via the network 300. The recording unit 402 records image data transmitted via the network 300. When an operation instruction is given from the user, the access reaction unit 403 performs control according to the instruction. Specifically, when an instruction to record image data is given from the user, the image data transmitted together with this instruction is recorded in the recording unit 402. On the other hand, when the user gives an instruction to reproduce the image data, the image selection unit 409 is instructed to select an image corresponding to the reproduction mode at that time. Here, the access reaction unit 403 has at least the digest reproduction mode described above as a reproduction mode.

  The face detection unit 404 detects a face part in the image recorded in the recording unit 402. Similar to the face detection circuit 114 of the digital camera 100, the face can be detected from the distribution of feature points in the image (for example, if the main subject is a person or the like, eyes, nose, mouth, etc.). . The facial expression detection unit 405 evaluates the display priority of the image in the digest reproduction mode by detecting the facial expression from the facial image detected by the facial detection unit 404.

  The background detection unit 406 detects a background part in the image recorded in the recording unit 402. Here, the background portion is a portion of the image 800 other than the person shown by the broken line frame 801 in the figure, for example. In order to simplify the processing, a part other than the face part of the person (including the person's body) may be detected as the background part. In order to simplify the description, in the following description, a part other than the face detected by the face detection unit 404 is detected as the background. The image determination unit 407 evaluates the display priority of the image in the digest playback mode by determining the presence or absence of a subject that easily recalls memories such as characters in the background portion detected by the background detection unit 406. The age-specific database 408 records various types of data necessary for determining the subject that easily recalls the memories in the image determination unit 407 by age. For example, in the age-specific database, character fonts and catchphrase data according to age, clothing, vehicle data, and the like are recorded.

  The image selection unit 409 has a function of selecting image data from the recording unit 402 according to an instruction from the access reaction unit 403, and is detected by the facial expression detection unit 405 when the digest reproduction mode is selected by the user. It has a function of selecting image data that easily recalls memories from the facial expression and the subject determined by the image determination unit 407. The display control unit 410 performs display control so that the image data selected by the image selection unit 409 can be viewed on the PC 500.

  Hereinafter, the operation of the communication system shown in FIG. 2 will be described. FIG. 3 is a flowchart showing the operation of the server 400 of FIG.

  When an operation instruction is given by the user, the access reaction unit 403 of the server 400 determines whether the instruction is an instruction to record image data (an instruction to upload image data) or an instruction to reproduce the image data (an instruction to download image data). Is determined (step S1). If it is an image recording instruction in the determination in step S1, the access reaction unit 403 records the image data transmitted together with the image recording instruction from the user in the recording unit 402 (step S2).

  The following processes in steps S3 to S7 are processes for smoothly reproducing the image data in the digest reproduction mode. These processes may be performed in a digest playback mode to be described later. After the image data is recorded in the recording unit 402 in step S2, the background portion of the image data recorded in the recording unit 402 is detected by the face detection unit 404 and the background detection unit 406 (step S3). Details of the processing in step S3 will be described later.

  After the background portion is detected, the image determination unit 407 determines whether there is an image including a subject that recalls nostalgia in the image (step S4). Details of the determination in step S4 will be described later. If it is determined in step S4 that an image including a subject that recalls nostalgia exists in the image, the image determination unit 407 uses, for example, step S4 as a code for identifying an image that includes a subject that recalls nostalgia. The “nostalgic degree” obtained at the time of the determination is associated with the image data and recorded in the recording unit 402 (step S5). As will be described in detail later, “nostalgic degree” is an index for digitizing nostalgia, and an image having a higher “nostalgic degree” has a higher display priority in the digest playback mode. On the other hand, if it is determined in step S4 that there is no subject reminiscent of nostalgia in the image, the process in step S5 is skipped.

  Next, the facial expression detection unit 405 of the server 400 determines whether or not the facial image detected by the face detection unit 404 is a facial expression image that invites nostalgia (step S6). Details of the determination in step S6 will be described later. In the determination in step S6, when the facial image is a facial expression image that invites nostalgia, the facial expression detection unit 405 uses, for example, the determination in step S6 as a code for identifying the facial expression image that invites nostalgia. The obtained “nostalgic degree” is recorded in the recording unit 402 in association with the image data (step S7). If the “nostalgic degree” is already recorded in step S5, the “nostalgic degree” obtained in step S6 is added to or multiplied by the result of step S5 and recorded. Thereafter, the processing of FIG. 3 is terminated. On the other hand, if it is determined in step S6 that the face image is not a facial expression image that invites nostalgia, the process of step S7 is skipped.

  Further, when it is an instruction to reproduce an image in the determination in step S1, the access reaction unit 403 determines whether or not the reproduction instruction is reproduction in the digest reproduction mode (step S8). If it is determined in step S8 that the reproduction is in the digest reproduction mode, the access reaction unit 403 sends an image selection instruction for the digest reproduction mode to the image selection unit 409. In response to this, the image selection unit 409 extracts image data attached with “nostalgic degree” from the image data recorded in the recording unit 402 (step S9). Then, the image selection unit 409 selects one having a high “nostalgic degree” from the extracted image data (step S10). This may be such that only image data having the highest “nostalgic degree” may be selected, or all image data having a “nostalgic degree” higher than a predetermined threshold may be selected. After image data with a high “nostalgic degree” is selected, the display control unit 410 controls the image data selected by the image selection unit 409 to be displayed on the PC 500 in order of age (step S11). Thereafter, the processing of FIG. 3 is terminated. Here, since it is not always possible to specify the exact age, the display order in step S11 may accept a certain degree of freedom. Further, an enlargement rate at the time of display may be increased for an image having a higher “nostalgic degree” at the time of display.

  If it is determined in step S8 that the playback is not in the digest playback mode, the access reaction unit 403 sends an instruction to the display control unit 410 to display a list display of images recorded in the recording unit 402 on the PC 500. (Step S12). In this case, after the image selected by the user is selected by the image selection unit 409, the image is displayed on the PC 500 under the control of the display control unit 410.

  FIG. 4 is a flowchart illustrating an example of a processing procedure of face detection processing for background portion detection in step S3 of FIG. The face detection process is not necessarily limited to the process shown in FIG. Further, the face detection process shown in FIG. 4 is the same as the process performed in the face detection circuit 114 of the digital camera 100.

  First, the face detection unit 404 reads out image data from the recording unit 402 and emphasizes a shadow in the image (step S101). This is a process for enhancing the contrast of an image. For example, when shadow enhancement processing is performed on the image 800 shown in FIG. 5, an image in the vicinity of the face 802 is an image in which a shadowed part or an outline part is emphasized as indicated by reference numeral 803.

  After emphasizing the shadow of the image, the face detection unit 404 sets the pattern of the eyes and nose (step S102). An example of an eye-nose pattern is indicated by reference numeral 804 in FIG. Face detection is performed by matching the eye-nose pattern 804 with the shadow-enhanced image 803. Usually, the face size and orientation in an image obtained by photographing vary, and therefore the eye-nose pattern 804 is also a face. Prepare a number corresponding to the direction and size of the. The process of step S102 is a process for setting one of a plurality of prepared eye / nose pattern 804.

  Next, the face detection unit 404 performs a matching process between the shadow-enhanced image (actually, other than the vicinity of the face 802 is included) 803 and the eye-nose pattern 804 (step S103). This is a process for obtaining the degree of coincidence between the images by scanning the shadow-enhanced image 803 with the eye-nose pattern 804, for example, by correlation calculation. After this matching processing, the face detection unit 404 determines whether there is a position where the eye / nose pattern 804 is matched, that is, a position where the degree of coincidence between the shadow enhanced image 803 and the eye / nose pattern 804 is higher than a predetermined amount. It is determined whether the image exists in the image (step S104). If it is determined in step S104 that a matching position exists, the face detection unit 404 detects the image at that position as a face image (step S105).

  On the other hand, if there is no matching position at the position of step S104, the orientation and size of the face in the image may be different from the eye-nose pattern 804. Further, when there are a plurality of face portions in the image, not all face portions can be detected with one eye-nose portion pattern 804. Therefore, if there is no matching position in the determination in step S104, or after step S105, the face detection unit 404 uses all the eye-nose pattern 804 to determine whether to end the face detection process. It is determined by determining whether or not it has been performed (step S106). If it is determined in step S106 that matching processing has not been performed using all the eye-nose pattern 804, the face detection unit 404 switches to another eye-nose pattern (step S107), and returns to step S103. The matching process is performed again. On the other hand, when the matching process is performed using all the eyes and nose pattern 804, the position information of the face is output to the background detection unit 406, and the face detection process is terminated. In response to this, the background detection unit 406 detects all images other than the face detected in step S105 as images of the background (step S108). When the background portion is detected in this way, the process proceeds to step S4 in FIG.

  FIG. 6 is a flowchart showing a process for determining whether or not there is a subject reminiscent of nostalgia in step S4 of FIG. Here, in the present embodiment, when a subject that recalls nostalgia exists in the background image, the “nostalgic degree” is set according to the ease of specifying the age of the subject. . The higher the “nostalgic” subject, the easier it is to specify the age, and the user who sees it is more likely to recall nostalgia.

  In FIG. 6, the image determination unit 407 first determines whether or not a subject including characters such as a station name and a signboard exists in the background image (step S201). For this purpose, the image determination unit 407 performs a character detection process shown in FIG.

  Here, the character detection process for performing the determination of step S201 will be described with reference to FIGS. FIG. 7 is a flowchart showing an example of character detection processing used in the determination in step S201 of FIG. FIG. 8 is a diagram for explaining the concept of the character detection process of FIG.

  Character detection is performed by detecting the line portion 805 and the blank portion 806 existing between the line portions 805 of the character image shown in FIG.

  First, the image determination unit 407 acquires a contrast-enhanced image I1 by enhancing the contrast of the background image (step S2011). Next, the image determination unit 407 binarizes the background image to obtain a binarized image I2 (step S2012).

  Thereafter, the image determination unit 407 obtains the degree of coincidence between the contrast enhanced image I1 and the binarized image I2 from the difference between the contrast enhanced image I1 and the binarized image I2 (step S2013). Then, the image determination unit 407 determines whether or not there is a portion where the degree of coincidence between the contrast-enhanced image I1 and the binarized image I2 is large (that is, a portion where the difference is small) in the background image (step S2014). If it is determined in step S2014 that there is no portion where the degree of coincidence between the contrast-enhanced image I1 and the binarized image I2 is large, the image determination unit 407 determines that there is no subject including characters in the background image (step S2014). S2017).

  On the other hand, if it is determined in step S2014 that there is a portion where the degree of coincidence between the contrast-enhanced image I1 and the binarized image I2 is large, the image determination unit 407 determines that the portion is configured by a line image. Then, the image determination unit 407 detects an image around a portion with a high degree of coincidence (step S2015). Then, it is determined whether or not many blank portions, that is, low contrast portions occupy the detected portion of the image (step S2016). If it is determined in step S2016 that the number of blank portions is small, the image determination unit 407 determines that the image of that portion does not have character image characteristics and that there is no character image (step S2017). On the other hand, if it is determined in step S2016 that there are many blank portions, the image determination unit 407 determines that the image of the portion has the characteristics of a character image and that a subject including characters exists (step S2018). ).

  Here, it returns to description of FIG. 6 again. If it is determined in step S201 that an object including characters is present in the background image, the image determination unit 407 adds 5 points to the “nostalgic level” (step S202). That is, the character information can be a clear material for reading the place and the atmosphere at that time, and is considered to be information that makes it easy to recall nostalgia. Therefore, the score of “nostalgic degree” is increased. After setting the “nostalgic degree”, the image determination unit 407 collates the detected character with the character image recorded in the age-specific database 408 and specifies the age (step S211). Then, the process of FIG. 6 is exited and the process proceeds to step S5 of FIG.

  On the other hand, in the determination in step S201, when there is no subject including characters in the background image, the image determination unit 407 determines whether or not the vehicle subject is included in the background image. For example, the determination is made by matching with a vehicle image recorded in the age-specific database 408 (step S203). When the vehicle subject is included in the background image in the determination in step S203, the image determination unit 407 adds 4 points to the “nostalgic degree” (step S204). In other words, vehicles such as trains and buses, like characters such as signs and signs, are easy to identify places and show changes over time, so the “nostalgic degree” is set next to character information. After setting the “nostalgic degree”, the image determination unit 407 identifies the age based on the detected vehicle subject (step S211). Then, the process of FIG. 6 is exited and the process proceeds to step S5 of FIG.

  On the other hand, in the determination in step S203, when the vehicle subject is not included in the background image, the image determination unit 407 includes the subject wearing fashionable clothes in the background image. For example, it is determined by matching with the trendy clothing image recorded in the age-specific database 408 (also using the shape of the collar, sleeve, tie thickness, color, etc.) (step S205). If it is determined in step S205 that a subject wearing fashionable clothes is included in the background image, the image determination unit 407 adds three points to the “degree of nostalgia” (step S206). After setting the “nostalgic degree”, the image determination unit 407 identifies the age of the detected fashionable clothes (step S211). Then, the process of FIG. 6 is exited and the process proceeds to step S5 of FIG.

  On the other hand, if it is determined in step S205 that the background image does not include a subject wearing fashionable clothes, the image determination unit 407 includes a building subject in the background image. It is determined whether the image is taken indoors (step S207). In the determination in step S207, if the background image includes a building subject or the image is taken indoors, the image determination unit 407 adds two points to the “nostalgic level” (step S208). . That is, even when a building or a room is clearly shown, a point is added to the “nostalgic degree” assuming that an element for specifying the location is reflected. After the “nostalgic degree” is set, the image determination unit 407 identifies the age from the detected building subject or indoor image (step S211). Then, the process of FIG. 6 is exited and the process proceeds to step S5 of FIG.

  On the other hand, if it is determined in step S207 that the building subject is not included in the background image or the image is not taken indoors, the image determination unit 407 determines that the ground in the background image is the ground image. It is determined whether or not is paved (step S209). If it is determined in step S209 that the ground in the background image is not paved, the image determination unit 407 adds one point to the “nostalgic level” (step S210). That is, an image in which the ground is not paved in an urban area or the like is considered an old image. In this case as well, a score is added to the “nostalgic degree”. After the “nostalgic degree” is set, the image determination unit 407 identifies the age based on the detected level of paving on the ground (step S211). Then, the process of FIG. 6 is exited and the process proceeds to step S5 of FIG.

  On the other hand, if it is determined in step S209 that the ground in the background image is paved, the process of FIG. In this case, in the determination in step S4 of FIG. 3, it is determined that there is no subject reminiscent of nostalgia in the background image.

  By determining “nostalgic degree” as in the process of FIG. 6, an image including a subject that recalls the age in the background can be preferentially displayed. In the process of step S4 in FIG. 3, the “nostalgic degree” does not increase for an image with a simple wall background, such as an ID photo, or an image with only the sky or mountains in the background. Further, in the process of step S4 in FIG. 3, the “nostalgic degree” does not increase even in a group photo or the like with nature in front. However, since such a group photo can be an image reminiscent of nostalgia, in order to determine such an image, it is determined whether or not the facial expression of the person in step S6 is an expression reminiscent of nostalgia.

  With reference to FIGS. 9-12, the determination process of whether it is the facial expression which invites the nostalgia of FIG.3 S6 is demonstrated. Here, “an expression that invites nostalgia” in the processing described below does not necessarily indicate only an expression such as an individual's laughter, but also includes a relationship with a person who is shown together. Therefore, even if there are no facial expressions, what is reflected in many people is a facial expression appropriate for the place, so that it can also be counted as a nostalgic facial expression.

  FIG. 9 is a flowchart for obtaining “a nostalgic degree corresponding to facial expressions” such as a person's crying and laughing. FIG. 12 is a flowchart for obtaining “a nostalgic degree corresponding to the relationship between the persons” when there are a plurality of persons in the image.

  First, FIG. 9 will be described. The facial expression detection unit 405 detects the facial expression from the facial image detected by the face detection unit 404 (step S301).

  Here, the facial expression detection process will be described with reference to FIGS. FIG. 10 is a flowchart showing an example of facial expression detection processing in step S301 of FIG. FIG. 11 is a diagram for explaining the concept of the facial expression detection process of FIG.

  First, the facial expression detection unit 405 detects an eye part, a nose part, and a mouth part from the shadow distribution of the face image (step S3011). Next, the facial expression detection unit 405 determines whether the face is facing the front from the detected positional relationship between the eyes, nose, and mouth (step S3012). In step S <b> 3012, for example, when both eyes are biased to any face screen, it is determined that the face is facing sideways. Further, when both eyes are evenly arranged in the left-right direction of the face and the nose part is arranged at a substantially central part in the up-down direction of the face, it is determined that the front is facing.

  If it is determined in step S3012, the facial expression detection unit 405 detects the center position L1 of the straight line connecting both ends of the lips in the mouth detected in step S3011 (see FIG. 11A). (Step S3013). Next, the facial expression detection unit 405 detects the upper center position L2 of the lips (see FIG. 11A) (step S3014). Further, the facial expression detection unit 405 detects the lower position N1 (see FIG. 11A) detected in step S3011 (step S3015).

  Next, the facial expression detection unit 405 obtains a difference between the distance between L1 and N1 and the distance between L2 and N1 (step S3016). Then, the facial expression is determined from this difference. For example, in the case of a smile, both ends of the lips are lifted as shown in FIG. 11B, and the distance between L1 and N1, the distance between L2 and N1, and the difference between them are the true faces shown in FIG. It is considered to be smaller than the case of. In the case of a crying face, both ends of the lips are lowered as shown in FIG. 11 (d), and the distance between L1 and N1 and the distance and difference between L2 and N1 are the true faces shown in FIG. 11 (c). It is thought that it becomes larger than the case of. Therefore, the facial expression can be determined by comparing the difference obtained in step S3016 with a predetermined threshold value indicating smile, true face, or crying face.

  Based on the above concept, the facial expression detection unit 405 determines whether or not the difference obtained in step S3016 is greater than a threshold value indicating a crying face (step S3017). If it is determined in step S3017 that the difference is larger than the threshold value indicating a crying face, the facial expression detection unit 405 determines that the facial expression is a crying face (step S3018). Thereafter, the process of FIG. 10 is exited. On the other hand, if it is determined in step S3017 that the difference is smaller than a threshold value indicating a crying face, the facial expression detection unit 405 determines whether the difference is smaller than a threshold value indicating a smile (step S3019). If the difference is smaller than the threshold value indicating smile in step S3019, the facial expression detection unit 405 determines that the facial expression is smiling (step S3020). Thereafter, the process of FIG. 10 is exited. On the other hand, if it is determined in step S3019 that the difference is larger than the threshold value indicating smile, the facial expression detection unit 405 determines that the facial expression is a true face (step S3021). Thereafter, the process of FIG. 10 is exited.

  In the determination in step S3012, if the face is not facing the front, it is difficult to determine the facial expression by the processing in steps S3013 to S3021. Therefore, in this case, the determination is simplified. That is, the facial expression detection unit 405 determines whether teeth (white portions) can be detected at the mouth (step S3022). If it is determined in step S3022 that teeth can be detected, the facial expression detection unit 405 determines that the facial expression is a smile (step S3023). Thereafter, the process of FIG. 10 is exited. On the other hand, if it is determined in step S3022 that teeth cannot be detected, it is determined that facial expression cannot be detected, and the process of FIG.

  Here, it returns to description of FIG. 9 again. In step S301, after detecting a facial expression, the facial expression detection unit 405 determines whether the detected facial expression is a smile of a plurality of persons (step S302). If it is determined in step S302 that the detected facial expression is a smile of a plurality of persons, the facial expression detection unit 405 adds 5 points to the “nostalgic level” (step S303). Then, the process of FIG. 9 is exited and the process of FIG. 12 is started. That is, for a photograph having a plurality of faces and facial expressions of smiling faces, the “nostalgic degree” is set higher because there was a fun at that time.

  On the other hand, if it is determined in step S302 that the detected facial expression is not a smile of a plurality of persons, the facial expression detection unit 405 determines whether or not the detected facial expression is such that a plurality of faces face each other (step S304). If it is determined in step S304 that the detected facial expression is such that a plurality of faces face each other, the facial expression detection unit 405 adds 4 points to the “nostalgic level” (step S305). Then, the process of FIG. 9 is exited and the process of FIG. 12 is started. In such a case as well, it can be considered that the image shows that there was some kind of exchange between the people at that time, so it can be considered as an image that can remember the time including the emotion at that time.

  On the other hand, if it is determined in step S305 that the detected facial expression does not face a plurality of faces, the facial expression detection unit 405 determines whether the detected facial expression is an up smile (step S306). If it is determined in step S306 that the detected facial expression is a smiling face, the facial expression detection unit 405 adds three points to the “nostalgic level” (step S307). Then, the process of FIG. 9 is exited and the process of FIG. 12 is started. In other words, an up smile is considered to be a photograph taken with great care by the photographer even if the subject is not conscious of it.

  On the other hand, if it is determined in step S306 that the detected facial expression is not an up smile, the facial expression detection unit 405 determines whether the detected facial expression is an up crying face (step S308). If it is determined in step S308 that the detected facial expression is an up-crying face, the facial expression detection unit 405 adds two points to the “nostalgic level” (step S309). Then, the process of FIG. 9 is exited and the process of FIG. 12 is started. In other words, the up crying face may not be interesting to the subject himself, but because the emotion is so high, there is a high possibility that some memory remains, and this is also an expression that can recall memories Since it can be considered, add points to the "nostalgic degree".

  On the other hand, if it is determined in step S308 that the detected facial expression is not an up-crying face, the facial expression detection unit 405 determines whether there is a human face that does not exist in the detected facial expression (step S310). . For this determination, an image of a face of a specific person such as a family is recorded in the facial expression detection unit 405, and this image is compared with the facial image input to the facial expression detection unit 405. . In the determination in step S310, if there is another human face that does not exist in the detected facial expression, the facial expression detection unit 405 adds one point to the “nostalgic degree” (step S311). Then, the process of FIG. 12 is started.

  Next, FIG. 12 will be described. The facial expression detection unit 405 determines whether or not there is a small person among a plurality of large men and women in the image from the face image detected by the face detection unit 404 (step S401). Here, the size of the face can be determined from the size of the face detected by the face detection unit 404, for example. Actually, zooming such as enlargement or reduction may be performed, and the influence on the size of the face due to such zooming is also considered. In addition, the determination of gender can be made from the difference in hairstyle and size. In the determination in step S401, when there is a small person among a plurality of large men and women in the image, the facial expression detection unit 405 adds 10 points to the “nostalgic degree” (step S402). Then, the process of FIG. 12 is exited and the process proceeds to step S7 of FIG. That is, if there are large male and female faces in the image, these are considered to be couples. Also, if there is a person with a small face in the meantime, it can be considered as a picture of parents and children. Such nostalgia for photos can be considered exceptional, so the “nostalgic level” is set high.

  On the other hand, in the determination of step S401, when there is no small person between the large men and women in the image, the facial expression detection unit 405 has a small person beside the large men and women in the image. It is determined whether or not (step S403). If it is determined in step S403 that there is a small person beside a plurality of large men and women in the image, the facial expression detection unit 405 adds 8 points to the “nostalgic degree” (step S404). Then, the process of FIG. 12 is exited and the process proceeds to step S7 of FIG. In such a case, it can be classified as an image reminiscent of nostalgia because it is thought that a close atmosphere is created following the presence of a small person between men and women.

  On the other hand, in the determination in step S404, when there is no small person next to a plurality of large men and women in the image, the facial expression detection unit 405 determines whether there are a large woman and a small person in the image. Determination is made (step S405). If it is determined in step S405 that there are a large woman and a small person in the image, the facial expression detection unit 405 adds 6 points to the “nostalgic level” (step S406). Then, the process of FIG. 12 is exited and the process proceeds to step S7 of FIG. In the determination in step S405, if there are no large woman and small person in the image, the facial expression detection unit 405 determines whether there are a large man and a small person in the image (step S405). S407). If it is determined in step S407 that there are a large man and a small person in the image, the facial expression detection unit 405 adds 5 points to the “nostalgic degree” (step S408). Then, the process of FIG. 12 is exited and the process proceeds to step S7 of FIG. That is, in the case of a photograph of an adult and a child, it can be considered as a photograph of a mother and a child, a father and a child, or a brother, cousin, uncle, aunt, great-grandparent, etc., not parents. Therefore, in step S405 and step S407, these are determined and added to the “nostalgic degree”.

  On the other hand, if it is determined in step S407 that neither a large man nor a small person exists in the image, the facial expression detection unit 405 determines whether the ratio between the face part and the body part is small (step S409). . If it is determined in step S409 that the ratio between the face part and the body part is small, the facial expression detection unit 405 adds one point to the “nostalgic degree” (step S410). Then, the process of FIG. 12 is exited and the process proceeds to step S7 of FIG. As the ratio of face to body size grows as it grows, a photo with a small ratio is considered a child's photo. And such a picture is added to "nostalgic degree" as something that can evoke the memories of a child. However, if the “nostalgic level” based on the ratio of face to body is too high, it will result in only pictures of children, so the score is lower than that of photographs with adults.

  As described above, according to the present embodiment, it is possible to provide a camera or system that can preferentially reproduce only images that arouse nostalgia from a large number of images. In addition, if a service on the Internet is used, for example, relatives who have been distant from various places like a family's chronological history can view it in a short time.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. For example, in the above-described embodiment, one “nostalgic degree” is set for each image data, but when a plurality of conditions overlap, the “nostalgic degree” corresponding to the plurality of conditions is added or multiplied. The final “nostalgic degree” may be obtained. Also, the score of “nostalgic degree” corresponding to the type of background subject and facial expression is an example, and can be determined as appropriate. The score may be set by the user.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

1 is a block diagram illustrating a configuration of a digital camera as an example of a photographing apparatus including an image display device according to an embodiment of the present invention. It is a figure which shows the structure of the communication system containing the image display apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows operation | movement of a server. It is a flowchart shown about the process sequence of an example of the face detection process for a background part detection. It is a figure shown about the concept of a face detection process. It is a flowchart shown about the determination process of whether the subject which recalls nostalgia exists. It is a flowchart which shows an example of a character detection process. It is a figure for demonstrating the concept of a character detection process. It is a flowchart for calculating the “nostalgic degree” according to the facial expression. It is a flowchart which shows an example of a facial expression detection process. It is a figure for demonstrating the concept of a facial expression detection process. It is a flowchart for calculating the “nostalgic degree” according to the relationship between persons.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Digital camera, 101 ... Micro processing unit (MPU), 102 ... Operation part, 103 ... Shooting lens, 104 ... Auto-focus (AF) circuit, 105 ... Aperture mechanism, 106 ... Aperture control circuit, 107 ... Image sensor, 108 DESCRIPTION OF SYMBOLS ... Analog front end (AFE) circuit, 109 ... Image processing circuit, 110 ... Display control circuit, 111 ... Display panel, 112 ... Shape detection circuit, 113 ... Distance detection circuit, 114 ... Face detection circuit, 115 ... Compression part, 116 ... Recording control circuit, 117 ... Recording medium, 118 ... Communication unit, 119 ... Auxiliary light emitting circuit, 300 ... Network, 400 ... Server, 500,600 ... Personal computer (PC), 700 ... Scanner

Claims (7)

In an image display device for displaying a plurality of images including face images in a predetermined priority order,
A face image detection unit for detecting a face portion in the image;
A background image detection unit for detecting an image of a background part other than the face part;
An image evaluation unit that evaluates the display priority of the image based on the image of the face portion and the background portion;
A display control unit that displays the images in a priority order according to the display priority;
An image display device comprising:   2. The image evaluation unit according to claim 1, wherein the display priority is evaluated by extracting a character feature from the background image and determining whether or not there is a character image. The image display device described. The image includes a plurality of face part images,
The image display device according to claim 1, wherein the image evaluation unit evaluates display priority of the image from a relationship between a relative position and a size of a plurality of faces in the image. The image evaluation unit includes a facial expression detection unit that detects facial expressions from the image of the face part,
The image display device according to claim 1, wherein the display priority is evaluated based on the detected facial expression and the background image. The image evaluation unit scores the images of the face part and the background part, and evaluates the display priority by evaluating the score,
The image display apparatus according to claim 1, wherein the display control unit preferentially displays the image having the high score. In an image display method for displaying a plurality of images including face images in a predetermined priority order,
Detect the face part in the above image,
Detect images of the background part other than the face part,
Evaluating the display priority of the image based on the image of the face portion and the background portion,
Displaying the images in a priority order according to the display priority;
An image display method characterized by the above. In an image display program for displaying a plurality of images including face images in a predetermined priority order,
Detecting a face portion in the image;
Detecting an image of a background portion other than the face portion;
Evaluating display priority of the image based on the image of the face portion and the background portion;
Displaying the images in a priority order according to the display priority;
Display program for causing a computer to execute.

Images