WO2018042633A1

WO2018042633A1 - Image management device, image management method, and image management program

Info

Publication number: WO2018042633A1
Application number: PCT/JP2016/075872
Authority: WO
Inventors: 桂一廣木
Original assignee: マクセル株式会社
Priority date: 2016-09-02
Filing date: 2016-09-02
Publication date: 2018-03-08

Abstract

The objective of the present invention is to provide an image management technique capable of reflecting a preference of a user in a rule for determining disclosure/non-disclosure. To this end, an image management device 100 includes: a behavior characteristic amount collection unit 202 for collecting a behavior characteristic amount to be used when calculating a disclosure determination score for classifying image data to be processed, for disclosure or non-disclosure, on the basis of information indicating a behavior of a person who captured an image relating to the image data or a viewer of the image data; a score calculation unit 205 for calculating the disclosure determination score of the image data by applying the behavior characteristic amount corresponding to the image data to a score calculation expression; a disclosure image determination unit 206 for provisionally determining that the image data is to be disclosed or non-disclosed on the basis of the disclosure determination score; a user correction unit 207 for receiving a correction instruction from the user with respect to a result of the determination of the disclosure or non-disclosure provisionally determined for the image data; a feedback unit 208 for changing the score calculation expression on the basis of a change history of the disclosure or non-disclosure indicated by the correction instruction; and an image disclosure unit 209 for disclosing the image if it has been determined that the image data is to be disclosed.

Description

Image management apparatus, image management method, and image management program

The present invention relates to an image management apparatus, an image management method, and an image management program, and more particularly to a technique for managing disclosure / non-disclosure of images.

Japanese Patent Laid-Open No. 2004-133867 provides “an image disclosing device capable of automatically determining whether each image is disclosed / not disclosed according to a predetermined rule without setting the user to be disclosed / not disclosed. "The image publishing apparatus includes an image storage unit that stores images owned by the user, and an image analysis unit that performs a plurality of image analysis processes on each image stored in the image storage unit." And a disclosure / non-disclosure determination unit that determines whether each image is disclosed or not based on the result of each image analysis process performed on each image by the image analysis unit (summary excerpt). Yes.

JP 2013-191035 A

In recent years, there are an increasing number of scenes where photos taken with an electronic camera, a smartphone, etc. are shared by a plurality of specific people or made public to an unspecified number of people. In addition, services and functions that automatically share captured photos have appeared. At that time, it is necessary to determine whether to open or close each image, and it is a problem that a lot of work is required when taking and managing a large number of photos.

According to Patent Document 1, it is possible to automatically determine whether or not to disclose each image according to a predetermined rule without setting the disclosure or non-disclosure by the user. However, the rule for deciding whether to disclose or not is fixed, and there remains a problem that the rule cannot be changed even if the rule does not match the user's preference or situation.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image management technique capable of reflecting a user's preference in a rule for determining whether to open or close.

In order to achieve the above object, the present invention has the structure described in the claims. As one aspect of the present invention, an image management device according to the present invention is based on an image acquisition unit that acquires image data to be processed, and information that represents a behavior of a photographer of the image data or a viewer of the image data. An action feature amount collection unit that collects an action feature amount used when calculating a disclosure determination score for classifying the image data as public or private, and a score calculation formula for calculating the disclosure determination score. A score calculation formula storage unit to be stored, and the score calculation formula is read from the score calculation formula storage unit, the behavior feature amount corresponding to the image data is applied to the score calculation formula, and the disclosure determination score of the image data And a score calculation unit that calculates whether the disclosure determination score satisfies a disclosure condition, and makes the image data public or private based on a result of the determination A tentatively determined public image determination unit, a user correction unit that accepts a correction instruction from a user for a determination result of disclosure or non-disclosure tentatively determined for the image data, and the correction instruction A feedback unit that changes the score calculation formula based on a public or non-public change history indicated by, and an image publishing unit that publishes the image data when it is decided to publish the image data. It is characterized by.

According to the present invention, it is possible to provide an image management technique capable of reflecting the user's preference in the rule for determining whether to open or close. Note that problems, configurations, and effects other than those described above will be clarified in the following embodiments.

Hardware configuration of image management device Functional block diagram of image management device The flowchart which shows the process of the first half in the image management apparatus which concerns on 1st embodiment. The flowchart which shows the process of the second half in the image management apparatus which concerns on 1st embodiment. The figure which shows an example of a score calculation process The figure which shows an example of the user correction screen which concerns on 1st embodiment. The figure which shows an example of the user correction screen which concerns on 2nd embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals, and redundant description is omitted. The image management apparatus according to the present embodiment may be of any type as long as it can store and output images, such as a smartphone, a digital camera, a tablet, and a digital photo frame. Hereinafter, a smartphone is an image management apparatus according to the present embodiment. An example used as will be described.

<First embodiment>
FIG. 1 is a diagram illustrating an example of a hardware configuration of the image management apparatus 100.

The image management apparatus 100 includes, for example, a communication I / F 113, a control unit 114, an input unit 115, a display unit 117, a signal separation unit 121, a tuner / demodulation unit 122, a storage 125, a mobile communication I / F 131, a memory 132, and an acceleration. A sensor unit 133, a geomagnetic sensor unit 134, a GPS receiver unit 135, a gyro sensor unit 136, an RTC (real-time clock) 137, a camera 140, a switch input unit 150, a voice input / output unit 160, and an external device I / F 170 are included. Each processing unit is connected via the bus 101.

Further, the image management apparatus 100 stores application programs in the storage 125, the control unit 114 expands the programs from the storage 125 to the memory 132, and the control unit 114 executes the programs to realize various functions. can do. In the following description, in order to simplify the description, various functions realized by the control unit 114 executing each application program will be described as being realized mainly by various program function units.

The application program may be stored in the storage 125 in advance before the image management apparatus 100 is shipped, or a medium such as a CD (Compact Disk) or a DVD (Digital Versatile Disk) or a semiconductor memory. And installed in the image management apparatus 100 via a medium connection unit (not shown).

Further, the application program may be downloaded and installed from an external network (not shown) via the communication I / F 113 and a wireless router (not shown). Or you may download and install from a delivery source via the base station which is not illustrated via mobile communication I / F131. Furthermore, it is also possible to connect to a personal computer (PC) that has acquired an application program via a network via an external device connection I / F (not shown), and move or copy from the PC to the image management apparatus 100 for installation. .

Also, the application program can be realized by hardware as a processing unit having the same function. When implemented as hardware, each processing unit takes the lead in realizing each function.

The communication I / F 113 is connected to a wireless router (not shown) via a wireless LAN or the like. The communication I / F 113 is connected to an external network via a wireless router, and transmits / receives information to / from a server on the external network. In addition to or instead of the communication function with the wireless router, it is possible to directly communicate with the server without using the wireless router by a method such as a wireless LAN such as Wi-Fi (registered trademark). The communication I / F 113 may be mounted with chips that perform different communication methods. Moreover, you may mount as one chip | tip which handles several communication systems.

The mobile communication I / F 131 is a third generation such as GSM (registered trademark) (Global System for Mobile Communications) system, W-CDMA (Wideband Code Division Multiple Access) system, CDMA2000 system, UMTS (Universal Mobile Telecommunications System) system, etc. A mobile communication system (hereinafter referred to as “3G”) or a mobile communication network such as LTE (Long Term Evolution) is used to connect to a communication network through a base station, and send and receive information to and from a server on the communication network. It can be carried out. In addition, the connection with the external network by the communication I / F 113 can be given priority over the communication network connection by the mobile communication I / F 131.

The control unit 114 is configured by, for example, a CPU (Central Processing Unit). The control unit 114 receives a user operation request via the input unit 115, and controls various program function units such as the signal separation unit 121, the display unit 117, and the communication I / F 113.

Further, the control unit 114 can acquire various information from a server on the external network via the communication I / F 113 and the wireless router, or from the server on the external network via the mobile communication I / F 131 and the base station. It also has a function to be passed to the functional part.

The storage 125 is configured using, for example, a ROM (Read Only Memory) or a HDD (Hard Disk Drive). The storage 125 is controlled by an instruction from the control unit 114 and can store an application program. In addition, various information created by the application program is stored.

Further, content such as a video / audio stream may be stored from a signal received from the tuner / demodulator 122, the communication I / F 113, or the mobile communication I / F 131. The storage 125 may be built in the image management apparatus 100 or may be a portable memory that can be attached to and detached from the image management apparatus 100.

A memory (RAM: Random Access Memory) 132 is controlled by an instruction from the control unit 114. The function unit of the application program stored in the storage 125 is expanded in the memory 132 by the control unit 114.

The display unit 117 is configured by a liquid crystal display, for example. The display unit 117 displays images and videos stored in the storage 125, captured and received images and videos, UI for various operations, and the like. The image or video to be displayed may be an image generated by an application program or a content image or video received via the tuner / demodulator 122, or an image received from a server on the external network via the communication I / F 113. It may be a video or an image or video distributed from a server on a communication network via the mobile communication I / F 131. Moreover, the display part 117 may be comprised integrally with the following touch panel etc., for example.

The input unit 115 is an input unit that receives an operation on the image management apparatus 100 from the user and inputs control information related to the input operation. For example, a touch panel stacked on the display unit 117 can be used. Also, voice input using the voice input / output unit 160 and gesture input using the camera 140 can be considered.

The acceleration sensor unit 133 measures the acceleration applied to the image management apparatus 100. The control unit 114 can know which part of the image management apparatus 100 is above, for example, by measuring the gravitational acceleration by the acceleration sensor unit 133.

The geomagnetic sensor unit 134 measures geomagnetism by using a plurality of magnetic sensors.

The GPS receiving unit 135 receives signals transmitted from a plurality of satellites using GPS (Global Positioning System). The control unit 114 can calculate the position information of the image management apparatus 100 based on the signal received by the GPS reception unit 135.

The gyro sensor unit 136 measures the angular velocity of the image management apparatus 100 that occurs when the user moves the image management apparatus 100.

The image management apparatus 100 can estimate the behavior and situation of the owner of the image management apparatus 100 such as walking, sleeping, and driving by using the measured acceleration, position information, and angular velocity.

The camera 140 includes an optical system such as a lens, an image sensor, a signal processing circuit, and the like. The control unit 114 controls exposure control, focus control, acquired image pixel number control, compression control of the camera 140, and recording control of an image captured by the camera 140 in the storage 125 according to a camera control program recorded in the storage 125. Control. The camera 140 includes an inner camera 141 provided on the same surface as the screen of the display unit 117 and an outer camera 142 provided on a surface opposite to the screen of the display unit 117 in the image management apparatus 100.

The switch input unit 150 takes in switch information in response to an operation of a physical button 151 (one or more), takes in switch information through the bus 101, and is used to control various application programs as necessary. The For example, as an example, the two buttons 151 are used to control the sound output magnitude adjustment. One or a plurality of physical buttons 151 may be provided.

The audio input / output unit 160 inputs / outputs an audio input signal from the microphone 161 provided in the image management apparatus 100 and an audio output signal to the speaker 162. Is controlled.

FIG. 2 is a functional block diagram of the image management apparatus 100. Part of the storage 125 constitutes a face image storage unit 221, an image storage unit 222, an operation / behavior history information storage unit 223, and a score calculation formula storage unit 224.

The face image storage unit 221 includes a face image of a person who allows the disclosure of an image or video (hereinafter collectively referred to as image data) in which he / she is photographed, or a person who does not permit the disclosure of an image in which he / she is photographed. Memorize face images.

The image storage unit 222 stores image data to be processed.

The operation / behavior history information storage unit 223 stores at least one of operation history information of the image management apparatus 100 and movement history information of the image management apparatus 100.

The score calculation formula storage unit 224 stores a score calculation formula for calculating a disclosure determination score for classifying image data as public or private.

The program for determining whether to open / close an image includes an image acquisition unit 201, an image feature amount collection unit 202, an image storage unit 203, an action feature amount collection unit 204, a score calculation unit 205, a public image determination unit 206, and a user correction unit. 207, a feedback unit 208, and an image disclosure unit 209. The image feature amount collection unit 202 includes a subject extraction unit 210, a face recognition unit 211, and an open information acquisition unit 212. Further, the behavior feature amount collection unit 204 includes a voice recognition unit 213.

The image acquisition unit 201 acquires image data to be processed. The image data may be an image (still image) or a video (moving image). Here, the image data may be taken by the camera 140 built in the image management apparatus 100, may be received from the outside by the communication I / F 113 and the mobile communication I / F 131, or may be tuner / demodulation. It may be received by the unit 122.

The image feature amount collection unit 202 collects image feature amounts used for calculating the public determination score from the analysis result of the image data or incidental information attached to the image data, for example, Exif (Exchangeable image file format) information. In the present embodiment, the collected image feature amount is digitized.

The subject extraction unit 210 extracts each subject region that is captured in the image data. The face recognition unit 211 recognizes whether or not a person's face is shot in the extracted subject area, and the subject area (face image) where the face is shot and the face image stored in the face image storage unit 221. , And digitizes information indicating whether a person who is permitted to publish is photographed or a non-public person is photographed, and outputs the information to the score calculation unit 205.

The face recognizing unit 211 identifies a person based on the face image in the subject area or the photographer's face image of the image data captured by the inner camera (the face only needs to be recognized), and the open information acquiring unit 212. Reads out the profile information of those persons from the storage of the image management apparatus 100 and digitizes them, or obtains them by connecting to an external network, for example, SNS (Social Networking Service), and digitizes the profile information according to the disclosure allowance. Or you may.

The open information acquisition unit 212 may refer to the shooting date / time and location information from the incidental information, and collect external information such as the date / time, weather conditions and map information at that location by connecting to an external network. Also good.

As described above, the image feature amount may be information on the date and time or place where the image data was shot, created, or edited, or the sex, age, clothes, posture, action, facial expression, etc. regarding the person who is the subject of the image data. Information, information such as the type, distance, motion, and brightness of the object to be imaged in the image data, or information related to the image capturing condition such as the image quality, focus, and image time of the image data may be used. Or, if the individual of the person who is the subject of the image data is identified by image analysis such as personal identification, the personal profile information, the relationship with the photographer, or when multiple persons are subjects May be information on the relationship between the persons.

The behavior feature amount collection unit 204 performs a process of quantifying information related to the behavior performed by the user on the image data. Here, in order to digitize the information, an operation history with respect to the input unit 115 may be used or analyzed, or the image input / output unit 160 may display the image data of the image management apparatus 100. The surrounding voice may be used or analyzed.

Further, the behavior feature amount collecting unit 204 may acquire visual information around the image management apparatus 100 when the image data is displayed by the camera 140 and collect the analysis result as the behavior feature amount.

Further, the behavior feature amount collection unit 204 may use or analyze values of various sensors (sensors that collect exercise state and position information) such as the GPS reception unit 135 and the acceleration sensor unit 133. For example, the behavior feature amount collection unit 204 analyzes various sensor information such as a GPS unit and an acceleration sensor, and then takes a photographer and a person who has viewed the screen of the display unit 117 that has moved the image data ( (Hereinafter referred to as “browser”) whether it was at home, whether it was in a public place, whether it was in a car or airplane, whether it was a selfie, Good.

In addition, the behavior feature amount is an operation history information, for example, copy, delete, rename, folder move, share, upload, download, operation history information, for example, image data (image data file) It may be a history of file operations such as email attachment, compression, decompression, etc., or a history of image editing or video editing such as editing, appending, image quality adjustment, image size change, and video effect addition. Further, it may be an operation history related to meta information such as favorite registration, tagging, and application registration.

In addition, the behavior feature amount collection unit 204 acquires the states of the control unit 114, the display unit 117, the memory 132, and the storage 125, and is an application that is used when or before or after browsing an image or video based on the image data. Alternatively, information related to a browsed file or Web page may be collected as an action feature amount.

Further, the inner camera 141 captures the face image of the viewer with the image data displayed on the display unit 117, and the face recognition unit 211 performs face recognition processing of the image data from the inner camera 141, Identify the photographer's face image. The behavior feature amount collection unit 204 collects a comparison result between the face image of the viewer or the photographer and the face image stored in the face image storage unit 221 as the behavior feature amount. For example, information such as the sex, age, clothes, posture, behavior, facial expression, etc. of the photographer or viewer may be collected as the behavior feature amount.

Similarly to the open information acquisition unit 212, the behavior feature amount collection unit 204 has a function of connecting to an external network, and includes profile information of the viewer, information regarding the relationship with the owner of the image management apparatus 100, Information regarding the relationship with the person who is the subject of the data may be retrieved and collected from an external network, for example, a social working service.

The image storage unit 203 has a function of associating image feature amount information and action feature amount information with the image data and writing them in the image storage unit 222 and reading out image data to be processed from the image storage unit 222.

The score calculation unit 205 reads the score calculation formula from the score calculation formula storage unit 224, applies the behavior feature quantity and the image feature quantity corresponding to the image data, and calculates the disclosure determination score of the image data.

The public image determination unit 206 tentatively determines whether the image data is public or non-public based on the comparison result of the score determination threshold and the public determination score indicating that the disclosure is permitted. In this embodiment, two states of public / non-public are assumed, but it is also possible to set the disclosure range to three or more states in stages, such as disclosure to acquaintances and disclosure to family members. is there. In the third embodiment, the public image determination unit 206 has a function of executing a masking process on a non-public face image, details of which will be described in the third embodiment.

The user correction unit 207 accepts a correction instruction from the user for the public or non-public determination result tentatively determined for the image data.

The feedback unit 208 changes the score calculation formula stored in the score calculation formula storage unit 224 based on the change history of image disclosure / non-publication, the image feature amount, and the behavior feature amount. The score calculation formula is changed using, for example, machine learning. In machine learning of class classification such as support vector machine and neural network, by inputting training data consisting of feature quantity represented by vector and correct answer class, calculation formula to classify unknown feature quantity into correct class is output can do. In this image management apparatus, the image feature value and the action feature value are regarded as feature values, and the public / non-disclosure change history of the image manually operated by the user is regarded as a correct class, and machine learning is performed. The calculation formula for classifying into two public classes can be updated.

The image publishing unit 209 publishes the image data when it is decided to publish. Prior to publication, a confirmation screen showing the determination result of the image data is displayed on the display unit 117. However, the image data determined to be private by the public image determination unit 206 is not displayed. Alternatively, it is displayed that the image data cannot be displayed.

Next, processing of the image management apparatus according to the first embodiment will be described with reference to FIGS. FIG. 3 is a flowchart showing the first half process in the image management apparatus according to the first embodiment. FIG. 4 is a flowchart illustrating the latter half of the process in the image management apparatus according to the first embodiment. FIG. 5 is a diagram illustrating an example of a score calculation process. FIG. 6 is a diagram illustrating an example of a user correction screen according to the first embodiment. In the following, description will be given along the order of steps in FIGS. 3 and 4.

As shown in FIG. 3, when the main power of the image management apparatus 100 is turned on (S01), the behavior feature amount collection unit 204 starts accumulation of behavior feature amounts (S02).

The behavior feature amount is information obtained by quantifying a factor (hereinafter referred to as “disclosure determination factor”) for determining whether or not to allow a predetermined captured image to be disclosed. As an example of the behavior feature amount, for example, information obtained by digitizing the operation history of the button 151 received by the switch input unit 150 may be used, or the acceleration sensor unit 133, the geomagnetic sensor unit 134, the GPS reception unit 135, and the gyro sensor unit 136 may be used. The sensor result (raw data) detected by may be stored as movement history information.

Further, the behavior feature amount collection unit 204 is based on the variation amount of the sensor result, and the movement state of the person carrying the image management apparatus 100 (hereinafter referred to as “user”), for example, stationary, walking, or moving by a vehicle. The moving state may be determined and the result may be temporarily stored.

In this embodiment, as an example of quantifying the behavior feature value, a flag “1” is used when corresponding to each disclosure determination factor, and “0” is set when the disclosure determination factor is not applicable. You may define action feature-value using the positive / negative number according to.

When the user performs shooting using the camera 140 of the image management apparatus 100 (S03), the image acquisition unit 201 acquires image data generated by shooting by the camera 140, and acquires date and time information acquired from the RTC 137, The position information of the shooting location is acquired from the GPS receiving unit 135, the Exif information is written, the image data is attached to the image storage unit 203, and is output. The Exif information includes various information defined by the Exif standard, such as device information for specifying a photographing device, in addition to photographing conditions such as screen brightness and image resolution.

At the same time, the behavior feature amount collection unit 204 outputs the behavior feature amount collection result regarding the captured image to the image storage unit 203. The image storage unit 203 attaches Exif information and behavior feature amount data to the image data and stores them in the storage 125 (S04).

In this step, the action feature quantity output by the action feature quantity collection unit 204 to the image storage unit 203 includes the action feature quantity continuously accumulated from step S01, information identifying the photographer, and environment information of the shooting location. May be included.

For example, the behavior feature amount collection unit 204 causes the inner camera 141 to simultaneously capture the photographer's image when the outer camera 142 captures the image, and causes the face recognition unit 211 to output the photographer image. The face recognition unit 211 compares the face image of the owner of the image management apparatus 100 stored in the storage 125 in advance with the photographer image to determine whether or not they are the same person, and outputs the determination result to the behavior feature amount collection unit 204. . The behavior feature amount collection unit 204 inputs “1” as the value of the owner flag if the owner of the image management apparatus 100 and “0” if it is a non-owner.

In addition, when the self-shooting stick control unit 171 is connected to the external device I / F 170 and the fact that the shooting instruction signal is output to the self-shooting stick control unit 171 is recorded as the operation information, behavior feature amount collection is performed. The unit 204 sets “1” as the flag value of the “selfie” item as a disclosure determination factor.

Further, the behavior feature amount collection unit 204 collects sounds around the time when the shooting button is pressed with the microphone 161 and inputs the collected sounds to the voice input / output unit 160. When the voice recognition unit 213 detects from the voice signals collected before and after the image is taken that a voice such as “upload” or “show to everyone” is spoken by the photographer's voice, or a cheering sound was raised during the shooting. If it is detected that the laughter has risen, the behavior feature amount collection unit 204 sets the value of the public recommendation flag to “1” for the image.

In addition, the behavior feature amount collection unit 204, for example, when there is an operation history of posting a photo to the SNS immediately after the image is taken, to send mail to a specific acquaintance, or to “share” or “good photo” in the content of the email If a specific phrase such as “I want to see” is included, it is determined that the image may be disclosed, and the value of the recommendation flag is set to “1”.

The score calculation unit 205 reads the score calculation formula stored in the score calculation formula storage unit 224, and calculates the score by substituting the image feature amount and behavior feature amount values attached to the image into the score calculation formula (S05). ). Then, the public image determination unit 206 compares the score calculated for determining whether the public condition is satisfied with the public determination threshold, and if the calculated score is equal to or greater than the public determination threshold (S06 / YES). It is tentatively determined to publish the image, and a value “1” is entered in the disclosure flag (S07). If the calculated score is less than the disclosure determination threshold value (S06 / NO), it is tentatively determined that the image is not disclosed and a value “0” is set in the disclosure flag (S08). Thereafter, the image storage unit 203 adds a public flag to the image and stores it in the storage (S09).

Taking the score calculation formula as an example, the following formula (1) may be used.

FIG. 5 shows the weighting coefficient of the equation (1) and the flag values of the images 1 to 6. Note that the items and weighting coefficients are stored in the score calculation formula storage unit 224, and the flag values of the items for the images 1 to 6 are attached to the images and stored. However, FIG. Therefore, they are shown in the same table.

As shown in FIG. 5, according to the equation (1), the score S1 of the image 1 “image taken by the owner with the child in the sightseeing spot” is 100 + 1 + 50 + 0.5 = 151.5. At this time, when the disclosure determination threshold value Sth1 is 50, the image 1 is determined to be disclosed.

As another example, the score S1 of the image 2 “image where the child is sleeping at home” is 1 + 0.5-10 = −8.5, and it is determined that the score is not disclosed.

Further, the score S1 of the image 3 “image taken by the owner while drinking at home” is 1 + 1 + 50-100 = −48, and is determined to be private.

The score calculation formula may be the following formula (2).

Expression (2) sets the disclosure determination threshold value Sth2 to a value higher than Sth1, that is, a value that further restricts the disclosure, but when the disclosure recommendation flag is set to “1”, It is a score calculation formula for determining. In this way, not only the score value alone but also the value of the recommendation recommendation flag digitized by the behavior feature amount collection unit 204 may be incorporated into the logical operation, and the disclosure determination may be performed with emphasis on the behavior feature amount.

For example, the image 4 is the same image as the image 3, but the cheering of the drinking scene in the image taken in the party scene at home is detected by the voice recognition unit 213. Suppose that In this case, although it is determined that it is not disclosed in the equation (1), it is determined that it is disclosed in the equation (2).

Further, the image 5 is an image that the behavior feature amount collection unit 204 determines to be an “in driving” image. In this case, the behavior feature amount collection unit 204 may be configured to set the value of the “non-public recommendation flag” to “1” and determine that the value is private regardless of the value of the score S1 according to the equation (2). .

In addition, the action feature amount collection unit 204 captures a through image from the inner camera 141 while the image is captured on the display unit 117 after capturing, and determines whether the non-owner of the image management apparatus 100 is captured in the face recognition unit 211. You may judge. When the non-owner is shown, it is determined that the captured image is an image that may be shown to others, and “1” may be set in the “public recommendation flag”.

In the above equation (2), AND may be used as a logical operation instead of OR. In this case, based on not only the score value but also the behavior feature amount collected when the image is taken, it is possible to determine that the image whose disclosure determination flag value is “1” is disclosed. Through the above process, a process of automatically tentatively determining whether an image is public / non-public is executed. Next, correction by the user is performed, and release / non-disclosure of the image is confirmed and the image is released. Note that “determined” here is not intended to prohibit further correction after user correction, but, for example, after tentatively determined to be released is corrected to open by user correction, then the user corrects it again to non-public You can also

As shown in FIG. 4, the public image determination unit 206 reads the public determination target image from the storage (S11), and reads the value of the public flag attached to each image (S12). Then, a “public” decision button is made close to each image (including superposition) for an image with the public flag value “1”, and a “non-public” decision button is made for an image with the public flag value “0”. A disclosure confirmation screen is displayed on the screen of the display unit 117 (S13).

FIG. 6 is a diagram illustrating an example of a user correction screen according to the first embodiment, and illustrates a state in which a UI for accepting a manual operation is displayed on a touch panel in which the display unit 117 and the input unit 115 are integrated.

The icon 301 indicates that the image 302 is determined to be public. An icon 311 indicates that whether to disclose the image 312 is automatically determined and manual correction is not performed. An icon 321 indicates that the image 322 is not disclosed. An icon 331 indicates that the disclosure / non-disclosure of the image 332 is automatically determined. The user can change the disclosure / non-disclosure of the image by touching the icon. For example, when the icon 301 is touched, that is, when manual correction is performed, the icon 301 changes to the appearance of the icon 321 and the image 302 is not disclosed.

When the same operation is performed on an image for which disclosure / non-disclosure is automatically determined, the

icons

311 and 331 indicating that disclosure / non-disclosure is automatically determined disappear. For example, when the icon 313 is touched to make the image 312 private, the icon 311 disappears.

Further, the user correction unit 207 need not be operated on the touch panel, and may be an operation such as an input using a physical switch, an audio input using the audio input / output unit 160, or a gesture input using the camera 140.

When the user correction unit 207 receives a manual correction operation by the user (S14 / YES), the user correction unit 207 rewrites and stores the value of the image disclosure flag in accordance with the content of the manual correction operation (S15).

The feedback unit 208 performs a so-called feedback process in which the score calculation formula used for the manually corrected image is corrected based on the manual correction content (S16). As an example of feedback processing, when the status is changed from non-public to public, the weight coefficient of an item whose public determination flag of the manually corrected image is “1” is increased. When it is changed from public to non-public, the weight coefficient of the item for which the public determination flag of the manually corrected image is “1” is decreased.

When the input unit accepts an image publishing operation from the user (S17 / YES), the image publishing unit 209 extracts and publishes only the images whose disclosure flag value is “1” from the storage (S18).

According to the present embodiment, it is possible to determine whether the image is open / closed according to the image feature amount and the behavior feature amount. Further, when manual correction is performed on the automatically / publicly disclosed result, the score calculation formula used for the public determination is corrected according to the manual correction content. While repeating this, the user's preference can be reflected in the score calculation formula.

<Second embodiment>
The second embodiment is an embodiment in which public / private information is input from a photographer at the time of shooting and is reflected in score calculation. Hereinafter, a second embodiment will be described with reference to FIG. The second embodiment is a diagram illustrating an example of a user correction screen according to the second embodiment.

Specifically, in the shooting step (S03) of the first embodiment, when a through image is displayed on the screen of the display unit 117 when shooting with the camera 140 activated, the display unit 117 as shown in FIG. A public icon 401 for designating a public image and a private icon 402 for designating a private image are displayed on the touch panel integrated with the input unit 115. The public icon 401 and the private icon 402 have a function as a shutter button for taking a picture with the camera 140. Therefore, the public icon 401 corresponds to the first shooting button, and the non-public icon 402 corresponds to the second shooting button. The user correction unit 207 executes the display process of the public icon 401 and the non-public icon 402 and the operation of writing the result of pressing the public icon 401 and the result of the pressing on the public flag.

Then, when the photographer touches the public icon 401, the action feature amount collection unit 204 writes 1 in the “public flag” in the photographed image, and when the photographer touches the private icon 402, the photograph is taken. 0 is written in the “public flag” of the image.

According to the present embodiment, a touch operation on the public icon 401 or the non-public icon 402 can simultaneously perform an operation of pressing the shutter button and a public / non-public determination process, thereby improving operability.

<Third embodiment>
The third embodiment is an embodiment in which when a private person, for example, an image showing a person other than himself / herself is disclosed, masking processing such as mosaic or blurring is performed on the face of the private person and then the public person is released.

The image 6 “image taken by the owner with a private person in the tourist spot” in FIG. 5 is 100 + 1 + 50−200 = −49 according to the above-described formula (1), and is provisional in step S08. Thus, “0” is written in the value of the public flag.

On the other hand, when the user manually corrects and changes from private to public in step S14, the face recognition unit 211 identifies a private person among the facial images recognized from the photographed image, and the identification result (of the photographed image). The position information of the area where the non-public person is photographed is output to the image disclosure unit 209. The image publishing unit 209 performs a processing process (masking process) for not identifying a face on the face image of a non-public person, and the image publishing unit 209 publishes the image subjected to the masking process.

The image storage unit 203 may store an image that has undergone masking processing, but the image data stores an image that has not been subjected to masking processing, and the image disclosure unit 209 writes “1” in the masking processing flag. You may save it. Then, when the image publishing unit 209 next publishes an image, the masking process flag may be referred to. If the value of the flag is “1”, the masking process may be performed and then the image may be published. .

According to the present embodiment, it is possible to prevent a private person's face from being disclosed even when an image of a private person being captured is handled as a public image by manual correction.

As a modification of the present embodiment, the image feature amount collection unit 202 acquires information specifying the destination to which the image publishing unit 209 publishes (outputs) an image until the actual publication after the publishing operation. To do. Then, the face image in the SNS designated as the disclosure destination may be searched, and a subscriber to the same SNS may be handled as a publicly permitted person (not a private person). In other words, when the subject in the captured image matches the face image of the member in the SNS designated as the connection destination, the SNS is treated as a public image and output without subjecting the subject's face image to masking processing. To do.

On the other hand, when attempting to upload the same image to another SNS, the subject is disclosed after being subjected to a masking process on the assumption that the subject is a private person.

According to this modified example, it is possible to dynamically switch between a private person and a publicly permitted person according to the publication destination. On the other hand, the image management apparatus can store an image that is not subjected to masking processing, so that the user can view the original image.

The above embodiments are merely examples of the present invention and are not intended to limit the present invention. Various changes and modifications can be made by those skilled in the art within the scope of the technical idea disclosed in this specification.

For example, if there is an operation history for taking in / out an image from / from a private folder, the behavior feature amount collection unit 204 determines that the image placed in a specific folder may be disclosed, and sets “1” in the “publication recommended flag” of the image. May be included.

Furthermore, a family face image is stored in advance in the face image storage unit 221. Then, the face recognizing unit 211 compares the face photographed in the image with the family face image, and determines that the image taken together with the opposite sex other than the family should not be disclosed. “1” may be entered in the “flag”.

In addition, the behavior feature amount collection unit 204 refers to the shooting date / time information and the position information, determines that a photograph shown in a place other than the home at night or in an inappropriate place should not be disclosed, “1” may be entered in the “flag”.

100: Image management apparatus, 313: Icon (public), 321: Icon (non-public), 331: Icon (automatic)

Claims

An image acquisition unit for acquiring image data to be processed;
Based on information representing the behavior of the photographer of the image data or the viewer of the image data, action feature amounts used for calculating a disclosure determination score for classifying the image data as public or private are collected. An action feature collection unit;
A score calculation formula storage unit that stores a score calculation formula for calculating the disclosure determination score;
A score calculation unit that reads the score calculation formula from the score calculation formula storage unit, applies the behavior feature amount corresponding to the image data to the score calculation formula, and calculates a disclosure determination score of the image data;
Determining whether or not the disclosure determination score satisfies a disclosure condition, and based on a result of the determination, a disclosure image determination unit that tentatively determines that the image data is to be disclosed or not disclosed;
A user correction unit that receives a correction instruction from a user for a determination result of disclosure or non-disclosure determined provisionally for the image data;
A feedback unit that changes the score calculation formula based on a public or private change history indicated by the correction instruction;
An image publishing unit for publishing the image data when the image data is decided to be published;
An image management apparatus comprising:
The image management apparatus according to claim 1,
An image feature amount collecting unit for collecting an image feature amount used for calculating the disclosure determination score from an image analysis result of the image data or incidental information attached to the image data;
The score calculation unit further applies the image feature amount to the score calculation formula to calculate a disclosure determination score of the image data;
An image management apparatus.
The image management apparatus according to claim 2,
A face image storage unit for storing a face image of a person who allows the release of image data showing himself or a person who does not allow the release of an image showing himself;
The image feature amount collection unit, a subject extraction unit that extracts a subject of the image data;
A face recognition unit that recognizes a face image included in the subject and compares the face image stored in the face image storage unit;
The score calculation unit calculates the disclosure determination score based on a comparison result of the face recognition unit;
An image management apparatus.
The face image management apparatus according to claim 3.
The image feature amount collection unit is connected to a social networking service that the image management device has connected in the past, and an open information acquisition unit that determines whether the face image recognized by the face recognition unit is uploaded to the social networking service Further comprising
The score calculation unit calculates the disclosure determination score based on a determination result of the open information acquisition unit;
An image management apparatus.
The image management apparatus according to claim 1,
An operation action history information storage unit that stores at least one of operation history information of the image management apparatus and movement history information of the image management apparatus;
The behavior feature amount collecting unit collects the movement feature amount based on the operation history information and the movement history information accumulated in the operation behavior history information storage unit;
An image management apparatus.
The image management apparatus according to claim 1,
A camera that shoots a subject and generates image data;
A face image storage unit for storing a face image of a person who allows the release of image data showing himself or a face image of a person who does not allow the release of an image showing himself;
The image acquisition unit acquires image data to be processed from the camera,
The behavior feature amount collection unit collects a comparison result between the photographer's face image and the face image stored in the face image storage unit as a behavior feature amount,
The image publishing unit tentatively determines that the image is to be published when the face image of the subject of the image data matches the image of the person permitted to publish stored in the face image storage unit.
An image management apparatus.
The image management apparatus according to claim 1,
A microphone,
The behavior feature amount collecting unit includes a voice recognition unit that analyzes voice data collected by the microphone, and collects a behavior feature amount based on an identification result of the voice recognition unit.
An image management apparatus.
The image management apparatus according to claim 1,
Comprising at least one sensor for collecting the motion state of the image management device;
The behavior feature amount collecting unit collects the behavior feature amount based on an output result of the sensor;
An image management apparatus.
The image management apparatus according to claim 1,
A display unit having a screen for displaying the image data;
The public image determination unit displays the image data and an icon indicating a determination result of disclosure or non-disclosure determined provisionally for the image data on the screen.
An image management apparatus.
The image management apparatus according to claim 1,
A camera that shoots a subject and generates image data;
A display unit having a screen for displaying the image data;
A first shooting button for simultaneously inputting that a through image of the subject generated by the camera displayed on the screen is generated and stored as a public image, the through image is captured and stored as a private image; A second shooting button for inputting simultaneously,
An image management apparatus.
The image management apparatus according to claim 6.
The user correction unit releases the image including the face image of the person who does not allow the image of the image in which the image is shown to be disclosed to the image data that the release image determination unit has determined to be unpublished provisionally. When receiving an input of a correction instruction to perform, the image publication unit publishes an image that has been subjected to masking processing for a face image that does not allow publication included in the image data,
An image management apparatus.
The image management apparatus according to claim 2,
The feedback unit changes the score calculation formula so that an image feature amount or an action feature amount used for calculating a score of image data having a change history from public to private does not contribute to the disclosure determination score. And changing the score calculation formula so that the image feature amount or the behavior feature amount used for calculating the score of the image data having the history of change from private to public has a high contribution to the public determination score. A featured image management apparatus.
Obtaining image data to be processed;
Based on information representing the behavior of the photographer of the image data or the viewer of the image data, action feature amounts used for calculating a disclosure determination score for classifying the image data as public or private are collected. Steps,
Applying the behavior feature amount corresponding to the image data to a score calculation formula for calculating the disclosure determination score, and calculating a disclosure determination score of the image data;
Determining whether or not the disclosure determination score satisfies a disclosure condition, and tentatively determining whether to make the image data public or private based on a result of the determination;
Accepting a correction instruction from a user for a determination result of disclosure or non-disclosure determined provisionally for the image data;
Changing the score calculation formula based on a public or non-public change history indicated by the correction instruction;
Publishing the image data when the image data is determined to be published;
An image management method comprising:
Obtaining image data to be processed;
Based on information representing the behavior of the photographer of the image data or the viewer of the image data, action feature amounts used for calculating a disclosure determination score for classifying the image data as public or private are collected. Steps,
Applying the behavior feature amount corresponding to the image data to a score calculation formula for calculating the disclosure determination score, and calculating a disclosure determination score of the image data;
Determining whether or not the disclosure determination score satisfies a disclosure condition, and tentatively determining whether to make the image data public or private based on a result of the determination;
Accepting a correction instruction from a user for a determination result of disclosure or non-disclosure determined provisionally for the image data;
Changing the score calculation formula based on a public or non-public change history indicated by the correction instruction;
Publishing the image data when the image data is determined to be published;
An image management program for causing a computer to execute the above.