JP6799660B2 - Image processing device, image processing method, program - Google Patents

Description

The present invention relates to an automatic photographing technique in an imaging device.

When shooting still images and movies with an image pickup device such as a camera, it is normal for the user to determine the shooting target through a viewfinder, etc., check the shooting status by himself, and adjust the framing of the shot image to shoot the image. is there. Such an imaging device is provided with a function of detecting an operation error of the user and notifying the user, detecting the external environment, and notifying the user when it is not suitable for shooting. In addition, there has been a mechanism for controlling the camera so as to be in a state suitable for shooting.

There is a life log camera that periodically and continuously shoots an imaging device that performs shooting by such a user operation without giving a shooting instruction by the user (Patent Document 1). The life log camera is used in a state of being worn on the user's body with a strap or the like, and records a scene that the user sees in daily life as an image at regular time intervals. When shooting with a life log camera, the user does not shoot at the intended timing such as when the shutter is released, but shoots at regular time intervals, so it is possible to leave an unexpected moment as an image that is not normally shot.

Special Table 2016-536868 Japanese Unexamined Patent Publication No. 2004-354251

However, when the user wears the life log camera and performs automatic shooting on a regular basis, an image that the user does not like may be acquired, and the image at the moment that the user really wants to obtain may not be acquired.

Further, even if the life log camera has a learning function and can learn the moment to be photographed and automatically photograph the moment, a large amount of teacher data is required for the learning. In Patent Document 2, in a defect inspection device that inspects the presence or absence of defects in a test object using a neural network, an artificial defect image of the test object is created by image processing to make up for the lack of learning patterns. The technology is disclosed. However, unlike the defect inspection in which the types of the test objects are limited, the types of subjects in the life log camera are infinite, and it is difficult to make up for the lack of learning patterns by image processing.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device capable of acquiring a suitable image for a user without performing a special operation. ..

The image processing apparatus according to the present invention includes an acquisition means for acquiring teacher data regarding a captured image captured by an imaging means and a learning means for generating a learning model for evaluating an image based on the teacher data. The learning means has, and the teacher data based on the recording image taken according to the user's instruction and the recording image taken according to the user's instruction are continuously photographed. It is characterized in that the learning model is generated by using the teacher data based on the learning image.

Further, the image processing apparatus according to the present invention communicates with a generation means for generating teacher data from a captured image taken by the imaging means and a communication for transmitting the teacher data generated by the generation means to the learning means for generating a learning model. The generation means includes means, and the generation means generates teacher data from a recording image taken according to a user's instruction, and with respect to the recording image taken according to the user's instruction. Teacher data is also generated from the images for learning taken continuously, and the communication means uses the teacher data generated from the image for recording and the teacher data generated from the image for learning. It is characterized in that it is transmitted to the learning means.

According to the present invention, it is possible to provide an image processing device capable of acquiring an image suitable for the user without performing a special operation by the user.

It is a figure which shows typically the image pickup apparatus. It is a figure which shows the structure of the image pickup apparatus. It is a figure which shows the structure of the image pickup apparatus and an external device. It is a figure which shows the structure of an external device. It is a figure which shows the structure of the image pickup apparatus and an external device. It is a figure which shows the structure of an external device. It is a flowchart explaining the 1st control circuit. It is a flowchart explaining the 2nd control circuit. It is a flowchart explaining the shooting mode processing. It is a figure explaining a neural network. It is a figure for demonstrating the area division in a photographed image. It is a flowchart explaining the learning mode determination. It is a flowchart explaining a learning process. It is a figure explaining the display process which concerns on this Embodiment.

[First Embodiment]
<Configuration of imaging device>
FIG. 1 is a diagram schematically showing an imaging device according to the first embodiment.

The image pickup device 101 shown in FIG. 1A is provided with an operating member capable of operating the power switch (hereinafter, referred to as a power button, but may be operated by tapping, flicking, swiping, etc. on the touch panel). There is. The lens barrel 102, which is a housing including a group of photographing lenses for imaging and an image sensor, is attached to the image pickup device 101 and is provided with a rotation mechanism capable of rotationally driving the lens barrel 102 with respect to the fixed portion 103. The tilt rotation unit 104 is a motor drive mechanism capable of rotating the lens barrel 102 in the pitch direction shown in FIG. 1B, and the pan rotation unit 105 is a motor drive mechanism capable of rotating the lens barrel 102 in the yaw direction. Therefore, the lens barrel 102 can rotate in one or more axes. Note that FIG. 1B is an axis definition at the fixed portion 103 position. Both the angular velocity meter 106 and the accelerometer 107 are mounted on the fixed portion 103 of the image pickup apparatus 101. Then, the vibration of the image pickup apparatus 101 is detected based on the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit and the pan rotation unit are rotationally driven based on the detected shaking angles. As a result, the lens barrel 102, which is a movable portion, is configured to correct the runout and the tilt.

FIG. 2 is a block diagram showing the configuration of the image pickup apparatus of the present embodiment.

In FIG. 2, the first control circuit 223 includes a processor (for example, CPU, GPU, microprocessor, MPU, etc.) and a memory (for example, DRAM, SRAM, etc.). These perform various processes to control each block of the image pickup apparatus 101, and control data transfer between each block. The non-volatile memory (EEPROM) 216 is a memory that can be electrically erased and recorded, and stores constants, programs, and the like for the operation of the first control circuit 223.

In FIG. 2, the zoom unit 201 includes a zoom lens that performs scaling. The zoom drive control circuit 202 drives and controls the zoom unit 201. The focus unit 203 includes a lens for adjusting the focus. The focus drive control circuit 204 drives and controls the focus unit 203.

The image pickup unit 206 includes an image pickup element and an A / D converter, and the image pickup element receives light incident through each lens group, and outputs charge information corresponding to the amount of light to the image processing circuit 207 as analog image data. The image processing circuit 207 is an arithmetic circuit equipped with a plurality of ALUs (Arithmetic and Logic Units), and is an image such as distortion correction, white balance adjustment, and color interpolation processing for digital image data output by A / D conversion. The processing is applied and the digital image data after application is output. The digital image data output from the image processing circuit 207 is converted into a recording format such as a JPEG format by the image recording circuit 208, and transmitted to the memory 215 and the video output circuit 217 described later.

The lens barrel rotation drive circuit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction.

The device shake detection circuit 209 is equipped with, for example, an angular velocity meter (gyro sensor) 106 that detects the angular velocity of the image pickup device 101 in the three axial directions, and an accelerometer (accelerometer) 107 that detects the acceleration of the device in the three axis directions. .. The device shake detection circuit 209 calculates the rotation angle of the device, the shift amount of the device, and the like based on the detected signal.

The audio input circuit 213 acquires an audio signal around the image pickup device 101 from a microphone provided in the image pickup apparatus 101, performs analog-to-digital conversion, and transmits the audio signal to the audio processing circuit 214. The voice processing circuit 214 performs voice-related processing such as optimization processing of the input digital voice signal. Then, the voice signal processed by the voice processing circuit 214 is transmitted to the memory 215 by the first control circuit 223. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing circuit 207 and the audio processing circuit 214.

The image processing circuit 207 and the audio processing circuit 214 read out the image signal and the audio signal temporarily stored in the memory 215, encode the image signal, encode the audio signal, and the like, and perform the compressed image signal and the compressed audio signal. To generate. The first control circuit 223 transmits these compressed image signals and compressed audio signals to the recording / reproducing circuit 220.

The recording / playback circuit 220 records the compressed image signal, the compressed audio signal, and other control data related to photographing on the recording medium 221 with the image processing circuit 207 and the audio processing circuit 214. When the audio signal is not compressed and encoded, the first control circuit 223 transfers the audio signal generated by the audio processing circuit 214 and the compressed image signal generated by the image processing circuit 207 to the recording / playback circuit 220. It is transmitted and recorded on the recording medium 221.

The recording medium 221 may be a recording medium built in the image pickup apparatus 101 or a removable recording medium. The recording medium 221 can record various data such as a compressed image signal, a compressed audio signal, and an audio signal generated by the image pickup apparatus 101, and a medium having a capacity larger than that of the non-volatile memory 216 is generally used. For example, the recording medium 221 includes all types of recording media such as hard disks, optical disks, magneto-optical disks, CD-Rs, DVD-Rs, magnetic tapes, non-volatile semiconductor memories, and flash memories.

The recording / reproducing circuit 220 reads (reproduces) a compressed image signal, a compressed audio signal, an audio signal, various data, and a program recorded on the recording medium 221. Then, the first control circuit 223 transmits the read compressed image signal and compressed audio signal to the image processing circuit 207 and the audio processing circuit 214. The image processing circuit 207 and the audio processing circuit 214 temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode them according to a predetermined procedure, and transmit the decoded signals to the video output circuit 217 and the audio output circuit 218. To do.

The voice input circuit 213 has a plurality of microphones mounted on the image pickup device 101, and the voice processing circuit 214 can detect the direction of sound on a plane on which the plurality of microphones are installed, for search and automatic shooting described later. Used. Further, the voice processing circuit 214 detects a specific voice command. The voice command may be configured so that the user can register a specific voice in the image pickup apparatus in addition to some commands registered in advance. It also recognizes sound scenes. In sound scene recognition, sound scene determination is performed by a network trained by machine learning based on a large amount of voice data in advance. For example, a network for detecting a specific scene such as "cheering", "applause", or "speaking" is set in the voice processing circuit 214. Then, when a specific sound scene or a specific voice command is detected, the detection trigger signal is output to the first control circuit 223 and the second control circuit 211.

A second control circuit 211, which is provided separately from the first control circuit 223 that controls the entire main system of the image pickup apparatus 101, controls the power supply of the first control circuit 223.

The first power supply circuit 210 and the second power supply circuit 212 supply electric power for operating the first control circuit 223 and the second control circuit 211, respectively. By pressing the power button provided on the image pickup apparatus 101, power is first supplied to both the first control circuit 223 and the second control circuit 211. As will be described later, the first control circuit 223 is the first power supply. It controls the circuit 210 to turn off its own power supply. Even while the first control circuit 223 is not operating, the second control circuit 211 is operating, and information from the device shake detection circuit 209 and the voice processing circuit 214 is input. The second control circuit is configured to perform a determination process of whether or not to start the first control circuit 223 based on various input information, and when the activation is determined, a power supply instruction is given to the first power supply circuit. ..

The audio output circuit 218 outputs a preset audio pattern from a speaker built in the image pickup apparatus 101, for example, during shooting.

The LED control circuit 224 controls the LED provided in the image pickup apparatus 101, for example, at the time of shooting, in a preset lighting / blinking pattern.

The video output circuit 217 is composed of, for example, a video output terminal, and transmits an image signal in order to display the video on a connected external display or the like. Further, the audio output circuit 218 and the video output circuit 217 may be one combined terminal, for example, a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

The communication circuit 222 communicates between the image pickup device 101 and the external device, and transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. In addition, it receives control signals related to shooting such as shooting start / end commands, pan / tilt, and zoom drive, and drives the imaging device 101 from instructions of an external device capable of intercommunication with the imaging device 101. In addition, information such as various parameters related to learning processed by the learning processing circuit 219, which will be described later, is transmitted and received between the image pickup device 101 and the external device. The communication circuit 222 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a WirelessUSB, and a GPS receiver.

<Configuration with external communication equipment>
FIG. 3 is a diagram showing a configuration example of a wireless communication system between the image pickup device 101 and the external device 301. The image pickup device 101 is a digital camera having a shooting function, and the external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

The image pickup device 101 and the smart device 301 are a master / slave of a control station and a subordinate station such as a communication 302 by a wireless LAN conforming to the IEEE802.11 standard series and a Bluetooth Low Energy (hereinafter referred to as “BLE”), for example. It is possible to communicate with the communication 303 having a relationship. The wireless LAN and BLE are examples of communication methods, and each communication device has two or more communication functions, for example, by one communication function that communicates in a relationship between a control station and a subordinate station. If it is possible to control the other communication function, another communication method may be used. However, without losing generality, the first communication such as wireless LAN can perform higher speed communication than the second communication such as BLE, and the second communication consumes more than the first communication. It shall be at least one of low power consumption and short communication range.

The configuration of the smart device 301 will be described with reference to FIG.

The smart device 301 has, for example, a wireless LAN control circuit 401 for wireless LAN, a BLE control circuit 402 for BLE, and a public line control circuit 406 for public wireless communication. In addition, the smart device 301 further includes a packet transmission / reception circuit 403. The wireless LAN control circuit 401 performs RF control of the wireless LAN, communication processing, a driver that performs various controls of communication by the wireless LAN conforming to the IEEE802.11 standard series, and protocol processing related to the communication by the wireless LAN. The BLE control circuit 402 performs a driver that performs RF control of BLE, communication processing, various controls of communication by BLE, and protocol processing related to communication by BLE. The public line control circuit 406 performs a driver for performing RF control of public wireless communication, communication processing, various controls of public wireless communication, and protocol processing related to public wireless communication. Public wireless communication is compliant with, for example, IMT (International Multimedia Telecommunication) standards and LTE (Long Term Evolution) standards. The packet transmission / reception circuit 403 performs processing for executing at least one of transmission and reception of packets related to communication by wireless LAN and BLE and public wireless communication. In this example, the smart device 301 is described as performing at least one of transmission and reception of packets in communication, but other communication formats such as circuit switching may be used in addition to packet switching. Good.

The smart device 301 further includes, for example, a control circuit 411, a storage circuit 404, a GPS receiving unit 405, a display device 407, an operating member 408, a voice input voice processing circuit 409, and a power supply circuit 410. The control circuit 411 controls the entire smart device 301, for example, by executing a control program stored in the storage circuit 404. The storage circuit 404 stores, for example, a control program executed by the control circuit 411 and various information such as parameters required for communication. Various operations described later are realized by the control circuit 411 executing the control program stored in the storage circuit 404.

The power supply circuit 410 supplies power to the smart device 301. The display device 407 has a function capable of outputting visually recognizable information such as an LCD or LED, or a sound output of a speaker or the like, and displays various information. The operation member 408 is, for example, a button or the like that receives an operation of the smart device 301 by a user. The display device 407 and the operation member 408 may be composed of a common member such as a touch panel.

The voice input voice processing circuit 409 may be configured to acquire the voice emitted by the user from, for example, a general-purpose microphone built in the smart device 301, and acquire the user's operation command by voice recognition processing.

In addition, voice commands are acquired by the user's pronunciation via a dedicated application in the smart device. Then, it can be registered as a specific voice command for causing the voice processing circuit 214 of the image pickup apparatus 101 to recognize the specific voice command via the communication 302 by the wireless LAN.

The GPS (Global Positioning System) 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude / latitude information) of the smart device 301. Alternatively, the position may be estimated by using WPS (Wi-Fi Positioning System) or the like to estimate the current position of the smart device 301 based on the information of the wireless network existing in the surrounding area. When the acquired current GPS position information is located within a preset position range (within a predetermined radius range), the movement information is notified to the image pickup apparatus 101 via the BLE control circuit 402, which will be described later. Used as a parameter for automatic shooting and automatic editing. Further, when the GPS position information has a position change of a predetermined value or more, the movement information is notified to the image pickup apparatus 101 via the BLE control circuit 402, and is used as a parameter for automatic shooting or automatic editing described later.

As described above, the image pickup device 101 and the smart device 301 exchange data with the image pickup device 101 by communication using the wireless LAN control circuit 401 and the BLE control circuit 402. For example, it transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. In addition, the smart device issues an operation instruction such as shooting of the image pickup device 101, transmits voice command registration data, and performs predetermined position detection notification and location movement notification based on GPS position information. It also sends and receives learning data via a dedicated application in the smart device.

<Structure of accessories>
FIG. 5 is a diagram showing a configuration example of an external device 501 capable of communicating with the image pickup device 101. The image pickup device 101 is a digital camera having a photographing function, and the external device 501 is a wearable device including various sensing units capable of communicating with the image pickup device 101 by, for example, a Bluetooth communication module.

The wearable device 501 is configured to be worn on the user's arm, for example, and is a sensor that detects biological information such as the user's pulse, heartbeat, and blood flow at a predetermined cycle, and an acceleration sensor that can detect the user's motion state. Etc. are installed.

The biological information detection circuit 502 detects that, for example, a pulse sensor for detecting a pulse, a heartbeat sensor for detecting a heartbeat, a blood flow sensor for detecting blood flow, and a change in potential due to contact with the skin by a conductive polymer are detected. Includes sensors to In this embodiment, a heartbeat sensor will be used as the biological information detection circuit 502. The heart rate sensor detects the user's heartbeat by irradiating the skin with infrared light using, for example, an LED or the like, detecting the infrared light transmitted through the body tissue with the light receiving sensor, and processing the signal. The biological information detection circuit 502 outputs the detected biological information as a signal to the control circuit 607 described later.

The shaking detection circuit 503 that detects the user's motion state is equipped with, for example, an acceleration sensor or a gyro sensor, and whether the user is moving based on the acceleration information or whether the user is swinging his / her arm to take an action. Motion such as can be detected.

Further, an operation member 505 that accepts the operation of the wearable device 501 by the user and a display device 504 that outputs visually recognizable information such as an LCD or an LED are mounted.

The configuration of the wearable device 501 will be described with reference to FIG.

The wearable device 501 includes, for example, a control circuit 607, a communication circuit 601, a biological information detection circuit 502, a shaking detection circuit 503, a display device 504, an operation member 505, a power supply circuit 606, and a storage circuit 608.

The control circuit 607 controls the entire wearable device 501, for example, by executing a control program stored in the storage circuit 608. The storage circuit 608 stores, for example, a control program executed by the control circuit 607 and various information such as parameters required for communication. Various operations described later are realized, for example, by the control circuit 607 executing the control program stored in the storage circuit 608.

The power supply circuit 606 supplies power to the wearable device 501. The display device 504 has a function capable of outputting visually recognizable information such as an LCD or LED, or a sound output of a speaker or the like, and displays various information. The operation member 505 is, for example, a button or the like that receives an operation of the wearable device 501 by the user. The display device 504 and the operation member 505 may be composed of a common member such as a touch panel.

Further, the operation member acquires the voice uttered by the user from, for example, a general-purpose microphone built in the wearable device 501, acquires the voice uttered by the user by voice processing, and performs the user's operation command by voice recognition processing. May be configured to acquire.

Various detection information processed by the control circuit 607 from the biological information detection circuit 502 and the shaking detection circuit 503 is transmitted to the image pickup apparatus 101 by the communication circuit 601.

For example, the detection information is transmitted to the imaging device 101 at the timing when the change in the user's heartbeat is detected, or the detection information is transmitted at the timing of the change in the moving state such as walking movement / running movement / stopping. Further, for example, the detection information is transmitted at the timing when the preset arm swing motion is detected, or the detection information is transmitted at the timing when the movement of the preset distance is detected.

<Sequence of imaging operation>
FIG. 7 is a flowchart illustrating an example of the operation undertaken by the first control circuit 223 of the image pickup apparatus 101 in the present embodiment.

When the user operates the power button provided on the image pickup device 101, the first power supply circuit 210 causes the power supply unit to supply power to each block of the first control circuit 223 and the image pickup device 101.

Similarly, in the second control circuit 211 as well, the power supply unit supplies power to the second control circuit by the second power supply circuit 212. For details of the operation of the second control circuit, refer to the flowchart of FIG. It will be described using.

When the electric power is supplied, the process of FIG. 7 starts. In step S701 (hereinafter, "step" is simply abbreviated as "S"), the start condition is read. In this embodiment, the activation conditions are as follows.
(1) Power is activated by manually pressing the power button (2) Power is activated by an instruction from an external device (for example, 301) by external communication (for example, BLE communication) (3) From the Sub processor (second control circuit 211) Power supply activation Here, in the case of power activation from the Sub processor of (3), the activation conditions calculated in the Sub processor are read, and the details will be described later with reference to FIG.

Further, the activation condition read here is used as one parameter element at the time of subject search or automatic shooting, and will be described later. When the start condition reading is completed, the process proceeds to S702.

In S702, various sensors are read. The sensor read here may be a vibration detecting sensor such as a gyro sensor or an acceleration sensor from the device shaking detection circuit 209. Further, it may be the rotation position of the tilt rotation unit 104 or the pan rotation unit 105. In addition, it may be a detection trigger for voice level or specific voice recognition detected by the voice processing circuit 214, or sound direction detection.

Further, although not shown in FIGS. 1 to 6, information is also acquired by a sensor that detects environmental information.

For example, there are a temperature sensor that detects the temperature around the image pickup device 101 at a predetermined cycle, and a barometric pressure sensor that detects a change in the atmospheric pressure around the image pickup device 101. Further, an illuminance sensor that detects the brightness around the image pickup device 101, a humidity sensor that detects the humidity around the image pickup device 101, a UV sensor that detects the amount of ultraviolet rays around the image pickup device 101, and the like may be provided. .. In addition to the detected temperature information, atmospheric pressure information, brightness information, humidity information, and UV information, the temperature change amount, atmospheric pressure change amount, brightness change amount, and humidity change obtained by calculating the rate of change at predetermined time intervals from various detected information. The amount and the amount of change in ultraviolet rays are used for judgment such as automatic shooting, which will be described later.

When various sensors are read in S702, the process proceeds to S703.

The S703 detects whether communication from an external device is instructed, and if there is a communication instruction, communicates with the external device.

For example, the smart device 301 may perform remote operation via wireless LAN or BLE, or transmit or receive data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. In addition, is there an operation instruction such as shooting of the image pickup device 101 from the smart device 301, a voice command registration data transmission, a predetermined position detection notification based on GPS position information, a location movement notification, or an instruction to send / receive learning data? Please read it.

Further, for example, when the wearable device 501 updates the biometric information such as the user's exercise information, arm action information, and heartbeat, the information is read via BLE. Further, the various sensors for detecting the above-mentioned environmental information may be mounted on the imaging device 101, or may be mounted on the smart device 301 or the wearable device 501. In that case, the environmental information is read via BLE. Also do. When the communication is read from the external device in S703, the process proceeds to S704.

In S704, the mode setting determination is performed. The mode set in S704 is determined and selected from the following.

(1) Automatic shooting mode [Mode judgment conditions]
From each detection information (image, sound, time, vibration, place, physical change, environmental change) set by learning described later, the elapsed time after shifting to the automatic shooting mode, past shooting information, etc. When it is determined that automatic shooting should be performed, the automatic shooting mode is set.

[Processing in mode]
In the automatic shooting mode processing (S710), the subject is automatically searched by driving pan / tilt and zoom based on each detection information (image, sound, time, vibration, place, body change, environmental change). Then, when it is determined that the timing is such that the user's favorite shooting can be performed, the shooting method is determined from various shooting methods such as single still image shooting, continuous still image shooting, moving image shooting, panoramic shooting, and time-lapse shooting. Processing is performed and shooting is performed automatically.

(2) Learning mode [Mode judgment conditions]
When it is determined that automatic learning should be performed based on the elapsed time since the last learning process and the number of information and learning data associated with the images that can be used for learning, the automatic learning mode is set. Will be done. Alternatively, this mode is also set when instructed to set the learning data via communication from the smart device 301.

[Processing in mode]
In the automatic learning mode processing (S712), learning is performed according to the user's preference. Learning is performed according to the user's preference by using a neural network based on information such as each operation on the smart device 301 and learning information notification from the smart device 301. The information of each operation on the smart device 301 includes, for example, image acquisition information from the image pickup device, information for which manual editing instruction is given via a dedicated application, and a determination value input by the user for the image in the image pickup device. There is information.

The details of the automatic shooting mode processing and the learning mode processing will be described later.

In S705, it is determined in S704 whether or not the mode setting determination is set to the low power consumption mode. The low power consumption mode determination is determined to be the low power consumption mode if it is not the determination condition of any of the "automatic shooting mode" and the "learning mode" described later. When the determination process is performed, the process proceeds to S705.

In S705, if it is determined that the power consumption mode condition is low, the process proceeds to S706.

In S706, the Sub processor (second control circuit 211) is notified of various parameters (sway detection determination parameter, sound detection parameter, time lapse detection parameter) related to the activation factor determined in the Sub processor. The values of various parameters change as they are learned in the learning process described later. When the processing of S706 is completed, the process proceeds to S707, the power of the Main processor (first control circuit 223) is turned off, and the processing is completed.

On the other hand, if it is determined in S705 that the power consumption mode is not low, the process proceeds to S709 to determine whether or not the mode setting is the automatic shooting mode, and if it is the automatic shooting mode, the process proceeds to S710 and the automatic shooting mode processing is performed. Will be. When the process is completed, the process returns to S702 and the process is repeated. If it is determined in S709 that the mode is not the automatic shooting mode, the process proceeds to S711.

In S711, it is determined whether or not the mode setting is the learning mode, and if it is the learning mode, the process proceeds to S712 and the learning mode processing is performed. When the process is completed, the process returns to S702 and the process is repeated. If it is determined in S711 that the learning mode is not set, the process returns to S702 and the process is repeated.

FIG. 8 is a flowchart illustrating an example of the operation undertaken by the second control circuit 211 of the image pickup apparatus 101 in the present embodiment.

When the user operates the power button provided on the image pickup apparatus 101, the second power supply is supplied to the second control circuit 211 in the same manner as the power supply unit supplies power to the first control circuit 223 by the first power supply circuit 210. Power is supplied from the power supply unit to the second control circuit 211 by the circuit 212. When the power is supplied, the Sub processor (second control circuit 211) is started, and the process of FIG. 8 starts.

In S801, it is determined whether or not a predetermined period, which is a sampling cycle, has elapsed. For example, when it is set to 10 msec, the process proceeds to S802 in a cycle of 10 msec. If it is determined that the predetermined period has not elapsed, the Sub processor returns to S801 without performing any processing and waits for the predetermined period to elapse.

In S802, the shaking detection value is acquired. The vibration detection value is an output value from a vibration detection sensor such as a gyro sensor or an acceleration sensor from the device vibration detection circuit 209.

When the shaking detection value is acquired in S802, the process proceeds to S803 to perform a preset shaking state detection process. Some examples will be described.

(1) Tap detection A state in which the user taps the image pickup device 101 with, for example, a fingertip (tap state) can be detected from an output value of an acceleration sensor attached to the image pickup device 101. By passing the output of the 3-axis acceleration sensor through a bandpass filter (BPF) set in a specific frequency region at a predetermined sampling cycle, the signal region of the acceleration change due to tapping can be extracted. Tap detection is performed depending on whether or not the number of times the acceleration signal after the BPF exceeds the predetermined threshold threshold A during the predetermined time TimeA is the predetermined number of times CountA. In the case of a double tap, CountA is set to 2, and in the case of a triple tap, CountA is set to 3.

(2) Detection of shaking state It is possible to detect the shaking state of the image pickup device 101 from the output values of the gyro sensor and the acceleration sensor attached to the image pickup device 101. The output of the gyro sensor or acceleration sensor is subjected to absolute value conversion after the high frequency component is cut by the HPF and the low frequency component is cut by the LPF. Vibration detection is performed depending on whether or not the number of times the calculated absolute value exceeds the predetermined threshold value ThrishB during the predetermined time TimeB is greater than or equal to the predetermined number of times CountB. For example, it is possible to determine whether the shaking is small, such as when the image pickup device 101 is placed on a desk or the like, or whether the shaking is large, such as when the image pickup device 101 is worn and walked. Further, by having a plurality of conditions of the determination threshold value and the number of determination counts, it is possible to detect a fine shaking state according to the shaking level.

When the specific shaking state detection process is performed in S803, the process proceeds to S804 to perform a preset specific sound detection process. Some examples will be described.

(1) Detection of specific voice command Detects a specific voice command. In addition to some voice commands registered in advance, the user can register a specific voice in the imaging device.

(2) Specific sound scene recognition Sound scene judgment is performed by a network trained by machine learning based on a large amount of voice data in advance. For example, it detects a specific scene such as "cheers", "applause", or "speaking".

(3) Sound level determination Detection is performed by sound level determination by a method such as adding the time when the magnitude of the sound level exceeds the predetermined level value within the predetermined time.

(4) Sound direction determination It is possible to detect the direction of sound on a plane on which a plurality of microphones are installed, and detect the direction of sound with respect to a sound level of a predetermined magnitude.

The above determination process is performed in the voice processing circuit 214, and it is determined in S804 whether or not the specific sound is detected.

When the specific sound detection process is performed in S804, the process proceeds to S805. In S805, the Main processor (first control circuit 223) determines whether or not it is in the OFF state, and if the Main processor is in the OFF state, the process proceeds to S806 to perform a preset time lapse detection process. The elapsed time from the transition from ON to OFF of the Main processor is measured, and if the elapsed time is equal to or greater than the parameter TimeC, it is determined to be time elapsed, and if it is smaller than TimeC, it is not determined to be time elapsed.

When the time lapse detection process is performed in S806, the process proceeds to S807 to determine whether the low power consumption mode release determination has been made. The low power consumption mode release condition is determined by the following.
(1) Judgment condition for specific shaking detection (2) Judgment condition for specific sound detection (3) Judgment condition for time lapse judgment Each of them has entered the judgment condition for specific shaking detection by the specific shaking state detection process in S803. Can be determined. Further, by the specific sound detection process in S804, it can be determined whether or not the determination condition for the specific sound detection is satisfied. Further, by the time lapse detection process in S806, it can be determined whether or not the time lapse detection determination condition is satisfied. Therefore, if any one or more of the conditions are met, a determination is made to cancel the low power consumption mode.

When the release condition is determined in S807, the process proceeds to S808 to turn on the power of the Main processor, and in S809, the condition (shaking, sound, time) determined to release the low power consumption mode is notified to the Main processor, and the process returns to S801. Loop the process.

If none of the release conditions are met in S807 and it is determined that the low power consumption mode release determination is not made, the process returns to S801 and the process is looped.

When it is determined in S805 that the Main processor is in the ON state, the information acquired in S802 to 805 is notified to the Main processor, and the process returns to S801 to loop the processing.

In the present embodiment, even when the Main processor is ON, the Sub processor performs shaking detection and specific sound detection, and notifies the Main processor of the detection result. However, when the Main processor is ON, the processing of S802 to 805 may not be performed, and the processing in the Main processor (S702 in FIG. 7) may be configured to detect the shaking detection and the specific sound detection.

Although the method of canceling the low power consumption mode due to shaking detection, sound detection, and the passage of time has been described in detail above, the low power consumption mode may be canceled based on environmental information. Environmental information can be determined by whether or not the absolute amount or change amount of temperature, atmospheric pressure, brightness, humidity, or ultraviolet ray amount exceeds a predetermined threshold value.

<Automatic shooting mode processing>
The details of the automatic shooting mode processing will be described with reference to FIG. As described above, the first control circuit 223 of the image pickup apparatus 101 in the present embodiment is in charge of the following processing.

In S901, the image processing circuit 207 causes the image processing circuit 207 to perform image processing on the signal captured by the imaging unit 206 to generate an image for subject recognition.

Subject recognition such as person or object recognition is performed from the generated image.

When recognizing a person, the face or human body of the subject is detected. In the face detection process, a pattern for determining the face of a person is predetermined, and a portion matching the pattern included in the captured image can be detected as a face image of the person.

At the same time, the reliability indicating the certainty of the subject's face is also calculated, and the reliability is calculated from, for example, the size of the face region in the image, the degree of coincidence with the face pattern, and the like.

Similarly, for object recognition, it is possible to recognize an object that matches a pre-registered pattern.

There is also a method of extracting a feature subject by using a histogram of hue, saturation, etc. in the captured image. In this case, regarding the image of the subject captured within the shooting angle of view, the process of dividing the distribution derived from the histogram of the hue, saturation, etc. into a plurality of sections and classifying the images captured in each section is executed. Will be done.

For example, a histogram of a plurality of color components is created for the captured image, the image is divided by the mountain-shaped distribution range, and the captured image is classified in the area belonging to the combination of the same sections, and the image area of the subject is divided. Be recognized.

By calculating the evaluation value for each image area of the recognized subject, the image area of the subject having the highest evaluation value can be determined as the main subject area.

By the above method, each subject information can be obtained from the imaging information.

In S902, the image shake correction amount is calculated. Specifically, first, the absolute angle of the image pickup device is calculated based on the angular velocity and acceleration information acquired by the device shake detection circuit 209. Then, the vibration isolation angle for moving the tilt rotation unit 104 and the pan rotation unit 105 in the angle direction for canceling the absolute angle is obtained, and the image shake correction amount is used. The calculation method of the image shake correction amount calculation process here can be changed by a learning process described later.

In S903, the state of the imaging device is determined. Based on the angle and movement amount detected by the angular velocity information, acceleration information, GPS position information, etc., it is determined what kind of vibration / movement state the image pickup device is currently in.

For example, when the image pickup device 101 is attached to a car for shooting, the subject information such as the surrounding landscape changes greatly depending on the distance traveled.

Therefore, it can be used for automatic subject search, which will be described later, by determining whether or not the vehicle is in a "vehicle moving state" in which the vehicle is mounted on a car or the like and is moving at a high speed.

Further, it is determined whether or not the change in the angle is large, and it is determined whether or not the image pickup apparatus 101 is in the “placed shooting state” where there is almost no shaking angle.

In the "place-shooting state", it can be considered that the angle of the image pickup apparatus 101 itself does not change, so that the subject search for the stand-alone shooting can be performed.

Further, when the angle change is relatively large, it is determined to be in the "handheld state", and the subject search for handheld can be performed.

In S904, subject search processing is performed. The subject search is composed of the following processes.

(1) Area division The area division will be described with reference to FIG. As shown in FIG. 11A, the area is divided around the entire periphery around the position of the image pickup device (origin O is the position of the image pickup device). In the example of FIG. 11A, the tilt direction and the pan direction are each divided by 22.5 degrees. When divided as shown in FIG. 11A, the circumference in the horizontal direction becomes smaller and the area area becomes smaller as the angle in the tilt direction deviates from 0 degrees. Therefore, as shown in FIG. 11B, when the tilt angle is 45 degrees or more, the area range in the horizontal direction is set to be larger than 22.5 degrees. 11 (c) and 11 (d) show examples of area division within the shooting angle of view. The axis 1101 is the direction of the image pickup apparatus 101 at the time of initialization, and the area division is performed with this direction angle as a reference position. 1102 shows the angle of view area of the image being captured, and an example of the image at that time is shown in FIG. 11 (d). In the image projected at the angle of view, the image is divided as shown in 1103 to 1118 in FIG. 11D based on the area division.

(2) Calculation of importance level for each area For each area divided as described above, the importance level indicating the priority of searching is calculated according to the subject existing in the area and the scene situation of the area. The importance level based on the situation of the subject is based on, for example, the number of people existing in the area, the size of the person's face, the face orientation, the certainty of face detection, the facial expression of the person, and the personal authentication result of the person. calculate. In addition, the importance level according to the situation of the scene is, for example, general object recognition result, scene discrimination result (blue sky, backlight, evening scene, etc.), sound level and voice recognition result from the direction of the area, motion detection in the area. Information etc. Further, the vibration state of the image pickup apparatus is detected by the state determination (S903) of the image pickup apparatus, and the importance level can be changed according to the vibration state. For example, when it is determined that the subject is in the "placed shooting state", the subject is searched for the subject having a high priority (for example, the user of the imaging device) registered by face recognition, so that the subject is searched for. When face recognition is detected, it is determined that the importance level is high. In addition, automatic shooting, which will be described later, will also be performed with priority given to the above-mentioned face, and even if the user of the image pickup device wears the image pickup device and takes a lot of time to carry around and take a picture, the image pickup device is removed from the desk. By placing it on the top, you can leave many images of the user. At this time, since it is possible to search by pan / tilt, it is possible to leave an image of the user or a group photo of many faces just by installing it properly without considering the placement angle of the image pickup device. it can. Under the above conditions alone, as long as there is no change in each area, the area with the highest importance level will be the same, and as a result, the area to be searched will not change forever. Therefore, the importance level is changed according to the past shooting information. Specifically, the importance level is lowered in the area that has been continuously designated as the search area for a predetermined time, and the importance level is lowered in the area that was photographed in S910 described later for a predetermined time. May be good.

(3) Determining the search target area After the importance level of each area is calculated as described above, the area with the high importance level is determined as the search target area. Then, the pan / tilt search target angle required to capture the search target area at the angle of view is calculated.

In S905, pan / tilt drive is performed. Specifically, the pan / tilt drive amount is calculated by adding the image shake correction amount and the drive angle in the control sampling based on the pan / tilt search target angle, and the tilt rotation is performed by the lens barrel rotation drive circuit 205. The unit 104 and the pan rotation unit 105 are driven and controlled, respectively.

In S906, the zoom unit 201 is controlled to drive the zoom. Specifically, the zoom is driven according to the state of the search target subject determined in S904. For example, when the subject to be searched is the face of a person, if the face on the image is too small, it may not be detected because it is smaller than the minimum detectable size, and the face may be lost. In such a case, the size of the face on the image is controlled to be increased by zooming to the telephoto side. On the other hand, if the face on the image is too large, the subject tends to deviate from the angle of view due to the movement of the subject or the imaging device itself. In such a case, zooming to the wide-angle side controls the size of the face on the screen to be smaller. By performing zoom control in this way, it is possible to maintain a state suitable for tracking the subject.

In S904 to S906, the method of searching for a subject by pan / tilt or zoom drive has been described, but the subject search may be performed by an imaging system that uses a plurality of wide-angle lenses to shoot in all directions at once. In the case of an omnidirectional camera, enormous processing is required when performing image processing such as subject detection using all the signals obtained by imaging as input images. Therefore, a part of the image is cut out, and the subject search process is performed within the cut out image range. Similar to the method described above, the importance level for each area is calculated, the cutting position is changed based on the importance level, and the automatic shooting determination described later is performed. This makes it possible to reduce power consumption by image processing and search for a subject at high speed.

In S907, it is determined whether or not there is a shooting instruction by the user (manual), and if there is a shooting instruction, the process proceeds to S910. At this time, the user (manual) shooting instruction may be performed by pressing the shutter button, tapping the housing of the imaging device with a finger or the like (tap), inputting a voice command, or instructing from an external device. The shooting instruction by tap operation is a shooting instruction method in which when the user taps the housing of the imaging device, the device shaking detection circuit 209 detects continuous high-frequency acceleration in a short period of time and triggers shooting. The voice command input is a shooting instruction method in which when the user utters a password (for example, "take a picture") instructing a predetermined shooting, the voice processing circuit 214 recognizes the voice and triggers the shooting. The instruction from the external device is a shooting instruction method triggered by a shutter instruction signal transmitted via a dedicated application, for example, from a smartphone or the like connected to the image pickup device via Bluetooth.

If the user gives a shooting instruction in S907, the process proceeds to S914. The processing of S914 and the subsequent processing of S915 will be described in detail later.

If there is no shooting instruction in S907, the process proceeds to S908 and an automatic shooting determination is performed. In the automatic shooting judgment, which of the judgment of whether to perform automatic shooting and the judgment of the shooting method (single still image shooting, continuous still image shooting (continuous shooting), movie shooting, panoramic shooting, time-lapse shooting, etc. are executed. Judgment).

(1) Judgment of whether to perform automatic shooting Judgment of whether to perform automatic shooting is performed based on the following two judgments. One is to make a determination to perform automatic shooting when the importance level exceeds a predetermined value based on the importance level for each area obtained in S904. The second is a judgment based on a neural network, which is one of machine learning. As an example of a neural network, an example of a network using a multi-layer perceptron is shown in FIG. A neural network is used to predict an output value from an input value, and by learning the input value and an output value that serves as a model for that input in advance, a new input value can be obtained. On the other hand, the output value that follows the learned model can be estimated. The learning method will be described later. In FIG. 10, 1001 and its vertical circles are neurons in the input layer, 1003 and its vertical circles are neurons in the middle layer, and 1004 are neurons in the output layer. Arrows such as 1002 indicate the connections that connect each neuron. In the judgment based on the neural network, the feature amount based on the subject in the current angle of view, the scene and the state of the imaging device is given as an input to the neurons in the input layer, and the calculation based on the forward propagation law of the multi-layer perceptron The value output from the output layer is obtained through. Then, if the output value is equal to or greater than the threshold value, it is determined to perform automatic shooting. The features of the subject are the current zoom magnification, general object recognition result at the current angle of view, face detection result, number of faces reflected in the current angle of view, face smile / eye closure, face angle, face authentication ID. The number, the line-of-sight angle of the subject person, the scene discrimination result, the detection result of a specific composition, etc. are used. In addition, the elapsed time from the previous shooting, the current time, GPS position information and the amount of change from the previous shooting position, the current voice level, the person making the voice, applause, whether or not cheers are raised, etc. are used. You may. Further, vibration information (acceleration information, state of the imaging device), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays) and the like may be used. Further, when there is information notification from the wearable device 501, notification information (user's movement information, arm action information, biological information such as heartbeat, etc.) may also be used as a feature. This feature is converted into a numerical value in a predetermined range and given to each neuron in the input layer as a feature quantity. Therefore, each neuron in the input layer needs as many features as the above-mentioned features.

In the judgment based on this neural network, the output value is changed by changing the connection weight between each neuron by the learning process described later, and the judgment result can be adapted to the learning result.

Further, the determination of automatic shooting also changes depending on the activation condition of the Main processor read in S702 of FIG. 7. For example, in the case of activation by tap detection or activation by a specific voice command, it is very likely that the operation is for the user to currently take a picture. Therefore, the shooting frequency is set to increase.

(2) Judgment of shooting method In the judgment of the shooting method, which of still image shooting, moving image shooting, continuous shooting, panoramic shooting, etc. is performed based on the state of the imaging device and the state of the surrounding subject detected in S901 to S904. To determine whether to execute. For example, when the subject (person) is stationary, still image shooting is executed, and when the subject is moving, moving image shooting or continuous shooting is executed. In addition, when a plurality of subjects exist so as to surround the image pickup device, or when it can be determined that the subject is a scenic spot based on the GPS information described above, the images taken sequentially while operating the pan / tilt are taken. A panoramic shooting process that synthesizes and generates a panoramic image may be executed.

In S909, if it is determined by the automatic shooting determination of S908 to shoot, the process proceeds to S910, and if not, the process proceeds to the end of the shooting mode process.

In S910, shooting is started. At this time, in the case of manual shooting, a still image is shot, or in the case of automatic shooting, shooting is performed by a shooting method manually set by the user, and in the case of automatic shooting, shooting by the shooting method determined in S908 is started. At that time, autofocus control is performed by the focus drive control circuit 204. Further, exposure control is performed so that the subject has an appropriate brightness by using an aperture control circuit (not shown), a sensor gain control circuit, and a shutter control circuit. Further, after shooting, the image processing circuit 207 performs various image processing such as auto white balance processing, noise reduction processing, and gamma correction processing to generate an image.

In the case of automatic shooting, when a predetermined condition is satisfied, the imaging device may take a means of notifying the person to be shot that the person is to be shot and then taking a picture. As the method of notification, for example, the voice from the voice output circuit 218 or the LED lighting light from the LED control circuit 224 may be used, or a motion operation that visually guides the line of sight of the subject by driving the pan / tilt. May be done. The predetermined conditions are, for example, the number of faces within the angle of view, the degree of smile / eye closure of the face, the line-of-sight angle and face angle of the subject person, the face recognition ID number, the number of persons registered for personal recognition, and the like. .. In addition, the general object recognition result at the time of shooting, the scene discrimination result, the elapsed time from the previous shooting, the shooting time, whether the current position based on GPS information is a scenic spot, the voice level at the time of shooting, and aloud. Whether or not there is a person, applause, whether or not cheers are rising, etc. Further, vibration information (acceleration information, state of imaging device), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays) and the like. By performing broadcast photography based on these conditions, it is possible to leave a preferable image of the camera's line of sight in a scene of high importance.

In addition, it has multiple predetermined conditions, and changes the sound according to each condition, changes the LED lighting method (color, blinking time, etc.), and pan / tilt motion method (movement method and drive speed). You may change it.

In S911, editing processing such as processing the image generated in S910 and adding it to a moving image is performed. Specific examples of image processing include trimming processing based on a person's face and in-focus position, image rotation processing, HDR (high dynamic range) effect, bokeh effect, and color conversion filter effect. In the image processing, a plurality of images may be generated based on the image generated in S910 by a combination of the above processes, and may be saved separately from the image generated in S910. Further, as for the moving image processing, the captured moving image or the still image may be added to the generated edited moving image while adding special effect processing of slide, zoom, and fade. Regarding editing in S911, the image processing method can also be determined by determining the captured image information or various information detected before imaging based on the neural network, and this determination process is performed by the learning process described later. , Judgment conditions can be changed.

In S912, learning information generation processing of the captured image is performed. Here, learning information used for the learning process described later is generated and recorded. Specifically, in the image taken this time, the zoom magnification at the time of shooting, the general object recognition result at the time of shooting, the face detection result, the number of faces in the photographed image, the degree of smile / eye closure of the face, the face angle, and the face. The authentication ID number, the line-of-sight angle of the subject person, and the like. In addition, the scene discrimination result, the elapsed time from the previous shooting, the shooting time, the GPS position information and the amount of change from the previous shooting position, the audio level at the time of shooting, the person making a voice, applause, and whether or not cheers are raised. Is it? In addition, vibration information (acceleration information, state of the imaging device), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), moving image shooting time, whether or not it is due to manual shooting instructions, and the like. Furthermore, the score, which is the output of the learning model that quantifies the user's image preference, is also calculated.

This information is generated and recorded as tag information in the captured image file. Alternatively, the information may be written to the non-volatile memory 216 or stored in the recording medium 221 in a format in which the information of each captured image is listed as so-called catalog data.

In S913, the past shooting information is updated. Specifically, the number of shots for each area described in the explanation of S908, the number of shots for each person registered for personal authentication, the number of shots for each subject recognized by general object recognition, and the number of shots for each scene for scene discrimination. The count of the number of images taken this time is incremented by one.

<Learning mode processing>
Next, learning according to the user's preference in the present embodiment will be described.

In the present embodiment, a neural network as shown in FIG. 10 is used, and a machine learning algorithm is used to perform learning according to the user's preference in the learning processing circuit 219 to generate a learning model. The learning processing circuit 219 uses, for example, Jetson TX2 manufactured by NVIDIA. A neural network is used to predict an output value from an input value, and by learning the actual value of the input value and the actual value of the output value in advance, the output is output for a new input value. The value can be estimated. By using a neural network, learning is performed according to the user's preference for the above-mentioned automatic shooting and subject search.

In addition, subject registration (face recognition, general object recognition, etc.), which is also feature data to be input to the neural network, is also registered.

In the present embodiment, the elements learned by the learning process are as follows.

(1) Automatic shooting Learn about automatic shooting. In automatic shooting, learning is performed to automatically shoot an image that suits the user's taste. As described above in the description using the flow of FIG. 9, the learning information generation process is performed after shooting (S912). An image to be trained is selected by a method described later, and learning is performed by changing the weight of the neural network based on the learning information contained in the image. The learning is performed by changing the neural network for determining the automatic shooting timing and changing the neural network for determining the shooting method (still image shooting, moving image shooting, continuous shooting, panoramic shooting, etc.).

(2) Automatic editing Learning for automatic editing will be described. In the automatic editing, learning is performed for the editing immediately after shooting in S911 of FIG. Editing immediately after shooting will be described. An image to be trained is selected by a method described later, and learning is performed by changing the weight of the neural network based on the learning information contained in the image. Various detection information obtained from shooting or information immediately before shooting is input to the neural network to determine the editing method (trimming processing, image rotation processing, HDR (high dynamic range) effect, blur effect, color conversion filter effect, etc.). I do.

(3) Subject search Learning for subject search will be described. In the subject search, learning is performed to automatically search for a subject that suits the user's preference. As described above in the description using the flow of FIG. 9, in the subject search process (S904), the importance level of each area is calculated, pan / tilt and zoom are driven, and the subject search is performed. The learning is learned by the captured image and the detection information during the search, and is learned by changing the weight of the neural network. Various detection information during the search operation is input to the neural network, the importance level is calculated, and the pan / tilt angle is set based on the importance level to perform the subject search reflecting the learning. In addition to setting the pan / tilt angle based on the importance level, for example, learning of pan / tilt drive (speed, acceleration, frequency of movement) is also performed.

(4) Subject registration The learning for subject registration will be described. In subject registration, learning is performed to automatically register and rank subjects according to the user's preference. As learning, for example, face recognition registration, general object recognition registration, gesture and voice recognition, and sound scene recognition registration are performed. Authentication registration is performed for people and objects, and rank is set based on the number and frequency of image acquisition, the number and frequency of manual shooting, and the frequency of appearance of the subject under search. The registered information will be registered as an input for determination using each neural network.

Next, the learning method will be described.

As a learning method, there are "learning in an imaging device" and "learning in cooperation with a communication device".

The method of learning in the imaging device will be described below. There are the following methods for in-image learning in the imaging device in this embodiment.

(1) Learning from Detection Information at the Time of Shooting Instruction by the User As described in S907 to S913 of FIG. 9, in the present embodiment, the image pickup apparatus 101 can perform two types of shooting, manual shooting and automatic shooting. When there is a manual shooting instruction (as described above, it is performed based on three determinations) in S907, information that the shot image is a manually shot image is added in S912. Further, when it is determined in S909 that the automatic shooting is ON and the image is taken, information is added in S912 that the shot image is an automatically shot image.

When manually shooting here, it is very likely that the picture was taken based on the user's favorite subject, favorite scene, favorite place and time interval. Therefore, learning is performed based on each feature data obtained at the time of manual shooting and learning information of the shot image.

In addition, from the detection information at the time of manual shooting, learning is performed regarding extraction of feature quantities in shot images, registration of personal authentication, registration of facial expressions for each individual, and registration of combinations of people. Further, from the detection information at the time of subject search, for example, learning is performed to change the importance of a nearby person or object from the facial expression of the personally registered subject.

(2) Learning based on detection information during subject search During the subject search operation, it is determined what kind of person, object, and scene the subject registered for personal authentication is being captured at the same time, and the time during which the subject is captured within the angle of view at the same time. Calculate the ratio.

For example, if the time ratio in which the person A of the personal authentication registered subject is photographed at the same time as the person B of the personal authentication registered subject is higher than the predetermined threshold value, it can be determined that the importance is high. Therefore, when the person A and the person B are within the angle of view, various detection information is saved as learning data and learned by the learning mode process 716 so that the score of the automatic shooting determination is high.

In another example, when the time ratio in which the person A of the personal authentication registered subject appears at the same time as the subject "cat" determined by the general object recognition is higher than the predetermined threshold value, it can be determined that the importance is high. Therefore, when the person A and the "cat" are within the angle of view, various detection information is saved as learning data so that the score of the automatic shooting determination is high. Then, learning is performed by the learning mode process 716.

In this way, when the subject being searched for appears frequently, if the score of the automatic shooting judgment is increased, the importance of people and objects near the subject registered for personal authentication will also be increased. Can be changed.

In addition, when the smile degree of the person A who is the subject registered for personal authentication is detected, or when "joy" or "surprise" is detected by detecting the facial expression, the subject that is simultaneously captured is learned to be important. Will be done. In addition, when facial expressions such as "anger" and "true face" are detected, it is unlikely that the subject in the picture at the same time is important, so processing such as not learning is performed.

Next, learning in cooperation with an external communication device in this embodiment will be described. There are the following methods for learning in cooperation with an external communication device in this embodiment.

(3) Learning by Acquiring an Image with an External Communication Device As described in FIG. 3, the image pickup device 101 and the external device 301 have communication means for communication 302 and 303. Images are transmitted and received mainly by communication 302, and images in the image pickup apparatus 101 can be communicated and acquired by the external device 301 via a dedicated application in the external device 301. Further, the thumbnail image of the image data stored in the image pickup apparatus 101 can be viewed via a dedicated application in the external device 301. As a result, the user can select an image that he / she likes from the thumbnail images, check the image, and operate the image acquisition instruction to acquire the image to the external device 301.

At this time, since the user selects an image, instructs transmission, and acquires the image, it is very likely that the acquired image is the user's favorite image. Therefore, it is determined that the acquired image is an image to be learned, and by learning based on the learning information of the acquired image, various learnings preferred by the user can be performed.

An operation example will be described. FIG. 14 shows an example in which an image in the image pickup apparatus 101 is viewed via a dedicated application of the external device 301 which is a smart device. Thumbnail images (1404 to 1409) of image data stored in the image pickup device are displayed on the display device 407, and the user can select an image that he / she likes and acquire the image. At this time, a display method changing unit (1401, 1402, 1403) for changing the display method is provided. When 1401 is pressed, the display order is changed to the date and time priority display mode, and the images are displayed on the display device 407 in the order of the shooting date and time of the images in the image pickup device 101. (For example, 1404 is displayed as having a new date and time, and 1409 is displayed as having an old date and time.) Pressing 1402 changes to the recommended image priority display mode. Based on the score for determining the user's preference for each image calculated in FIG. 9S912, the images are displayed on the display device 407 in descending order of the scores of the images in the image pickup device 101. (For example, 1404 is displayed as having a high score and 1409 is displayed as having a low score.) Pressing 1403 allows you to specify a person or object subject, and then specifying a specific person or object subject displays only a specific subject. You can also do it.

The settings of 1401 to 1403 can be turned on at the same time. For example, when all the settings are turned on, only the specified subject is displayed, the image with the newest shooting date and time is prioritized, and the image has a high score. Will be given priority and displayed.

In this way, since the user's preference is also learned for the captured image, it is possible to easily extract only the user's favorite image from a large number of captured images by a simple confirmation work. ..

(4) Learning by inputting a determination value into an image via an external communication device As described above, the image pickup device 101 and the external device 301 have communication means and are stored in the image pickup device 101. This is a configuration in which the image can be viewed via a dedicated application in the external device 301. Here, the user may configure the image to be scored. A high score (for example, 5 points) can be given to an image that the user thinks he / she likes, and a low score (for example, 1 point) can be given to an image that the user does not like. The structure is such that The score of each image is used for re-learning together with the learning information in the imaging device. The output of the neural network, which inputs the feature data from the specified image information, is learned so as to approach the score specified by the user.

In the present embodiment, the user inputs the determination value to the captured image via the communication device 301, but the image pickup device 101 may be operated to directly input the determination value to the image. .. In that case, for example, the image pickup device 101 is provided with a touch panel display, and the user presses the GUI button displayed on the touch panel display screen display device to set the mode for displaying the captured image. Then, the user can perform the same learning by a method such as inputting a determination value to each image while checking the captured image.

(5) Learning by changing parameters in an external communication device As described above, the image pickup device 101 and the external device 301 have communication means, and the learning parameters currently set in the image pickup device 101. Can be communicated to the external device 301 and stored in the storage circuit 404 of the external device 301. As the learning parameters, for example, the weight of the neural network and the selection of the subject to be input to the neural network can be considered. Further, the learning parameter set in the dedicated server can be acquired via the public line control circuit 406 via the dedicated application in the external device 301 and set as the learning parameter in the imaging device 101. And. As a result, the parameters at a certain point in time can be saved in the external device 301 and the learning parameters can be returned by setting the image pickup device 101, or the learning parameters possessed by other users can be returned via a dedicated server. It can also be acquired and set in its own imaging device 101.

Next, the learning processing sequence will be described.

In the mode setting determination of S704 of FIG. 7, it is determined whether or not the learning process should be performed, and when the learning process is performed, it is determined that the learning mode is used, and the learning mode process of S712 is performed.

The judgment conditions of the learning mode will be described. Whether or not to shift to the learning mode is determined from the elapsed time since the last learning process, the number of information that can be used for learning, and whether or not a learning process instruction was given via a communication device. FIG. 12 shows a determination processing flow for determining whether or not to shift to the learning mode, which is determined in the mode setting determination process of S704.

When the learning mode determination is instructed to start in the mode setting determination process of S704, the process of FIG. 12 starts. In S1201, it is determined whether or not there is a registration instruction from the external device 301. The registration here is a determination as to whether or not there is a registration instruction for learning described above. For example, there are <learning by image information acquired by an image acquired by a communication device> and <learning by inputting a determination value into an image via a communication device>. When there is a registration instruction from an external device in S1201, the process proceeds to S1208, the learning mode determination is set to TRUE, and the processing of S712 is set. If there is no registration instruction from the external device in S1201, the process proceeds to S1202. In S1202, it is determined whether or not there is a learning instruction from an external device. The learning instruction here is a determination as to whether or not there is an instruction to set the learning parameter, such as <learning by changing the image pickup device parameter in the communication device>. When there is a learning instruction from an external device in S1202, the process proceeds to S1208, the learning mode determination is set to TRUE, the process of S712 is set to be performed, and the learning mode determination process is terminated. If there is no learning instruction from the external device in S1202, the process proceeds to S1203.

In S1203, the elapsed time TimeN since the previous learning process (recalculation of the neural network weight) is acquired, and the process proceeds to S1204. In S1204, the number of new data to be learned DN (the number of images designated to be learned during the elapsed time TimeN since the last learning process was performed) is acquired, and the process proceeds to S1205. In S1205, the threshold value DT is calculated from TimeN. For example, the threshold DTa when TimeN is smaller than the predetermined value is set to be larger than the threshold DTb when it is larger than the predetermined value, and the threshold is set to become smaller with the passage of time. As a result, even when the learning data is small, the imaging device is made easy to change in learning according to the usage time by learning again when the passage of time is large.

When the threshold value DT is calculated in S1205, the process proceeds to S1206, and it is determined whether or not the number of data DN to be learned is larger than the threshold value DT. When the DN is larger than the threshold value DT, the process proceeds to S1207, the DN is set to 0, then the process proceeds to S1208, the learning mode determination is set to TRUE, the processing of S712 is set to be performed, and the learning mode determination processing is completed. To do.

If the DN is equal to or less than the threshold value DT in S1206, the process proceeds to S1209. Since there is no registration instruction from an external device, no learning instruction from an external device, and the number of learning data is less than or equal to a predetermined value, the learning mode determination is set to FALSE, the S712 process is set not to be performed, and the learning mode determination is performed. End the process.

Next, the processing in the learning mode processing (S712) will be described. The detailed flow of the learning mode processing is shown in FIG.

When the learning mode is determined in S711 of FIG. 7 and the process proceeds to S712, the process of FIG. 13 starts. In S1301, it is determined whether or not there is a registration instruction from the external device 301. If there is a registration instruction from an external device in S1301, the process proceeds to S1302. In S1302, various registration processes are performed.

Various registrations are registrations of features to be input to the neural network, such as face recognition registration, general object recognition registration, sound information registration, and location information registration.

When the registration process is completed, the process proceeds to S1303, and the element to be input to the neural network is changed from the information registered in S1302.

When the process of S1303 is completed, the process proceeds to S1307.

If there is no registration instruction from the external device 301 in S1301, the process proceeds to S1304 to determine whether or not there is a learning instruction from the external device 301. When there is a learning instruction from the external device, the process proceeds to S1305, the learning parameters communicated from the external device are set in each determination device (weight of the neural network, etc.), and the process proceeds to S1307.

If there is no learning instruction from an external device in S1304, learning (recalculation of the neural network weight) is performed in S1306. The process of S1306 is entered under the condition that the number of data to be learned DN exceeds the threshold value and each determination device can be relearned, as described with reference to FIG. Re-learning is performed using a method such as an error back propagation method or a gradient descent method, the weight of the neural network is recalculated, and the parameters of each judge are changed. When the learning parameter is set, the process proceeds to S1307.

In S1307, the images in the file are rescored. In the present embodiment, scores are assigned to all captured images stored in the file (recording medium 221) based on the learning result, and automatic editing or automatic file deletion is performed according to the assigned scores. It is composed. Therefore, when re-learning or setting of learning parameters from an external device is performed, it is necessary to update the score of the captured image as well. Therefore, in S1307, the recalculation for adding a new score to the captured image stored in the file is performed, and when the processing is completed, the learning mode processing is terminated.

In the present embodiment, the description has been made based on the configuration of learning in the image pickup apparatus 101, but the learning process is performed on the external device 301 side, the data necessary for learning is communicated to the external device 301, and only on the external device side. A similar learning effect can be achieved with a configuration that executes learning. In that case, as described in <Learning by changing parameters in the communication device>, learning is performed by setting parameters such as the weight of the neural network learned on the external device side in the image pickup device 101 by communication. It may be configured.

Further, the learning process may be provided in both the image pickup apparatus 101 and the external device 301. For example, the learning information held by the external device 301 may be communicated to the imaging device 101 at the timing when the learning mode processing 716 is performed in the imaging device 101, and the learning may be performed by merging the learning parameters.

Next, a method of compensating for the lack of teacher data in the learning of the neural network will be described.

A sufficient number of teacher data is required to accurately estimate the output value from the input value in the neural network. It is difficult to improve the estimation accuracy if the neural network model is complicated and has a high degree of freedom with respect to the number of teacher data. In the field of machine learning, even data that is slightly different from the teacher data may be subjected to a process called Data Augmentation so that it can be estimated robustly. This is often done by adding image processing such as aspect ratio change, rotation (roll, pitch, yaw), blurring, noise addition, and shifting to the teacher data (in this case, the image). However, it does not always match the image that can be taken by the camera. For example, even if blurring is added by image processing, the same blurring cannot always be achieved even if the aperture is actually opened or the focus is shifted by the camera.

If the data to be estimated by the neural network and the teacher data are not similar, this teacher data may be a factor that lowers the estimation accuracy of the neural network. Moreover, even if a predetermined rotation (roll, pitch, yaw) is applied, it is not always possible to reproduce the angle actually taken by a human being with a camera. Specifically, even if the image is simply rotated by 45 degrees or 90 degrees from the center of the image, the user has few chances to take a picture in which the subject is not upright, so it is used as teacher data for learning the user's preference. Has a low degree of contribution.

As described above, it is difficult to make up for the shortage of teacher data by image processing, and it is preferable to increase the teacher data by actual shooting. Alternatively, when performing Data Augmentation by image processing, it is better to have an image close to the image that can be taken by the camera rather than an image that cannot be taken by the camera. Therefore, in the present embodiment, a method of automatically performing actual shooting for learning to increase teacher data will be described.

As described above, when it is determined in S907 of FIG. 9 that there is a shooting instruction by the user, the process proceeds to S910 and S914.

In S914, it is determined whether or not the current number of teacher data is smaller than a predetermined number N (N is a natural number). Then, only when the current number of teacher data is smaller than a predetermined number N, the process proceeds to S915 assuming that the teacher data is insufficient, and automatic learning imaging for supplementing the teacher data is performed. This N may be changed according to the complexity and the degree of freedom (the number of nodes and the number of layers of the intermediate layer) of the neural network. If the neural network is complicated or the degree of freedom is high, the number of teacher data required increases, so N is increased. If the current number of teacher data ≥ N in S914 and it is determined that sufficient teacher data has been stored, S915 is skipped and the process proceeds to S912.

As described above, the automatic shooting for learning is performed only when the manual shooting instruction is given in S907. When the manual shooting is performed, the shooting is performed based on the user's favorite subject, favorite scene, favorite place and time interval. This is because it is very likely. Therefore, if automatic shooting for learning is performed at this time, there is a high possibility that teacher data that reflects the user's preference can be acquired.

Further, since the automatic learning shooting of S915 and the manual shooting of S910 cannot be performed at the same time, the timings are staggered. Whichever comes first, make sure that automatic learning and manual shooting are performed continuously. If the automatic learning shooting is delayed, the user may move the camera and move away from the preferable composition for manual shooting. On the other hand, if the manual shooting is delayed, the shutter timing will shift. In the following description, unless otherwise specified, manual photography is performed first, and automatic learning photography is performed immediately afterwards.

In S915, automatic shooting for learning is performed. There are several possible methods for automatic learning photography. The first is continuous shooting. After manual shooting, continuous shooting is automatically performed for learning, and continuous shooting images are acquired. If the timing is close to that of manual shooting, it is possible to acquire multiple teacher data that are close to the user's favorite image. In this case, since the image obtained by manual shooting is treated as a recorded image, it is recorded on the recording medium 221. However, the image obtained by automatic learning shooting is used only for learning and is visible to the user's eyes. It doesn't stick.

The second is video recording. Movie shooting is automatically performed before or after manual shooting, and a function that combines still images and movies and is provided to users may be installed in general cameras and life log cameras. Alternatively, the camera is equipped with a function that constantly overwrites a moving image for a certain period of time in a memory such as a ring buffer and provides the user with a moving image for a predetermined period before and after the timing at which the still image was taken. There are times. This automatically acquired moving image is decomposed into still images and used as teacher data. This is also valuable as teacher data for the same reason as the first continuous shooting. It should be noted that this function is not limited when it is used, and a moving image may be shot only for learning purposes. In that case, the video is not provided to the user.

The third is bracket photography. Bracket shooting is performed by gradually changing the shooting conditions for manual shooting. The shooting conditions to be changed may be parameters that can be changed by the camera, such as focus, exposure, white balance, strobe light emission, zoom, and sharpness. The same effect as Data Augmentation can be expected by changing these shooting conditions. If data augmentation that cannot be realized by a camera is performed and used as teacher data, the neural network learned from it can only estimate data that is close to the teacher data. This makes it unsuitable as a neural network for cameras. Therefore, if the teacher data is increased by bracket shooting that can be realized by the camera, the effect as Data Augmentation can be expected.

It should be noted that some bracket shooting must be performed immediately after manual shooting, and some can be performed even after a certain amount of time has passed. The former involves mechanical movements such as focus and zoom. If these are not taken continuously with manual shooting, the composition will change and they will not be valid as teacher data. On the other hand, the latter is due to image processing such as white balance, sharpness, and development conditions of RAW image data. These can be generated based on the manually photographed image even if they cannot be continuously photographed. In this case, the RAW data of the manually captured image may be recorded. When the image is generated based on the manually captured image, it does not have to be at the time of shooting, and may be generated while the camera is on standby.

In this way, some bracket shooting needs to be performed continuously with manual shooting, and some does not need to be performed continuously. Therefore, even if the bracket shooting type is prioritized and automatic shooting is performed. Good. This means that bracket shooting, which must be performed in succession with manual shooting, is performed first.

In addition, if it is determined from the information of the camera's angular velocity meter 106 and accelerometer 107 that the user has moved the camera between manual shooting and automatic learning shooting, the automatic learning shooting should be stopped. You may.

In S912, learning information for manual shooting and automatic learning shooting is generated, and teacher data is created. For the image obtained by the automatic learning shooting, the learning information can be generated in the same manner as the image obtained by the manual shooting. Since the image obtained by manual shooting is likely to be the user's preference, a predetermined high score is given. Then, the score is also attached to the teacher data generated from the image obtained by the automatic shooting for learning.

Alternatively, the image obtained by the automatic learning photography may be given a score according to the relationship with the image obtained by the manual photography. For example, if the learning automatic shooting is performed without a gap from the manual shooting, the image obtained by the learning automatic shooting is given a high score equivalent to the image obtained by the manual shooting. .. Then, as the interval between the manual shooting and the automatic learning shooting increases, the score for the image obtained by the automatic learning shooting can be lowered. As a result, the score of the manually shot image at the best shot timing instructed by the user becomes the highest, and as the score shifts from that, the score becomes lower, so that the user's preference for shutter timing can be learned. Alternatively, the similarity between the manually captured image and the image obtained by the automatic learning image may be compared with each other, and a score may be given according to the similarity. Furthermore, when the subject is a moving object or the scene including the subject is changing, the images before and after the image taken at the timing of manual shooting are intentionally used for learning as negative teacher data. You may. By doing so, it can be expected that the user's shutter timing preference can be learned more strictly. Further, instead of the images before and after, the images having a similarity lower than the threshold value with the images obtained by manual shooting among the images continuously captured may be used as negative teacher data.

Further, with the same idea for the bracket image, the score can be lowered as the shooting conditions set by the bracket deviate from the shooting conditions set by the manual shooting. For example, give the highest score to an image obtained by manual shooting, give the second highest score to an image with exposure compensation +1 in bracket shooting, and give the third highest score to an image with exposure compensation +2. is there. This makes it possible to learn about the user's favorite shooting conditions.

The learning information of the image obtained by the automatic learning shooting may be diverted from the learning information of the image obtained by the manual shooting. For example, it is highly possible that the subject to be photographed is the same in automatic learning shooting and manual shooting, so general object recognition results and face detection results generated from images obtained by manual shooting are for learning. It can be diverted as learning information for images obtained by automatic shooting. As a result, the time for generating learning information can be shortened.

Further, the shooting instruction by the user to be determined in S907 may include the above-mentioned voice command, tap operation to the device 101, and shooting instruction from the external devices 301 and 501.

Further, since the learning automatic shooting itself is not instructed by the user, it is desirable to perform shooting with an electronic shutter having a low shutter sound in the learning automatic shooting.

Further, in the automatic learning shooting, since the shooting is performed at a timing different from the timing intended by the user, there is a possibility that personal information not intended by the user is saved. This can be a problem when considering privacy. Therefore, the image obtained by the automatic learning shooting may not be saved, but only the learning information generated from this image may be saved. The learning information is, for example, a parameter corresponding to the input layer of the neural network, and since it is in a format other than an image, it is difficult to specify the privacy information. Alternatively, the learning information may not record information related to a person such as a personal authentication ID, and may be replaced with a predetermined specified value instead.

Further, the condition for performing the automatic learning imaging of S914 does not have to be the number of teacher data. For example, if it can be determined that the estimation accuracy of the neural network has improved, the determination of S914 may be NO. Whether or not the estimation accuracy has improved is verified by the following method. When teacher data is acquired by automatic learning photography, it is input to the neural network to obtain the output value. If the difference between the output value and the teacher value is smaller than a predetermined value, it can be judged that the accuracy of the neural network has improved. In other words, even if new data is input, the output value is close to the model value, so it can be judged that the accuracy has improved.

Further, by using this "difference between the neural network output value and the teacher value", it is possible to remove the teacher data acquired by the automatic learning imaging that is not suitable as the teacher data as an outlier. If the difference between the neural network output value and the teacher value is larger than a predetermined value, it means that the estimation has not been performed, and this teacher data can be said to be teacher data whose properties are significantly different from the teacher data learned in the past. In this case, it is highly possible that the camera has already been moved by the user immediately after manual shooting and is facing an unintended direction such as the sky or the ground, and is removed as an outlier. That is, it is not registered as teacher data.

In addition, it is possible to verify outliers of automatically captured images for learning without passing through a neural network. If the difference between the learning automatic image and the manually photographed image is larger than a predetermined value in the feature vector that combines the features of the input layer of the neural network, it may be removed as an outlier.

It is possible to increase teacher data by these automatic shooting for learning. These teacher data will be used for learning when the next learning mode is executed. As the teacher data increases, the estimation accuracy of the neural network can be expected to improve.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The present invention is not limited to shooting with a digital camera or a digital video camera, but can be mounted on a shooting device such as a surveillance camera, a Web camera, or a mobile phone.

101: Imaging device, 301: Smart device, 501: Wearable device, 104: Tilt rotation unit, 105: Pan rotation unit

Claims (21)

An acquisition means for acquiring teacher data regarding a captured image captured by an imaging means, and
It has a learning means for generating a learning model for evaluating an image based on the teacher data.
The learning means is for learning, which is continuously captured with respect to the teacher data based on the recording image taken according to the user's instruction and the recording image captured according to the user's instruction. An image processing apparatus characterized in that the learning model is generated by using the teacher data based on the image of. The image processing apparatus according to claim 1, wherein the teacher value in the image for recording and the teacher value in the image for learning are different. The image processing apparatus according to claim 2, wherein the teacher value in the image for learning is smaller than the teacher value in the image for recording. The claim is characterized in that the larger the difference between the shooting conditions of the image for recording and the image for learning, the larger the difference between the teacher value in the image for recording and the teacher value in the image for learning. The image processing apparatus according to 2 or 3. The larger the difference between the timing of shooting the image for recording and the timing of shooting the image for learning, the larger the difference between the teacher value in the image for recording and the teacher value in the image for learning. The image processing apparatus according to claim 2 or 3. The image processing apparatus according to any one of claims 1 to 5, wherein the learning means uses at least a part of the image for learning as negative teacher data to generate the learning model. .. 6. The learning means is characterized in that, among the images for learning, an image having a similarity lower than the threshold value with the image for recording is used as negative teacher data to generate the learning model. The image processing apparatus according to. The image for learning is an image taken by changing the shooting conditions of the image for recording and at least one of the conditions of focus, exposure, white balance, strobe light emission, and zoom. Item 2. The image processing apparatus according to any one of Items 1 to 7. The learning image is an image generated from a moving image taken immediately before or after the shooting of the recording image, or a continuous shooting image taken immediately before or after the shooting of the recording image. The image processing apparatus according to any one of claims 1 to 8, wherein the image processing apparatus is provided. The image processing apparatus according to any one of claims 1 to 9, further comprising a communication means for transmitting the learning model generated by the learning means to an external device. A generation means for generating teacher data from a captured image taken by an imaging means,
It has a communication means for transmitting the teacher data generated by the generation means to the learning means for generating the learning model.
The generation means generates teacher data from a recording image taken according to a user's instruction, and continuously takes a picture of the recording image taken according to the user's instruction. Teacher data is also generated from images for learning,
The communication means is an image processing apparatus characterized in that the teacher data generated from the image for recording and the teacher data generated from the image for learning are transmitted to the learning means. The image processing apparatus according to claim 11, wherein the generation means makes the teacher value in the image for recording different from the teacher value in the image for learning. The image processing apparatus according to claim 12, wherein the generation means makes the teacher value in the image for learning smaller than the teacher value in the image for recording. In the generation means, the larger the difference between the shooting condition of the image for recording and the shooting condition of the image for learning, the larger the difference between the teacher value in the image for recording and the teacher value in the image for learning. The image processing apparatus according to claim 12 or 13, wherein the teacher value in the image for learning is determined so as to be such. In the generation means, the larger the difference between the timing of shooting the image for recording and the timing of shooting the image for learning, the difference between the teacher value in the image for recording and the teacher value in the image for learning. The image processing apparatus according to claim 12 or 13, wherein the teacher value in the image for learning is determined so that The image processing apparatus according to any one of claims 11 to 15, wherein the generation means generates at least a part of the image for learning as negative teacher data. The image processing apparatus according to any one of claims 11 to 16, wherein the image for learning is generated from a moving image taken immediately before or immediately after the image for recording is taken. The image processing apparatus according to any one of claims 11 to 16, wherein the image for learning is a continuous shooting image taken immediately before or immediately after taking the image for recording. The acquisition process to acquire teacher data related to the captured image captured by the imaging means,
It has a generation step of generating a learning model for evaluating an image based on the teacher data.
In the generation step, the teacher data based on the recording image taken according to the user's instruction and the learning image taken continuously with respect to the recording image taken according to the user's instruction. An image processing method characterized by generating the learning model using the teacher data based on the image of. A generation process that generates teacher data from captured images taken by the imaging means,
It has a communication step of transmitting the teacher data generated in the generation step to a learning means for generating a learning model.
In the generation step, teacher data is generated from the recording image taken according to the user's instruction, and the recording image taken according to the user's instruction is continuously photographed. Teacher data is also generated from the image for learning,
The image processing method, characterized in that, in the communication step, the teacher data generated from the image for recording and the teacher data generated from the image for learning are transmitted to the learning means. A program for causing a computer to execute each step of the image processing method according to claim 19 or 20.

Images