JP5200821B2 - Imaging apparatus and program thereof - Google Patents

Description

  The present invention relates to an imaging apparatus that acquires a captured image by an imaging element, and a program executed by a computer built in the imaging apparatus.

  Conventionally, techniques for detecting a gaze direction of a person and capturing an image in the gaze direction have been proposed (see, for example, Patent Document 1 to Patent Document 4).

Specifically, Patent Literature 1 discloses a technique for detecting the gaze direction of a person, directing the camera in the gaze direction according to the detected gaze direction, and controlling the gaze direction of the agent. In Patent Document 2, it is determined whether or not the person is looking at the imaging device by detecting the line-of-sight direction of one or more persons included in the input image. Discloses a technique for recording an image that is determined to be looking at an imaging device. Further, Patent Document 3 discloses a technique for detecting the state of an eyeball by a sensor and changing the focal length of the camera based on the detection result. Furthermore, Patent Document 4 discloses a technique for detecting a person's line-of-sight direction with an image sensor and enlarging and displaying a line-of-sight direction portion in a captured image obtained by another camera.
JP-A-11-73289 JP 2006-319610 A JP 2001-281520 A JP-A-7-77665

  However, the conventional techniques described in Patent Document 1 to Patent Document 4 described above have a problem in that an image in the detected line-of-sight direction is captured as it is, and a wasteful image is recorded.

  Therefore, the present invention has been proposed in view of the above-described circumstances, and an object of the present invention is to provide an imaging apparatus capable of avoiding useless image recording and effectively using a memory and a program thereof. .

An imaging apparatus according to the present invention specifies a range in which gazes of a predetermined ratio or more concentrate among gazes of a plurality of gaze detectors in a captured image, and sets a partial region in an imaging frame corresponding to the identified range. The above problem is solved by trimming and recording the trimmed image.

In addition, the imaging apparatus according to the present invention trims a partial area in an imaging frame corresponding to the gaze direction of the person to be gaze detected in the captured image detected by the gaze direction detection unit, and records the trimmed image. Thus, the above-mentioned problem is solved.
In the imaging apparatus according to the present invention, the entire first imaging range corresponding to the first line-of-sight direction including the direction in which lines of sight of a predetermined first ratio or more of the lines of sight of a plurality of line-of-sight detectors are concentrated is an imaging frame. The second imaging range corresponding to the second line-of-sight direction including the direction in which the line-of-sight of a predetermined second ratio or more of the line-of-sight of the plurality of line-of-sight detectors is concentrated. When it is a part in the imaging frame, the above-described problem is solved by trimming to include an area in the imaging frame corresponding to the second imaging range.

Program of an imaging apparatus according to the present invention, the portion of the imaging frame specifies a range to focus a predetermined ratio or more of the line of sight of the plurality of the visual axis detection line of sight in the captured image, corresponding to the particular range The above-described problem is solved by causing a computer to execute a trimming process for trimming an area and a recording process for recording a trimmed image.

Furthermore, the program of the imaging apparatus according to the present invention is trimmed with a trimming process for trimming a partial area in the imaging frame corresponding to the gaze direction of the subject-eye-detected person in the captured image detected by the gaze direction detection process. The above-described problem is solved by causing a computer to execute a recording process for recording a recorded image .
In addition, the program of the imaging apparatus according to the present invention includes an entire first imaging range corresponding to the first line-of-sight direction including a direction in which lines of sight of a predetermined first ratio or more are concentrated among the lines of sight of a plurality of line-of-sight detectors. The imaging operation is started when it is included in the imaging frame, and the second imaging corresponding to the second gaze direction including the direction in which gazes of a predetermined second ratio or more concentrate among gazes of a plurality of gaze detection persons When the range is a part of the imaging frame, the above-described problem is solved by causing the computer to perform a trimming process for trimming to include an area in the imaging frame corresponding to the second imaging range.

  According to the present invention, the trimmed image is recorded by automatically determining the region of interest to the line-of-sight detector, so that it is possible to avoid the useless recording of the image and effectively use the memory.

  Hereinafter, an imaging apparatus as a preferred embodiment of the present invention will be specifically described.

[Configuration of imaging device]
As shown in FIG. 1, an imaging apparatus shown as an embodiment of the present invention includes an imaging optical system 11 and an imaging unit 12 that performs photoelectric conversion output corresponding to an optical image based on external light acquired by the imaging optical system 11. An imaging processing unit 13 that performs a predetermined process on the photoelectric conversion output output from the imaging unit 12, a data memory 14 that stores data supplied from the imaging processing unit 13, and a program that stores a predetermined program A memory 15, a main display processing unit 16 for generating a video signal to be displayed on the main display unit 17, a main display unit 17 for displaying various information including the video signal, a key input unit 18 for inputting a user operation, An image compression unit 19 that compresses image data, a memory card 20 that records data compressed by the image compression unit 19, and the overall operation of the imaging apparatus are controlled. And a control unit 21, a communication unit 22 for communicating with other imaging devices.

  The imaging optical system 11 is an optical system for imaging outside light in an imaging region within a predetermined range on an imaging element of the imaging unit 12. The imaging optical system 11 includes a plurality of lens groups for entering external light, a motor for driving these lens groups, and the like. The imaging optical system 11 has a shutter unit and an aperture unit (not shown), and moves the in-focus position of the shutter unit and the aperture position of the aperture unit by driving a motor. The imaging optical system 11 also includes a zoom lens that can freely zoom out and zoom in under the control of the control unit 21 and change the imaging range.

  The imaging unit 12 includes, for example, a CMOS (Complementary Metal-Oxide Semiconductor) imaging element on which RGB primary color filters are formed. The imaging unit 12 is scan-driven by a timing generator and a vertical driver (not shown). And the imaging part 12 outputs the photoelectric conversion output corresponding to the optical image imaged for every fixed period for 1 screen.

  The imaging processing unit 13 performs a predetermined process on the photoelectric conversion output output from the imaging unit 12. Specifically, the imaging processing unit 13 appropriately adjusts the gain for each primary color component of RGB in the analog signal state with respect to the photoelectric conversion output from the imaging unit 12, and then performs sample holding by the sample hold circuit. The digital data is converted by an A / D converter. In addition, the imaging processing unit 13 performs color process processing including pixel interpolation processing, γ correction processing, and matrix calculation on the digital data, and generates a luminance signal Y and color difference signals Cb and Cr as digital values. Then, the imaging processing unit 13 refers to the composite synchronization signal, the memory write enable signal, and the clock signal, and once writes the luminance signal Y and the color difference signals Cb and Cr into a buffer inside the DMA (Direct Memory Access) controller. Thereafter, the data is directly transferred to the data memory 14 used as a buffer memory.

  The data memory 14 is composed of a DRAM (Dynamic Random Access Memory) or the like, and stores the luminance signal Y and the color difference signals Cb and Cr supplied from the imaging processing unit 13 under the control of the control unit 21. The data stored in the data memory 14 is read by the control unit 21 and written in a VRAM (Video Random Access Memory) (not shown) for display on the main display unit 17.

  The program memory 15 is a ROM (Read Only Memory) that stores a predetermined program such as an imaging program based on eye-gaze detection described later. The program stored in the program memory 15 is executed by the control unit 21.

  The main display processing unit 16 periodically reads out the luminance signal Y and the color difference signals Cb and Cr from the VRAM, generates a video signal based on these data, and outputs the video signal to the main display unit 17.

  The main display unit 17 is composed of a liquid crystal display or the like provided on the back side of the casing of the imaging device. The main display unit 17 functions as a monitor display unit (electronic finder) in the imaging mode. The main display unit 17 performs display based on the video signal supplied from the main display processing unit 16 in the imaging mode. Thereby, the imaging device can display the image information fetched from the VRAM at that time in real time. That is, the imaging apparatus can display the captured image captured by the imaging unit 12 on the main display unit 17. In this way, the imaging device is in a state in which imaging can be performed in a so-called through image display state in which an image at that time is displayed on the main display unit 17 in real time as a monitor image. In the imaging apparatus, when a shutter button that is one of the key input units 18 is operated at a timing at which imaging is desired, a trigger signal is generated for the control unit 21.

  The key input unit 18 is a group of operators for inputting user operations collectively including a power button, a shutter button, a mode switching button, a menu button, a determination button, a cross key, and the like. The key input unit 18 is connected to the control unit 21 and supplies an operation command signal to the control unit 21 instructing that the key input unit 18 has been operated in accordance with the operation of each key.

  Under the control of the control unit 21, the image compression unit 19 performs ADCT (Adaptive Discrete Cosine Transform) or entropy according to the JPEG (Joint Photograph Coding Experts Group) standard for the written image. Data compression is performed by performing processing such as Huffman encoding, which is an encoding method. The image compression unit 19 supplies the code data subjected to data compression to the memory card 20.

  The memory card 20 is a recording medium that is detachable from the imaging apparatus. Under the control of the control unit 21, the code data obtained by the image compression unit 19 is written in the memory card 20 as a data file of one image.

  The control unit 21 includes a CPU (Central Processing Unit) and the like, and controls the overall operation of the imaging apparatus. Specifically, the control unit 21 reads the luminance signal Y and the color difference signals Cb and Cr transferred from the data memory 14 by controlling the imaging processing unit 13 and transferred to the data memory 14 which is a DRAM. Then, the control unit 21 causes the read signal to be written in a VRAM (not shown).

  In addition, the control unit 21 performs data compression / decompression, recording / reproduction control for the memory card 20, and the like. That is, in the above-described imaging mode, the control unit 21 is based on the trigger signal generated in response to the operation of the shutter button, and the luminance signal Y for one screen captured from the imaging unit 12 at that time and The color difference signals Cb and Cr are supplied to the image compression unit 19. At this time, the imaging processing unit 13 supplies the luminance signal Y and the color difference signals Cb and Cr written in the data memory which is a DRAM to the image compression unit 19 for each component. Specifically, the imaging processing unit 13 reads out the luminance signal Y and the color difference signals Cb and Cr, for example, in a unit called a basic block of 8 vertical pixels × 8 horizontal pixels, and supplies it to the image compression unit 19.

  In addition, when capturing a still image, the control unit 21 charges a large-capacitance capacitor for a strobe (not shown) and then drives a strobe light emitting unit (not shown) of the imaging unit 12 to flash. Further, the control unit 21 records the still image data file obtained by compressing the above-described still image data by the image compression unit 19 to the memory card 20 at the time when the first shutter button is operated during moving image capturing. To start. Thereafter, the control unit 21 continuously executes this processing at a predetermined frame rate, for example, 30 (frames / second), and the shutter button is operated for the second time, or a predetermined time limit, for example, 30 seconds is reached. When the time has elapsed, the series of still image data files are collectively set as a motion JPEG data file (AVI file).

  Furthermore, the control unit 21 selectively reads out image data recorded on the memory card 20 first in the reproduction mode. Then, the control unit 21 controls the image compression unit 19 to expand the compressed image data in a procedure completely opposite to the procedure for data compression in the imaging mode, and the decompressed image data is stored in the imaging processing unit 13. It is held in DRAM. Then, the control unit 21 stores the data held in the DRAM in the VRAM. The image data stored in the VRAM is periodically read and displayed as a video on the main display unit 17 as described above.

  When the selected image data is not a still image but a moving image, the control unit 21 continuously reproduces the individual still image data constituting the selected moving image file in terms of time. When the reproduction of all the still image data is finished, only the still image data located at the head is reproduced and displayed until the next reproduction instruction is given.

  The communication unit 22 communicates with other imaging devices under the control of the control unit 21 and exchanges gaze direction information of the detected person.

[Operation of imaging device]
Such an imaging device can capture an image of the detected person's line-of-sight direction by executing a predetermined imaging program stored in the program memory 15 under the control of the control unit 21. Is done. At this time, the imaging apparatus trims a partial area in the imaging frame corresponding to the detected line-of-sight direction under the control of the control unit 21 and records the trimmed image data in the memory card 20.

  Here, the line-of-sight direction information can be acquired by receiving information from another imaging device for line-of-sight detection. For example, as shown in FIG. 2, let us consider a case where a spectator C in the audience seats is watching actors A and B acting on the stage. In this case, the imaging device C1 whose imaging range is the stage receives the gaze direction information of the spectator C detected by the other imaging device C2 whose imaging range is the passenger seat via the communication unit 22. Then, for example, as illustrated in FIG. 3, the imaging device C <b> 1 cuts out the partial region T in the imaging frame F as a trimming range based on the received gaze direction information under the control of the control unit 21, and the trimming thereof. The recorded image data is recorded in the memory card 20.

  In addition, as shown in FIG. 4, for example, the imaging apparatus captures other actors A and B acting on the stage from the back of the stage when the audience C is watching the audience. It is also possible to perform line-of-sight detection and subject imaging and recording without using an imaging device. That is, for example, as illustrated in FIG. 5, the imaging device C <b> 1 detects gaze direction information of the spectator C from within the imaging frame F under the control of the control unit 21, and performs imaging based on the detected gaze direction information. The partial region T in the frame F can be cut out as a trimming range, and the trimmed image data can be recorded in the memory card 20.

  Hereinafter, as illustrated in FIGS. 2 and 3, a case in which a trimming range is determined based on a person's line-of-sight direction detected by another imaging apparatus will be described.

  Here, the trimming range specifies a range in which the line-of-sight directions of a plurality of persons are concentrated, and is a partial region in the imaging frame corresponding to the specified range. Specifically, the imaging apparatus determines, as a trimming range, a range in which gazes of a predetermined ratio or more are concentrated based on gaze direction information received or detected.

  For example, a case is considered in which a condition is set in which a region including the gaze direction of 60% or more of the persons whose face and gaze direction are recognized with a predetermined accuracy or more is set as a trimming range. In this case, for example, as shown in FIG. 6, when there are seven spectators C whose faces and line-of-sight directions can be recognized with a predetermined accuracy or more, the imaging apparatus has four spectators whose lines of sight are facing the actor A. When it is detected that there are three spectators who are looking at the actor B, as shown in FIG. 7, a region T in which both the actors A and B are reflected is determined as a trimming range. . Further, for example, as shown in FIG. 8, when there are seven spectators C whose face and line-of-sight directions can be recognized with a predetermined accuracy or more, the imaging apparatus has five spectators whose line of sight faces the actor A. When it is detected that there are two spectators whose line of sight faces the actor B, as shown in FIG. 9, the area T in which the actor A appears is determined as a trimming range.

  As described above, the imaging apparatus determines the position and area of the trimming range according to the distribution of the viewpoint direction of the person and the degree of concentration under the control of the control unit 21 and stores the trimmed image data in the memory. By recording on the card 20, it is possible to avoid the useless recording of images and to effectively use the memory. At this time, the imaging apparatus may perform processing with a degree of freedom for the user by allowing the user to select a trimming range via the key input unit 18.

  In the case where the trimming range is determined by exchanging gaze direction information between the two imaging devices, the gaze position detected by the gaze detection imaging device prior to the start of imaging, It is necessary to set a correspondence relationship with the position in the imaging frame that is imaged by the imaging device for subject imaging. Therefore, the two imaging devices set the correspondence between the line-of-sight position and the position in the imaging frame according to a series of procedures as shown in FIG.

  First, as shown in FIG. 10, the imaging device for line-of-sight detection sets the imaging device as an imaging device for line-of-sight detection in response to the user operating the menu button of the key input unit 18 in step S1. At the same time, in step S2, communication information for communicating with the imaging device for imaging the subject is set.

  In response to this, in step S3, the imaging device for line-of-sight detection causes the main display unit 17 to display guidance that prompts the user to install the imaging device at the actual imaging position under the control of the control unit 21. In step S4, the above-described display of the through image is started.

  On the other hand, in the same way, in the imaging device for subject imaging, in step S11, the imaging device is set as the imaging device for subject imaging in response to the user operating the menu button of the key input unit 18, and in step S12. The communication information for communicating with the imaging device for detecting the line of sight is set.

  In response to this, the imaging device for subject imaging causes the main display unit 17 to display guidance for urging the installation of the imaging device at the actual imaging position under the control of the control unit 21 in step S13. In step S14, display of the above-described through image is started.

  In step S5, the imaging device for line-of-sight detection that has started displaying the through image displays guidance on the main display unit 17 for instructing the line-of-sight detector such as a spectator to view the subject direction of interest. The user instructs the line-of-sight detector to see the subject direction according to this guidance.

  Thereafter, in step S6, the imaging device for line-of-sight detection processes the captured image to detect the face and line-of-sight direction in the imaging frame under the control of the control unit 21, and in step S7, the communication unit 22 is turned on. Then, the detected gaze direction information is transmitted to the imaging device for imaging the subject. The line-of-sight direction detection process will be described in detail later.

  Then, in step S8, the imaging device for line-of-sight detection inquires of the user whether or not to set another line-of-sight position via the main display unit 17 or the like. On the other hand, if it is not set, a series of processing ends.

  On the other hand, the imaging device for imaging a subject that has started through-image display causes the main display unit 17 to display guidance for instructing the position of the target subject on the screen of the main display unit 17 in step S15. In accordance with this guidance, the user instructs the position of the target subject on the screen of the main display unit 17 by an operation via the key input unit 18.

  Thereafter, in step S16, the imaging device for subject imaging detects the subject position on the screen of the main display unit 17 instructed by the user under the control of the control unit 21, that is, in the imaging frame. In step S <b> 17, the line-of-sight direction information transmitted from the line-of-sight detection imaging device is received via the communication unit 22.

  Then, in step S18, the imaging device for subject imaging, under the control of the control unit 21, determines the gaze direction indicated by the received gaze direction information and the subject position within the imaging frame detected in step S16. The data is stored in the data memory 14 or the like in association with each other.

  Subsequently, in step S19, the imaging apparatus for imaging the subject inquires the user whether or not to set another line-of-sight position via the main display unit 17 or the like, and in the case of setting, the processing from step S15 is performed. On the other hand, if not set, the process proceeds to step S20.

  Then, in step S20, the imaging device for imaging the subject is based on the correspondence between the plurality of gaze directions stored in step S18 and the subject position in the imaging frame under the control of the control unit 21. Conversion information for converting an arbitrary line-of-sight direction into a position in the imaging frame is generated, and a series of processing ends. This conversion information is used when trimming is performed during actual imaging.

  In the imaging apparatus, the correspondence relationship between the line-of-sight position and the subject position in the imaging frame can be set according to such a series of procedures, and imaging with trimming can be performed.

  Note that the detection of the gaze direction by the above-described gaze detection imaging apparatus is performed according to a series of procedures as shown in FIG.

  That is, as shown in FIG. 11, when the imaging apparatus starts imaging from the through image display state and acquires an image under the control of the control unit 21 in step S31, in step S32 Detect multiple faces reflected in.

  Subsequently, in step S33, the imaging apparatus selects one of the unprocessed faces as a processing target under the control of the control unit 21, and in step S34, the position of the hair is selected with respect to the selected face portion. And how much the position of the eye is shifted in the left-right direction from the center of the face. Then, in step S35, the imaging apparatus refers to the correspondence table created based on statistics stored in advance in the program memory 15 under the control of the control unit 21, and the deviation amount specified in step S34. The direction of the face corresponding to, that is, the angle in the horizontal direction from the center of the face is specified. Note that when the eye position cannot be detected, the imaging device determines that the person to be detected is sideways or backward and further detects the position of the ear or the like to specify the face direction.

  When the imaging device detects the orientation of the face in this way, in step S36, the imaging device determines whether it is possible to specify the amount of deviation of the position of the black eye among the white eyes.

  Here, if the imaging device cannot specify the amount of deviation of the position of the black eye, the process proceeds to step S37, and the direction of the face specified in step S35 is controlled under the control of the control unit 21. It is stored in the data memory 14 as the line-of-sight direction of the face to be processed (line-of-sight detection person), and the process proceeds to step S40.

  On the other hand, when the amount of deviation of the position of the black eye can be specified, the imaging device is created based on the statistics stored in advance in the program memory 15 under the control of the control unit 21 in step S38. With reference to the correspondence table, the line-of-sight direction, that is, the vertical and horizontal angles from the center of the face is specified for the deviation amount specified in step S36, and in step S39, the specified line-of-sight direction and the face to be processed (covered object) are specified. It is stored in the data memory 14 as the line-of-sight direction of the line-of-sight detector.

  When the imaging device specifies the line-of-sight direction for one face in this way, in step S40, the imaging device determines whether or not the line-of-sight direction for all the faces reflected in the image has been specified, and is unprocessed. If there is a face, the processing from step S33 is repeated. On the other hand, when the imaging device specifies the line-of-sight direction for all the faces reflected in the image, in step S41, a predetermined one of the detected line-of-sight directions is controlled under the control of the control unit 21. Extract those that can be identified with the accuracy of.

  Then, in step S42, the imaging apparatus specifies a minimum region where the line of sight of the first ratio or more is concentrated under the control of the control unit 21, and in step S43, the minimum line of sight of the second ratio or more is concentrated. An area is specified, and a series of processing is completed. Note that the minimum area where the line of sight of the first ratio or more concentrates corresponds to a rough imaging range when the imaging device for imaging the subject performs imaging, and the minimum area where the line of sight of the second ratio or more concentrates is rough. This corresponds to the trimming range within the imaging range.

  In this way, the imaging apparatus can detect the line-of-sight direction of the line-of-sight detector. The imaging device for eye gaze detection uses first gaze direction information including a direction in which gazes of a first ratio or more concentrate and second gaze direction information including a direction in which gazes of a second percentage or more concentrate. To the image pickup apparatus.

  Now, when a moving image is picked up using a line-of-sight detection imaging apparatus that performs such a line-of-sight detection process and a subject imaging apparatus, these are followed according to a series of procedures as shown in FIG. Two imaging devices will cooperate.

  First, as shown in FIG. 12, when the imaging device for line-of-sight detection acquires a new captured image under the control of the control unit 21 in step S51, the captured image is reflected in the acquired image in step S52. Detects multiple faces that are exposed.

  Subsequently, in step S53, the imaging device for line-of-sight detection detects the line-of-sight direction of each face according to a series of procedures as shown in FIG. 11 under the control of the control unit 21, and then in step S54. The first line-of-sight direction including the direction in which the lines of sight greater than or equal to the first ratio are identified, and the second line-of-sight direction including the direction in which the lines of sight greater than or equal to the second ratio are concentrated is identified in step S55.

  In step S56, the imaging device for line-of-sight detection transmits the identified first line-of-sight direction information and second line-of-sight direction information to the imaging device for imaging the subject via the communication unit 22. Thereafter, in step S57, the imaging device for line-of-sight detection repeats the processing from step S51 until the end determination is performed, and when the end determination is performed, the series of processing ends.

  On the other hand, when the imaging device for imaging the subject receives the first gaze direction information and the second gaze direction information transmitted from the imaging device for gaze detection via the communication unit 22 in step S61, in step S62. The first gaze direction and the second gaze direction indicated by each of the first gaze direction information and the second gaze direction information based on the above-described conversion information generated prior to imaging under the control of the control unit 21. The direction is converted into a first imaging range and a second imaging range.

  Subsequently, in step S63, the imaging device for subject imaging determines whether or not the entire first imaging range is included in the current imaging frame under the control of the control unit 21.

  Here, when the entire first imaging range is not included in the current imaging frame, imaging with trimming cannot be performed. Therefore, in step S64, the imaging device for subject imaging determines whether or not another moving image is being captured and recorded under the control of the control unit 21. On the other hand, if the process from step S61 is repeated, if the image is being captured and recorded, in step S65, the capturing and recording of the moving image is terminated, and the process from step S61 is repeated.

  On the other hand, when the entire first imaging range is included in the current imaging frame, the imaging device for subject imaging shifts the process to step S66 under the control of the control unit 21. In step S67, it is determined whether or not the second imaging range is a part of the current imaging frame.

  When the second imaging range is not part of the current imaging frame, the imaging device for subject imaging cannot perform imaging with trimming, and thus repeats the processing from step S61, while If the imaging range is a part of the current imaging frame, in step S68, the area in the imaging frame corresponding to the second imaging range is set with a predetermined margin under the control of the control unit 21. The trimmed moving image data is recorded on the memory card 20.

  Then, in step S69, the imaging device for subject imaging repeats the processing from step S61 until the end determination is performed, and when the end determination is performed, the series of processing ends.

  In accordance with such a series of procedures, the imaging apparatus can capture a moving image and generate moving image data obtained by trimming a region where the line-of-sight of the person to be detected is concentrated. Note that the imaging apparatus can generate still image data obtained by trimming an area where the line of sight of the subject line-of-sight detection person concentrates by performing the same processing when capturing a still image.

  As described above, the imaging apparatus automatically determines a region where the line-of-sight of the subject-to-be-detected person concentrates, that is, a region focused on by a person, generates image data trimmed so as to include the region, and stores the memory Record on the card 20. Therefore, the imaging apparatus can obtain an image in which a region where the line of sight of the subject's eye gaze is concentrated is close-up, and avoids wasteful image recording by recording the entire region where the eye gaze of the subject's gaze detector is concentrated. Thus, the memory can be effectively used.

  Here, actual imaging is started with the case where the entire first imaging range is included in the current imaging frame as a trigger, and the second imaging range is part of the current imaging frame. Although it has been described that trimming is performed using the case as a trigger, in the imaging apparatus, a technique other than trimming can be applied in order to effectively use the memory.

  Specifically, the imaging apparatus can effectively use the memory by determining whether or not to record a captured image according to the detected situation of the plurality of gaze directions. As a specific example, processing in the case of capturing a still image using an imaging device for eye-gaze detection and an imaging device for subject imaging will be described with reference to FIG. Note that since the imaging device for line-of-sight detection performs the same processing as the processing shown in FIG. 12, only the processing performed by the imaging device for subject imaging is shown in FIG.

  First, as shown in FIG. 13, the imaging device for subject imaging waits until the user presses the shutter button of the key input unit 18 in step S71, and when the shutter button is pressed, in step S72, The first gaze direction information and the second gaze direction information transmitted from the gaze detection imaging device are received via the communication unit 22, and based on the conversion information described above, the first gaze direction information and the second gaze direction are received. The first line-of-sight direction and the second line-of-sight direction indicated by each of the direction information are converted into a first imaging range and a second imaging range.

  Subsequently, in step S73, the imaging device for imaging an object determines whether the entire first imaging range is included in the current imaging frame under the control of the control unit 21.

  Here, in the case where the entire first imaging range is included in the current imaging frame, the imaging device for subject imaging performs autofocus operation and control under the control of the control unit 21 in step S74. An automatic exposure operation or the like is performed, a still image capturing preparation operation is started, and the processing from step S72 is repeated.

  On the other hand, when the entire first imaging range is not included in the current imaging frame, the imaging device for subject imaging shifts the process to step S75 under the control of the control unit 21, It is determined whether the second imaging range is a part of the current imaging frame.

  When the second imaging range is part of the current imaging frame, the imaging device for subject imaging captures a still image and the memory card 20 under the control of the control unit 21 in step S76. The process from step S72 is repeated.

  On the other hand, when the second imaging range is not part of the current imaging frame, the imaging device for subject imaging captures, for example, 10 seconds from the pressing of the shutter button under the control of the control unit 21 in step S77. It is determined whether or not a predetermined time has passed, and the processing from step S72 is repeated until the predetermined time has passed. The imaging device for subject imaging ends the series of processes as it is after a predetermined time has elapsed. Note that the imaging device for subject imaging does not end the process due to such a timeout, but loops a series of processes so that the entire first imaging range is included in the current imaging frame, and A still image when the second imaging range is part of the current imaging frame may be automatically recorded on the memory card 20.

  According to such a series of procedures, the imaging device determines whether to record a still image according to the detected state of the line-of-sight direction, and records only still image data of a situation in which a person is paying attention. can do. Therefore, the imaging apparatus can avoid the useless recording of the image by recording the image in the state where the line of sight of the person to be detected is not concentrated, and can effectively use the memory.

  Here, assuming that a still image is captured, an imaging preparation operation with an autofocus operation and / or an automatic exposure operation is triggered by the case where the entire first imaging range is included in the current imaging frame. It has been described that the actual imaging is started with a trigger when the second imaging range is a part of the current imaging frame. However, when the imaging device captures a moving image, What is necessary is just to perform the imaging preparation operation | movement which starts the circular recording imaging of the past for the case where the whole one imaging range is contained in the present imaging frame as a trigger. In this circular recording imaging, the imaging results are sequentially and cyclically recorded in a predetermined memory, and the recording is cyclically triggered when the entire first imaging range is included in the current imaging frame. This is an operation for recording a moving image on the memory card 20.

  The imaging device may also perform trimming when starting the actual imaging using the case where the second imaging range is a part of the current imaging frame as a trigger.

  Furthermore, when the entire first imaging range is included in the current imaging frame, the imaging device controls the imaging unit 12 to continuously record a plurality of still images in the ring buffer. When a transition is made to the so-called past imaging state and the second imaging range is a part of the current imaging frame, past continuous shooting is performed from the past imaging state, and the continuously shot image data is stored in the memory card 20. It may be recorded. In the imaging apparatus, by adopting the past continuous shooting in this way, it is possible to capture and record the decisive scene that the person is paying attention to with a time width, which is necessary depending on the judgment of the user. Post-processing such as extracting only image data can be performed.

  Furthermore, the imaging apparatus may not only electronically trim within the imaging frame but also mechanically change the imaging direction and zoom magnification.

  In addition, the imaging apparatus may perform trimming not only when imaging is finished, but also when editing in the playback mode or when changing the image display by a slide show.

  Furthermore, the imaging device may perform different processing depending on the type of the eye-gaze detection person, such as the sex of adults and children, sex, animals other than humans, and the like.

[Effect of the embodiment]
As described above in detail, the imaging apparatus shown as the embodiment of the present invention is within the imaging frame corresponding to the gaze direction of the subject-line-of-sight detection person in the detected captured image under the control of the control unit 21. Are trimmed, and the trimmed image is recorded in the memory card 20.

  As a result, the imaging apparatus automatically determines the region of interest of the subject-of-sight detection person and records the trimmed image on the memory card 20, thereby avoiding useless image recording and effective use of the memory. Can be planned.

  In addition, the imaging apparatus identifies a range in which the gaze directions of a plurality of gaze detectors are concentrated under the control of the control unit 21, and trims a partial region in the imaging frame corresponding to the identified range. To do. More specifically, this imaging apparatus, under the control of the control unit 21, selects a partial area in the imaging frame corresponding to a range in which gazes of a predetermined ratio or more concentrate among gazes of a plurality of gaze detection persons. Trim.

  Thereby, this imaging device can reliably determine the region of interest to the line-of-sight detector, and effectively uses the memory while reliably recording the decisive scene that the non-line-of-sight detector is paying attention to. be able to.

  Furthermore, this imaging device detects the line-of-sight direction of the line-of-sight detector in the captured image under the control of the control unit 21 and corresponds to the line-of-sight direction of the line-of-sight detector in the detected captured image. The inner partial area can be trimmed. In other words, this imaging apparatus can perform the gaze direction detection process of the gaze detection person and the imaging process of the subject with a single unit.

  Furthermore, the imaging device is configured to transmit gaze direction information indicating the gaze direction of the gaze detection person in the captured image detected by another imaging device for gaze detection under the control of the control unit 21. It is also possible to trim a partial region in the imaging frame corresponding to the gaze direction of the person to be detected in the captured image indicated by the received gaze direction information. That is, this imaging apparatus can perform imaging processing of a subject in cooperation with another imaging apparatus for detecting a line of sight.

  In this case, prior to the start of imaging, the imaging apparatus is configured to control the control unit 21 to detect the line-of-sight position of the subject-to-be-detected person detected by the other imaging apparatus and the imaging frame captured by the imaging apparatus. A correspondence relationship with the inner position is set, and conversion information for converting an arbitrary sight line direction into a position within the imaging frame is generated based on the correspondence relationship between the set plurality of sight line directions and the imaging frame position. And this imaging device specifies the range where the gaze directions of a plurality of gaze detection persons concentrate based on conversion information under control of control part 21, and the imaging frame corresponding to this specified range Trim a partial area inside.

  Thereby, when this imaging device cooperates with other imaging devices for eye-gaze detection, it is possible to reliably determine the region that the eye-gaze detector is paying attention to, and the determination that the eye-gaze detector is paying attention to. This makes it possible to effectively use the memory while reliably recording typical scenes.

  In addition, the imaging device is configured to control a first imaging range corresponding to a first line-of-sight direction including a direction in which lines of sight of a first ratio or more concentrate among the lines of sight of a plurality of line-of-sight detectors under the control of the control unit 21. When the entire image is included in the imaging frame, the imaging operation is started, and the second corresponding to the second gaze direction including the direction in which the gazes of the second ratio or more concentrate among the gazes of the plurality of gaze detection persons. When the imaging range is a part of the imaging frame, trimming is performed so as to include an area in the imaging frame corresponding to the second imaging range.

  Thereby, this imaging device can avoid the situation where it image | photographs when the whole 1st imaging range is not contained in an imaging frame, and also the area | region where a to-be-detected person is paying attention can be ensured. This makes it possible to make effective use of the memory while reliably recording the decisive scene that the non-line-of-sight detector is paying attention to.

  In addition, the imaging apparatus determines whether or not to record the captured image in the memory card 20 according to the state of the line-of-sight direction of the person to be detected in the detected captured image under the control of the control unit 21. to decide.

  As a result, the imaging apparatus automatically determines a situation that is noticed by a plurality of eye-gaze detectors, and determines whether or not imaging and recording are performed. Therefore, it is possible to avoid useless image recording and effectively use the memory. Can be planned.

  Furthermore, this imaging device is configured to control a first imaging range corresponding to a first line-of-sight direction including a direction in which lines of sight of a first ratio or more concentrate among the lines of sight of a plurality of line-of-sight detectors under the control of the control unit 21. The imaging preparation operation is started when the whole is included in the imaging frame, and the second corresponding to the second gaze direction including the direction in which the gazes of the second or higher ratio concentrate among the gazes of the plurality of gaze detection persons. When the two imaging ranges are a part of the imaging frame, imaging is started so as to include an area in the imaging frame corresponding to the second imaging range, and an image is recorded in the memory card 20.

  Thereby, this imaging device can avoid the situation where it image | photographs when the whole 1st imaging range is not contained in an imaging frame, Furthermore, the situation which a to-be-detected person notices reliably This makes it possible to make effective use of the memory while reliably recording the decisive scene that the non-line-of-sight detector is paying attention to.

  Furthermore, in the case of capturing a moving image, the imaging apparatus captures the past cyclic recording imaging when the entire first imaging range is included in the imaging frame under the control of the control unit 21. Therefore, when the second imaging range is included in the imaging frame, it is possible to reliably shift to the moving image imaging operation.

  In addition, in the case of capturing a still image, the imaging apparatus performs an autofocus operation and / or when the entire first imaging range is included in the imaging frame under the control of the control unit 21. Since the imaging preparation operation with the automatic exposure operation is started, it is possible to surely shift to the still image imaging operation when the second imaging range is included in the imaging frame.

  Furthermore, this imaging device detects the line-of-sight direction of the line-of-sight detector in the captured image under the control of the control unit 21, and according to the state of the line-of-sight direction of the line-of-sight detector in the detected captured image. Then, it is determined whether or not to record the captured image on the memory card 20. In other words, this imaging apparatus can perform the gaze direction detection process of the gaze detection person and the imaging process of the subject with a single unit.

  Furthermore, the imaging device is configured to transmit gaze direction information indicating the gaze direction of the gaze detection person in the captured image detected by another imaging device for gaze detection under the control of the control unit 21. It is also possible to determine whether or not to record the captured image on the memory card 20 according to the state of the visual line direction of the person to be detected in the captured image indicated by the received visual line direction information. That is, this imaging apparatus can perform imaging processing of a subject in cooperation with another imaging apparatus for detecting a line of sight.

  In addition, prior to the start of imaging, the imaging apparatus is configured to control the control unit 21 to control the line-of-sight position of a subject-to-be-detected person detected by another imaging apparatus and the imaging frame captured by the imaging apparatus. A correspondence relationship with the position is set, and conversion information for converting an arbitrary gaze direction into a position within the imaging frame is generated based on the correspondence relationship between the set plurality of viewing directions and the position within the imaging frame. And this imaging device specifies the range where the gaze directions of a plurality of gaze detection persons concentrate based on conversion information under control of control part 21, and the imaging frame corresponding to this specified range Imaging is started so as to include the inner area, and an image is recorded in the memory card 20.

  Thereby, when this imaging device cooperates with other imaging devices for eye-gaze detection, it is possible to reliably determine the region that the eye-gaze detector is paying attention to, and the determination that the eye-gaze detector is paying attention to. This makes it possible to effectively use the memory while reliably recording typical scenes.

  The above-described embodiment is an example of the present invention. For this reason, the present invention is not limited to the above-described embodiment, and even if it is a form other than this embodiment, as long as it does not depart from the technical idea according to the present invention, the design and the like Of course, various modifications are possible.

It is a block diagram shown about the composition of the imaging device shown as an embodiment of the present invention. It is a figure which shows a mode that the detection of a gaze direction of a to-be-detected person and imaging of a to-be-photographed object are performed using two imaging devices. It is a figure which shows the specific example of the imaging frame imaged by each of two imaging devices in the condition shown in FIG. It is a figure which shows a mode that the detection of a gaze direction of a to-be-photographed person and imaging | photography of a to-be-photographed object are performed using one imaging device. It is a figure which shows the specific example of the imaging frame imaged with an imaging device in the condition shown in FIG. It is a figure which shows a mode that four spectators are looking at one actor, and three spectators are looking at the other actor. It is a figure which shows the specific example of the imaging frame for demonstrating the trimming range determined in the condition shown in FIG. It is a figure which shows a mode that there are five spectators who look at one actor, and two spectators who look at the other actor. It is a figure which shows the specific example of the imaging frame for demonstrating the trimming range determined in the condition shown in FIG. It is a flowchart which shows a series of procedures at the time of setting the correspondence of the gaze position detected by the imaging device for gaze detection, and the position in the imaging frame imaged by the imaging device for imaging a subject. It is a flowchart which shows a series of procedures at the time of detecting a gaze direction of a to-be-sighted person by the imaging device for gaze detection. It is a flowchart which shows a series of procedures at the time of imaging a moving image using two imaging devices. It is a flowchart which shows a series of procedures at the time of imaging a still image using two imaging devices.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Imaging optical system 12 Imaging part 13 Imaging process part 14 Data memory 15 Program memory 16 Main display process part 17 Main display part 18 Key input part 19 Image compression part 20 Memory card 21 Control part 22 Communication part

Claims (15)

An imaging device that acquires a captured image by an imaging element,
Trimming means for trimming a partial region in the imaging frame corresponding to the line-of-sight direction of the person to be detected in the captured image;
Recording means for recording the image trimmed by the trimming means;
With
The trimming means identifies a range in which gazes of a predetermined ratio or more concentrate among gazes of a plurality of gaze detectors, and trims a partial region in the imaging frame corresponding to the identified range. Imaging device. Receiving means for receiving gaze direction information indicating the gaze direction of the gaze-detected person in a captured image detected by another imaging device for gaze detection;
The trimming means, claims, characterized in that the viewing direction of a plurality of the visual axis detected individual in line-of-sight direction information in the captured image showing received by said receiving means to trim the partial areas in the imaging frame to focus predetermined ratio or more Item 2. The imaging device according to Item 1 . Prior to the start of imaging, setting means for setting a correspondence relationship between the line-of-sight position of the subject-to-be-detected person detected by the other imaging device and the position in the imaging frame captured by the imaging device;
Conversion information generating means for generating conversion information for converting an arbitrary line-of-sight direction into a position in the imaging frame based on a correspondence relationship between the plurality of line-of-sight directions set by the setting means and the position in the imaging frame. Prepared,
The trimming unit identifies a range in which the gaze directions of a plurality of gaze detectors are concentrated by a predetermined ratio or more based on the conversion information generated by the conversion information generation unit, and an imaging frame corresponding to the specified range The imaging apparatus according to claim 2 , wherein a partial region in the inside is trimmed. An imaging device that acquires a captured image by an imaging element,
Trimming means for trimming a partial region in the imaging frame corresponding to the line-of-sight direction of the person to be detected in the captured image;
A gaze direction detection means for detecting the visual line direction of the visual axis detected individual in the recording means and the image pickup image that records the cropped image by said trimming means
With
The trimming means, imaging apparatus characterized by trimming the partial areas in the imaging frame corresponding to the viewing direction of the sight line detecting people in the viewing direction detecting means is detected by the imaging image. An imaging device that acquires a captured image by an imaging element,
Trimming means for trimming a partial region in the imaging frame corresponding to the line-of-sight direction of the person to be detected in the captured image;
Recording means for recording the image trimmed by the trimming means;
With
The trimming means includes an entire first imaging range corresponding to a first gaze direction including a direction in which gazes of a predetermined first ratio or more concentrate among gazes of a plurality of gaze detectors. The second imaging range corresponding to the second line-of-sight direction including the direction in which the lines of sight of a predetermined second ratio or more of the lines of sight of the plurality of line-of-sight detectors are concentrated is one in the imaging frame. An image pickup apparatus that performs trimming so as to include an area in an image pickup frame corresponding to the second image pickup range. Depending on the circumstances of the viewing direction of the sight line detected individual in the captured image, of claims 1 to 5, characterized in that it comprises a determining means for determining whether or not to record the captured image in the recording means The imaging device of any one of them. The determination means includes an entire first imaging range corresponding to a first visual line direction including a direction in which visual lines of a predetermined first ratio or more are concentrated among the visual lines of a plurality of visual line detectors. The second imaging range corresponding to the second gaze direction including a direction in which gazes of a predetermined second ratio or more concentrate among gazes of the plurality of gaze detectors within the imaging frame. The imaging apparatus according to claim 6, wherein if it is a part, imaging is started so as to include an area in an imaging frame corresponding to the second imaging range, and the recording unit records an image. In the case of capturing a moving image, the determination means performs an imaging preparation operation for starting the past circular recording when the entire first imaging range is included in the imaging frame. The imaging device according to claim 7 . When the still image is captured, the determination unit starts an imaging preparation operation with an autofocus operation and / or an automatic exposure operation when the entire first imaging range is included in the imaging frame. The imaging apparatus according to claim 7 . Gaze direction detection means for detecting the gaze direction of the gaze detector in the captured image ;
And a judging unit for judging whether or not to record the captured image in the recording unit according to the state of the gaze direction of the person to be detected in the captured image detected by the gaze direction detecting unit. The imaging device according to claim 1 or 5 . Receiving means for receiving gaze direction information indicating the gaze direction of the gaze-detected person in a captured image detected by another imaging device for gaze detection;
The determining means determines whether or not to record the captured image on the recording means according to the state of the gaze direction of the person to be detected in the captured image indicated by the visual line direction information received by the receiving means. The imaging apparatus according to claim 10, characterized in that: Prior to the start of imaging, setting means for setting a correspondence relationship between the line-of-sight position of the subject-to-be-detected person detected by the other imaging device and the position in the imaging frame captured by the imaging device;
Conversion information generating means for generating conversion information for converting an arbitrary line-of-sight direction into a position in the imaging frame based on a correspondence relationship between the plurality of line-of-sight directions set by the setting means and the position in the imaging frame. Prepared,
The determination unit specifies a range in which the line-of-sight directions of a plurality of eye-gaze detectors are concentrated based on the conversion information generated by the conversion information generation unit, and an area in the imaging frame corresponding to the specified range The imaging apparatus according to claim 11, wherein imaging is started so as to include an image, and the recording unit records an image. A program executed by a computer built in an imaging apparatus that acquires a captured image by an imaging element,
A trimming process for trimming a partial region in the imaging frame corresponding to the line-of-sight direction of the person to be detected in the captured image;
A recording process for recording the trimmed image in the recording means;
Including
The trimming process specifies a range in which gazes of a predetermined ratio or more concentrate among gazes of a plurality of gaze detectors, and trims a partial region in the imaging frame corresponding to the specified range. program. A program executed by a computer built in an imaging apparatus that acquires a captured image by an imaging element,
A trimming process for trimming a partial region in the imaging frame corresponding to the line-of-sight direction of the person to be detected in the captured image;
A recording process for recording the trimmed image in the recording means;
Gaze direction detection processing for detecting the gaze direction of the gaze detector in the captured image;
Including
The program for trimming, wherein the trimming process trims a partial region in an imaging frame corresponding to a gaze direction of a person to be detected in the captured image detected by the gaze direction detection process. A program executed by a computer built in an imaging apparatus that acquires a captured image by an imaging element,
A trimming process for trimming a partial region in the imaging frame corresponding to the line-of-sight direction of the person to be detected in the captured image;
A recording process for recording the trimmed image in the recording means;
Including
In the trimming process, the entire first imaging range corresponding to the first gaze direction including the direction in which gazes of a predetermined first ratio or more out of gazes of a plurality of gaze detectors are included in the imaging frame. The second imaging range corresponding to the second line-of-sight direction including the direction in which the lines of sight of a predetermined second ratio or more of the lines of sight of the plurality of line-of-sight detectors are concentrated is one in the imaging frame. A program for trimming to include an area in an imaging frame corresponding to the second imaging range.

Images