JP4923517B2 - Imaging device, imaging method, and semiconductor device - Google Patents

Description

The present invention relates to an imaging device , an imaging method, and a semiconductor device , and more particularly to an improvement in data processing for setting an edit point for audio video data obtained by photographing using an imaging device.

  In recent years, digital imaging apparatuses that photograph a subject, encode digital video data, and record the data on a recording medium have become widespread, and opportunities for handling a large amount of digital video data have increased even in general households.

  However, once you shoot the video, you want to edit it and put it together, but it is troublesome to find the starting point of editing. For example, video data taken at an athletic meet or wedding may be temporarily recorded on a recording medium, but may be left unattended for reasons such as troublesome editing. .

  Even when the editing start point is found, the picture corresponding to the start point is an inter-screen prediction picture in the predictive coding process, and thus cannot be easily used as a start position.

  As described above, the video data shot by the conventional imaging device requires troublesome editing work in order to easily view only the important part or leave it on the recording medium. It was.

  By the way, Japanese Patent Laid-Open No. 2003-299010 discloses a video content editing support system. This system captures an image and records video content data, and obtains it by shooting with the imaging device. And an editor terminal device for receiving and displaying the received video content data in real time via a network or the like.

  An imaging device of this editing support system includes an electronic mark generator that generates electronic mark data based on user operations and the like, and describes the generated electronic mark data in association with the time code of video content data obtained by shooting And an electronic mark insertion portion. The editor terminal device of the editing support system includes a list creation unit that creates electronic mark list data based on the electronic mark data from the imaging device, and a display unit that displays video content data from the imaging device. The video content data image synchronized with the timing corresponding to the electronic mark data is displayed on the display unit.

In such a video content editing support system, by adding electronic mark data to video content data which is imaging data of a subject by user operation during shooting, video content data obtained by imaging is stored in a personal computer or the like. It can be automatically edited on the editor terminal device based on the electronic mark data.
JP 2003-299010 A

  However, in the video content editing support system described in the above document, electronic mark data indicating the editing position needs to be added to the video content data during shooting in order to automatically perform editing after shooting. When shooting, there is a problem that it is necessary to perform troublesome operations such as adding a marker to an important shooting part that should be edited and left.

The present invention has been made in order to solve the above-described conventional problems, and enables an imaging unit that is important for a photographer to be edited automatically or by a simple selection operation for guidance. An object is to obtain an imaging method and a semiconductor device .

The invention according to claim 1 of the present application is an imaging apparatus that acquires image information and audio information by photographing a subject, and records an audio video stream including the image information and audio information. An image capturing unit that outputs a signal, an image processing unit that performs image processing on the image signal obtained by capturing the subject and extracts image information including image feature amounts indicating the characteristics of image change, and obtains sound A voice acquisition unit that outputs a voice signal, a voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition, and extracts voice information including a voice feature amount indicating a feature of voice change; , when the image feature amount or the audio feature amount is greater than a predetermined threshold value, the characteristic amount determination unit for determining a photographing timing of the image or voice has changed is valid for editing point, knitted And an information generation unit that generates an edit point information indicating a photographing timing is determined to be appropriate as a point, stored in a recording medium an audio video stream including the image information, audio information, and an editing point information, further The information generation unit determines whether or not buffer data that is image information before encoding is held in the image processing unit, and when the buffer data before encoding is held, When the edit point is set to a picture corresponding to the image, sound, or shooting timing when the shooting state has changed, and the buffer data before encoding is not held, the edit point is set to the image processing unit. This is the unit of random access that is closest to the shooting timing at which the image or sound changes in the stream obtained by encoding the image signal by If the pre-encoding buffer data is held in the image processing unit, the picture processing unit sets the picture corresponding to the editing point as the first picture of the VOB unit. The VOB unit is formed so that

The invention according to claim 2 of the present application is the imaging apparatus according to claim 1, wherein the unique identification information acquisition unit that acquires the unique identification information indicating the shooting state and the acquired unique identification information are subjected to signal processing to obtain the shooting state. A unique identification information processing unit that extracts a unique feature amount indicating a feature of the change, and the feature amount determination unit, when the image feature amount, the audio feature amount, or the unique feature amount is larger than a predetermined threshold, It is determined that the shooting timing at which the image, the sound, or the shooting state has changed is appropriate as the editing point.

  The invention according to claim 3 of the present application is the imaging device according to claim 2, wherein the unique feature amount is determined by a magnitude of a physiological change of a photographer that occurs during photographing or a magnitude of adjustment by an operation of the photographer. It was meant to be shown.

  According to a fourth aspect of the present invention, in the imaging device according to the third aspect, the photographer's physiological changes that occurred during the photographing are a change in the amount of sweating of the photographer, a change in α wave, a change in the number of blinks It is at least one of a change in the pupil and a change in the pulse, and the unique identification information acquisition unit has a sensor according to the type of the physiological change for measuring the physiological change of the photographer. .

  The invention according to claim 5 of the present application is the imaging apparatus according to claim 1, wherein the image processing unit has already encoded a picture to be encoded with respect to an image signal obtained by imaging a subject. Inter-picture predictive coding processing for predictive coding with reference to the picture, and extracting the image feature amount based on a motion vector indicating the magnitude of image motion used in the inter-picture predictive coding processing. The audio processing unit performs an encoding process corresponding to the encoding process for the image signal on the audio signal obtained by acquiring the audio, and the information generation unit is valid as the editing point. Based on the determined shooting timing, a specific picture in the image stream obtained by encoding the image signal is set as the edit point.

  The invention according to claim 6 of the present application is the imaging apparatus according to claim 1, wherein the sound processing unit extracts the sound feature amount based on a change amount of the sound signal.

  According to a seventh aspect of the present invention, in the imaging device according to the second aspect, a threshold level is set for each of the image feature amount or the sound feature amount and the unique feature amount based on a user manual operation signal. A control unit, wherein the feature amount determination unit determines each feature amount based on a corresponding threshold level set by the control unit, and the image, sound, or shooting state in which the shooting state is changed This is to determine whether the timing is valid as an edit point.

  The invention according to claim 8 of the present application is the imaging device according to claim 2, wherein each of the plurality of scenarios is associated with a combination of a threshold level with respect to each of the image feature amount, the sound feature amount, and the unique feature amount. And a control unit that sets threshold levels of the various feature amounts based on the scenario specified by the user's manual operation and the table information, and the feature amount determination unit includes: Each feature amount is determined based on a corresponding threshold level set by the control unit, and it is determined whether the shooting timing at which the image, sound, or shooting state has changed is appropriate as an editing point. To do.

  The invention according to claim 9 of the present application is the imaging apparatus according to claim 8, wherein the table information is information obtained by downloading from an information terminal on a network.

The invention according to claim 10 of the present invention is the imaging apparatus according to claim 1, wherein said information generating unit, the editing point, the image or from the time when the event that changes occur in the voice, actually image or, The shooting timing is set according to the delay time until the sound changes.

The invention according to claim 11 of the present application is an imaging apparatus that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information. An image capturing unit that outputs a signal, an image processing unit that performs image processing on the image signal obtained by capturing the subject and extracts image information including image feature amounts indicating the characteristics of image change, and obtains sound A voice acquisition unit that outputs a voice signal, a voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition, and extracts voice information including a voice feature amount indicating a feature of voice change; A feature amount determination unit that determines that a shooting timing at which the image or sound has changed is appropriate as an edit point when the image feature amount or the sound feature amount is greater than a predetermined threshold; An information generation unit that generates edit point information indicating shooting timing determined to be valid as a point, and stores an audio video stream including the image information, audio information, and edit point information in a recording medium, and The information generation unit determines whether or not buffer data that is image information before encoding is held in the image processing unit, and when the buffer data before encoding is held, When the edit point is set to the picture corresponding to the shooting timing when the image or sound changes, and the buffer data before the encoding is not held, it can be used for the encoding process in the image processing unit. If the remaining time and the time required for re-encoding are compared and the time required for re-encoding exceeds the remaining time available for the encoding process in the image processing unit, The edit point is set to the first picture of a VOB unit, which is a unit of random access, closest to the shooting timing at which the image or sound changes in the image stream obtained by encoding the image signal by the image processing unit. If the time required for the re-encoding does not exceed the remaining time available for the encoding process in the image processing unit, the image processing unit is instructed to re-encode the image stream, and the image processing When the pre-encoding buffer data is held, the unit forms a VOB unit so that the picture corresponding to the edit point becomes the first picture of the VOB unit, and the pre-encoding buffer data When the time required for the re-encoding does not exceed the remaining time available for the encoding process in the image processing unit. Is on the outs picture stream, the picture corresponding to the edit point, be re-encoded to be an I-picture positioned at the head of the VOB unit is intended.

According to a twelfth aspect of the present invention, in the imaging apparatus according to the first or eleventh aspect , the time at which an event that changes the image or the sound occurs is recorded in the audio video stream as the edit point. is there.

According to a thirteenth aspect of the present invention, in the imaging device according to the twelfth aspect , the occurrence time of the event is recorded in the audio video stream as a play list indicating a reproduction condition.

According to a fourteenth aspect of the present invention, in the imaging device according to the twelfth aspect, information indicating whether the edit point is caused by an image or a sound is embedded in the audio video stream. It is.

According to a fifteenth aspect of the present invention, in the imaging device according to the first or eleventh aspect , the information generation unit is configured to edit a picture corresponding to a time at which an event that changes the image or the sound occurs at the time of editing. It is embedded in the audio video stream as an out-of-sequence picture used for thumbnail display.

The invention according to claim 16 of the present application is an imaging method for acquiring image information and audio information by photographing a subject and recording an audio video stream including the image information and audio information. An imaging step for outputting a signal, an image processing step for performing signal processing on the image signal obtained by imaging the subject, and extracting image information including an image feature amount indicating a feature of an image change, and acquiring sound A voice acquisition step for outputting a voice signal, and a voice processing step for performing signal processing on the voice signal obtained by the voice acquisition to extract voice information including a voice feature amount indicating a feature of voice change; When the image feature amount or the sound feature amount is larger than a predetermined threshold, it is determined that the shooting timing at which the image or sound has changed is appropriate as the editing point. A feature amount determination step to be determined, an information generation step for generating edit point information indicating a photographing timing determined to be appropriate as the edit point, and an audio video stream including the image information, audio information, and edit point information The information generating step determines whether or not buffer data that is image information before encoding is held in the image processing unit that executes the image processing step. If the buffer data before encoding is held, the editing point is set to the picture corresponding to the image, audio, or shooting timing when the shooting state has changed, and the buffer data before encoding is set. Is not held, the edit point is the stream in the stream obtained by encoding the image signal by the image processing unit. Set to the first picture of the VOB unit, which is the unit of random access, closest to the shooting timing at which the image or sound has changed, and the image processing step stores the buffer data before encoding in the image processing unit. If so, the VOB unit is formed so that the picture corresponding to the edit point becomes the first picture of the VOB unit.

The invention according to claim 17 of the present application is a semiconductor device that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information, and is obtained by imaging the subject. An image processing unit that performs signal processing on the received image signal to extract image information including an image feature amount indicating a change feature of the image, an audio acquisition unit that acquires audio and outputs an audio signal, and the audio A voice processing unit that performs signal processing on the voice signal obtained by acquiring the voice information including the voice feature quantity indicating the feature of the voice change, and the image feature quantity or the voice feature quantity is greater than a predetermined threshold value. when is large, the characteristic amount determination unit for determining a photographing timing of the image or voice has changed is valid for editing point, shot is determined to be appropriate as the edit point in Thailand And an information generation unit that generates an edit point information indicating the ring, and stores the image information, audio information, and the audio video stream including an edit point information on a recording medium, further, the information generating unit, the image processing Whether or not buffer data that is image information before encoding is held, and if the buffer data before encoding is held, the edit point is set to the image, audio, or When the picture corresponding to the shooting timing at which the shooting state has changed is set and the buffer data before encoding is not held, the editing point is obtained by encoding the image signal by the image processing unit. Set the first picture of the VOB unit, which is the unit of random access, closest to the shooting timing at which the image or sound changes in the stream. Image processing unit, if the coding buffer before the data is held in the image processing section, a picture corresponding to the edit point, to form a VOB unit to be a head picture VOB unit, stuff It is.

According to the first aspect of the present invention, there is provided an imaging apparatus that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information. An image capturing unit that outputs a signal, an image processing unit that performs image processing on the image signal obtained by capturing the subject and extracts image information including image feature amounts indicating the characteristics of image change, and obtains sound A voice acquisition unit that outputs a voice signal, a voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition, and extracts voice information including a voice feature amount indicating a feature of voice change; , when the image feature amount or the audio feature amount is greater than a predetermined threshold value, the characteristic amount determination unit for determining a photographing timing of the image or voice has changed is valid for editing point, knitted And an information generation unit that generates an edit point information indicating a photographing timing is determined to be appropriate as a point, stored in a recording medium an audio video stream including the image information, audio information, and an editing point information, further The information generation unit determines whether or not buffer data that is image information before encoding is held in the image processing unit, and when the buffer data before encoding is held, When the edit point is set to a picture corresponding to the image, sound, or shooting timing when the shooting state has changed, and the buffer data before encoding is not held, the edit point is set to the image processing unit. This is the unit of random access that is closest to the shooting timing at which the image or sound changes in the stream obtained by encoding the image signal by If the pre-encoding buffer data is held in the image processing unit, the picture processing unit sets the picture corresponding to the editing point as the first picture of the VOB unit. The VOB unit is formed so that the edit point is accurately set when there is buffer data before encoding, and the edit point is easily set when there is no buffer data before encoding. The portion of the audio / video stream obtained by shooting can be edited automatically or by a simple selection operation for guidance.

According to the second aspect of the present invention, in the imaging apparatus according to the first aspect, the unique identification information acquisition unit that acquires the unique identification information indicating the shooting state, and the acquired unique identification information are subjected to signal processing to obtain the shooting state. A unique identification information processing unit that extracts a unique feature amount indicating a feature of the change, and the feature amount determination unit, when the image feature amount, the audio feature amount, or the unique feature amount is larger than a predetermined threshold, Since it is determined that the shooting timing at which the image, sound, or shooting state has changed is appropriate as the editing point, the shooting timing at which the shooting state has greatly changed can be set as the editing point.

  According to a third aspect of the present invention, in the imaging apparatus according to the second aspect, the characteristic feature amount is determined by a magnitude of a photographer's physiological change occurring during photographing or a magnitude of adjustment by a photographer's operation. Therefore, the shooting timing when the photographer unconsciously operates the imaging apparatus and the shooting timing when the photographer concentrates or gets excited can be set as editing points.

  According to the invention of claim 4 of the present application, in the imaging apparatus of claim 3, at least one of a change in the amount of sweat of the photographer, a change in α wave, a blinking frequency, a change in pupil, and a change in pulse Therefore, it is possible to edit an important scene for the photographer based on the physiological change of the photographer. It becomes possible.

  According to a fifth aspect of the present invention, in the imaging apparatus according to the first aspect, the image processing unit has encoded a picture to be encoded with respect to an image signal obtained by imaging a subject. Inter-picture predictive coding processing for predictive coding with reference to the picture is extracted, and the image feature amount is extracted based on a motion vector indicating the magnitude of image motion used in the inter-picture predictive coding processing. Therefore, the image feature amount related to the motion of the image can be accurately extracted based on the motion vector used in the predictive encoding process.

  According to the invention of claim 6 of the present application, in the imaging apparatus according to claim 1, the sound processing unit extracts the sound feature amount based on the amount of change in the sound signal. The voice feature amount can be accurately extracted based on the voice signal.

  According to the seventh aspect of the present invention, in the imaging apparatus according to the second aspect, the image processing apparatus includes a control unit that sets a threshold level for each of the image feature amount, the sound feature amount, and the unique feature amount based on a manual operation signal. Therefore, the user can set the detection intensity of the image feature amount or the voice feature amount and the unique feature amount, and thus the user's preference can be reflected in the automatic editing of the captured video data.

  According to the invention of claim 8 of the present application, in the imaging apparatus according to claim 2, the correspondence relationship between each of a plurality of different scenarios and a combination of a threshold level with respect to an image feature quantity or a voice feature quantity and a unique feature quantity is shown. The table level is stored, and the threshold level for each feature is set by selecting the scenario by manual operation of the user. Therefore, by selecting the scenario according to the shooting location, such as athletic meet or wedding, Automatic editing is possible depending on the location where the wedding is taken.

  According to the ninth aspect of the present invention, in the imaging apparatus according to the eighth aspect, the table information is downloaded and acquired from an information terminal on a network. Appropriate judgment that suits the scenario for determining whether the shooting timing at which the image, sound, or shooting state has changed is appropriate as an edit point for each feature that is a change in the image, sound, or shooting state The intensity can be set.

According to a tenth aspect of the present invention, in the imaging apparatus according to the first aspect , the information generation unit may actually change the edit point from the point in time when an event that changes the image or the sound occurs. Since the shooting timing is set according to the delay time until the sound changes, the editing point can be set almost at the timing when the event actually occurs.

According to the invention of claim 11 of the present application, there is provided an imaging apparatus that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information. An image capturing unit that outputs a signal, an image processing unit that performs image processing on the image signal obtained by capturing the subject and extracts image information including image feature amounts indicating the characteristics of image change, and obtains sound A voice acquisition unit that outputs a voice signal, a voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition, and extracts voice information including a voice feature amount indicating a feature of voice change; A feature amount determination unit that determines that a shooting timing at which the image or sound has changed is appropriate as an edit point when the image feature amount or the sound feature amount is greater than a predetermined threshold; An information generation unit that generates edit point information indicating shooting timing determined to be valid as a point, and stores an audio video stream including the image information, audio information, and edit point information in a recording medium, and The information generation unit determines whether or not buffer data that is image information before encoding is held in the image processing unit, and when the buffer data before encoding is held, When the edit point is set to the picture corresponding to the shooting timing when the image or sound changes, and the buffer data before the encoding is not held, it can be used for the encoding process in the image processing unit. If the remaining time and the time required for re-encoding are compared and the time required for re-encoding exceeds the remaining time available for the encoding process in the image processing unit, The edit point is set to the first picture of a VOB unit, which is a unit of random access, closest to the shooting timing at which the image or sound changes in the image stream obtained by encoding the image signal by the image processing unit. If the time required for the re-encoding does not exceed the remaining time available for the encoding process in the image processing unit, the image processing unit is instructed to re-encode the image stream, and the image processing When the pre-encoding buffer data is held, the unit forms a VOB unit so that the picture corresponding to the edit point becomes the first picture of the VOB unit, and the pre-encoding buffer data When the time required for the re-encoding does not exceed the remaining time available for the encoding process in the image processing unit. Is on the outs picture stream, the picture corresponding to the edit point, since the re-encoded to be an I-picture positioned at the head of the VOB unit, if there is a pre-encoding of the buffer data, and coding Even if there is no previous buffer data, if the time required for re-encoding does not exceed the remaining time available for encoding, the above edit point is set correctly and there is no buffer data before encoding. When the time required for re-encoding exceeds the remaining time available for encoding, the edit point can be set easily, and it seems that it is important for the photographer of the audio video stream obtained by shooting The part can be edited automatically or by a simple selection operation for guidance.

According to the twelfth aspect of the present invention, in the imaging apparatus according to the first or eleventh aspect , the time when the event that changes the image or the sound is recorded is recorded in the audio video stream as the edit point. The point can be set very easily.

According to the thirteenth aspect of the present invention, in the imaging device according to the twelfth aspect , since the event occurrence time is recorded in the audio-video stream as a playlist indicating the reproduction condition, the editing point can be set very easily. it can.

According to the invention of claim 14, in the imaging apparatus according to claim 12, wherein said edit point, information indicating if by any cause of images, or voice, embeds the audio video stream, At the time of editing, it is possible to thin out the editing points depending on what factors cause the editing points.

According to the fifteenth aspect of the present invention, in the imaging device according to the first or eleventh aspect , the picture corresponding to the event occurrence time is embedded in the stream as an out-of-sequence picture used for thumbnail display at the time of editing. A picture suitable as a point can be confirmed at a glance by a thumbnail display.

According to the sixteenth aspect of the present invention, there is provided an imaging method for acquiring image information and audio information by photographing a subject and recording an audio video stream including the image information and audio information. An imaging step for outputting a signal, an image processing step for performing signal processing on the image signal obtained by imaging the subject, and extracting image information including an image feature amount indicating a feature of an image change, and acquiring sound A voice acquisition step for outputting a voice signal, and a voice processing step for performing signal processing on the voice signal obtained by the voice acquisition to extract voice information including a voice feature amount indicating a feature of voice change; When the image feature amount or the sound feature amount is larger than a predetermined threshold, it is determined that the shooting timing at which the image or sound has changed is appropriate as the editing point. A feature amount determination step to be determined, an information generation step for generating edit point information indicating a photographing timing determined to be appropriate as the edit point, and an audio video stream including the image information, audio information, and edit point information The information generating step determines whether or not buffer data that is image information before encoding is held in the image processing unit that executes the image processing step. If the buffer data before encoding is held, the edit point is set to the picture corresponding to the image, audio, or shooting timing when the shooting state has changed, and the buffer data before encoding is set. Is not held, the edit point is the stream in the stream obtained by encoding the image signal by the image processing unit. Set to the first picture of the VOB unit, which is the unit of random access, closest to the shooting timing at which the image or sound has changed, and the image processing step stores the buffer data before encoding in the image processing unit. If so, the VOB unit is formed so that the picture corresponding to the edit point becomes the first picture of the VOB unit. Therefore, when there is buffer data before encoding, the edit point is accurately set and encoded. When there is no previous buffer data, the edit point can be set easily, and the part of the audio / video stream obtained by shooting is considered to be important for the photographer automatically or by simple selection operation for guidance. It becomes possible to edit.

According to the invention of claim 17 of the present application, a semiconductor device that acquires image information and audio information by photographing a subject and records an audio-video stream including the image information and audio information is obtained by imaging the subject. An image processing unit that performs signal processing on the received image signal to extract image information including an image feature amount indicating a change feature of the image, an audio acquisition unit that acquires audio and outputs an audio signal, and the audio A voice processing unit that performs signal processing on the voice signal obtained by acquiring the voice information including the voice feature quantity indicating the feature of the voice change, and the image feature quantity or the voice feature quantity is greater than a predetermined threshold value. when is large, the characteristic amount determination unit for determining a photographing timing of the image or voice has changed is valid for editing point, shot is determined to be appropriate as the edit point in Thailand And an information generation unit that generates an edit point information indicating the ring, and stores the image information, audio information, and the audio video stream including an edit point information on a recording medium, further, the information generating unit, the image processing Whether or not buffer data that is image information before encoding is held, and if the buffer data before encoding is held, the edit point is set to the image, audio, or When the picture corresponding to the shooting timing at which the shooting state has changed is set and the buffer data before encoding is not held, the editing point is obtained by encoding the image signal by the image processing unit. Set the first picture of the VOB unit, which is the unit of random access, closest to the shooting timing at which the image or sound changes in the stream. Image processing unit, if the coding buffer before the data is held in the image processing section, a picture corresponding to the edit point, because it forms a VOB unit to be a head picture VOB unit, If there is buffer data before encoding, or if there is no buffer data before encoding and the time required for re-encoding does not exceed the remaining time that can be used for encoding, the above edit points are used. If there is no buffer data before encoding and the time required for re-encoding exceeds the remaining time that can be used for encoding, the edit point can be set easily and obtained by shooting. The recorded audio-video stream can be edited as a stream that can be edited automatically or by a simple selection operation for guidance. A semiconductor device that can be obtained can be obtained.

Embodiments of the present invention will be described below.
(Embodiment 1)
1 and 2 are diagrams for explaining an imaging apparatus according to Embodiment 1 of the present invention. FIG. 1 shows the overall configuration of the imaging apparatus of Embodiment 1, and FIG. Fig. 2 shows an audio video stream obtained by the device.
The imaging apparatus 101 according to the first embodiment obtains an image signal Sim and an audio signal Sau by photographing a subject, and performs signal processing based on information indicating a shooting state on the obtained image signal Sim and audio signal Sau. Thus, an MPEG-2 compliant stream (hereinafter also referred to as audio video data) D that can be edited automatically or by a simple selection operation with respect to guidance is generated.

  That is, the imaging apparatus 101 includes an imaging unit 11 that captures a subject and outputs an image signal Sim, an image signal Sim obtained by capturing the subject, filter processing, compression encoding processing, feature amount extraction processing, and the like. The image processing unit 11a extracts the image information including the image feature amount indicating the feature of the image change. Here, the change in the image is a change in the image of the subject caused by the occurrence of the event, and the image feature amount is the magnitude of the change in the image, the length of the period in which the image is not changed substantially or substantially, etc. It is.

  The imaging device 101 acquires sound and outputs it as a sound signal Sau, and performs signal processing such as filter processing, compression coding processing, and feature amount extraction processing on the sound signal Sau, And a voice processing unit 12a that extracts voice information including a voice feature amount indicating a feature of voice change. Here, the change in the sound is a change in the sound from the subject caused by the occurrence of the event, and the sound feature amount depends on the magnitude of the change in the sound or the length of the period in which the sound does not change substantially or substantially. is there.

  The imaging device 101 performs unique identification information acquisition unit 10 that acquires unique identification information Did for identifying a photographer's shooting state, and performs signal processing such as filtering and feature amount extraction processing on the acquired unique identification information Did. And a unique identification information processing unit 10a that extracts information including a unique feature amount indicating a feature of a change in shooting state. Here, the change in the shooting state is a photographer's physiological change caused by the occurrence of an event or an operation of the imaging device by the photographer, and the unique feature amount is a magnitude of the photographer's physiological change or a zoom adjustment by the photographer. , And the size of focus adjustment.

  The imaging apparatus 101 is configured such that the shooting timing at which the shooting situation has changed is an edit point based on the feature amount obtained by the feature amount extraction processing in the image processing unit 11a, the sound processing unit 12a, and the unique identification information processing unit 10a. And an edit point information generation unit 22a that generates edit point information indicating the shooting timing determined to be appropriate as the edit point. Here, the shooting timing at which the shooting situation has changed includes the timing at which the subject image has changed during shooting, the timing at which the sound from the subject has changed during shooting, and the timing at which the shooting state has changed. The change in the shooting state includes a change in the photographer's physiology that occurs during shooting, and a change in zoom, focus, and the like due to the operation of the photographer.

  The image pickup apparatus 101 uses an image stream obtained by compressing and encoding the image signal Sim in the image processing unit 11a based on information from the image processing unit 11a, the sound processing unit 12a, and the unique identification information processing unit 10a. A system processing unit 13 for creating an audio video stream including the audio stream obtained by the compression encoding process of the audio signal in the audio processing unit 12a and the editing point information generated by the editing point information generating unit 22a; A series of recordings based on a recording medium 30a for storing the audio-video stream, a recording medium interface unit 30 connected between the recording medium 30a and the data bus Dbus, and a user operation signal generated by a user operation And a control unit 20a for controlling the above-described units so that the reproduction process is performed.

Hereinafter, signal processing performed in each of the above-described units will be described in detail.
The filter process for the image signal Sim performed by the image processing unit 11a is a process of extracting only a signal in a specific frequency band. The compression encoding process for the image signal Sim performed in the image processing unit 11a is an intra-screen and inter-screen predictive encoding process corresponding to MPEG-2. The predictive encoding process is not limited to that corresponding to MPEG-2, but may be one corresponding to MPEG-4 or MPEG-4 AVC. Further, here, the feature amount extraction processing for the image signal Sim is performed by calculating the magnitude of the change in the sudden change portion where the captured image has changed suddenly, the duration of the state in which the image does not change at all or substantially, and the like. This is a process of extracting as a feature amount based on a motion vector indicating the motion of an image used in the inter-screen predictive encoding process. The sudden change portion of the image is, for example, a shooting portion when the photographer is pointed at the camera toward a specific subject, and the non-changeable portion of the image is, for example, a direction in which the photographer's viewpoint is in a specific direction This is the shooting part when it does not move.

  The filtering process on the audio signal Sau performed by the audio acquisition unit 12 is a process of extracting only a signal in a specific frequency band. The compression encoding process for the audio signal Sau performed by the audio acquisition unit 12 is a process corresponding to an encoding process for an image signal such as MPEG-2 or MPEG-4, which generates audio compression data by compressing the audio signal. is there. Further, here, the feature amount extraction processing for the audio signal Sau is based on the magnitude of the change in the audio signal, the magnitude of the change at the sudden change portion where the voice has changed greatly, or the state in which the voice does not change at all or substantially. This is a process for extracting the duration of the process as a feature amount. For example, the sudden change in the sound occurred when the person being photographed started talking, when music performance began at a concert, or when a pistol or whistle sound was used as a start signal at an athletic meet, etc. The recording part of the time. In addition, the non-changeable part of the voice is a silent recording part of an intermediate curtain such as a play.

  The filtering process for the unique identification information Did performed by the unique identification information processing unit 10a is a process of extracting only the specific frequency component of the output signal of the unique identification information acquiring unit 10 that is the unique identification information Did. In the feature quantity extraction processing for the unique identification information Did performed by the unique identification information processing unit 10a, the magnitude of the change at the sudden change portion where the value of the unique identification information Did has changed suddenly or greatly, and the value of the unique identification information Did are completely This is a process of extracting the duration of the state that has ceased to change as a characteristic feature. The sudden change portion of the unique identification information corresponds to, for example, a characteristic change of a physiological phenomenon that occurs when a photographer's tension due to concentration of consciousness starts. Here, the characteristic change in the physiological phenomenon is, for example, a large change in the physiological phenomenon of the photographer that occurs during the shooting. The physiological phenomena to be detected are sweating action, blinking, pupil change, and pulse, and the unique identification information processing unit 10a detects changes in various physiological phenomena such as sweating action and blinking. It has a sensor according to. For example, the sweating action can be monitored by a sensor that measures the thermal conductivity of the photographer's hand. The photographer's physiological phenomenon as the unique identification information is not limited to the above.

In addition, the process for determining the edit point is for the feature amount determination unit 21 to determine whether or not the shooting timing at which the shooting situation has changed is appropriate as the edit point. Specifically, the following six determination processes are performed. It is.
In the first determination process, the size of the motion vector of all macroblocks in the screen, or the size of the motion vector of a specific macroblock in the screen, which is a feature amount from the image processing unit 11a, is determined. It is determined whether or not the threshold value is exceeded, and the shooting timing at which the magnitude of the motion vector is determined to exceed the threshold value is determined to be appropriate as the editing point.

  If the size of the motion vector of all macroblocks in the screen exceeds a predetermined threshold, the amount of shaking of the imaging device's housing exceeds a certain threshold, or the screen brightness level suddenly It may have changed.

  In the second determination process, it is determined whether or not a state in which the magnitude of the change in the motion vector or the change in the focus distance, which is the feature amount from the image processing unit 11a, is equal to or less than a certain threshold value continues for a certain period of time. Then, the shooting timing determined to have continued for a certain time or more is determined to be appropriate as the editing point.

  If the change in the magnitude of the motion vector or the change in the focus distance remains below a certain threshold, it is considered that the photographer's viewpoint has not changed.

  In the third determination process, it is determined whether or not the magnitude of the change in the dynamic range of the sound, which is the feature amount from the sound processing unit 12a, exceeds a predetermined threshold value. The timing is determined to be appropriate as an editing point. Note that the change in sound includes not only a change in sound from the subject but also a sound emitted by the photographer, for example, a coughing sound.

  The fourth determination process detects whether or not the unchanged state of the sound dynamic range has continued for a certain period of time, and the shooting timing determined that the unchanged state has continued for a certain period of time is appropriate as an editing point. It is determined. In this case, the start position of the unchanged part can be set as the editing point.

  In the fifth determination process, whether the magnitude of the change in the heart rate of the photographer or the change in the thermal conductivity of the photographer's hand, which is a feature amount from the unique identification information processing unit 10a, exceeds a certain threshold value. Whether or not the image capturing timing is determined to have exceeded the threshold is determined to be appropriate as the editing point.

  The sixth determination process detects whether or not the change in the number of blinks or the size of the pupil, which is a feature amount from the unique identification information processing unit 10a, exceeds a certain threshold value, and determines that the threshold value has been exceeded. The obtained shooting timing is determined to be appropriate as an editing point.

  For operations that are performed unconsciously such as focus and zoom, the shooting timing at which these operations are performed is appropriate as an edit point based on the output level of the dedicated sensor, not the feature amount from the image processing unit 11a. It may be determined whether or not. In this case, specifically, the unique identification information acquisition unit 10 acquires the output of the dedicated sensor as unique identification information indicating a shooting state, and the unique identification information processing unit 10a performs shooting based on the unique identification information. A unique feature amount indicating the magnitude of adjustment such as focus and zoom by the photographer's operation, which is the magnitude of the state change, is acquired. Then, the feature amount determination unit 21 determines the unique feature amount, and determines whether or not the shooting timing at which the shooting state such as focus or zoom has changed is appropriate as the editing point. In addition, a sensor that measures a photographer's brain wave, for example, an α wave may be provided, and based on the output level of the sensor, it may be determined whether or not the imaging timing at which the α wave has changed is appropriate as an editing point. In this case, specifically, the unique identification information acquisition unit 10 acquires the output level of the α wave measurement sensor as unique identification information indicating a shooting state, and the unique identification information processing unit 10a is based on the unique identification information. Thus, the characteristic feature amount indicating the magnitude of the change of the α wave of the photographer representing the photographing state is acquired. Then, the feature amount determination unit 21 determines the unique feature amount, and determines whether or not the shooting timing at which the photographer's α wave representing the shooting state has changed is appropriate as the editing point. In addition, the image and sound feature quantities can be detected by using a dedicated sensor instead of signal processing and extracting image signals and sound signals obtained by photographing.

Further, in the first embodiment, the editing point information generation unit 22a includes information indicating the photographing timing determined to be appropriate as the editing point, and the photographing timing includes, for example, a sound change, a video change, or a photographing state. Information indicating what kind of feature quantity is determined based on the feature quantity such as a change in the amount of the change, and outputs the information to the system processing unit 13. The editing point information generation unit 22a sets the first I picture of the VOB unit closest to the shooting timing determined as the editing point before the shooting timing as a picture to be used as an access point at the time of editing. Information indicating that the edit point is set in the I picture is output to the system processing unit 13. Further, the system processing unit 13 updates a playlist, which is management information included in the audio video stream D, based on information from the editing point information generation unit 22a.
In other words, the playlist of the audio video stream created by the system processing unit 13 has the edit point information indicating the shooting timing determined to be appropriate as the edit point and the shooting timing determined to be appropriate as the edit point. It includes information indicating whether the determination is based on the feature amount, and information indicating which picture is set as a picture used as an access point at the time of editing.

  However, the picture used as the access point is not limited to the I picture before the shooting timing, which is the closest to the shooting timing at which the shooting situation such as the image or sound, which has been determined as the editing point, has changed. The amount of feature attributed to this event from the timing of the event that changes the image or sound of the subject or the shooting state of the photographer, depending on factors such as the change in the image and the change in the sound. A picture to be an edit point may be determined in consideration of a delay time until detection of. For example, the I picture closest to the shooting timing that is backed by the delay time from the timing when the shooting situation changes may be set as the picture that is used as the editing point. In this case, the delay time may be a time determined according to the focus information or the like, or may be a fixed time determined in advance uniformly for all factors or separately for each factor.

  In the first embodiment, since the audio / video stream corresponds to MPEG-2, the time at which the event for changing the image, sound, or shooting state occurs is set as the image, sound, or shooting state. As the time when the feature amount indicating the feature of the change is detected, it is included in the playlist and written in the management information recording area of the stream. However, the stream may correspond to MPEG-4 AVC, and in this case Only the feature amount detection time may be recorded in the additional information recording area (SEI) of the stream.

  Further, in the first embodiment, the control unit 20a outputs a command signal to the feature amount determination unit 21 based on a scenario selected by the user before shooting, and becomes a factor for setting an edit point. In addition to determining sound, video, photographer's physiological phenomenon, and the like, determination intensity, that is, a threshold value of a feature amount, when determining a shooting timing when an editing point setting factor has changed as an editing point is determined.

  In addition, when the user selects a shooting location such as an athletic meet, a concert, a wedding, or a trip according to the guidance from the imaging apparatus, the control unit 20a changes a shooting situation change pattern, for example, a change in volume. In accordance with the pattern, change pattern of brightness, change pattern of photographer's physiological phenomenon, etc., the determination strength for various factors for setting the edit point is set to one of a plurality of predetermined default values that have been facilitated in advance. To do. However, the default values prepared as determination strengths for various factors may be independently adjustable according to user preferences and the like.

  Although not shown, the photographing apparatus 101 includes a playback unit that decodes and plays back an audio / video stream recorded on a recording medium based on a control signal from the control unit 20a.

Next, the structure of AV data recorded on the recording medium will be briefly described.
FIG. 2 is a diagram for explaining the structure of AV data recorded on a recording medium.
Here, the recording medium is a disc-shaped recording medium such as a DVD (Digital Versatile Disk) disc. However, the recording medium is not limited to a disk-shaped recording medium such as a DVD, and may be, for example, an HDD (Hard Disk Drive), a memory card, or a magnetic tape. In addition, an audio video stream D including a stream Ds obtained by encoding an image signal Sim and an audio signal Sau corresponding to one content and management information Dm corresponding to these contents is written on the recording medium. It is. The management information Dm is written in an inner area near the center of the disc-shaped recording medium, and the stream Ds is written in an area outside the inner area. The stream Ds is divided by the VOB unit VOBU.

  The management information Dm includes a playlist Dmp, and the playlist Dmp includes a plurality of auxiliary information playitems [0], [1], [2],..., [N],. It is included.

  For example, the stream Ds of the encoded data D shown in FIG. 2 includes a VOB unit VOBU (m−k), a VOB unit VOBU (m), and a VOB unit VOBU (m + q), and a specific VOB unit VOBU (m ) Corresponding to the playlist includes time information Dtm, AV information Dav, operation information Dop, physiological information Dph, and an edited flag Def. Here, the time information Dtm includes information Dst indicating the start time of the VOB unit VOBU (m) and information Det indicating the end time of the VOB unit VOBU (m). The AV information Dav includes information Dvi indicating a feature amount related to an image and information Dau indicating a feature amount related to a sound. The operation information Dop includes information Dhm indicating the degree of camera shake, information Dfo indicating the operation amount during the focus operation, and information Dzm indicating the operation amount during the zoom operation. The physiological information Dph includes sweat information Dsu indicating the amount of sweat of the photographer, α-wave information Dαw indicating the α-wave intensity of the photographer, blink information Dbk indicating the frequency of the operator's blink, and the degree of pupil change of the operator. It includes pupil information Dpu to be displayed and pulse information Dps to indicate the pulse rate of the operator. As described above, the time at which the event for changing the image, sound, or shooting state occurs is substantially the same as the time at which the feature amount indicating the change feature of the image, sound, or shooting state is detected. It is included in the list and written in the management information recording area of the stream.

Next, the operation will be described.
[Setting operation before shooting]
First, a manual setting operation before photographing will be described.
The photographer sets a determination strength used for determining whether or not the shooting timing at which the shooting situation has changed is appropriate as an editing point in accordance with an event such as an athletic meet or a wedding.

  For this judgment strength, select the judgment strength for each factor for editing point setting by selecting the one corresponding to athletic meet or wedding from a plurality of scenarios preset in the imaging device Although it can be set to a value according to the scenario, the operation manually set by the operator will be described here.

FIG. 3 shows a screen for performing edit point insertion setting of the imaging apparatus 100.
On this setting screen 100a, an AV information setting button 110, an operation information setting button 120, and a physiological information setting button 130 are displayed. Further, a detailed setting screen 130a for performing more detailed setting of physiological information is displayed in the lower right portion of the setting screen 100a. On the detailed setting screen 130a, a sweat information setting button 131, pupil information, and the like are displayed. A setting button 132 and a pulse information setting button 133 are displayed. Although not shown in FIG. 3, a detailed setting screen for performing more detailed settings of AV information and a detailed setting screen for performing more detailed settings of operation information can be displayed.

  Each button can set the determination strength for each element to an arbitrary level between the minimum level indicated by “−” display and the maximum level indicated by “+” display. The “0” display indicates an intermediate level between these.

  Here, for example, the fact that the level of the determination intensity related to sweat information is high means that even when the change in the amount of sweat is relatively small, the shooting timing at which the change in the amount of sweat occurs is appropriate as the editing point. That's what it means. On the other hand, when the level of the determination intensity related to sweat information is small, even if the change in the amount of sweat is relatively large, it is determined that the shooting timing at which the change in the amount of sweat occurs is not appropriate as an edit point. .

  For example, when shooting at a competition such as an athletic meet, it is considered that a large change in sound such as a cue occurs at the start of performance or competition, so the determination strength for the audio element of AV information is stronger than the average level. It is considered that it is better to set the strength of the determination for the pulse element of the physiological information because the photographer may be injured during the competition.

  When shooting a landscape during a trip or the like, the photographer may set the determination strength for the image element of the AV information to be higher than the average level, and may shoot a distant landscape. It is considered that the determination strength for the zoom operation amount may be increased.

  In weddings, the photographer sets both the image feature amount determination strength and the sound feature amount determination strength of AV information to be stronger than the average level, and the feature amount of each element of physiological information is relatively high. A case where the determination intensity is set high can be considered.

  Such setting of the determination strength is performed by the control unit 20a according to a user operation, that is, a manual operation of the photographer, and the control unit 20a indicates the determination strength for each element set according to the user operation. A control signal is supplied to the feature amount determination unit 21. Thereafter, when the photographer performs photographing, the feature amount determination unit 21 determines the feature amounts based on the corresponding determination strength (threshold level) set by the control unit 20a, and the image Then, it is determined whether or not the shooting timing at which the sound or the shooting state has changed is appropriate as the editing point.

[Operation during shooting]
Next, the operation of the imaging apparatus at the time of shooting will be specifically described.
FIG. 4 is a diagram illustrating an operation flow of the imaging apparatus according to the first embodiment.
When shooting is started (step S1), the imaging apparatus 101 acquires image information, audio information, and information regarding the shooting state (step S2).

  Specifically, in step S2, the imaging unit 11 outputs the image signal Dim by imaging the subject, the audio acquisition unit 12 acquires the audio and outputs the audio signal Dau, and the unique identification information. The acquisition unit 10 detects an operation of the imaging device by the photographer and a physiological change of the photographer, and performs processing of outputting unique identification information Did regarding the operation amount and the physiological change in parallel.

  Then, the unique identification information processing unit 10a, based on the unique identification information Did from the unique identification information acquisition unit 10 and the control signal from the control unit 20a, the operation amount in the focus or zoom operation, and the physiological change of the photographer. Features such as the amount of sweating, the frequency of blinking, and the amount of fluctuation of the pulse rate are detected (step S2a). Further, the image processing unit 11a generates an image stream by performing an MPEG-2 compatible predictive coding process on the image signal based on the image signal Dim from the photographing unit 11 and the control signal from the control unit 20a. At the same time, based on the motion vector used in the predictive encoding process, image information including the image feature amount such as the magnitude of the image change at the portion where the image suddenly changed is acquired (step S2b). The audio processing unit 12a generates an audio stream by performing an encoding process on the audio signal based on the audio signal Dim from the audio acquisition unit 12 and the control signal from the control unit 20a. On the basis of the voice information, the voice information including the voice feature amount such as the magnitude of the voice change at the portion where the voice is suddenly changed is acquired (step S2c).

  Next, the feature amount determination unit 21 is based on a feature amount related to a user operation or a physiological change of a photographer, a feature amount related to an image, a feature amount related to sound, and a determination intensity set for each feature amount. Then, it is determined whether the shooting timing at which the feature amount is detected is appropriate as an editing point (step S3).

  Subsequently, the edit point information generation unit 22a generates edit point information indicating the shooting timing determined to be appropriate as the edit point, and the picture used as the edit point is the most before this shooting timing. Edit point picture information indicating that the first I picture of the near VOB unit is set is generated (step S4).

  Thereafter, the system processing unit 13 creates an audio video stream including the image stream, the audio stream, the editing point information, and the editing point picture information based on the control signal from the control unit 20a and outputs the audio video stream to the recording medium interface 30. To do. Then, the recording medium interface records the input audio video stream on the recording medium (step S5).

Hereinafter, the process of step S3 for determining the edit point will be described.
Specifically, the feature amount determination unit 21 uses the unique feature amount detected by the unique identification information processing unit 10a and the image detected by the image processing unit 11a based on a control signal indicating the determination strength from the control unit 20a. For each of the feature quantity and the voice feature quantity detected by the voice processing unit 12a, it is determined whether or not the shooting timing at which each feature quantity is detected is appropriate as an edit point.

  For example, the feature amount related to camera shake detected by the unique identification information processing unit 10a is the size of the camera shake of the photographer. If the detected camera shake size is equal to or greater than the judgment intensity set in advance before shooting, that is, the camera shake threshold value, the shooting timing at which the feature amount related to the camera shake is detected is the editing point. If the magnitude of camera shake is smaller than the above-described determination strength, it is determined that the photographing timing is not appropriate as an editing point (step S3a). Further, the feature quantity related to focus and the feature quantity related to zoom detected by the unique identification information processing unit 10a are a focus fluctuation amount changed by the focus operation and a zoom fluctuation amount changed by the zoom operation, respectively. And, for these feature amounts as well as the feature amounts related to camera shake, the shooting timing at which the feature amounts are detected depends on whether the magnitude is equal to or greater than the determination intensity set before shooting. It is determined whether or not the edit point is valid (step S3a).

  Further, the feature amount related to sweating detected by the unique identification information processing unit 10a is the sweating amount of the photographer. If the detected perspiration amount is equal to or higher than the determination intensity set in advance before photographing, that is, the perspiration amount threshold, it is determined that the photographing timing at which the feature amount related to perspiration is detected is appropriate as an editing point. If the perspiration amount is smaller than the detected intensity, it is determined that the photographing timing is not appropriate as an editing point. Further, the feature quantity related to α wave, the feature quantity related to blink, the feature quantity related to pupil, and the feature quantity related to pulse detected by the unique identification information processing unit 10a are the magnitude of change of α wave, the frequency of blink, The magnitude of the change and the magnitude of the change in the pulse rate. As for the feature quantities related to the physiological changes of these photographers, each feature quantity depends on whether or not the value is equal to or higher than the determination strength set in advance before shooting, as is the feature quantity related to sweating. It is determined whether or not the detected shooting timing is valid as an editing point (step S3a).

  The feature amount related to the image detected by the image processing unit 11a is the magnitude of the change at the portion where the image suddenly changes, or the time when the portion where the image does not change substantially or substantially continues. The magnitude of the change in the detected image sudden change portion or the duration of the no-image-change state is determined in advance by the judgment intensity set before shooting, that is, the threshold of the magnitude of change or the state duration. If it is equal to or greater than the threshold value, it is determined that the shooting timing at which these feature amounts are detected is appropriate as an editing point, and otherwise, it is determined that the shooting timing is not appropriate as an editing point (step S3b).

  The feature amount related to the sound detected by the sound processing unit 12a is the magnitude of the change at the portion where the sound has changed greatly, or the time during which the state where the sound does not change at all or substantially does not change. Then, the magnitude of the change in the detected sudden voice change portion or the duration of the voice no-change state is determined in advance by the judgment intensity set before shooting, that is, the threshold of the magnitude of change, or the state duration. If it is equal to or greater than the threshold value, it is determined that the shooting timing at which these feature amounts are detected is appropriate as an editing point, and otherwise, it is determined that the shooting timing is not appropriate as an editing point (step S3c).

  Thereafter, the editing point information generation unit 22a uses the feature amount determination unit 21 as the editing point based on the respective feature amounts supplied from the processing units 10a, 11a, and 12a. Each time it is determined to be valid, edit point information indicating the shooting timing is generated, and the picture used as the edit point is set to the first I picture of the VOB unit that is before the shooting timing and is closest thereto. Edit point picture information indicating that this has been done is generated (step S4).

  5A and 5B are diagrams for specifically explaining the edit point setting process. FIG. 5A shows a processing flow, and FIG. 5B shows the shooting timing and the breaks of VOB units in the image stream. Shows the relationship.

  In the first embodiment, the editing point information generation unit 22a calculates a delay time due to the feature amount, that is, a time period that varies depending on the feature amount from when the event occurs until the shooting situation changes due to the occurrence of the event (step S1). S11).

  Next, the edit point information generation unit 22a edits the cut of the nearest VOB unit (i) before the shooting timing Tep that is backed by the calculated delay time Δt from the shooting timing Tcp at which the feature amount is detected. A point is set (step S12a).

  After that, the edit point information generation unit 22a generates edit point information indicating the photographing timing Tep determined to be valid as the edit point, and, as shown in FIG. Edit point setting information indicating that the first I picture F1 of the VOB unit VOBU (f) closest to the shooting timing is set is generated. Then, the play item of the audio video stream is changed so as to indicate that the shooting timing Tep is set as the edit point (step S13).

  In FIG. 5 (b), the picture F1 is an I picture that does not refer to other pictures at the time of encoding or decoding, and the pictures F4, F7, and F10 are at the time of encoding or decoding. A P picture that refers to a forward I picture or P picture, and the pictures F2, F3, F5, F6, F8, and F9 are, when encoded or decoded, a forward I picture or P picture and a backward P picture. B picture that refers to

  Each VOB unit is composed of a plurality of pictures, and an I picture is located at the head of the VOB unit. Two B pictures are arranged between adjacent I and P pictures or between two adjacent P pictures. Has been. The VOB units VOBU (f−1) and VOBU (f + 1) are VOB units positioned before and after the VOB unit VOBU (f).

[Operation during playback]
At the time of reproduction, the audio video stream recorded on the recording medium is based on embedded edit point information, that is, the start time and end time of the playlist corresponding to the VOB unit in which the first picture is set as the edit point. Automatically edited and played.

  Note that the playback of the audio video stream recorded on the recording medium is not limited to the one that is automatically edited by picking up the editing point, but in the recorded audio video stream based on the editing conditions set by the user, It may be performed by editing only the portion satisfying the set editing conditions.

FIG. 6 is a diagram for explaining processing for automatically editing and reproducing a recorded audio-video stream based on setting conditions, for example.
In Embodiment 1, when a playback unit (not shown) of the imaging apparatus 101 starts playback of an audio video stream recorded on a recording medium, the playback unit (not shown) is based on each item of a playlist included in the audio video stream. It is determined whether the process has been completed (step S21). If the process has been completed, the reproduction is terminated.

  On the other hand, if the result of determination in step S <b> 21 is that processing has not ended, the playback unit determines whether or not the edit point satisfies editing conditions (step S <b> 22), and satisfies the setting condition regarding the feature amount. The current VOB unit VOBU is reproduced (step S23).

  Note that the information related to the editing points included in the audio video stream can be used not only for automatic editing but also for editing audio video streams by the user.

FIG. 7 is a diagram for explaining how to use such editing points, and shows a display screen for editing an audio video stream recorded on a recording medium.
Here, the display device 200 is a display unit of a television set or a personal computer, and the display screen 210 has a strip-shaped indicator 211 indicating the entire audio-video stream corresponding to one content recorded on the recording medium. , A band-shaped indicator 212 showing a specific VOB unit 211a in the audio / video stream in an enlarged manner; It is shown.

  Further, the display screen 210 includes a processing thumbnail display area 220 in which a picture of an edit point for which the user adjusts the editing conditions is displayed. On the display screen 210, adjacent to the processing thumbnail display area 220, operation areas 230 and 240 corresponding to each element for adjusting the determination strength of the feature amount, which is an editing condition to be satisfied by the editing point picture, are displayed. Has been.

  In this way, the user can adjust the editing condition to be satisfied by the picture set at each editing point, that is, the determination amount of the feature amount, on the display screen 210.

  Note that the display for editing support can display all the I pictures on the thumbnail screen, and the thumbnail screen of the picture to be edited can be made larger than the thumbnail screens of the other I pictures.

  In addition, the display for the above-mentioned editing support is such that the order in which the pictures to be edited are displayed as thumbnails may be the order corresponding to the type of the cause of the feature quantity, or the feature quantity normalized with respect to all the factors. It may be in the order of size.

  Further, the editing support may be displayed by sequentially displaying pictures set as editing points in a slide show format. In this case, primary editing is performed by selecting whether or not the necessary editing points are necessary. It is also possible to provide support for subsequent editing.

  The display for editing support may be one in which several seconds from the editing point are automatically connected, and the editing point is switched according to the favorite BGM pitch and tempo and displayed as a digest. In this case, the recorded audio-video stream may be re-edited so as to have such a digest version, or such display may be performed without any particular editing.

Furthermore, the control unit 20a or the editing point information generation unit 22a also manages a flag for recognizing whether or not editing has been completed, and has information on whether or not the recorded audio-video stream has been edited. It may be a thing.
The edited audio / video stream may be one in which only the playlist is changed without changing the actual data portion.

  As described above, in the imaging apparatus 101 according to the first embodiment, the image processing unit 11a that extracts the image feature amount representing the feature of the image change from the image signal Dim obtained by photographing the subject, and obtained by photographing the subject. An audio processing unit 12a that extracts a voice feature amount representing a feature of voice change from the obtained voice signal Dau, and a unique feature amount representing a feature of a shooting state change based on information Did indicating a photographer's physiological change Whether or not the shooting timing at which the image or sound changes is appropriate as an editing point by comparing the extracted feature quantity with a preset determination intensity. Therefore, it is possible to generate an audio video stream that can automatically edit a shooting portion that is considered important for the photographer.

  In the first embodiment, the edit point is set at the break of the VOB unit in the AV encoded data close to the shooting timing determined to be valid as the edit point. Even when the signal is encoded, the edit point can be set without processing the encoded image signal.

  In the first embodiment, the editing point is set to the shooting timing retroactive from the shooting timing at which the shooting situation has changed by the delay time from the event occurrence time until the shooting situation changes due to the event occurrence. The point can be set at the timing when the event actually occurs.

  In the first embodiment, the case where the operator manually sets the determination strength when determining that the shooting timing at which the shooting situation has changed is appropriate as the edit point has been described. The determination strength for each of the factors may be set by selecting one corresponding to an athletic meet or wedding from a plurality of scenarios set in advance in the imaging apparatus.

  As described above, the imaging device that determines the determination strength for each factor by selecting a scenario, for example, in the imaging device of Embodiment 1, the control unit is configured to each of a plurality of scenarios, the image feature amount, and the audio feature. The table information indicating the correspondence relationship between the amount and the combination of the threshold level for each of the unique feature amounts is held, and corresponds to each of the above factors based on the scenario specified by the user's manual operation and the table information. It is assumed that a threshold level of a feature amount is set, and further, the feature amount determination unit is configured based on the threshold level set by the control unit corresponding to the image feature amount, the audio feature amount, and the unique feature amount, respectively. This is realized by determining whether or not the shooting timing when the image, sound, and shooting state have changed is appropriate as an editing point. Rukoto is possible.

  In this case, table information indicating a correspondence relationship between each of the plurality of scenarios and a combination of threshold levels for each of the image feature amount, the sound feature amount, and the unique feature amount is downloaded from an information terminal on the network. It is also possible to use what has been acquired.

  The combination of threshold levels of each feature amount included in the table information may be two of an image feature amount, an audio feature amount, and a unique feature amount, and the table information includes each of a plurality of scenarios. And a threshold level corresponding to any one of the image feature value, the sound feature value, and the unique feature value.

(Embodiment 2)
FIG. 8 is a diagram for explaining an imaging apparatus according to Embodiment 2 of the present invention.
In the imaging apparatus 102 according to the second embodiment, instead of the editing point information generation unit 22a in the imaging apparatus 101 according to the first embodiment, when a picture that is an editing point is not an I picture, a picture that is an editing point and its vicinity The editing point information generation unit 22b that instructs the image processing unit 11a to perform re-encoding so as to change the picture type of this picture. The control unit 20b is different from the control unit 20a of the first embodiment only in that the image processing unit 11a is controlled at the time of re-encoding. The other configuration of the imaging apparatus 102 according to the second embodiment is the same as that of the imaging apparatus 101 according to the first embodiment.

Next, the operation will be described.
In the imaging apparatus 102 according to the second embodiment, the manual setting operation before photographing is performed in the same manner as in the first embodiment.

  When shooting is started, the feature amount determination unit 21 of the imaging apparatus 102 acquires image information, audio information, and information regarding the shooting state, and obtains the acquired information from the acquired information, as in the imaging apparatus 101 of the first embodiment. It is determined whether or not the shooting timing at which the shooting situation has changed is appropriate as an edit point based on the user operation and the feature amount of the photographer's physiological change, the image feature amount, and the audio feature amount. .

  In the second embodiment, the editing point information generation unit 22b is the feature amount determination unit 21, and the shooting state is changed based on the feature amounts supplied from the processing units 10a, 11a, and 12a. Each time it is determined that the timing is valid as an edit point, edit point information indicating the edit point is generated, and if the picture corresponding to the edit point is other than an I picture, a re-encoding command is issued to the image processing unit. 11a.

FIG. 9 shows a flow of edit point information generation processing and re-encoding processing.
In the second embodiment, the control unit 20b calculates a delay time corresponding to the type of feature quantity, that is, a time period that varies depending on the feature quantity from when the event occurs until the shooting situation changes due to the event occurrence (step S20). S11).

  Next, the editing point information generation unit 22b compulsorily creates a VOB unit that starts the picture corresponding to the shooting timing Tep that is backed by the delay time from the shooting timing Tcp at which the feature amount is detected. Command to 11a. Then, the image processing unit 11a performs re-encoding processing for forcibly re-creating the VOB unit VOBU (step S12b).

  Thereafter, the editing point information generation unit 22b generates editing point information indicating the photographing timing Tep determined to be valid as the editing point, and also displays the editing point as shown in FIGS. 10 (b) to 10 (d). Then, edit point picture information indicating that the first I picture of the forcibly created VOB unit VOBU is set is generated. Then, the play item of the audio video stream is changed so as to indicate that the shooting timing Tep is set as the edit point (step S13).

Hereinafter, a re-encoding process for forcibly re-creating a VOB unit VOBU will be described.
FIG. 10A shows one VOB unit VOBU (f) composed of a plurality of pictures F1, F2, F3, F4, F5, F6, F7, F8, F9, F10,.

  Here, the picture F1 is an I picture that does not refer to other pictures at the time of encoding and decoding, and the pictures F4, F7, and F10 are the front I picture or P at the time of encoding and decoding. P pictures that refer to pictures, and pictures F2, F3, F5, F6, F8, and F9 are B pictures that refer to forward I pictures or P pictures and backward P pictures at the time of encoding and decoding. Each picture shown in FIG. 10A has an original reference relationship defined by MPEG-2.

  In FIG. 10B, the picture to be edited is the fourth picture F4 of the VOB unit VOBU (f), the picture type of this picture F4 is changed, and the two previous B pictures F2 and F3 are changed. The case where the reference relationship is changed is shown.

  In this case, the picture F4 is changed from the P picture to the I picture, and the B pictures F2 and F3 are re-encoded to refer only to the front I picture F1. Also, a new VOB unit VOBU (fb1) starting from the picture F4 is created, and processing such as changing the index of pictures after the picture F4 is performed. The VOB unit VOBU (fa1) is a new VOB unit immediately before the VOB unit VOBU (fb1), in which the reference relationship between the B pictures F2 and F3 is changed.

  In FIG. 10C, the picture to be edited is the fifth picture F5 of the VOB unit VOBU (f), the reference relationship between this picture F5 and the subsequent B picture F6 is changed, and the picture type of the P picture F7 Shows the case of changing.

  In this case, the picture F7 is changed from the P picture to the I picture, and the pictures F5 and F6 are re-encoded to refer only to the rear I picture F7 whose picture type is changed. Also, a new VOB unit VOBU (fb2) starting from the picture F5 is created, and processing such as changing the index of pictures after the picture F8 is performed. The VOB unit VOBU (fa2) is a new VOB unit immediately before the VOB unit VOBU (fb2) with the P picture F4 as the last picture.

  In FIG. 10D, the picture that is the editing point is the sixth picture F6 of the VOB unit VOBU (f), and the reference relationship of this picture F6 and the reference relationship of the B pictures F5 and F7 before and after that are changed. Shows the case.

  In this case, the picture F7 is changed from the P picture to the I picture, the picture F5 is re-encoded to refer only to the front P picture F4, and the picture F6 is referred to only the rear P picture F7. Is done. Also, a new VOB unit VOBU (fb3) starting from the picture F6 is created, and processing such as changing the index of pictures after the picture F8 is performed. The VOB unit VOBU (fa3) is a new VOB unit immediately before the VOB unit VOBU (fb3) with the P picture F5 as the final picture.

  In the second embodiment having such a configuration, an image processing unit 11a that extracts an image feature amount representing a feature of an image change from an image signal obtained by photographing a subject, and an audio signal obtained by photographing the subject. From the voice processing unit 12a that extracts a voice feature amount that represents the feature of a change in voice, and unique identification information that extracts a unique feature amount that represents a feature of a change in shooting state based on information indicating a physiological change of the photographer The processing unit 10a is provided, and the extracted feature value is compared with a predetermined determination strength, and it is determined whether or not the shooting timing at which the image or the sound has changed is appropriate as the editing point. As in the first embodiment, it is possible to generate an audio video stream that can automatically edit a shooting portion that is considered important for the photographer.

  In the second embodiment, the edit point is a reference relationship between the picture type and surrounding pictures so that the picture corresponding to the shooting timing determined to be valid as the edit point becomes a break in the VOB unit. Since these pictures are re-encoded so as to be changed, edit points can be set accurately even when the image signal obtained by shooting is encoded.

  Further, in the second embodiment, the editing point is set to the shooting timing retroactive from the shooting timing when the shooting situation such as the image and the sound changes by the delay time from the occurrence of the event until the shooting state changes due to the event occurrence. Since it is set, the edit point can be set at the timing when the event actually occurs.

In the second embodiment, when the picture set at the edit point is an inter-picture prediction picture, the picture is transcoded and recorded so as to be an in-plane prediction picture. The in-plane prediction picture obtained by the above may be recorded as a sub-picture separately from the inter-picture prediction picture set at the editing point.
In this case, at the time of editing, the inter-screen prediction picture set at the editing point is replaced with the intra-screen prediction picture recorded as the sub picture, and the replaced intra-screen prediction picture is replaced with the head of the VOB unit that is the editing point. It can be used for playback as a picture.

(Embodiment 3)
FIG. 11 is a diagram for explaining an imaging apparatus according to Embodiment 3 of the present invention.
The imaging apparatus 103 according to the third embodiment replaces the editing point information generation unit 22a in the imaging apparatus 101 according to the first embodiment and inserts editing points depending on whether there is buffer data before encoding. An edit point information generation unit 22c that switches between a process of generating a new VOB unit VOBU corresponding to the edit point of the first picture and a process of setting the edit point at the break of the VOB unit VOBU closest to the edit point It is. Further, the control unit 20c is different from the control unit 20a of the first embodiment only in that the image processing unit 11a is controlled in accordance with switching of the editing point setting process. The other configuration of the imaging apparatus 103 according to the third embodiment is the same as that of the imaging apparatus 101 according to the first embodiment.

Next, the operation will be described.
In the imaging apparatus according to the third embodiment, the manual setting operation before shooting is performed in the same manner as in the first embodiment.

  When shooting starts, the imaging device 103 acquires image information, audio information, and information indicating the shooting state, as with the imaging device 101 of the first embodiment, and a user operation obtained from the acquired information. And whether or not the shooting timing at which the shooting situation such as the image or sound changes is appropriate as the editing point based on the feature amount of the photographer's physiological change, the feature amount of the image, and the feature amount of the sound .

  In the third embodiment, the editing point information generation unit 22c is a feature amount determination unit 21 that captures a change in the shooting state based on the feature amounts supplied from the processing units 10a, 11a, and 12a. Each time it is determined that the timing is appropriate as an edit point, edit point information indicating the shooting timing at which the edit point is set is generated, and edit point setting processing is performed.

FIG. 12 shows a flow of edit point setting processing.
In the third embodiment, the control unit 20c calculates a delay time corresponding to the type of feature quantity, that is, a time period that varies depending on the feature quantity from the time when the event occurs until the shooting situation changes due to the event occurrence (step S1). S11).

  Next, the edit point information generation unit 22c determines whether or not there is buffer data that is an image signal before encoding at the time of calculation of the delay time (step S11a), and if there is buffer data before encoding. If it is determined, the VOB unit VOBU being created is closed and a new VOB unit VOBU is created (step S12c). On the other hand, if it is determined in step S11a that there is no buffer data before encoding, the shooting timing Tep is set to the shooting timing Tep before the shooting timing Tep, which is retroactive by the calculated delay time from the timing Tcp at which the shooting situation has changed. Processing for setting the cut point of the nearest VOB unit VOBU as an edit point is performed (step S12a). The process of step S12a is the same as the process of step S12a of the first embodiment.

  Thereafter, the editing point information generation unit 22c generates editing point information indicating the photographing timing Tep determined to be valid as the editing point, and whether or not the editing point has been set by any one of the steps S12a and S12c. As shown, the play item of the system stream is changed (step S13).

Hereinafter, FIG. 13 is a diagram for explaining the process of creating a new VOB unit VOBU in which the first picture is set as an edit point in step S12c.
FIG. 13A shows one VOB unit VOBU (j) composed of a plurality of pictures J1, J2, J3, J4, J5, J6, J7, J8, J9, J10,.

  Here, the picture J1 is an I picture that does not refer to other pictures at the time of encoding and decoding, and the pictures J4, J7, and J10 are the preceding I picture or P at the time of encoding and decoding. P pictures that refer to pictures, and pictures J2, J3, J5, J6, J8, and J9 are B pictures that refer to a forward I picture or P picture and a backward P picture at the time of encoding and decoding. Each picture of the VOB unit VOBU (j) has an original reference relationship defined by MPEG-2.

FIG. 13B shows a case where a new VOB unit VOBU is generated with the fourth picture J4 of the VOB unit VOBU (j) as an editing point.
In this case, the picture J4 to be encoded as the P picture in the VOB unit VOBU (j) is encoded as the first I picture Ja1 of the new VOB unit VOBU (ja). In the VOB unit VOBU (j), the second picture J2 and the third picture J3, which are B pictures, are encoded as B pictures that refer to only the front I picture J1. Note that the pictures Ja4 and Ja7 in the VOB unit VOBU (ja) are P pictures that refer to the preceding I picture or P picture in encoding and decoding, and the pictures Ja2 and Ja3 in the VOB unit VOBU (ja). , Ja5, and Ja6 are B pictures that refer to the front I picture or P picture and the rear P picture at the time of encoding and decoding.

FIG. 13C shows a case where a new VOB unit VOBU is generated with the fifth picture J5 of the VOB unit VOBU (j) as an editing point.
In this case, the picture J5 to be encoded as the B picture in the VOB unit VOBU (j) is encoded as the first I picture of the new VOB unit VOBU (jb). Note that the picture J8 in the VOB unit VOBU (jb) is a P picture that refers to the front I picture at the time of encoding and decoding, and the pictures J6, J7, J9, and J10 in the VOB unit VOBU (jb) are The B picture refers to a front I picture or P picture and a rear P picture at the time of encoding and decoding.

FIG. 13D shows a case where a new VOB unit VOBU is generated with the sixth picture J6 of the VOB unit VOBU (j) as an editing point.
In this case, the picture J6 to be encoded as the B picture in the VOB unit VOBU (j) is encoded as the first I picture of the new VOB unit VOBU (jc). The fifth picture J5 that is a B picture in the VOB unit VOBU (j) is encoded as a B picture that refers to only the front P picture J4. Note that the picture J9 in the VOB unit VOBU (jc) is a P picture that refers to the front I picture at the time of encoding and decoding, and the pictures J7, J8, and J10 in the VOB unit VOBU (jc) are code This is a B picture that refers to a forward I picture or P picture and a backward P picture at the time of encoding and decoding.

  In the third embodiment having such a configuration, an image processing unit 11a that extracts an image feature amount representing a feature of an image change from an image signal obtained by photographing a subject, and an audio signal obtained by photographing the subject. From the voice processing unit 12a that extracts a voice feature amount that represents the feature of a change in voice, and unique identification information that extracts a unique feature amount that represents a feature of a change in shooting state based on information indicating a physiological change of the photographer A processing unit 10a, and the extracted feature value is compared with a predetermined determination strength to determine whether the shooting timing at which the feature value is generated is valid as an edit point. As in the first mode, it is possible to generate an audio / video stream that can automatically edit a shooting portion that seems to be important for the photographer.

  In the third embodiment, when an edit point is inserted, processing for generating a new VOB unit VOBU having the edit point as the first picture and occurrence of an event depending on whether or not there is buffer data before encoding. Since the process of changing the cut point of the VOB unit VOBU closest to the timing to the edit point is switched, if the image signal obtained by shooting is not encoded, the VOB unit VOBU is accurately generated by using the edit point as a reference. Edit points can be set at various positions, and when the image signal obtained by shooting is encoded, the edit points can be easily set without processing the audio-video stream. it can.

  In the third embodiment, the editing point is set to the shooting timing retroactive from the timing at which the shooting situation has changed by the delay time from the occurrence of the event to its detection, that is, until the shooting situation changes due to the event occurrence. The editing point can be set to the shooting timing when the event actually occurs.

  In the third embodiment, the shooting timing for setting the edit point is determined according to the delay time from the occurrence of the event to the actual change of the image, sound, or shooting state. May occur after the image, sound, or shooting state changes. In such a case, the edit point is set according to the time from the change of the image, sound, or shooting state to the event occurrence. The shooting timing may be determined.

  In the third embodiment, when the image signal obtained by shooting is encoded, the cut point of the VOB unit VOBU closest to the timing at which the event occurs is set as the editing point. In this case, These pictures are re-encoded so that the picture type and the reference relationship of the surrounding pictures are changed so that the picture corresponding to the shooting timing determined to be valid as the editing point becomes a break in the VOB unit. You may make it do.

  In this case, as shown in FIG. 14, when setting an edit point, a new VOB unit VOBU having the edit point as the first picture is determined based on the result of determination as to whether there is buffer data before encoding (step S11a). And the picture corresponding to the shooting timing determined to be valid as the edit point become a break in the VOB unit, and the reference relationship between the picture type and the surrounding pictures is changed. As described above, the process of re-encoding these pictures (step S12b) is switched.

  Furthermore, when the image signal obtained by shooting is encoded, processing for setting the break of the VOB unit VOBU closest to the event occurrence timing as an edit point and forcibly generating the VOB unit VOBU by re-encoding You may make it switch to the process to perform according to whether the time which re-encoding requires exceeds the remaining time which can be utilized for the encoding process in the said image process part.

FIG. 15 shows a flow for switching processing for setting an edit point according to the remaining time available for encoding when there is no buffer data before encoding.
In this case, when the edit point is inserted, it is determined whether there is buffer data before encoding (step S12). If it is determined that there is buffer data, the edit point is the same as in the third embodiment. A process for forcibly generating a new VOB unit VOBU with the first picture as the first picture is performed (step S13a).

  On the other hand, if it is determined that there is no buffer data before encoding, it is determined whether or not the time required for re-encoding exceeds the remaining time available for encoding processing in the image processing unit 11a at that time. Determination is made (step S12a). If it is determined that the time required for re-encoding exceeds the remaining time that can be used for the encoding process at that time, the edit point is set at the break of the VOB unit close to the event occurrence timing (step S13c) On the other hand, if it is determined in step S12a that the time required for re-encoding does not exceed the remaining time usable for the encoding process at that time, the picture corresponding to the event occurrence timing is set as the head. A re-encoding process for forcibly creating a VOB unit to be performed is performed (step S13b). In the processing flow shown in FIG. 15, when there is no buffer data before encoding, the processing in step S13b and the processing in step S13c are performed using the time required for re-encoding and the encoding processing in the image processing unit. The processing flow is changed according to either one of steps S13b and S13c previously set by the photographer when there is no buffer data before encoding. It may be what performs.

  Further, in each of the above embodiments, the audio video stream is assumed to be a system stream corresponding to MPEG-2, but the audio video stream is assumed to be a system stream corresponding to MPEG-4 or MPEG-4 AVC. There may be.

  However, in an MPEG-4AVC-compatible system stream, there are I pictures that cannot be randomly accessed and I pictures (IDR) that can be accessed randomly, so that the I picture set as the edit point is determined from the event occurrence timing. It is the closest, randomly accessible I picture (IDR).

  In addition, since an auxiliary information writing area (SEI) is set in the MPEG-4 AVC compatible system stream, it indicates what causes the generation of the feature amount in this writing area. You can also embed information.

  In each of the above embodiments, the audio / video stream includes picture data corresponding to one sequence. This stream is used for thumbnail editing selection in addition to picture data corresponding to one sequence. The sub-picture data outside the sequence may be embedded. In this case, at the time of editing, an appropriate picture as an editing point can be confirmed at a glance by displaying thumbnails.

  In each of the above embodiments, all the pictures set as edit points are used for editing. However, it may be difficult to edit if there are many edit points. The editing points may be thinned out for each setting factor, that is, for each change of the image or the change of the sound. For example, by deleting an edit point set by a change in sound from a plurality of set edit points, it is possible to reduce information on the edit point used at the time of editing.

  In addition, in the MPEG-4 AVC system stream, randomly accessible I pictures (IDR) are arranged with a gap from I pictures that cannot be randomly accessed, so such I pictures (IDR) are edited. By setting as a point, the number of editing points can be reduced.

  Furthermore, in each of the above-described embodiments, the edit point is set by determining whether or not the feature amount at the time of the event occurrence is equal to or higher than a predetermined determination strength and setting the event occurrence timing as the edit point. Alternatively, all the event occurrence timings may be set as edit points, and at the time of editing, it may be determined whether or not the edit points are actually used.

  Specifically, the image capturing apparatus configured as described above is configured to edit the information generation unit of any of the image capturing apparatuses according to any one of the first to third embodiments using an image, a sound, or a shooting timing at which the shooting state is changed as an edit point. Point information is generated, and the feature amount determination unit determines the image feature amount, the audio feature amount, or the unique feature amount when the audio video stream is edited, and the edit point information is determined to be an edit point. It can be realized by determining whether or not to use the photographing timing shown as for editing.

  In this case, specifically, the occurrence time of all the events that change the image, sound, and shooting state is embedded in the audio video stream as editing point information indicating the shooting timing as an editing point. For this reason, when the event occurrence timing is set to the edit point, it is not necessary to determine in real time whether the event occurrence timing is used as the edit point.

  Further, in each of the above embodiments, the editing point is not only the shooting timing when the subject image or sound changes, but also the shooting timing when the photographer's physiological phenomenon changes or the shooting when the photographer operates the camera device. Although the photographing apparatus that also sets the timing is shown, the editing point may be set only at the photographing timing when the image or sound of the subject has changed. In this case, the imaging apparatus does not include the unique identification information acquisition unit 10 and the unique identification information processing unit 10a of the above embodiment.

  Although not particularly mentioned in the description of each of the above embodiments, the imaging device 101 according to the first embodiment shown in FIG. 1, the imaging device 102 according to the second embodiment shown in FIG. 8, and the embodiment shown in FIG. In the imaging apparatus 103 according to the third aspect, each functional unit excluding the imaging unit 11, the recording medium 30a, and the recording medium interface 30 is typically realized as an LSI that is an integrated circuit. These functional units may be individually made into one chip, or some of them may be made into one chip so as to include all of them.

  For example, the plurality of functional units in the imaging apparatus of each of the above embodiments may be a single-chip functional unit other than the memory corresponding to the recording medium 30a and the recording medium interface 30.

  Here, the integrated circuit is referred to as an LSI, but the integrated circuit may be referred to as an IC, a system LSI, a super LSI, or an ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to one in which one or more functional units are realized as an LSI, and the functional unit may be realized in a dedicated circuit or a general-purpose processor. Further, as the LSI, a Field Programmable Gate Array (FPGA) that can be programmed after its manufacture or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used. .

Furthermore, if integrated circuit technology that replaces LSI emerges as a result of advances in semiconductor technology or other technologies derived from it, naturally, functional units may be integrated using this technology. There is a possibility that the technology of circuit integration is one that is applied biotechnology.
In recent years, digital cameras and portable terminals capable of photographing a subject and recording a moving image have been developed, and the functional units constituting the imaging devices of the first to third embodiments are added to such a digital camera and a portable terminal. With this digital audio recording, an audio / video stream that can be edited automatically or with a simple selection operation for guidance is recorded, and the required part of the stream is automatically edited and played back. A camera or a mobile terminal can be realized.

  The imaging apparatus of the present invention generates an audio / video stream that can be edited automatically or by a simple selection operation with respect to a guidance, which is considered to be important for a photographer, and in particular, a digital video camera for home use. Furthermore, it is useful in digital cameras, portable terminals, and the like.

It is a block diagram explaining the imaging device 101 by Embodiment 1 of this invention. 6 is a diagram illustrating an audio / video stream stored in a recording medium in the imaging apparatus 101 according to Embodiment 1. FIG. 6 is a diagram for explaining an operation for setting a condition relating to automatic edit point insertion in the imaging apparatus 101 according to Embodiment 1. FIG. It is a figure explaining operation | movement of the imaging device 101 of the said Embodiment 1. FIG. It is a figure explaining the edit point setting process in the imaging device 101 of the said Embodiment 1 concretely, and is the relationship between a processing flow (FIG. (A)) and imaging | photography timing, and the break of the VOB unit in an audio video stream ( The figure (b)) is shown. It is a figure which shows the flow of a process which automatically edits and reproduces the audio video stream obtained by the imaging device 101 of the said Embodiment 1. FIG. It is a figure explaining the utilization method of an edit point in the imaging device 101 of the said Embodiment 1. FIG. It is a figure for demonstrating the imaging device 102 by Embodiment 2 of this invention. It is a figure which shows the flow of the edit point setting process in the imaging device 102 of the said Embodiment 2. FIG. It is a figure explaining the re-encoding process which forcibly recreates VOB unit VOBU in the imaging device 102 by the said Embodiment 2, and the normal reference relationship at the time of encoding (figure (a)), re-encoding Three modification examples (FIG. (B) to (d)) of the picture type and the reference relationship according to FIG. It is a figure for demonstrating the imaging device 103 by Embodiment 3 of this invention. It is a figure which shows the flow of an edit point insertion process in the imaging device 103 of the said Embodiment 3. FIG. It is a figure explaining the process which produces the new VOB unit VOBU in the imaging device 103 by the said Embodiment 3, the normal reference relationship at the time of an encoding (figure (a)), and the newly created VOB unit Three modification examples (FIGS. (B) to (d)) are shown. It is a figure which shows the modification of the edit point insertion process flow in the imaging device 103 of the said Embodiment 3. FIG. It is a figure which shows the other modification of the edit point insertion processing flow in the imaging device 103 of the said Embodiment 3. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Unique identification information acquisition part 10a Unique identification information processing part 11 Imaging part 11a Image processing part 12 Audio | voice acquisition part 12a Audio | voice processing part 20a, 20b, 20c Control part 21 Feature-value determination part 22a, 22b, 22c Edit point information generation part 30 Recording medium interface unit 30a Recording medium 101, 102, 103 Imaging device

Claims (17)

An imaging device that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information,
An imaging unit for imaging a subject and outputting an image signal;
An image processing unit that performs signal processing on an image signal obtained by imaging the subject and extracts image information including an image feature amount indicating a feature of an image change;
An audio acquisition unit that acquires audio and outputs an audio signal;
A voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition and extracts voice information including a voice feature amount indicating a feature of voice change;
A feature amount determination unit that determines that a shooting timing at which the image or sound has changed is appropriate as an editing point when the image feature amount or the sound feature amount is larger than a predetermined threshold;
An information generation unit that generates edit point information indicating a shooting timing determined to be appropriate as the edit point;
An audio video stream including the image information, audio information, and editing point information is stored in a recording medium;
The information generation unit
It is determined whether or not buffer data that is image information before encoding is held in the image processing unit,
When the buffer data before encoding is held, the editing point is set to a picture corresponding to the shooting timing at which the image, sound, or shooting state has changed,
When the buffer data before the encoding is not held, the edit point is set to the shooting timing at which the image or the sound in the stream obtained by encoding the image signal by the image processing unit changes. Set it to the first picture of the VOB unit that is the nearest random access unit,
The image processing unit
When the buffer data before encoding is held in the image processing unit, the VOB unit is formed so that the picture corresponding to the edit point becomes the first picture of the VOB unit.
An imaging apparatus characterized by that. The imaging device according to claim 1,
A unique identification information acquisition unit for acquiring unique identification information indicating a shooting state;
A unique identification information processing unit that performs signal processing on the acquired unique identification information and extracts a unique feature amount indicating a characteristic of a change in a shooting state;
The feature amount determination unit determines that the shooting timing at which the image, sound, or shooting state has changed is appropriate as an edit point when the image feature amount, the audio feature amount, or the unique feature amount is larger than a predetermined threshold. To
An imaging apparatus characterized by that. The imaging apparatus according to claim 2, wherein
The unique feature amount indicates the magnitude of the photographer's physiological change that occurred during shooting, or the magnitude of adjustment by the photographer's operation.
An imaging apparatus characterized by that. The imaging device according to claim 3.
The physiological change of the photographer that occurred during the photographing is at least one of a change in the amount of sweat of the photographer, a change in α wave, a change in the number of blinks, a change in the pupil, and a change in the pulse,
The unique identification information acquisition unit has a sensor that measures the photographer's physiological change and that corresponds to the type of the physiological change.
An imaging apparatus characterized by that. The imaging device according to claim 1,
The image processing unit performs an inter-screen predictive encoding process for predictively encoding a picture to be encoded with reference to an encoded picture with respect to an image signal obtained by imaging a subject, The image feature amount is extracted based on a motion vector indicating the magnitude of motion of the image used in the inter-screen predictive encoding process,
The audio processing unit performs an encoding process corresponding to the encoding process for the image signal on the audio signal obtained by acquiring the audio,
The information generation unit sets a specific picture in an image stream obtained by encoding an image signal as the edit point, based on a shooting timing determined to be appropriate as the edit point.
An imaging apparatus characterized by that. The imaging device according to claim 1,
The audio processing unit extracts the audio feature amount based on a magnitude of a change in the audio signal;
An imaging apparatus characterized by that. The imaging apparatus according to claim 2, wherein
Based on a user's manual operation signal, a control unit that sets a threshold level for each of the image feature amount or the sound feature amount, and the unique feature amount,
The feature amount determination unit determines each feature amount based on a corresponding threshold level set by the control unit, and the shooting timing at which the image, sound, or shooting state changes is valid as an editing point. To determine whether or not
An imaging apparatus characterized by that. The imaging apparatus according to claim 2, wherein
Table information indicating a correspondence relationship between each of a plurality of scenarios and a combination of threshold levels corresponding to each of the image feature amount, the sound feature amount, and the unique feature amount, and a scenario designated by a user's manual operation And a control unit for setting threshold levels of the various feature amounts based on the table information,
The feature amount determination unit determines each feature amount based on a corresponding threshold level set by the control unit, and the shooting timing at which the image, sound, or shooting state changes is valid as an editing point. To determine whether or not
An imaging apparatus characterized by that. The imaging apparatus according to claim 8.
The table information is obtained by downloading from an information terminal on the network.
An imaging apparatus characterized by that. The imaging device according to claim 1,
The information generation unit
The edit point is set to a shooting timing according to a delay time from when an event that changes the image or sound occurs until the image or sound actually changes.
An imaging apparatus characterized by that. An imaging device that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information,
An imaging unit for imaging a subject and outputting an image signal;
An image processing unit that performs signal processing on an image signal obtained by imaging the subject and extracts image information including an image feature amount indicating a feature of an image change;
An audio acquisition unit that acquires audio and outputs an audio signal;
A voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition and extracts voice information including a voice feature amount indicating a feature of voice change;
A feature amount determination unit that determines that the shooting timing at which the image or sound has changed is appropriate as an edit point when the image feature amount or the sound feature amount is greater than a predetermined threshold;
An information generation unit that generates edit point information indicating a shooting timing determined to be appropriate as the edit point;
An audio video stream including the image information, audio information, and editing point information is stored in a recording medium;
The information generation unit
It is determined whether or not buffer data that is image information before encoding is held in the image processing unit,
If the buffer data before encoding is held, the editing point is set to the picture corresponding to the shooting timing at which the image or sound has changed,
If the buffer data before encoding is not held, the remaining time available for the encoding process in the image processing unit is compared with the time required for re-encoding,
When the time required for the re-encoding exceeds the remaining time available for the encoding process in the image processing unit, the edit point is an image obtained by encoding the image signal by the image processing unit. Set the first picture of the VOB unit, which is the unit of random access, closest to the shooting timing at which the image or sound changes in the stream,
If the time required for the re-encoding does not exceed the remaining time available for the encoding process in the image processing unit, the image processing unit is instructed to re-encode the image stream,
The image processing unit
When the buffer data before encoding is held, a VOB unit is formed so that the picture corresponding to the edit point becomes the first picture of the VOB unit,
Not held above before encoding buffered data, and the case where the time required for re-encoding does not exceed the remaining time available for the encoding process in the image processing section, the upper Kiga image stream Are re-encoded so that the picture corresponding to the edit point is the I picture located at the head of the VOB unit.
An imaging apparatus characterized by that. The imaging device according to claim 1 or 11,
Recording the time at which an event giving a change to the image or sound occurs as an edit point in an audio video stream;
An imaging apparatus characterized by that. The imaging device according to claim 12, wherein
Recording the time of occurrence of the event in the audio-video stream as a playlist indicating playback conditions;
An imaging apparatus characterized by that. The imaging device according to claim 12, wherein
Embedding in the audio video stream information indicating whether the edit point is due to image or audio factors;
An imaging apparatus characterized by that. The imaging device according to claim 1 or 11,
The information generation unit
Embed a picture corresponding to the time at which an event giving a change to the image or sound occurs as an out-of-sequence picture used for thumbnail display at the time of editing in the audio video stream.
An imaging apparatus characterized by that. An imaging method for acquiring image information and audio information by photographing a subject and recording an audio video stream including the image information and audio information,
An imaging step of imaging a subject and outputting an image signal;
An image processing step of performing signal processing on an image signal obtained by imaging the subject and extracting image information including an image feature amount indicating a feature of image change;
An audio acquisition step of acquiring audio and outputting an audio signal;
A voice processing step of performing voice signal processing on the voice signal obtained by the voice acquisition to extract voice information including a voice feature amount indicating a feature of voice change;
A feature amount determination step for determining that a shooting timing at which the image or sound has changed is appropriate as an edit point when the image feature amount or the sound feature amount is larger than a predetermined threshold;
An information generation step for generating edit point information indicating a photographing timing determined to be appropriate as the edit point;
Storing an audio video stream including the image information, audio information, and editing point information in a recording medium, and
The information generation step includes
It is determined whether or not buffer data that is image information before encoding is held in the image processing unit that executes the image processing step,
When the buffer data before encoding is held, the editing point is set to a picture corresponding to the shooting timing at which the image, sound, or shooting state has changed,
When the buffer data before the encoding is not held, the edit point is set to the shooting timing at which the image or the sound in the stream obtained by encoding the image signal by the image processing unit changes. Set it to the first picture of the VOB unit that is the nearest random access unit,
The image processing step includes
When the buffer data before encoding is held in the image processing unit, the VOB unit is formed so that the picture corresponding to the edit point becomes the first picture of the VOB unit.
An imaging method characterized by the above. A semiconductor device that acquires image information and audio information by photographing a subject and records an audio video stream including the image information and audio information,
An image processing unit that performs signal processing on an image signal obtained by imaging the subject and extracts image information including an image feature amount indicating a feature of an image change;
An audio acquisition unit that acquires audio and outputs an audio signal;
A voice processing unit that performs signal processing on the voice signal obtained by the voice acquisition and extracts voice information including a voice feature amount indicating a feature of voice change;
A feature amount determination unit that determines that a shooting timing at which the image or sound has changed is appropriate as an editing point when the image feature amount or the sound feature amount is larger than a predetermined threshold;
An information generation unit that generates edit point information indicating a shooting timing determined to be appropriate as the edit point;
An audio video stream including the image information, audio information, and editing point information is stored in a recording medium;
The information generation unit
It is determined whether or not buffer data that is image information before encoding is held in the image processing unit,
When the buffer data before encoding is held, the editing point is set to a picture corresponding to the shooting timing at which the image, sound, or shooting state has changed,
When the buffer data before the encoding is not held, the edit point is set to the shooting timing at which the image or the sound in the stream obtained by encoding the image signal by the image processing unit changes. Set it to the first picture of the VOB unit that is the nearest random access unit,
The image processing unit
When the buffer data before encoding is held in the image processing unit, the VOB unit is formed so that the picture corresponding to the edit point becomes the first picture of the VOB unit.
A semiconductor device.

Images