JP2012253692A - Imaging apparatus, reproducer, data structure, control method of imaging apparatus and imaging apparatus program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus for reproducing atmosphere upon photographing and for reproducing an image without giving discomfort to an appreciator when the image is reproduced with sound, and to provide a reproducer, a data structure, a control method of the imaging apparatus and an imaging apparatus program.SOLUTION: The imaging apparatus includes: an imaging section for imaging an image and generating electronic data of the image; a sound input section for inputting sound and converting it into an electric signal; a sound processing section for generating sound data by performing processing on the electric signal which the sound input section converts, and separating the sound data into a plurality of pieces of partial sound data on the basis of a characteristic of the sound; and a control section for selecting the sound data or the partial sound data corresponding to the sound outputted in synchronization with the image and making the data correspond to the image in accordance with a display mode when the image corresponding to the image data which the imaging section generates is reproduced.

Description

  The present invention relates to an imaging apparatus that acquires an image and generates electronic image data, a playback apparatus that reproduces an image, a data structure including image data, a control method for the imaging apparatus, and a program for the imaging apparatus.

  2. Description of the Related Art Conventionally, in a digital camera having a function of shooting a moving image, a technique for adding and reproducing audio data before and after a still image shot during moving image shooting has been disclosed (for example, see Patent Document 1). .

  In addition, in an imaging / playback apparatus having a function of shooting and playing back a movie, an imaging / playback apparatus capable of setting a snap movie shooting mode in which shooting automatically ends after a predetermined time has elapsed as a mode other than the normal movie shooting mode. Are known. In this imaging / playback apparatus, it is possible to switch and play back images with high tempo while outputting audio prepared separately from the audio at the time of shooting in the snap movie shooting mode (see, for example, Patent Document 2).

JP 2008-22246 A JP 2010-141414 A

  However, in Patent Document 1 described above, when the voice of the subject is included in the sound during reproduction, there is a possibility that the voice is interrupted in the middle. When the subject's voice was interrupted, the viewer of the replayed image was concerned about the content of the interrupted sound, and had to continue watching with a sense of discomfort.

  Further, in Patent Document 2 described above, since there is no relationship between playback and video, in some cases, a mismatch between sound and video may occur, and the viewer may feel uncomfortable. .

  The present invention has been made in view of the above, and an imaging apparatus capable of reproducing the atmosphere at the time of shooting and reproducing the image without making the viewer feel uncomfortable when reproducing an image together with sound. An object is to provide an apparatus, a data structure, an imaging apparatus control method, and an imaging apparatus program.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging unit that captures an image and generates electronic data of the image, and inputs audio and converts it into an electrical signal. A voice input unit that generates voice data by processing the electrical signal converted by the voice input unit, and that separates the voice data into a plurality of partial voice data based on the characteristics of the sound; In accordance with a display mode when reproducing an image corresponding to the image data generated by the imaging unit, audio data or partial audio data corresponding to audio output in synchronization with the image is selected and associated with the image And a control unit.

  The imaging apparatus according to the present invention is characterized in that, in the above invention, the sound characteristic includes a sound intensity.

  In the imaging device according to the present invention, in the above invention, the sound characteristic includes a frequency, and the sound processing unit calculates a frequency spectrum of the sound data, and the frequency spectrum is calculated based on the sound characteristic. The plurality of partial sound data is generated from a partial frequency spectrum obtained by separating.

  Moreover, the imaging apparatus according to the present invention is characterized in that, in the above-mentioned invention, the voice processing unit can generate partial voice data including a human voice.

  In the image pickup apparatus according to the present invention, in the above invention, when the control unit reproduces and displays an image in a slide show mode in which a plurality of images are continuously displayed for a predetermined time, audio output in synchronization with the image is output. As described above, partial voice data not including a human voice is selected with the highest priority.

  In the image pickup apparatus according to the present invention, the display unit capable of displaying an image corresponding to the image data and the sound corresponding to the sound data or the partial sound corresponding to the partial sound data can be output. An audio output unit, and the control unit selects audio or partial audio corresponding to the display mode of the image on the display unit, and causes the audio output unit to output in synchronization with the display of the image. Features.

  In the image pickup apparatus according to the present invention, in the above-described invention, a reproduction apparatus capable of displaying an image and outputting audio, a communication unit that transmits and receives information via a network, and information to be output to the reproduction apparatus Transmission that creates data and a transmission file that is determined according to the display mode of the image corresponding to the image data and includes audio data that is output in synchronization with the image and that is transmitted to the playback device via the communication unit And a file creation unit.

  In addition, the playback device according to the present invention includes a display unit capable of displaying an image, a sound output unit capable of outputting sound suitable for an image displayed on the display unit, image data, and an image corresponding to the image data. An interpreter that interprets the description of a data structure that includes audio data including audio that is output in synchronization with the image, and the display unit based on the result of interpretation by the interpreter And a reproduction control unit that causes the audio output unit to output audio while displaying an image.

  Further, the playback device according to the present invention further includes a communication unit capable of communicating with the imaging device via the network and receiving the transmission file as the data structure from the imaging device in the above invention. It is characterized by.

  In addition, the data structure according to the present invention outputs image data, mode information that defines a display mode of an image corresponding to the image data, and outputs the image in synchronization with the image when the image is displayed according to the display mode. And voice data including voice.

  According to another aspect of the present invention, there is provided a method for controlling an imaging apparatus, wherein an imaging step of capturing an image and generating electronic data of the image, and inputting an audio signal during a period including the time of imaging of the image in the imaging step. A voice input step for converting into a signal; and processing the electrical signal converted in the voice input step to generate voice data, and separating the voice data into a plurality of partial voice data based on the characteristics of the sound Depending on the display mode when reproducing the image corresponding to the image data generated in the audio processing step and the imaging step, the audio data or the partial audio data corresponding to the audio output in synchronization with the image is selected. And an associating step for associating with the image.

  The imaging apparatus program according to the present invention captures an image to generate electronic data of the image, and processes the electrical signal to an imaging apparatus that inputs sound and converts it into an electrical signal. In accordance with the sound processing step for generating sound data and separating the sound data into a plurality of partial sound data based on sound characteristics, and the display mode when the image is reproduced, the sound data is synchronized with the image. And selecting a voice data or partial voice data corresponding to the voice to be output and associating it with the image.

  According to the present invention, audio data is separated into a plurality of partial audio data, and audio data or partial audio data suitable for the display mode of the image data to be displayed is selected and associated with the image. In this case, it is possible to reproduce the atmosphere at the time of photographing the image and allow the viewer to appreciate the reproduced image in a comfortable state.

FIG. 1 is a diagram illustrating a configuration of a communication system including an imaging device and a playback device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an outline of processing performed by the audio processing unit of the imaging apparatus according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a temporal change in the maximum sound pressure level of the audio data. FIG. 4 is a diagram showing an outline of the sound separation process performed by the sound processing unit of the imaging apparatus according to the embodiment of the present invention. FIG. 5 is a diagram schematically showing the data structure of the transmission file generated by the transmission file creation unit of the imaging apparatus according to the embodiment of the present invention. FIG. 6 is a schematic diagram showing an outline of characteristic processing performed by the imaging apparatus according to the embodiment of the present invention. FIG. 7 is a flowchart showing an outline of processing performed by the imaging apparatus according to the embodiment of the present invention. FIG. 8 is a flowchart showing an outline of audio data recording processing performed by the imaging apparatus according to the embodiment of the present invention. FIG. 9 is a flowchart showing an overview of the moving image shooting process performed by the imaging apparatus according to the embodiment of the present invention. FIG. 10 is a flowchart showing an outline of the reproduction process performed by the imaging apparatus according to the embodiment of the present invention. FIG. 11 is a flowchart showing an outline of a slide show display process performed by the imaging apparatus according to the embodiment of the present invention. FIG. 12 is a diagram schematically illustrating a situation in which playback is performed using the second partial sound during operation playback in the slide show display. FIG. 13 is a flowchart showing an overview of display processing performed by the playback apparatus according to the embodiment of the present invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as “embodiments”) will be described with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a configuration of an imaging device and a playback device according to an embodiment of the present invention, and a configuration of a communication system including the imaging device and the playback device. The communication system 100 shown in the figure includes an imaging device 1 that captures an image of a predetermined visual field and generates image data, and a playback device 3 that displays the image. The imaging device 1 and the playback device 3 have a function of communicating information including image data with each other.

  First, the configuration of the imaging device 1 will be described. The imaging device 1 includes an imaging unit 11, an operation input unit 12, a voice input unit 13, a display unit 14, a voice output unit 15, a communication unit 16, a storage unit 17, and a control unit 18.

  The imaging unit 11 captures an image of a predetermined visual field and generates digital image data. The imaging unit 11 includes one or more lenses, an optical system that collects light from a subject existing in a predetermined visual field region, a diaphragm that adjusts an incident amount of light collected by the optical system, and a diaphragm An image sensor such as a CCD (Charge Coupled Device) that receives light passing through and converts it into an electrical signal, and an A / D converter that converts an analog signal output from the image sensor into image data composed of a digital signal. Have.

  The operation input unit 12 receives input of various types of information including operation instructions for the imaging apparatus 1. The operation input unit 12 is a power button of the imaging device 1, a still image release button that gives a still image shooting instruction, a moving image release button that gives a moving image shooting instruction, and mode switching that switches among various operation modes that can be set by the imaging device 1. Buttons and control buttons including instructions for reproducing and editing image data. In addition, the operation input unit 12 includes a touch panel provided by being stacked on the display unit 14.

  The voice input unit 13 is configured using a microphone, and has a function of inputting voice and converting it into an electrical signal.

  The display unit 14 is configured using liquid crystal, organic EL, or the like, and has a function of appropriately displaying operation information of the imaging apparatus 1 and information related to photographing in addition to image data.

  The audio output unit 15 is configured using a speaker and has a function of outputting audio to the outside.

  The communication unit 16 transmits / receives information to / from the playback device 3 via the network. As this network, a wireless communication network such as a wireless local area network (LAN) or infrared communication may be applied, or a network including wired communication may be applied.

  The storage unit 17 includes an image data storage unit 171 that stores image data, an audio data storage unit 172 that stores audio data, and a program storage unit 173 that stores various programs executed by the imaging apparatus 1. The storage unit 17 is configured using a semiconductor memory such as a flash memory or a RAM (Random Access Memory) that is fixedly provided inside the imaging apparatus 1. Note that the storage unit 17 may have a function as a recording medium interface that reads information recorded on a recording medium while recording information on a recording medium such as a memory card mounted from the outside.

  The image data storage unit 171 stores a moving image file in which moving image data encoded based on a standard such as MPEG (Moving Picture Experts Group) is stored in a container together with audio data, and also JPEG (Joint Photographic Experts Group) or the like. Still image data encoded based on the format is stored.

  The control unit 18 is configured using a CPU (Central Processing Unit) or the like, and is connected to each component of the imaging device 1 that is a control target via a bus line. The control unit 18 includes an image processing unit 181 that performs predetermined image processing on the image data generated by the imaging unit 11, and an audio processing unit that performs predetermined signal processing on the electrical signal output from the audio input unit 13. 182, a transmission file creation unit 183 that creates a transmission file to be transmitted to the playback device 3, and a timer 184 having a timekeeping function.

  The audio processing unit 182 samples and quantizes the audio signal and performs A / D conversion to generate audio data, and performs sound processing by performing FFT (Fast Fourier Transformation) and filtering on the generated audio data. Separation is performed at a predetermined frequency, the sound pressure level of the separated audio data is determined, and a plurality of partial audio data is generated by performing inverse FFT on the audio data after separation that satisfies a predetermined standard.

FIG. 2 is a diagram for explaining an overview of processing performed by the audio processing unit 182. In FIG. 2, the horizontal axis f represents frequency, and the vertical axis L represents sound pressure level (SPL). A curve 200 shown in FIG. 2 is a frequency spectrum of audio data (hereinafter referred to as “frequency spectrum 200”). In the present embodiment, the audio processing unit 182 separates the frequency spectrum 200 into two with the reference frequency f ₀ as a boundary. Thereafter, the sound processing unit 182 determines the level of the sound pressure level by comparing the maximum sound pressure level L _max and the reference sound pressure level L _{0 in the} two separated partial frequency spectra. Here, the sound pressure level L is an amount proportional to the common logarithm of the value of the ratio of the sound pressure P to the reference sound pressure P ₀ = 2 × 10 ⁻⁵ (Pa), and L = 20 × log ₁₀ (P / P ₀ ) (dB).

In the case illustrated in FIG. 2, the frequency spectrum 200 is separated into two partial frequency spectra 201 and 202 by the audio processing unit 182. Further, as a result of the sound pressure level determination by the sound processing unit 182, the magnitude relationship between the maximum sound pressure levels L ₁ and L ₂ and the reference sound pressure level L ₀ in the partial frequency spectra 201 and 202 is L ₁ > L _0. , L ₂ <L ₀ .

For example, in FIG. 2, if the reference frequency f ₀ = 600 (Hz) and the reference sound pressure level L ₀ = 50 (dB), there is a high possibility that the partial frequency spectrum 201 includes a human voice. This is because the frequency of voices in which a person normally talks is 150 to 600 Hz, and the sound pressure level is 60 to 90 dB. On the other hand, the partial frequency spectrum 202 includes sound having a relatively high frequency and a low sound pressure level. Therefore, it is highly possible that the sound given by the partial frequency spectrum 202 includes a sound that has a high healing effect, such as a sound of a river, a song of a bird, or a leaf touching a tree. Thus, by appropriately determining the reference frequency f ₀ and the reference sound pressure level L ₀ , it is possible to clearly separate the partial sound including the human voice and the partial sound including the natural background sound.

FIG. 3 is a diagram illustrating an example of a temporal change in the maximum sound pressure level L _max of the audio data. In FIG. 3, the horizontal axis t represents time, and the vertical axis L _max represents the maximum sound pressure level. A curve 300 shown in FIG. 3 indicates a time change of the maximum sound pressure level of the audio data corresponding to one moving image data, and time T indicates moving image shooting time. In FIG. 3, time T ₀ indicates the playback time when the playback time of the moving image is set to be substantially equal to the playback time of the still image when performing a slide show display in which a moving image and a still image are mixed. Yes. By setting the playback time T ₀ , playback with a good tempo can be performed when performing a slide show in which moving images and still images are mixed.

On the other hand, in the case of the moving image shown in FIG. 3, the maximum sound pressure level L _max exceeds the reference sound pressure level L ₀ immediately after the start of shooting, and still exceeds the reference sound pressure level L ₀ even at time T _0. . For this reason, when the reference sound pressure level L ₀ = 50 (Hz), if the image is switched at this point, it is switched in a state in which a human voice enters, and the image is interrupted while something is being spoken. There is a possibility that. In the present embodiment, in order to avoid such a situation and realize a slideshow with less sense of incongruity, when a voice of a person is included in the sound output in synchronization with the image, the voice of the person is not output. To output other partial audio.

FIG. 4 is a diagram showing an outline of the sound separation process performed by the sound processing unit 182. The first partial sound corresponds to sound data whose frequency f is smaller than the reference frequency f ₀ (f <f ₀ ) and whose maximum sound pressure level L _max is equal to or higher than the reference sound pressure level L ₀ (L _max ≧ L ₀ ). is doing. Therefore, the partial frequency spectrum 201 shown in FIG. 2 corresponds to the first partial sound.

On the other hand, in the second partial sound, the frequency f is equal to or higher than the reference frequency f ₀ (f ≧ f ₀ ), and the maximum sound pressure level L _max is smaller than the reference sound pressure level L ₀ (L _max <L ₀ ). Supports audio data. Therefore, the partial frequency spectrum 202 shown in FIG. 2 corresponds to the second partial sound.

When the frequency f and the maximum sound pressure level L _max do not satisfy the above-described conditions, the sound processing unit 182 does not record the separated sound data as partial sound data.

  By the way, in order to enable the voice processing unit 182 to clearly recognize the voice of a person who is a subject, the imaging device 1 is provided with a face detection unit that can detect a face image included in the image data. The audio data may be limited to a band suitable for the subject's voice or the like according to the characteristics determined when the detection unit detects the face image. In this case, the audio processing unit 182 can limit the band of the audio data by including a band limiting filter circuit (BPF: Band-Pass Filter) that is a filter circuit that limits the band, for example.

  Note that the sound processing unit 182 may record sound data, and may generate sound to be output by extracting only data of a portion related to a necessary band from the recorded sound data.

  In addition, the audio processing unit 182 may amplify audio data of a certain band from the audio data, and perform audio data band limitation by attenuating audio data of other bands.

  Further, the sound processing unit 182 may have a configuration in which the filter characteristics can be switched. For example, the voice processing unit 182 may have a configuration capable of switching between a filter characteristic optimized for an adult voice band and a filter characteristic optimized for a child voice band. Here, the frequency band of an adult voice is 100 to 8000 Hz, while the frequency band of a child's voice is 150 to 10000 Hz. For this reason, when recording the voice of an adult clearly, only the voice data of 100 to 8000 Hz is extracted and output from the input voice data, while when recording the voice of a child clearly, the input is performed. Only the audio data of 150 to 10000 Hz is extracted from the audio data and output. In addition, you may make it extract the audio | voice data according to sex by switching to the filter characteristic according to sex.

  Further, the audio processing unit 182 may extract and output audio data in a frequency band different from that of human voice. Specifically, the voice processing unit 182 may extract and output an animal voice, a car, an airplane engine sound, and the like.

  The transmission file creation unit 183 creates a transmission file to be transmitted to the playback device 3. FIG. 5 is a diagram schematically showing the data structure of the transmission file. The transmission file 400 shown in the figure has text information 401, image data 402, and audio data 403. The text information 401 includes, as information to be interpreted by the playback device 3, information related to the type of transmission file, mode information that determines the display mode of the image, and character information that is displayed together with the image. The information regarding the type of transmission file includes information indicating that the data is for a slide show, information indicating the last data in a series of data for a slide show, and the like. The audio data 403 is either audio data or first and second partial audio data.

  FIG. 6 is a schematic diagram illustrating an outline of characteristic processing performed by the imaging apparatus 1 having the above configuration. Specifically, FIG. 6 is a diagram schematically showing a difference between a situation at the time of photographing by the imaging apparatus 1 and an audio output at the time of reproduction. Note that the characters described in FIG. 6 are for representing the characteristics of the sound corresponding to the image, and the characters are not displayed on the actual screen. As shown in FIG. 6A, when the subject 700 is uttering the word “tight” before and after shooting, the audio data storage unit 172 stores the word of the subject 700 as the first partial audio data. In addition, the sound of “Zaza” (sound of the river flow) is stored as the second partial audio data as the background sound. When the imaging apparatus 1 captures and stores audio data as described above, when the image is reproduced, as shown in FIG. 6B, only the second partial audio is reproduced, and the words uttered by the subject 700 are reproduced. Has the function of not playing.

  Next, the configuration of the playback device 3 will be described. The playback device 3 includes a communication unit 31, an operation input unit 32, a display unit 33, an audio output unit 34, a storage unit 35, and a control unit 36. The communication unit is a communication interface that transmits and receives information including image data to and from the imaging device 1. The operation input unit 32 receives input of various types of information including operation instructions for the playback device 3. The display unit 33 is configured using liquid crystal, organic EL, or the like, and can display various information. The audio output unit 34 is configured using a speaker or the like and can output audio. The storage unit 35 is configured using a semiconductor memory or the like, and stores various types of information including an operation program for the playback device 3. The control unit 36 is configured using a CPU or the like, and comprehensively controls the operation of the playback device 3.

  The control unit 36 interprets the information written in the transmission file sent from the imaging device 1, displays the image on the display unit 33 based on the result interpreted by the interpretation unit, and outputs the sound. A reproduction control unit 362 that outputs sound to the unit 34, and a timer 363 having a timekeeping function.

  The playback apparatus 3 having the above configuration is realized by any one of a television, a projector, a personal computer, a digital photo frame, and the like. When the playback device 3 is realized by such an electronic device, the display switching means for switching between the display of the image based on the image data sent from the imaging device 1 and the display of the image based on other functions is played back. The device 3 may be provided.

  FIG. 7 is a flowchart illustrating an outline of processing performed by the imaging apparatus 1. First, if the power supply of the imaging device 1 is on (step S101: Yes), the imaging device 1 proceeds to step S102. On the other hand, if the power supply of the imaging device 1 is not turned on (step S101: No), the imaging device 1 ends a series of processes.

  In step S102, when the imaging device 1 is set to the shooting mode (step S102: Yes), the control unit 18 controls the imaging unit 11 and the voice input unit 13 to start acquiring images and sounds, respectively. This is performed (step S103).

  Thereafter, the control unit 18 causes the display unit 14 to display an image corresponding to the image data generated by the imaging unit 11 and subjected to image processing by the image processing unit 181 as a through image (step S104).

  Subsequently, when the still image release button is operated and a still image release signal is input (step S105: Yes), the imaging apparatus 1 performs still image shooting (step S106). Thereafter, the control unit 18 compresses the captured image and records it as still image data in the image data storage unit 171 (step S107). In addition, the control unit 18 performs processing such as classification on the audio data and records it in the audio data storage unit 172 (step S108). After step S108, the imaging apparatus 1 returns to step S101.

  FIG. 8 is a flowchart showing an overview of the audio data recording process in step S108. The imaging device 1 repeatedly performs the processing of steps S201 to S204 described below for audio data acquired in a predetermined time zone before and after image shooting.

First, the control unit 18 records audio data in the audio data storage unit 172 (step S201). Thereafter, the voice processing unit 182 separates the voice data into two partial voice data based on the magnitude of the reference frequency f ₀ (step S202). Subsequently, the sound processing unit 182 determines the sound pressure level of each sound data by comparing the maximum sound pressure level L _max of the two separated partial sound data with the reference sound pressure level L ₀ ( Step S203).

  Thereafter, the voice processing unit 182 records the partial voice data in the voice data storage unit 172 according to the level determination result (step S204). For example, in the case of audio data having the frequency spectrum 200 shown in FIG. 2, the first partial audio data corresponding to the partial frequency spectrum 201 and the second partial audio data corresponding to the partial frequency spectrum 202 are stored in the audio data storage unit 172. To be recorded. If the audio data level determination result does not satisfy the condition shown in FIG. 4, the audio processing unit 182 deletes the audio data without recording it in the audio data storage unit 172.

  When all the loops are completed, the imaging apparatus 1 returns to the main routine shown in FIG.

  Next, a case where the still image release signal is not input in step S105 of FIG. 7 (step S105: No) will be described. In this case, when the moving image release button is pressed and a moving image release signal is input (step S109: Yes), the imaging apparatus 1 performs moving image shooting (step S110). On the other hand, when the moving image release signal is not input in step S109 (step S109: No), the imaging device 1 returns to step S101.

  FIG. 9 is a flowchart showing an overview of the moving image shooting process in step S110. First, the control unit 18 starts recording moving image data and audio data (step S301).

  Thereafter, the control unit 18 records the moving image data in the image data storage unit 171 (step S302). Thereafter, when a still image release signal is input (step S303: Yes), the control unit 18 records still image data (step S304). On the other hand, when the still image release signal is not input (step S303: No), the imaging apparatus 1 proceeds to step S309 described later.

  The imaging device 1 also performs audio recording processing in parallel with the processing in steps S302 to S304. The processes in steps S305 to S308 corresponding to the voice recording process sequentially correspond to the processes in steps S201 to S204 described above.

  After step S304 and S308, when an instruction to end moving image shooting is input by the operation input unit 12 (step S309: Yes), the control unit 18 ends recording of moving image data and audio data (step S310). Thereafter, the imaging apparatus 1 returns to the main routine shown in FIG.

  In step S309, when the operation input unit 12 does not input a moving image shooting end instruction (step S309: No), the imaging apparatus 1 returns to step S301.

  Next, the case where the imaging device 1 is not set to the shooting mode in step S102 of FIG. 7 (step S102: No) will be described. In this case, the imaging apparatus 1 performs a reproduction process (step S111) and returns to step S101.

  FIG. 10 is a flowchart illustrating an outline of the reproduction process performed by the imaging apparatus 1. In FIG. 10, the control unit 18 displays a predetermined number of reproduction target images on the display unit 14 (step S401). Subsequently, when the imaging apparatus 1 is set to the slide show mode (step S402: Yes), the control unit 18 causes the display unit 6 to perform a slide show display (step S403). Thereafter, when an instruction to end the reproduction process is input by the operation input unit 12 (step S404: Yes), the imaging apparatus 1 returns to the main routine illustrated in FIG. On the other hand, when the operation input unit 12 does not input a reproduction processing end instruction (step S404: No), the imaging apparatus 1 returns to step S401.

  FIG. 11 is a flowchart showing an outline of the slide show display in step S403. In FIG. 11, the imaging apparatus 1 repeatedly performs the processes of steps S501 to S513 described below for all target images. In the slide show display described below, it is assumed that the playback time is the same for moving images and still images. However, if the moving image ends earlier than the playback time of the still image, the image is switched at that time. By performing such a slide show display, it is possible to reproduce an image with a sense of tempo even when moving images and still images are mixed.

  First, a case where the imaging device 1 has not established communication with the playback device 3 (step S501: No) will be described. In this case, when there is the second partial audio data corresponding to the image data to be reproduced (step S502: Yes), the control unit 18 reproduces and displays the image on the display unit 14 for a predetermined time, and the second partial audio is audio. The output unit 15 is caused to reproduce only the same time as the image display time (step S503), and one loop is terminated. On the other hand, if there is no second partial audio data corresponding to the image data to be reproduced (step S502: No), the control unit 18 reproduces and displays the image on the display unit 14 for a predetermined time, and also outputs the audio corresponding to the image. Playback is performed for a predetermined time (step S504), and one loop is terminated. When there is no second partial audio data, only the image may be reproduced and the audio may not be reproduced.

  Next, a case where the imaging device 1 has established communication with the playback device 3 (step S501: Yes) will be described. In this case, if there is second partial audio data corresponding to the image data to be reproduced (step S505: Yes), the transmission file creation unit 183 creates a transmission file including the image data and the second partial audio data corresponding to each other. (Step S506). The transmission file generated here has a configuration in which the second partial audio data is applied as the audio data 403 of the transmission file 400 shown in FIG.

  Thereafter, the communication unit 16 performs streaming transmission of the transmission file to the playback device 3 under the control of the control unit 18 (step S507).

  When the transmission of the transmission file by the communication unit 16 is completed (step S508: Yes), the imaging device 1 proceeds to step S511 described later. On the other hand, when transmission of the transmission file by the communication unit 16 is not completed (step S508: No), when the communication unit 16 receives an image switching request from the playback device 3 (step S509: Yes), the imaging device 1 The transmission of the image data being transmitted and the second partial audio data is stopped (step S510), and the process proceeds to step S511 described later. In step S509, when an image switching request is not received from the playback device 3 (step S509: No), the imaging device 1 returns to step S507.

  In step S511, when the transmission file transmitted immediately before is data to be transmitted at the end of a series of slide shows (step S511: Yes), the communication unit 16 transmits an end signal indicating the end of the slide show to the playback device 3 (step S511). Step S512). Thereafter, the imaging apparatus 1 ends one loop process. On the other hand, in step S511, when the data transmitted immediately before is not the last data of the slide show (step S511: No), the imaging device 1 ends one loop process.

  If there is no second partial audio data corresponding to the image data to be reproduced in step S505 (step S505: No), the transmission file creation unit 183 creates a transmission file including the image data and audio data corresponding to each other (step S505). S513). Thereafter, the imaging apparatus 1 proceeds to step S507.

  When the loop processing for all target images is completed, the imaging apparatus 1 returns to the reproduction processing shown in FIG.

FIG. 12 is a diagram schematically showing a situation in which the second partial sound is played during operation playback in the above-described slide show display. More specifically, in FIG. 12, after the display unit 14 displays the still image 501 for the time T ₀ , the moving image typically described in the moving image frames 502 to 505 is displayed together with the second partial sound for the time T _0. It shows a situation in which a still image 506 is displayed for a time T ₀ after that. In this case, since the voice of the subject 600 is not output during moving image reproduction (schematically described with an X mark in the balloon), even if the subject 600 is talking about something, the moving image is changed to a still image. The story is not interrupted when migrating. Therefore, the viewer of the slide show can appreciate the images without a sense of incongruity.

  In the case shown in FIG. 12, the second partial sound is output for the still images 501 and 506 if corresponding second partial sound data exists. Note that when displaying a still image in a slide show display in which a moving image and a still image are mixed, no sound may be output.

  Next, the case where the imaging device 1 is not set to the slide show mode in step S402 in FIG. 10 (step S402: No) will be described. In this case, when reproduction of a moving image is selected by the operation input unit 12 (step S405: Yes), the control unit 18 determines whether or not communication with the reproduction device 3 is established (step S406). When the control unit 18 determines that communication with the playback device 3 has been established (step S406: Yes), the communication unit 16 plays the video file of the selected video under the control of the control unit 18. Streaming transmission is performed to the apparatus 3 (step S407). On the other hand, when the control unit 18 determines in step S406 that communication with the playback device 3 has not been established (step S406: No), the control unit 18 causes the display unit 14 to play back and display the selected video ( Step S408). After step S407 or S408, the imaging apparatus 1 proceeds to step S404.

  A case where reproduction of a moving image is not selected in step S405 (step S405: No) will be described. In this case, when playback of a still image is selected (step S409: Yes), the imaging device 1 determines whether or not communication with the playback device 3 has been established (step S410). When the control unit 18 determines that communication with the playback device 3 has been established (step S410: Yes), when there is second partial audio data corresponding to the image data to be played back (step S411: Yes). The transmission file creation unit 183 creates a transmission file including the corresponding image data and second partial audio data (step S412).

  After step S412, the communication unit 16 performs streaming transmission of the transmission file to the playback device 3 under the control of the control unit 18 (step S413).

  Thereafter, when transmission of the transmission file by the communication unit 16 is completed (step S414: Yes), the imaging device 1 proceeds to step S404. On the other hand, when the transmission of the transmission file by the communication unit 16 is not completed (step S414: No), when the communication unit 16 receives an end instruction from the playback device 3 (step S415: Yes), the imaging device 1 Transmission of the still image data and the second partial audio data is stopped (step S416), and the process proceeds to step S404. When the transmission of the transmission file by the communication unit 16 has not been completed (step S414: No), when the communication unit 16 does not receive an end instruction from the playback device 3 (step S415: No), the imaging device 1 goes to step S413. Return.

  If there is no second partial audio data corresponding to the still image data to be reproduced in step S411 (step S411: No), the transmission file creation unit 183 creates a transmission file including the corresponding still image data and audio data. (Step S417). Thereafter, the imaging apparatus 1 proceeds to step S413. If there is no second partial audio data corresponding to still image data to be played back, the transmission file creation unit 183 may create a transmission file using still image data without including audio data.

  Next, the case where the control unit 18 determines in step S410 that communication with the playback device 3 has not been established (step S410: No) will be described. In this case, when there is the first partial audio data corresponding to the still image data to be reproduced (step S418: Yes), the control unit 18 reproduces the still image and the first partial audio (step S419). That is, the control unit 18 displays a still image on the display unit 14 and causes the audio output unit 15 to output the first partial sound corresponding to the still image. Thereafter, the imaging apparatus 1 proceeds to step S404.

  On the other hand, when there is no first partial audio data corresponding to the still image data to be reproduced in step S418 (step S418: No), the control unit 18 reproduces the still image and the audio (step S420). That is, the control unit 18 displays a still image on the display unit 14 and causes the audio output unit 15 to output sound corresponding to the still image. Thereafter, the imaging apparatus 1 proceeds to step S404. In step S420, the display unit 14 may only display a still image, and the audio output unit 15 may not output the corresponding audio.

  If reproduction of a moving image is not selected in step S405 (step S405: No) and reproduction of a still image is not selected in step S409 (step S409: No), the imaging apparatus 1 proceeds to step S404.

  Next, an outline of processing performed by the playback apparatus will be described. FIG. 13 is a flowchart illustrating an outline of processing performed by the playback device 3. First, when the power of the playback device 3 is on (step S601: Yes), the playback device 3 proceeds to step S602. On the other hand, when the playback apparatus 3 is not turned on (step S601: No), the imaging apparatus 1 ends the series of processes.

  Subsequently, when the communication unit 31 receives a transmission file from the imaging device 1 (step S602: Yes), the interpretation unit 361 refers to the text information of the transmission file and the display target data forms part of the slide show. It is determined whether the file is a transmission file (step S603). When the interpretation unit 361 determines that the data is slide show data (step S603: Yes), the reproduction control unit 362 causes the display unit 33 to display an image corresponding to the received image data and causes the audio output unit 34 to display the image. On the other hand, the streaming reproduction is started by outputting the audio corresponding to the received audio data (step S604). If the communication unit 31 does not receive a transmission file from the imaging device 1 in step S602 (step S602: No), the imaging device 1 returns to step S601.

  When the predetermined time has elapsed since the playback device 3 started streaming playback (step S605: Yes), the communication unit 31 transmits a switching request to the imaging device 1 under the control of the control unit 36 ( Step S606). Thereafter, when the communication unit 31 receives a new transmission file from the imaging device 1 (step S607: Yes), the reproducing device 3 returns to step S604. If the predetermined time has not elapsed since the playback device 3 started streaming playback (step S605: No), the playback device 3 repeats step S605.

  When a new transmission file is not received within the predetermined time in step S607 (step S607: No), when the slide show end signal is received within the predetermined time (step S608: Yes), the playback device 3 performs step S601. Return to. On the other hand, when the slide show end signal is not received within the predetermined time (step S608: No), the control unit 36 displays an error display on the display unit 33 (step S609), and returns to step S601. Note that the playback device 3 may display a message prompting the user to select whether to continue the slide show after the error display. In this case, the playback device 3 performs processing according to the input content of the operation input unit 12.

  In step S603, if the transmitted image data is not slide show data (step S603: No), the playback device 3 performs normal playback processing (step S610), and the process returns to step S601.

  According to the embodiment of the present invention described above, the audio data is separated into a plurality of partial audio data, and the audio data or the partial audio data suitable for the display mode of the image data to be displayed is selected to correspond to the image. Therefore, when the image is reproduced, the atmosphere at the time of shooting the image can be reproduced and the viewer can view the reproduced image in a comfortable state.

  Further, according to the present embodiment, when generating partial sound data, the frequency spectrum of the sound data is separated based on the characteristics of the sound including the frequency, so that, for example, a human voice or background sound is separated. Is possible. Therefore, it is possible to output sound suitable for the situation during reproduction.

  Further, according to the present embodiment, since partial voice data (first partial voice data) including a human voice is generated, it is included in partial voice data (second partial voice data) having a higher frequency than that. The sound that is heard is a sound with a high healing effect. Therefore, by reproducing an image using such partial audio data, it is possible to perform reproduction with a background sound with a high healing effect that eliminates human voice and other noises.

  Further, according to the present embodiment, when an image is reproduced and displayed in a slide show mode in which a plurality of images are continuously displayed for a predetermined time, a portion that does not include a human voice as a sound output in synchronization with the image Since audio data is selected with the highest priority, it is possible to avoid as much as possible that a human voice is interrupted and output even when a moving image is switched halfway.

  Up to this point, one mode for carrying out the present invention has been described, but the present invention should not be limited only by the above-described embodiment.

  For example, in the present invention, the transmission file creation unit may create a transmission file including image data, audio data corresponding to the image data, and partial audio data. In this case, the playback device may be provided with a function of determining and outputting a sound or partial sound suitable for the image.

  In the present invention, the transmission file created by the transmission file creation unit may be recorded on a recording medium, and the recording medium may be inserted into the playback device and read by the playback device. In this case, the reproducing apparatus is provided with a recording medium interface for reading information recorded by the recording medium, and has a function of determining and outputting sound or partial sound suitable for the image to be reproduced based on the read data. The imaging device and the playback device need not be connected for communication.

  In the present invention, when separating voice data, the voice data may be separated according to a temporal change pattern of the sound intensity, or from the voice data using characteristics of a voice print of a human voice. You may make it isolate | separate the partial audio | voice data containing a human voice.

DESCRIPTION OF SYMBOLS 1 Imaging device 3 Reproduction | regeneration apparatus 11 Imaging part 12, 32 Operation input part 13 Audio | voice input part 14, 33 Display part 15, 34 Audio | voice output part 16, 31 Communication part 17, 35 Storage part 18, 36 Control part 100 Communication system 171 Image Data storage unit 172 Audio data storage unit 173 Program storage unit 181 Image processing unit 182 Audio processing unit 183 Transmission file creation unit 184, 363 Timer 200 Frequency spectrum 201, 202 Partial frequency spectrum 361 Interpretation unit 362 Playback control unit

Claims (12)

An imaging unit that captures an image and generates electronic data of the image;
An audio input unit that inputs audio and converts it into an electrical signal;
An audio processing unit that generates audio data by performing processing on the electrical signal converted by the audio input unit, and that separates the audio data into a plurality of partial audio data based on the characteristics of the sound;
Control for selecting audio data or partial audio data corresponding to audio to be output in synchronization with the image and associating it with the image according to a display mode when reproducing an image corresponding to the image data generated by the imaging unit And
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the sound characteristics include sound intensity. The sound characteristics include frequency,
The voice processing unit
3. The partial audio data is generated from a partial frequency spectrum obtained by calculating a frequency spectrum of the audio data and separating the frequency spectrum based on a characteristic of the sound. Imaging device. The voice processing unit
The imaging apparatus according to claim 3, wherein partial sound data including a human voice can be generated. The controller is
When playing back and displaying images in the slide show mode in which a plurality of images are continuously displayed for a predetermined time, it is necessary to select, with highest priority, partial audio data that does not include human voices as audio to be output in synchronization with the images. The imaging apparatus according to claim 4, wherein the imaging apparatus is characterized. A display unit capable of displaying an image corresponding to the image data;
An audio output unit capable of outputting audio corresponding to the audio data or partial audio corresponding to the partial audio data;
Further comprising
The controller is
The sound or the partial sound corresponding to the display mode of the image on the display unit is selected and output to the sound output unit in synchronization with the display of the image. The imaging device described. A communication unit that transmits and receives information via a network with a playback device capable of displaying images and outputting sound;
The information to be output to the playback device includes image data and audio data that is determined in accordance with the display mode of the image corresponding to the image data and is output in synchronization with the image, and the playback is performed via the communication unit. A transmission file creation unit for creating a transmission file to be transmitted to the device;
The imaging apparatus according to claim 1, further comprising: A display unit capable of displaying an image;
An audio output unit capable of outputting audio suitable for an image displayed by the display unit;
An interpretation unit for interpreting the description content of the data structure having image data and sound data including sound that is defined in accordance with a display mode of the image corresponding to the image data and output in synchronization with the image;
A reproduction control unit for displaying an image on the display unit based on a result of interpretation by the interpretation unit and outputting sound to the audio output unit;
A playback apparatus comprising:   9. The communication apparatus according to claim 8, further comprising a communication unit capable of communicating with the imaging apparatus according to claim 7 via the network and receiving the transmission file as the data structure from the imaging apparatus. Playback device. Image data,
Mode information that defines a display mode of an image corresponding to the image data;
Audio data including audio output in synchronization with the image when the image is displayed according to the display mode;
A data structure characterized by comprising: An imaging step of capturing an image and generating electronic data of the image;
An audio input step of inputting an audio of a period including the time of imaging of the image in the imaging step and converting it into an electrical signal;
A voice processing step of generating voice data by performing processing on the electrical signal converted in the voice input step, and separating the voice data into a plurality of partial voice data based on a characteristic of the sound;
Correspondence of selecting audio data or partial audio data corresponding to audio output in synchronization with the image and corresponding to the image according to the display mode when reproducing the image corresponding to the image data generated in the imaging step Attaching step,
A method for controlling an imaging apparatus, comprising: An imaging device that captures an image and generates electronic data of the image, and also inputs sound and converts it into an electrical signal.
A voice processing step of generating voice data by performing processing on the electrical signal, and separating the voice data into a plurality of partial voice data based on a sound characteristic;
An association step of selecting audio data or partial audio data corresponding to audio output in synchronization with the image and associating it with the image according to a display mode when reproducing the image;
A program for an imaging apparatus, characterized in that

Images