JP2020092381A - Sound acquisition device, sound acquisition method, and sound acquisition program - Google Patents

Abstract

To emphasize a sound emitted by a sound collection target located at a predetermined distance by using first and second microphones.SOLUTION: A sound acquisition device includes a first microphone that is arranged at a first distance from a sound collection target and collects a sound emitted from the sound collection target to acquire a first sound, a second microphone that is arranged apart from the sound collection target by a second distance different from the first distance and that collects a second sound by collecting the sound emitted from the sound collection target, and a voice emphasizing unit that performs emphasizing processing for emphasizing a component based on a distance difference between the first distance and the second distance from among the first and second sounds.SELECTED DRAWING: Figure 1

Description

The present invention relates to a voice acquisition device, a voice acquisition method, and a voice acquisition program for acquiring voice by using first and second microphones.

Conventionally, many mobile devices such as digital cameras and smartphones have a recording function and a recording function. This type of mobile device includes an image pickup unit for picking up an image of a subject and a built-in microphone for picking up surrounding sounds, and there are some that can record AV data including video and sound when shooting a moving image.

Furthermore, some portable devices can acquire recording data picked up by an external microphone through a cable or wireless communication, and can record the sound acquired by the external microphone and the image captured by the built-in image pickup unit. Is also commercialized. For example, when an external microphone can be placed at a position away from a mobile device that images a subject, by placing the external microphone in the vicinity of the subject that is the sound collection target, a sound with good S/N can be obtained. Can also be obtained from an external microphone.

JP, 2005-151471, A

However, for example, when collecting the bark of a wild bird, it is not always possible to install a microphone in the immediate vicinity of the wild bird that is the target of sound collection, and the microphone is located relatively far from the target of sound collection. May need to be installed. Then, not only the sound from the sound pickup target but also the sound near the microphone and the sound emitted from other than the sound pickup target are picked up as noise in the microphone.

In addition, in patent document 1, there is known a TV conference system using a plurality of microphones and cameras, which accurately selects a speaker. However, this system has a voiceprint authentication unit that authenticates conference participants who have registered voiceprints, an image pickup adjustment unit that controls the TV camera device so as to optimally capture the speaker, and the device scale is large. Moreover, it is not possible to select and record an arbitrary sound to be picked up.

The present invention provides a voice acquisition device, a voice acquisition method, and a voice acquisition program capable of emphasizing a voice emitted by a sound collection target located at a predetermined distance by using the first and second microphones. With the goal.

A voice acquisition device according to an aspect of the present invention includes a first microphone that is arranged at a first distance from a sound collection target and that collects a sound emitted from the sound collection target to acquire a first sound. A second microphone disposed apart from the sound collecting target by a second distance different from the first distance to collect a sound emitted from the sound collecting target to obtain a second sound; and the first microphone. And a voice enhancement unit that performs enhancement processing for enhancing the component based on the distance difference between the first distance and the second distance in the second voice.

A sound acquisition method according to an aspect of the present invention acquires a first sound by collecting a sound emitted from the sound collection target by a first microphone arranged at a first distance from the sound collection target. , A second microphone arranged apart from the sound collecting target by a second distance different from the first distance collects a sound emitted from the sound collecting target to obtain a second sound, and An emphasis process for emphasizing a component based on the distance difference between the first distance and the second distance of the first and second voices is performed.

A sound acquisition program according to an aspect of the present invention collects a sound emitted from the sound collection target by a first microphone arranged in a computer at a first distance from the sound collection target to generate a first sound. Is acquired, and a second microphone arranged apart from the sound collecting target by a second distance different from the first distance collects a sound emitted from the sound collecting target to obtain a second sound. Then, the procedure of performing the emphasis process for emphasizing the component based on the distance difference between the first distance and the second distance of the first and second voices is executed.

According to the present invention, a voice acquisition device, a voice acquisition method, a voice acquisition program, and a voice that can emphasize a voice emitted by a sound collection target located at a predetermined distance by using the first and second microphones. The purpose is to provide an acquisition system.

The block diagram which shows the audio|voice acquisition apparatus which concerns on the 1st Embodiment of this invention. Explanatory drawing which shows the state of sound collection. Explanatory drawing for demonstrating the time gap of the audio|voice which was picked up by taking time on a horizontal axis and showing an amplitude on a vertical axis. Explanatory drawing for demonstrating the time gap of the audio|voice which was picked up by taking time on a horizontal axis and showing an amplitude on a vertical axis. Explanatory drawing for demonstrating an example of the selection method of target audio|voice. 6 is a flowchart for explaining the operation of the camera 1. 6 is a flowchart for explaining the operation of the recorder 2. 6 is a flowchart for explaining the operation of the playback device 3. Explanatory drawing for demonstrating the state at the time of photography of the camera 1. Explanatory drawing which shows the state at the time of reproduction|regeneration of the reproducing|regenerating apparatus 3. The block diagram which shows the 2nd Embodiment of this invention. Explanatory drawing which shows the positional relationship of the bird 41, the grass 42, the recorder 2, and the camera 1 on XY coordinates. Explanatory drawing for demonstrating the time gap of the audio|voice which was picked up by taking time on a horizontal axis and showing an amplitude on a vertical axis. Explanatory drawing for demonstrating the time gap of the audio|voice which was picked up by taking time on a horizontal axis and showing an amplitude on a vertical axis. The flowchart which shows control of a reproducing|regenerating apparatus. The block diagram which shows the audio|voice acquisition apparatus which concerns on the 3rd Embodiment of this invention. Explanatory drawing which shows a mode at the time of sound collection and sound recording. The flowchart which shows control of a camera. The flowchart which shows control of a recorder.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a voice acquisition device according to a first embodiment of the present invention. The present embodiment employs first and second microphones that collect a sound (hereinafter, referred to as a target sound) that is a target of sound collection from the sound collection target, and the sound from the sound collection target to the first microphone is used. Emphasizing the sound to be collected, which is located at a predetermined distance from the first and second microphones, by delaying the collected sound based on the distance difference between the distance and the distance to the second microphone. It realizes the processing. Further, it is possible to separate and extract the target sound emitted from the sound collection target located at a predetermined distance from the first microphone, separately from the noise generated from other positions, and selectively collect the sound.

In the example of FIG. 1, one of the first and second microphones is configured as a camera as an imaging device having a sound collection function, and the other is configured as a recorder. The first and second microphones may be devices having any configuration. The audio acquisition device according to the present embodiment may be configured in the camera, in the recorder, or distributed in the camera and the recorder. It may be configured as an independent device. It should be noted that FIG. 1 shows an example in which the audio acquisition device is distributed and configured into a camera, a recorder, and a playback device.

First, a sound collecting method according to the first embodiment will be described with reference to FIGS. 2, 3A, 3B, and 4. FIG. 2 is an explanatory diagram showing a state of picked up sound, and FIGS. 3A and 3B are explanatory diagrams for explaining a time lag of picked up sound with time on the horizontal axis and amplitude on the vertical axis. Is. Further, FIG. 4 is an explanatory diagram for explaining an example of a method of selecting a target voice.

The camera 1 shown in FIG. 2 has a housing 1a in which each circuit of FIG. 1 is housed. A shutter button 15a that constitutes an operation unit 15 described later is provided on the upper surface of the housing 1a. Further, the recorder 2 shown in FIG. 2 has a housing 2a in which each circuit of FIG. 1 is housed.

In the example of FIG. 2, the bird 41 remains on the branch of the tree 43a. For example, when performing shooting and sound collection of wild birds, it is conceivable to install the recorder 2 in advance near a tree where wild birds are likely to stay. On the other hand, the user carrying the camera 1 shoots at a position relatively distant from the wild bird in order to prevent the bird from escaping and to avoid obstacles.

For video, it is possible to obtain a sufficiently high-quality image from a position relatively far from the subject by adopting a telephoto lens, but for audio, noise increases as the distance from the subject increases. , The sound collection quality of the target voice is deteriorated. For this reason, the recorder 2 is installed at a position closer to the sound collection target.

In the example of FIG. 2, the user holding the camera 1 uses the gap between the trees 43a and 43b and tries to capture and pick up the sound of the bird 41 at a position relatively distant from the bird 41 that is the subject. .. On the other hand, the recorder 2 is arranged on a tripod installed relatively close to the tree 43a and picks up the bark of the bird 41 as a target sound at a position relatively close to the bird 41. The frame surrounding the bird 41 in FIG. 2 indicates the shooting range of the camera 1, and the image 41a of the shot bird 41 is displayed on the display screen 16a of the display unit 16 of the camera 1 which will be described later. ing.

Grass 42 grows on the ground between the camera 1 and the bird 41. Therefore, in the ST sound collecting unit 12 of the camera 1 which will be described later, in addition to the sound of the bird 41, which is the target sound, the sound of the grass 42 fluttering in the wind and other sounds (hereinafter, sounds other than the target sound are called noise ) Is picked up. Also in the recorder 2, not only the sound of the bird 41, which is the target voice, but also other ambient noises are picked up. In order to simplify the description, it is assumed that, in the range in which the camera 1 and the recorder 2 can collect sound, only the target voice and the noise due to the grass 42 are generated.

The target sound emitted from the bird 41 is picked up by the camera 1 and the recorder 2. Considering the target voice A1 which is the direct sound mainly reaching the camera 1 and the target voice A2 which is the direct sound reaching the recorder 2, among these target voices, these target voices A1 and A2 are different from each other in amplitude. It is considered that they have the same frequency and substantially the same frequency characteristic. That is, it is considered that the target voices A1 and A2 have only the difference in arrival time due to the difference in the propagation distance, when the amplitude is ignored.

FIGS. 3A and 3B describe the waveforms at the time when these sounds reach the camera 1 and the recorder 2, assuming that the sound of the bird 41 that is the target sound and the noise such as the sound of the grass 42 are separated. To do. FIG. 3A shows the target voice, and the target voice at the position of the camera 1 has only the arrival time delay and the decrease in amplitude based on the difference in distance with respect to the target voice at the position of the recorder 2. In the example of FIG. 3A, the arrival time difference is Ta, and the target voice reaching the recorder 2 and the camera 1 is out of phase by an amount corresponding to the arrival time difference Ta.

Further, FIG. 3B shows the noise caused by the grass 42. The noise at the position of the recorder 2 and the noise at the position of the camera 1 have a delay in arrival time and a change in amplitude due to the difference in distance. In the example of FIG. 3B, the arrival time difference is Tb, and the noise reaching the recorder 2 and the camera 1 has a waveform shifted by an amount corresponding to the arrival time difference Tb.

Therefore, in the present embodiment, the sound acquired by the recorder 2 is delayed by the arrival time difference Ta and added to the sound acquired by the camera 1. As a result, among the components included in the sound picked up by the camera 1 and the recorder 2, the target sound having the arrival time difference Ta is added in a phase-matched state and strengthened (amplifies the amplitude). When recording the sound picked up by the camera 1 and the recorder 2, the reproduction timing of the recorded sound is shifted by the arrival time difference Ta so that the target sound can be reproduced in phase. It is possible. In the following description, a process of delaying at least one of the sound picked up by the camera 1 and the sound picked up by the recorder 2 to cancel the arrival time difference and match the phase of the target sound This is referred to as the above-mentioned shift processing (time-axis shift processing).

FIG. 4 illustrates this process, showing the waveform of the voice picked up by the recorder 2 and the waveform of the voice picked up by the camera 1 at the arrival time difference Ta from this voice, and showing these waveforms as the arrival time difference. A waveform obtained by shifting by Ta and adding on the time axis is shown. In the camera 1 and the recorder 2, the target voice picked up with the arrival time difference Ta is shifted on the time axis and synthesized (added) to obtain a relatively large peak value of the synthesized voice.

The arrival time difference can be calculated based on the distance and the speed of sound. For example, assuming that the sound collection target, the recorder 2 and the camera 1 are located on a straight line, by obtaining the distance between the camera 1 and the recorder 2, the arrival time difference can be calculated from the obtained distance and the sound velocity. it can. When the camera 1 is configured by a digital camera or the like, the camera 1 is often capable of distance measurement, and the arrival time difference can be calculated by obtaining the distance to the recorder 2 by the camera 1. The example of FIG. 2 shows that the distance between the camera 1 and the recorder 2 thus obtained is 8 m.

Note that, for example, when the recorder 2 is arranged at a position of several meters near the sound collection target and the camera 1 is separated from the sound collection target by, for example, about 100 m, the sound collection target, the recorder 2 and the camera 1 are on a substantially straight line. If it is located at, it is considered that the calculation error of the arrival time difference is relatively small.

In the present embodiment, a process of shifting at least one of the sounds picked up by the camera 1 and the recorder 2 on the time axis to cancel the arrival time difference, and then performing addition (hereinafter referred to as time axis shift). By performing an addition process), a process of increasing the amplitude of the target voice (emphasis process) may be performed and output.

Furthermore, in the present embodiment, by extracting only the components that strengthen each other by such time-axis shift addition processing, the components of the target speech are extracted from the synthesized speech obtained by the time-axis shift addition processing, The emphasis processing may be performed.

For example, if it is assumed that the frequency of the target voice does not change in a predetermined short period, it may be considered that the frequency components that strengthen each other in a predetermined cycle by the time-axis shift addition process in this period are the components of the target voice. By extracting the component, it is possible to extract the synthesized sound of only the target sound from the collected sound.

Also, for example, when the target voice is a song of a bird that is discontinuous with Chun-Chun, the target voice is mainly included by extracting only a predetermined period including the peak period of the components that strengthen each other by the time axis shift addition process. It is also possible to perform emphasis processing for extracting a synthesized sound to be generated. For example, the predetermined period including the peak position of the synthesized waveform of FIG. 4 is output as the extraction result of the target voice.

In FIG. 1, a control unit 10 is provided in the camera 1 that constitutes the image pickup apparatus. The control unit 10 may be configured by a processor using a CPU, an FPGA, or the like, may be a unit that operates according to a program stored in a memory (not shown) to control each unit, or an electronic circuit of hardware. May realize some or all of the functions.

The camera 1 includes an imaging unit 11 and an ST sound pickup unit 12. The image pickup unit 11 as an image acquisition unit has an optical system 11a and an image pickup device (not shown). The optical system 11a includes a lens, a diaphragm, and the like (not shown) for zooming and focusing. The optical system 11a includes a zoom (variable magnification) mechanism, a focus and a diaphragm mechanism (not shown) that drives these lenses. The image pickup device is composed of a CCD, a CMOS sensor, or the like, and the optical system 11a guides the subject optical image to the image pickup surface of the image pickup device. The image pickup device photoelectrically converts the subject optical image to obtain a picked-up image (image pickup signal) of the subject.

The imaging control unit 10a configured in the control unit 10 is configured to drive and control the zoom mechanism, the focus mechanism, and the aperture mechanism of the optical system 11a to adjust the zoom, aperture, and focus. The focus/angle-of-view information unit 10c is configured to acquire information regarding zoom, aperture, and focus from the optical system 11a and output the information to the photographing control unit 10a. By this feedback, the photographing control unit 10a can set the zoom, aperture, and focus to desired setting values. The imaging unit 11 is controlled by the imaging control unit 10a to perform imaging, and outputs an imaging signal of a captured image (moving image and still image) to the control unit 10.

The control section 10 includes a sound collection control/processing section 10d, and the sound collection control/processing section 10d controls the ST sound collection section 12. The ST sound collection unit 12 is configured by a stereo microphone or the like, and is controlled by the sound collection control and processing unit 10d to collect the sound around the camera 1 to acquire a sound signal, and acquire the acquired sound (hereinafter , Also referred to as the first voice) can be output to the sound collection control and processing unit 10d that also functions as an input unit. It is preferable that the ST sound collecting unit 12 has a peak of sensitivity in the shooting direction of the camera 1, that is, in the optical axis direction of the optical system 11a. A monaural microphone may be adopted instead of the ST sound collecting unit 12.

The camera 1 is provided with an operation unit 15. The operation unit 15 includes a release button, a function button, various switches for shooting mode setting, parameter operation, etc., a dial, a ring member, etc. (not shown), and outputs an operation signal based on a user operation to the control unit 10. The control unit 10 controls each unit based on an operation signal from the operation unit 15.

The control unit 10 captures the captured image (moving image and still image) from the image capturing unit 11. The image processing unit 10b of the control unit 10 performs predetermined signal processing on the captured image, such as color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing. Further, the sound collection control and processing unit 10d can take in the sound from the ST sound collecting unit 12 and perform a predetermined signal processing on the taken-in sound.

A display unit 16 is provided in the camera 1, and the display unit 16 has a display screen configured by, for example, an LCD (liquid crystal display device) or the like. This display screen is provided, for example, on the back surface of the housing of the camera 1. The control unit 10 causes the display unit 16 to display the captured image signal-processed by the image processing unit 10b. The control unit 10 can also display various menu displays, warning displays, etc. of the camera 1 on the display unit 16.

The camera 1 is provided with communication units 18a and 18b. The communication units 18a and 18b are communication devices that comply with a predetermined communication standard, and are controlled by the control unit 10 so as to be able to send and receive information to and from the recorder 2. The communication unit 18a can perform short-range wireless communication such as Bluetooth (registered trademark), and the communication unit 18b can perform wireless LAN communication such as Wi-Fi (registered trademark). Note that the communication units 18a and 18b are not limited to Bluetooth and Wi-Fi, but can employ communication in various communication systems. The control unit 10 can transmit the information recorded in the recording unit 14 to the reproduction device 3 via the communication unit 18a or 18b.

The camera 1 is provided with a recording unit 14. The recording unit 14 is configured by a predetermined recording medium such as a hard disk or a memory medium, and can record the information given from the control unit 10 and output the recorded information to the control unit 10. A card interface, for example, can be used as the recording unit 14, and the recording unit 14 can record image data on a recording medium such as a memory card.

The control unit 10 can compress the picked-up image after the signal processing and give the compressed image to the recording unit 14 to record the image. Further, the control unit 10 can compress the sound after the signal processing by the sound collection control and processing unit 10d, and give the compressed sound or the uncompressed sound to the recording unit 14 to record the sound. The camera 1 is provided with a clock unit 19, and the control unit 10 uses the time information from the clock unit 19 to add the time information to the sound acquired by the ST sound collecting unit 12 and attach the time information. It can be recorded in the recording unit 14 in association with the selected image.

The camera 1 is also provided with an image feature extraction unit 13. The image feature extraction unit 13 may be configured by a processor using a CPU or the like, performs image analysis on a captured image of a subject captured by the imaging unit 11, extracts the image feature, and controls the extraction result. The data is output to the section 10. Further, the camera 1 is provided with a position/direction sensor unit 17. The position/direction sensor unit 17 is composed of a position sensor, a gyro sensor, a magnetic sensor, and the like, and detects the position and direction of the camera 1 and outputs the detection result to the control unit 10.

In the present embodiment, the recording unit 14 has a cooperation information unit 14a. Information related to communication between the recorder 2 and the reproduction device 3 is recorded in the cooperation information unit 14a, and the control unit 10 controls the communication units 18a and 18b based on the information read from the cooperation information unit 14a. As a result, information can be exchanged between the recorder 2 and the playback device 3 by communication. The control unit 10 can transmit the first sound obtained by the ST sound collection unit 12 and the image obtained by the image pickup unit 11 to the reproduction device 3.

In the present embodiment, the control unit 10 includes a distance and arrival time difference determination unit 10e. The distance and arrival time difference determination unit 10e employs various distance measuring methods to obtain the distance from the camera 1 to the recorder 2, and determines the arrival time difference by calculating the obtained distance and the sound velocity. Note that the control unit 10 may perform distance measurement by the image plane phase difference method by imaging the recorder 2, for example, and performs distance measurement using a hill-climbing focus process that determines contrast from a captured image. May be. Further, the control unit 10 may obtain the distance to the recorder 2 using the detection result of the position and orientation sensor unit 17.

The control unit 10 may set the sound collection target to a subject in focus by the focus and angle-of-view information unit 10c, or may set a subject located in the center of the captured image. You may set to the to-be-photographed object designated by operation. When the sound collection target, the recorder 2 and the camera 1 are not located on a substantially straight line, the control unit 10 obtains the position of the camera 1, the distance to the recorder 2 and the sound collection target and the direction thereof, and collects the sound collection target. To the camera 1, the distance from the sound collection target to the recorder 2, and the distance from the camera 1 to the recorder 2 may be obtained.

The control unit 10 records the sound from the ST sound collecting unit 12 in the recording unit 14 in association with the information on the arrival time difference (arrival time difference information) obtained by the distance and arrival time difference determination unit 10e.

Further, the control unit 10 is provided with an occurrence timing estimation unit 10f. The generation timing estimation unit 10f receives the captured image from the imaging unit 11 and the image feature information from the image feature extraction unit 13, and estimates the generation timing of the sound emitted by the sound collection target by image analysis. For example, when the sound collection target is a bird and the sound of the bird is collected as the target sound, the generation timing estimation unit 10f causes the timing at which the bird's beak opens and closes and the timing at which the bird calls out according to the degree of opening and closing. Guess and output the guess result.

The control unit 10 associates the sound from the ST sound collecting unit 12 with the information of the estimation result of the sound generation timing obtained by the generation timing estimation unit 10f (time information of the generation timing) and records it in the recording unit 14. Is becoming

The recorder 2 is provided with a controller 20. The control unit 20 may be configured by a processor using a CPU or the like, may be a unit that operates according to a program stored in a memory (not shown) to control each unit, or may function as an electronic circuit of hardware. May be realized partially or entirely. The recorder 2 has communication units 21a and 21b. The communication units 21a and 21b are communication devices that comply with a predetermined communication standard, and are controlled by the control unit 20 so that information can be transmitted and received between the camera 1 and the playback device 3. The communication unit 21a is capable of short-range wireless communication such as Bluetooth (registered trademark), and the communication unit 21b is capable of wireless LAN communication such as Wi-Fi (registered trademark). Note that the communication units 21a and 21b are not limited to Bluetooth and Wi-Fi, but can employ communication in various communication systems.

The recorder 2 is provided with the ST sound collecting unit 22, and the ST sound collecting unit 22 can be configured by, for example, a stereo microphone (not shown). A monaural microphone may be adopted instead of the ST sound collecting unit 22. The ST sound collecting unit 22 is adapted to acquire a voice (hereinafter, also referred to as a second voice) with a stereo microphone. The control unit 20 is configured with a sound collection control unit 20a, and the sound collection control unit 20a controls the sound collection of the ST sound collection unit 22. The sound collection control unit 20a that functions as an input unit is configured to take in the second sound from the ST sound collection unit 22.

The recorder 2 is provided with an operation unit 23. The operation unit 23 includes various switches, dials, ring members and the like (not shown) for recording mode setting, parameter operation, etc., and outputs an operation signal based on a user operation to the control unit 20. The control unit 20 controls each unit based on an operation signal from the operation unit 23. Further, when the control information is given from the control unit 10 of the camera 1 via the communication units 21a and 21b, the control unit 20 may control each unit based on this control information. In this case, the recording section of the recorder 2 can be controlled by the control section 10 of the camera 1.

The recorder 2 is provided with a recording unit 25. The recording unit 25 is configured by a predetermined recording medium such as a hard disk or a memory medium, and can record the information given from the control unit 20 and output the recorded information to the control unit 20. A card interface, for example, can be used as the recording unit 25, and the recording unit 25 can record audio data on a recording medium such as a memory card. The control unit 20 can give the second sound after the signal processing to the sound recording unit 25 to record the second sound. The recorder 2 is provided with a clock unit 24, and the control unit 20 uses the time information from the clock unit 24 to add the time information to the second sound acquired by the ST sound collecting unit 22, and the recording unit 25. Can be recorded in.

The recording unit 25 also has a cooperation information unit 25a. Information related to communication between the camera 1 and the playback device 3 is recorded in the cooperation information unit 25a, and the control unit 20 controls the communication units 21a and 21b based on the information read from the cooperation information unit 25a. As a result, information can be exchanged between the camera 1 and the reproduction device 3 by communication. The control unit 20 can transmit the second sound acquired by the ST sound collecting unit 22 to the reproducing device 3.

The playback device 3 may be configured by a computer, a smartphone, a tablet terminal, or the like. The playback device 3 includes a control unit 30. The control unit 30 that functions as a voice emphasizing unit may be configured by a processor that uses a CPU, an FPGA, or the like, and may operate according to a program stored in a memory (not shown) to control each unit. Alternatively, some or all of the functions may be realized by an electronic circuit of hardware.

The playback device 3 is provided with an operation unit 32. The operation unit 32 includes various switches, dials, ring members and the like (not shown) for setting a reproduction mode and operating parameters, and outputs an operation signal based on a user operation to the control unit 30. The control unit 30 controls each unit based on an operation signal from the operation unit 32. The communication unit 31 is a communication device compatible with a predetermined communication standard, and is controlled by the control unit 30 to communicate with the camera 1 and the recorder 2 to exchange information. .. The reproducing device 3 is provided with a recording unit 34. The recording unit 34 is configured by a predetermined recording medium such as a hard disk or a memory medium, and records the information given from the control unit 30.

The control unit 30 receives the image and the first sound from the camera 1 and the second sound from the recorder 2 via the communication unit 31. As described above, the time information is added to the image, the first sound, and the second sound. The communication unit 31 also receives the arrival time difference information associated with the first voice and the information about the estimation result of the sound generation timing. These pieces of information added to the first and second voices are called cooperation information. The control unit 30 supplies the information received from the camera 1 and the recorder 2 to the recording unit 34 to record the information.

The control unit 30 is provided with a time axis shift addition processing unit 30a. The time-axis shift addition processing unit 30a reads out the information recorded in the recording unit 34 and performs time-axis shift processing on at least one of the received first voice and second voice, so that the target voice is collected. The difference between the arrival time from the target to the recorder 2 and the arrival time from the arrival at the camera 1 is canceled, and the phase of the target voice is matched. The time-axis shift addition processing unit 30a performs time-axis shift addition processing for synthesizing the first and second voices after the time-axis shift processing to obtain a synthetic sound.

The time axis shift addition processing unit 30a may adjust the sound delay time in the time axis shift processing according to the estimation result of the sound generation timing based on the image. The image and the sound acquired by the camera 1 are recorded in synchronization with each other, and it is considered that the estimation result of the sound generation timing based on the image matches the rising timing of the waveform of the first sound. Therefore, the time-axis shift addition processing unit 30a can perform the time-axis shift processing that reliably cancels the arrival time difference by adjusting the audio delay time according to the estimation result of the sound generation timing.

The control unit 30 is provided with a target voice extraction unit 30b. The target speech extraction unit 30b detects a frequency component (a target speech component) that reinforces each other in a predetermined cycle among the frequency components of the synthetic sound obtained by the time-axis shift addition processing, and performs filter processing for extracting the frequency component. To do. The target speech extraction unit 30b outputs the filtered speech (synthesized sound) to the reproduction unit 33 and the recording unit 34.

Further, the target voice extraction unit 30b performs a filter process of extracting a synthesized sound only for a predetermined period including the peak period of the components that strengthen each other in a predetermined cycle by the time-axis shift addition process, and the filtered voice (synthesized voice ) May be output to the reproducing unit 33 and the recording unit 34.

The recording unit 34 receives the voice from the target voice extracting unit 30b and records the voice. The reproduction unit 33 includes a speaker (not shown), and is controlled by the control unit 30 to reproduce and output the sound from the target sound extraction unit 30b.

It should be noted that the reproduction device 3 may omit the target voice extraction unit 30b and output the synthesized sound after the time-axis shift addition processing by the time-axis shift addition processing unit 30a to the reproduction unit 33 and the recording unit 34. Good.

Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 5 is a flow chart for explaining the operation of the camera 1, FIG. 6 is a flow chart for explaining the operation of the recorder 2, and FIG. 7 is a flow chart for explaining the operation of the reproducing apparatus 3. Further, FIG. 8 is an explanatory diagram for explaining a state of the camera 1 at the time of shooting, and FIG. 9 is an explanatory diagram showing a state of the reproducing device 3 during reproduction.

(recording)
First, the operation of collecting and recording sound will be described. As shown in the example of FIG. 2, it is assumed that the bird 41 remains on the branch of the tree 43a and the recorder 2 is installed relatively close to the tree 43a. In addition, the user carries the camera 1 and photographs the bird 41 at a position relatively distant from the tree 43a. In both the ST sound collecting unit 12 of the camera 1 and the ST sound collecting unit 22 of the recorder 2, the bark of the bird 41 that is the target of sound collection (target sound) and other ambient noise (sound from the grass 42) And are picked up.

A display screen 16a of the display unit 16 is provided on the back surface of the housing 1a of the camera 1. The user, for example, grips the casing 1a with his/her hand and, while watching the display screen 16a, captures the bird 41, which is the subject, in the visual field range, and presses the shutter button 15a to perform photographing.

When the power is turned on, the control unit 10 of the camera 1 determines whether the imaging mode is instructed in step S1 of FIG. When the imaging mode is not instructed, the control unit 10 shifts to a designated mode, for example, a cooperation mode for performing settings and transmission/reception for cooperation with the recorder 2 or a reproduction mode for reproducing recorded images. ..

For example, in the cooperation mode, the control unit 10 measures the distance between the camera 1 and the recorder 2. As a result, the control unit 10 obtains the distance difference between the distance between the bird 41, which is the sound collection target, and the recorder 2 and the distance between the bird 41 and the camera 1. For example, when not only the bird 41 but also the recorder 2 can be photographed by the camera 1, the control unit 10 can measure the distance and can obtain the distance difference in the focusing operation at the time of photographing. Further, the control unit 10 can obtain the distance difference by using the information on the position and the shooting direction from the position and direction sensor unit 17. If the recorder 2 has a positioning function, the control unit 30 may calculate the distance difference by acquiring the position information from the recorder 2.

The control unit 10 may display information on the distance difference on the display unit 16. For example, assuming that the bird 41, the recorder 2 and the camera 1 are arranged substantially linearly, the control unit 10 determines the distance from the camera 1 to the recorder 2 as the transmission distance between the first voice and the second voice. You may display as a distance difference. FIG. 8 shows that the display 41b indicating that the distance from the camera 1 to the recorder 2 is 8 m is displayed on the display screen 16a.

When the imaging mode is instructed, the control unit 10 starts moving image recording in the next step S2, and starts sound collection by the ST sound collection unit 12 in step S3. When the moving image and the sound are acquired, the clock unit 19 starts clocking, and the control unit 10 records the time information from the clock unit 19 at the same time.

In the next step S4, the control unit 10 determines whether or not the cooperation with the recorder 2 is designated. When the cooperation is not designated, the control unit 10 moves the process to step S8 and determines whether the recording end operation is performed. If cooperation is designated, the control unit 10 transmits information requesting cooperation to the recorder 2 to collect sound in cooperation with the recorder 2 in step S5, and proceeds to step S6. ..

When the power is turned on, the control unit 20 of the recorder 2 determines whether or not the recording mode is designated in step S21 of FIG. When the recording mode is not designated, the control unit 20 shifts to a designated mode, for example, a collaborative mode for performing setting and transmission/reception for cooperation with the recorder 1 or a reproduction mode for reproducing recorded sound. .. Note that the control unit 20 may determine that the recording is instructed by transmitting the information indicating the recording start from the camera 1.

When the recording mode is designated, the control unit 20 starts recording the voice in the next step S22. That is, the control unit 20 causes the ST sound collecting unit 22 to start collecting sound, gives the collected sound as the second sound to the recording unit 25, and starts recording. It should be noted that the control unit 20 causes the clock unit 24 to start timekeeping and records the time information from the clock unit 24 at the same time when the voice is acquired.

In the next step S23, the control unit 20 determines whether or not the cooperation with the camera 1 is designated. When the cooperation is not designated, the control unit 20 shifts the processing to step S27 and determines whether or not the end of recording is designated.

When the cooperation is designated, the control unit 20 adds the cooperation information to the second sound and records the second sound in the recording unit 25 in step S24. The control unit 20 also controls the sound collection control unit 20a to perform recording in accordance with the sensitivity setting designated by the control unit 10 of the camera 1. Thereby, the sound collection control unit 20a performs sensitivity setting such as automatically setting the microphone sensitivity so that the recording level becomes appropriate, or increasing the sensitivity as the distance from the sound collection target to the recorder 2 becomes longer. ..

Next, the control unit 20 transmits a cooperation response to the control unit 10 of the camera 1. In step S6, the control unit 10 of the camera 1 receives the cooperation response from the control unit 20 of the recorder 2, adds the cooperation information to the first voice, and records it in the recording unit 14. In steps S6 and S26, the control unit 10 of the camera 1 and the control unit 20 of the recorder 2 transmit and receive the cooperation response to each other to establish the synchronization of the voices to be recorded. For example, a process of matching the time information of the camera 1 and the time information of the recorder 2 with a predetermined time reference is performed. This makes it possible to reliably cancel the arrival time difference by the time axis shift processing.

Note that the examples of FIGS. 5 to 7 show an example in which the camera 1 and the recorder 2 perform recording independently of each other, and the recorded audio data is transmitted to the reproducing device 3 to reproduce. , When transmitting the second sound picked up by the recorder 2 to the camera 1 as it is, neglecting the delay due to the communication, the synchronization between the first sound and the second sound has been established. The process of matching the information and the time information of the recorder 2 with a predetermined time reference can be omitted. Further, in the reproducing apparatus 3, there is some error between the time information of the camera 1 and the time information of the recorder 2 by performing the time axis shift processing using the waveform shapes of the first sound and the second sound. However, it is possible to reliably cancel the arrival time difference.

The distance and arrival time difference determination unit 10e of the control unit 10 determines the arrival time difference based on information such as the distance to the sound pickup target and the microphone distance in step S7 subsequent to step S6. The control unit 10 gives the information of the arrival time difference to the recording unit 14 and records the information in association with the first voice. As a result, the arrival time difference can be calculated successively even when the distance between the camera 1 and the recorder 2 is changed by taking an image while moving the camera 1. Therefore, it is possible to reliably cancel the arrival time difference by the time axis shift processing. It will be possible. In step S7, the generation timing estimation unit 10f estimates the generation timing of the target voice, and the estimation result is recorded in association with the first voice.

In step S8, the control unit 10 determines whether or not a recording operation or a recording ending operation has been performed. If the ending operation has not been performed, the control unit 10 returns the process to step S1. When the ending operation is performed, the control unit 10 determines whether or not the recorder cooperation is performed in the next step S9. When the cooperation with the recorder 2 is being performed, the control unit 10 proceeds from step S9 to step S10, performs the transmission indicating the termination of the collaboration to the recorder 2, and proceeds the post-processing to step S11. ..

The control unit 20 of the recorder 2 determines whether or not the end of recording is instructed in step S27 following step S26. When the controller 10 is notified of the termination of the cooperation or when the user's operation instructs the end of the recording, the control unit 20 determines that the end of the recording is instructed, and the next step S28. Then, the second voice is converted into a file. Note that time information is added to the second sound as metadata or is associated with the second sound and made into a file as an independent file. If the control unit 20 determines that the end of recording is not instructed, the process returns to step S21.

When the control unit 10 of the camera 1 determines in step S9 that the cooperation with the recorder 2 is not performed, or after the end of step S9, the control unit 10 moves to step S11 to save the recorded image and first audio file. And returns the process to step S1. In this case, the time information, the arrival time difference information, and the occurrence timing estimation result information are also associated with the first voice and are filed as metadata of the first voice or as an independent file.

(Playback)
Next, it is assumed that the first voice and the second voice picked up by the camera 1 and the recorder 2 are reproduced. For example, it is assumed that the playback device 3 plays back a moving image acquired by the camera 1. In step S31, the control unit 30 of the playback device 3 determines whether or not the linked moving image playback mode is designated. The cooperative moving image reproduction mode is a mode in which, when reproducing a moving image, the second sound is used when the first sound corresponding to the moving image is reproduced, and the sound is reproduced in cooperation with the first sound and the second sound.

When the linked moving image playback mode is not designated, the control unit 30 shifts to a designated mode such as a normal playback mode. When the linked moving image reproduction mode is designated, the control unit 30 of the reproduction device 3 receives the image and the first sound from the camera 1 via the communication unit 31, and receives the second sound from the recorder 2. To do. Time information is added to the image, the first sound, and the second sound, arrival time difference information and information on the estimation result of the sound generation timing are added to the first sound, and the control unit 30 links these information. Information is also received (step S31). The control unit 30 gives the received information to the recording unit 34 and records it.

The time-axis shift addition processing unit 30a of the control unit 30 reads the information recorded in the recording unit 34. The time axis shift addition processing unit 30a calculates the arrival time difference (time axis shift amount) based on the information recorded in the recording unit 34 (step S33). The time-axis shift addition processing unit 30a performs time-axis shift processing on at least one of the received first voice and second voice using the calculated time-axis shift amount, so that the target voice is the target sound pickup target. From the arrival time to the recorder 2 and the arrival time difference to reach the camera 1 are canceled, and the phases of the target voices are matched. Further, the time-axis shift addition processing unit 30a synthesizes the first and second voices after the time-axis shift processing to obtain a synthetic sound (step S34). Regarding the target voice, the first voice and the second voice have the same phase with respect to the target voice, so that they are mutually strengthened (the amplitude of the synthesized voice increases). The time axis shift addition processing unit 30a synchronizes the synthetic sound and the moving image using the time information.

The control unit 30 may give the moving image combined by the time-axis shift addition processing unit 30a to the reproducing unit 33 to reproduce the moving image. In this case, the target sound is emphasized and reproduced, and the user can clearly hear the cry of the bird 41 from the reproduced sound.

The time-axis shift addition processing unit 30a may adjust the time for delaying the sound in the time-axis shift processing according to the estimation result of the sound generation timing based on the image. The image and the sound acquired by the camera 1 are recorded in synchronization with each other, and by adjusting the delay time of the sound according to the estimation result of the sound generation timing, the time axis shift processing that reliably cancels the arrival time difference. Is possible.

FIG. 9 shows a captured image displayed on the display screen 16a. The example of FIG. 9 shows an example in which the playback device 3 is configured by the tablet terminal 3a, and an area 35 for displaying a captured image captured by the camera 1 is provided on the display screen 33a that configures the playback unit 33. The image 35 a of the bird 41 is displayed in the area 35. Further, on the display screen 33a, the microphone distance display 36 indicates that the distance to the microphone is 8 m, and the sound collection target distance display 37 indicates that the distance to the bird 41 that is the sound collection target is 10 m. ing.

The control unit 30 may give the synthesized sound from the time-axis shift addition processing unit 30a to the speaker of the reproduction unit 33 to reproduce and output the sound in synchronization with the image on the display screen 33a.

Further, in step S34, the control unit 30 causes the target voice extraction unit 30b to extract the target voice from the synthesized sound from the time-axis shift addition processing unit 30a, and then outputs the target voice to the reproduction unit 33 in synchronization with the captured image. Good. The target speech extraction unit 30b detects, for example, a frequency component that strengthens in a predetermined cycle among the frequency components of the synthetic sound obtained by the time-axis shift addition process, and performs a filtering process that extracts the frequency component. By this filtering process, only the target voice is extracted, and the target voice is further emphasized. As a result, the user can very clearly hear the bark of the bird 41 by the reproduced sound output from the speaker of the reproduction unit 33 as sound.

The control unit 30 records the synthesized sound and the image from the time-axis shift addition processing unit 30a in the recording unit 34 in association with each other. Further, the control unit 30 may record the synthesized sound and the image from the target voice extraction unit 30b in the recording unit 34 in association with each other.

As described above, in the present embodiment, the first microphone is arranged apart from the sound collection target by the first distance and the sound emitted from the sound collection target is collected to acquire the first sound from the first microphone. A second microphone that captures one voice, is arranged at a second distance different from the first distance from the sound collection target, and collects the sound emitted from the sound collection target to obtain the second sound. And an audio enhancement unit that performs enhancement processing for enhancing the component based on the distance difference between the first distance and the second distance of the first and second voices. It is possible to obtain a voice acquisition device characterized by the above.

That is, in the present embodiment, the target sound is picked up by the first and second microphones, and based on the distance difference between the distance from the sound pickup target to the first microphone and the distance to the second microphone. It is possible to emphasize the target voice by deriving the arrival time difference and shifting and synthesizing at least one of the first voice and the second voice picked up by the shift amount according to the arrival time difference by time-axis shifting. .. As a result, the target voice can be emphasized with respect to noise generated from a position other than the sound pickup target, and the sound pickup target can be clearly heard with high sound quality. Furthermore, only the target voice can be extracted by detecting the frequency components that reinforce each other in a predetermined cycle among the frequency components of the synthesized voice after the time-axis shift addition processing and performing the filter processing to extract the frequency components. It is also possible to further emphasize the target voice. Further, when the sound pickup target and the first and second microphones are arranged on a substantially straight line, the arrival time difference is calculated from the distance between the first microphone and the second microphone, Distance measurement etc. can be simplified.

(Second embodiment)
FIG. 10 is a block diagram showing the second embodiment of the present invention. 10, the same components as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted. This embodiment differs from the first embodiment in the emphasis processing in the reproducing apparatus.

Note that FIG. 10 also shows an example in which the voice acquisition device is configured by being distributed to a camera as an imaging device, a recorder, and a playback device, but in the present embodiment also, the voice acquisition device is configured in the camera. Alternatively, it may be configured as a recorder, may be configured as distributed in the camera and the recorder, and may be configured as a device independent of these devices.

In the first embodiment, in order to simplify the description, it is assumed that only the target voice and the noise due to the grass 42 are generated in the range in which the camera 1 and the recorder 2 can collect sound. However, in the first embodiment, the time base shift addition processing is executed by obtaining the arrival time difference based on the distance difference between the distance from the sound pickup target to the camera 1 and the distance to the recorder 2. The sound that can be picked up by the cooperation of the recorder 2 is all sounds that arrive with a difference in arrival time. For example, not only the sound emitted from all positions on the straight line connecting the camera 1 and the recorder 2 is emphasized as the target sound, but also the sound from all positions where the sound is picked up by the arrival time difference is targeted. It will be emphasized as voice.

Therefore, in the present embodiment, it is possible to acquire, as the target voice, only the sound emitted from a predetermined one position on the straight line connecting the camera 1 and the sound pickup target. Further, in the present embodiment, by defining the position of the recorder 2, it is possible to acquire only the sound emitted from one known place on the straight line connecting the camera 1 and the sound collection target as the target sound. is there.

In the first embodiment, a monaural microphone can be adopted as the ST sound collecting unit 12 of the camera 1, but in the present embodiment, the ST sound collecting unit 12 needs to be a stereo microphone. There is.

First, a sound collecting method in the second embodiment will be described with reference to FIGS. 11 to 13. FIG. 11 is an explanatory diagram showing the positional relationship among the bird 41, the grass 42, the recorder 2 and the camera 1 on the XY coordinates. FIG. 12 and FIG. 13 are explanatory diagrams for explaining the time lag of the collected voice, with the horizontal axis representing time and the vertical axis representing amplitude.

In the example of FIG. 11, the bird 41, the recorder 2, and the camera 1 are not arranged on a straight line. The ST sound collecting unit 12 is a stereo microphone having a right microphone 12R and a left microphone 12L. Similar to a stereo microphone used in a general digital camera or the like, the microphones 12R and 12L of the ST sound pickup unit 12 have the same directional characteristics along the optical axis VC of the image pickup unit 11 and have the same sensitivity. It is common. Therefore, of the right first voice and the left first voice obtained by picking up by the microphones 12R and 12L, the components of the voices emitted from the position on the optical axis VC of the imaging unit 11 are the same. ..

Therefore, the target sound from the bird 41 is acquired by the microphones 12R and 12L so that the bird 41 is located on the optical axis VC when the camera 1 captures the image, that is, when the bird 41 is captured in the center of the screen. The generated components A1R and A1L have the same waveform. For example, if the frequency of the target voice does not change in a predetermined short period, it may be considered that the frequency components having the same waveform (same phase) in this period are the components of the target voice, and the frequency component is extracted. It is possible to extract only the target voice from the left and right first voices (the left first voice and the right first voice) that are collected by performing the processing (hereinafter, referred to as the same phase component extraction processing). As for sounds emitted from all positions on the optical axis VC of the image pickup unit 11, the first right sound and the first left sound have the same phase. In other words, the sound obtained by the in-phase component extraction processing (hereinafter referred to as the third sound) can be considered as the sound emitted from the optical axis VC of the image pickup unit 11.

12 and 13 show that the sound of the bird 41 and the noise of the sound of the grass 42, which are the target sounds, are separated, respectively, and the third sound based on the sound collected by the camera 1 and the recorder 2 collect the sound. The waveform of the voice is shown. FIG. 12 shows the target voice. The components A1R and A1L of the target voice of the third voice at the position of the camera 1 with respect to the target voice A2 at the position of the recorder 2 are the arrival time delay and the amplitude based on the difference in distance. Has only a decrease in In the example of FIG. 12, the arrival time difference is To, and the target voice reaching the recorder 2 and the target voice component in the third voice are out of phase by an amount corresponding to the arrival time difference To.

Further, FIG. 13 shows the noise caused by the grass 42. The noise AN2 at the position of the recorder 2 and the noise components ANL and ANR in the third voice based on the voice picked up by the camera 1 have the arrival time delay and the amplitude change based on the difference in distance. In the example of FIG. 13, the arrival time difference is Tb, and the noise AN2 reaching the recorder 2 and the noises ANL and ANR in the third voice are out of phase by an amount corresponding to the arrival time difference Tb.

Therefore, also in the present embodiment, the sound obtained by the recorder 2 is delayed by the arrival time difference To and is added to the sound obtained by the camera 1. As a result, among the components included in the voices collected by the camera 1 and the recorder 2, the target voice having the arrival time difference To is added in a phase-matched state and strengthened (the amplitude increases).

By the way, the recorder 2 is arranged at an arbitrary position. In the present embodiment, recorder 2 may or may not be arranged on optical axis VC. Letting LA be the distance between the bird 41 and the recorder 2 that are the sound pickup targets, and let LC be the distance between the bird 41 and the camera 1, the arrival time difference of the target voice is calculated from the distance difference (LC-LA). be able to. Assuming that the recorder 2 is fixed at a predetermined position, the distance from the recorder 2 is LA, and the position on the optical axis VC in the direction away from the camera 1 is uniquely determined to the position of the bird 41. ..

Therefore, a time axis shift for delaying and adding at least one of the second sound picked up by the recorder 2 and the third sound obtained by the camera 1 with a delay time corresponding to the arrival time difference between the target sounds at the camera 1 and the recorder 2. By performing the addition processing, it is possible to emphasize the sound emitted from the position of the bird 41, that is, the target sound emitted from the sound collection target.

Further, by designating the position where the recorder 2 is installed, it is possible to emphasize only the sound emitted from the position at a certain distance to the camera 1 as the target voice.

The audio acquisition device in FIG. 10 employs a reproduction device 50 instead of the reproduction device 3. The playback device 50 employs, instead of the control unit 30, a control unit 51 in which the same phase component extraction processing unit 30c is added to the control unit 30. The control unit 51 may be configured by a processor using a CPU, an FPGA, or the like, and may be a unit that operates according to a program stored in a memory (not shown) to control each unit, or a hardware unit. An electronic circuit may realize some or all of the functions.

As described above, the camera 1 picks up an image while picking up the sound pickup target in the center of the screen. The captured image thus obtained from the camera 1, the first sound, the time information added to the first sound, the arrival time difference information, and the information of the estimation result of the sound generation timing are given to the reproducing apparatus 50 and are stored in the recording unit 34. To be recorded. Further, the second sound from the recorder 2 and the time information added to the second sound are also given to the reproducing apparatus 50 and recorded in the recording unit 34.

During reproduction, the in-phase component extraction processing unit 30c of the control unit 51 performs in-phase component extraction processing on the left audio and the right audio included in the first audio from the recording unit 34 to obtain the third audio. The in-phase component extraction processing unit 30c gives the third voice to the time-axis shift addition processing unit 30a. The time-axis shift addition processing unit 30a is configured to perform the time-axis shift addition processing by using the third voice instead of the first voice.

Next, the operation of the embodiment thus configured will be described with reference to FIG. FIG. 14 is a flowchart showing the control of the playback device. 14, the same steps as those in FIG. 7 are designated by the same reference numerals and the description thereof will be omitted.

The operations of the camera 1 and the recorder 2 are the same as those in FIGS. In the present embodiment, when capturing an image at the time of cooperation with the recorder, the user captures the sound pickup target in the center of the screen. As a result, of the voices picked up by the microphones 12R and 12L of the ST sound pickup unit 12, the waveforms of the left voice and the right voice are the same for the components of the voice emitted from the position on the optical axis of the image pickup unit 11. Becomes

In step S41 of FIG. 14, the reproducing device 50 acquires the third audio by the same phase component extraction process for the left and right first audio recorded in the recording unit 34. The third sound corresponds to the sound emitted from the position on the optical axis of the image pickup unit 11.

The time-axis shift addition processing unit 30a emphasizes the target voice by performing at least one of the third voice and the second voice by performing the time-axis shift processing with the shift amount based on the arrival time difference and adding the result. Further, the target voice extraction unit 30b performs a process of extracting the emphasized target voice.

The synthetic sound from the time-axis shift addition processing unit 30a or the target voice extraction unit 30b is given to the reproduction unit 33 and output as a sound.

Other actions are similar to those of the first embodiment.

As described above, in the present embodiment, the target voice is picked up by the stereo microphone to obtain the first voice. Then, frequency components having the same phase are extracted from the left and right first voices to obtain the third voice. The third voice corresponds to the sound emitted from the position on the straight line. Then, the arrival time difference is obtained based on the distance difference between the distance from the sound pickup target to the stereo microphone and the distance to the second microphone, and the third voice and the second voice acquired with the shift amount according to the arrival time difference. The target voice is emphasized by shifting at least one of the above and synthesizing it. As a result, the target voice can be reliably emphasized against noise generated from a position other than the sound pickup target located at a predetermined distance from the stereo microphone, and the sound pickup target can be clearly heard with high sound quality. Becomes

In addition, in the above-mentioned embodiment, it has been described that the sound is collected in a state where the sound collection target is captured in the center of the screen. However, for example, the arrival time difference between the left and right first voices can be calculated based on the position of the sound collection target on the screen or the position of the sound collection target acquired by the position and direction sensor unit 17. In this case, even when sound is not collected with the sound collection target being captured in the center of the screen, at least one of the left and right first sounds is delayed according to the arrival time difference, and then the same phase component extraction process is performed. As a result, it is possible to obtain the third sound based on the sound emitted from the predetermined position on the optical axis of the imaging unit 11. On the contrary, by delaying at least one of the left and right first sounds by the predetermined arrival time difference and performing the same phase component extraction process, the sound emitted from the predetermined angular direction with respect to the optical axis of the imaging unit 11 is detected. It is possible to acquire the third sound based on

(Third Embodiment)
FIG. 15 is a block diagram showing a voice acquisition device according to the third embodiment of the present invention. 15, the same components as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted.

In each of the above-described embodiments, the camera 1 and the recorder 2 acquire the first and second sounds and the cooperation information, and the reproducing apparatus 3 automatically emphasizes the target sound by using the acquired information. It was something to do. However, in this case, the information on the arrival time difference is required as the cooperation information, the distance between the camera 1 and the recorder 2 needs to be calculated, and depending on the sound pickup target, the positional relationship between the camera 1 and the recorder 2, It was also necessary to acquire the position of the sound collection target. Therefore, in each of the above-described embodiments, the camera 1 determines the distance to the recorder 2 and the distance and direction to the sound pickup target, and the positioning function of the recorder 2 is used to detect the sound pickup target and the camera. The positional relationship between 1 and recorder 2 was obtained.

On the other hand, in the present embodiment, the calculation of the arrival time difference can be omitted. In the present embodiment, the shift amount in the time axis shift process for the enhancement process is acquired by utilizing human hearing. Note that FIG. 15 shows an example applied to the first embodiment, but this embodiment is also applicable to the second embodiment of FIG.

In the present embodiment, a camera 60 is used instead of the camera 1 and the reproducing apparatus 3 is omitted, and the camera 60 has the function of the reproducing apparatus 3 built therein. The camera 60 includes a control unit 61. The control unit 61 may be configured by a processor using a CPU, an FPGA, or the like, may be a unit that operates according to a program stored in a memory (not shown) to control each unit, or an electronic circuit of hardware. May realize some or all of the functions.

The control unit 61 omits the distance/arrival time difference determination unit 10e and the occurrence timing estimation unit 10f from the control unit 10 of FIG. 1, and the time-axis shift addition processing unit 30a and the target voice extraction unit of the control unit 30 of FIG. The configuration of 30b is added. Further, the control unit 61 includes a shift amount adjusting unit 61a. Note that the camera 60 omits the image feature extraction unit 13 and the position and orientation sensor unit 17 from the camera 1 of FIG. 1, uses the reproduction unit 33 instead of the display unit 16, and records instead of the recording unit 14. The part 62 is adopted.

The recording unit 62 is configured by a recording medium (not shown) such as a hard disk or a memory medium, records the information given from the control unit 61, reads the recorded information and outputs it to the control unit 61. The recording unit 62 has a second audio information unit 14b in addition to the cooperation information unit 14a in which information about communication with the recorder 2 is recorded. The second voice information section 14b is adapted to record the second voice given from the recorder 2.

The shift amount adjusting unit 61a of the control unit 61 is given an operation signal based on a user operation of the operation unit 15. The shift amount adjustment unit 61a is configured to change the shift amount of the time axis shift processing in the time axis shift addition processing unit 30a according to the operation signal.

Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 16 to 18. FIG. 16 is an explanatory diagram showing a state during sound pickup and recording. 17 is a flowchart showing the control of the camera, and FIG. 18 is a flowchart showing the control of the recorder. 17 and 18, the same steps as those in FIG. 5 or 6 are designated by the same reference numerals and the description thereof will be omitted.

In this embodiment, when the camera 60 collects sound in cooperation with the recorder 2, the sound collected by the recorder 2 is transferred to the camera 60. That is, when the control unit 61 of the camera 60 determines in step S4 of FIG. 17 that recorder cooperation is designated, cooperation control communication is performed in step S5. When the camera cooperation is designated in step S23 of FIG. 18, the control unit 20 of the recorder 2 shifts the processing to step S61 and transmits the audio signal. That is, the second sound controlled by the sound collection control unit 20a and collected by the ST sound collection unit 22 is transmitted to the camera 60 via the communication unit 21a or 21b. The second voice may be supplied to the recording unit 25 and recorded. In step S51 subsequent to step S5, the control unit 61 of the camera 60 receives the second sound from the recorder 2 via the communication unit 18a or 18b.

In the present embodiment, the control unit 61 gives the received second voice to the second voice information unit 14b of the recording unit 62 to record the received second voice, and the first voice and the second voice currently recorded in the recording unit 62. The second voice being recorded in the information section 14b is supplied to the time-axis shift addition processing section 30a to execute the time-axis shift addition processing. For example, the shift amount of the time axis shift process may be 0 in the initial state. In this case, if the delay at the time of recording and the transmission delay of the second voice are ignored, the first voice and the second voice in the sound pickup are subjected to the time axis shift processing in the time axis shift addition processing section 30a. It will be synthesized as it is. The synthetic sound from the time-axis shift addition processing unit 30a is supplied to the reproducing unit 33 as it is, or is supplied to the reproducing unit 33 via the target voice extracting unit 30b. The synthetic sound from the time axis shift addition processing unit 30a or the target voice extraction unit 30b is supplied to the recording unit 14 and recorded therein.

FIG. 16 shows how the sound is collected and recorded. On the back surface of the housing 60a of the camera 60, a display unit 33b that constitutes the reproduction unit 33 is provided. The user, for example, grips the housing 30a with his/her hand and, while watching the display on the display unit 33b, captures the bird 41, which is a subject, in the field of view, and presses the shutter button 15a to perform shooting. .. FIG. 16 shows that the image 41a of the bird 41 is displayed on the display unit 33b. Further, a mode switching button 15c and a shift amount adjusting button 15b, which form the operation unit 15, are arranged on the upper surface of the housing 60a.

In the present embodiment, the reproduction unit 33 has headphones 33c that output the synthesized sound as sound. The reproducing unit 33 may have a speaker (not shown) built in the housing 60a, not limited to the headphones 33c, and output the synthesized sound as sound from the speaker. The user operates the button 15b while listening to the synthetic sound from the headphones 33c. An operation signal based on the operation of the button 15b is supplied to the shift amount adjustment unit 61a.

The shift amount adjustment unit 61a outputs an instruction to change the shift amount of the time axis shift processing to the time axis shift addition processing unit 30a according to the user's operation of the button 15b (step S52). The time axis shift addition processing unit 30a performs the time axis shift processing on at least one of the first voice and the second voice read from the recording unit 14 by the shift amount according to the instruction of the shift amount adjusting unit 61a. Then, the first voice and the second voice are added.

The shift amount of the time axis shift process changes according to the operation amount of the user's button 15b. The user operates the button 15b so that the sound heard from the headphones 33c is best heard as the sound considered to be the target sound. When the user determines that the target voice is best heard from the headphones 33c, that is, the target voice is most appropriately emphasized, the operation of the button 15b is stopped. The shift amount corresponding to the operation of the button 15b corresponds to the arrival time difference between the time when the target voice reaches the recorder 2 and the time when the target voice reaches the camera 60. That is, in the present embodiment, by utilizing the hearing of the user, it is possible to acquire the information on the arrival time difference without performing distance measurement or the like.

The camera 60 gives not only the first and second sounds but also the synthesized sound to the recording unit 62 and records it (step S53). In this case, the camera 60 may record the information regarding the arrival time difference in association with the synthetic sound.

Other functions are similar to those of the first and second embodiments.

As described above, in the present embodiment, the same effect as in the first and second embodiments can be obtained, and the shift amount in the time axis shift process for enhancing the target voice can be used as the human hearing. It is possible to omit the calculation of the arrival time difference and the processing such as distance measurement necessary for this calculation.

It should be noted that in the above-described embodiment, the description has been made assuming that the transmission delay of the second sound during the sound collection is ignored and the sound is recorded in the recording unit of the camera. Since the adjustment is performed, there is no particular problem even if there is a transmission delay.

If the first sound picked up by the camera and the second sound picked up by the recorder can be synchronized to some extent, the first and second sounds recorded in the recording section of the camera or the recording section of the camera can be recorded. The recorded first sound and the second sound recorded in the recording unit of the recorder are read out after recording, and are appropriately emphasized by adjusting the shift amount of the time-axis shift process by utilizing the hearing of the user during reproduction. It is also possible to hear the target voice.

In the above embodiment, a digital camera is used as an image capturing device, but the camera may be a digital single lens reflex camera, a compact digital camera, a video camera, a movie camera, or a mobile phone. Of course, a camera built in a personal digital assistant (PDA) such as a smartphone or a smart phone may be used.

The present invention is not limited to the above embodiments as they are, and can be embodied by modifying the constituent elements within a range not departing from the gist of the invention in an implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in each of the above embodiments. For example, some of all the constituent elements shown in the embodiment may be deleted. Furthermore, the constituent elements of different embodiments may be combined appropriately.

It should be noted that even if the description of the claims, the description, and the operation flow in the drawings is made by using “first,” “next,” and the like for convenience, it is essential that they are performed in this order. It does not mean. Further, it goes without saying that the steps constituting these operation flows can be appropriately omitted as long as they do not affect the essence of the invention.

Of the techniques described here, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording medium and the recording unit may be recorded at the time of product shipment, may be distributed recording medium, or may be downloaded via the Internet.

It should be noted that, in the embodiments, a portion described as a "section" (section or unit) may be configured by a dedicated circuit or a combination of a plurality of general-purpose circuits, and if necessary, preprogrammed software may be used. It may be configured by combining a microcomputer that operates according to the above, a processor such as a CPU, or a sequencer such as an FPGA. It is also possible to design such that a part or all of the control is taken over by an external device, and in this case, a wired or wireless communication circuit intervenes. The communication may be performed using Bluetooth, WiFi, a telephone line, or the like, and may be performed using USB or the like. A dedicated circuit, a general-purpose circuit, and a control unit may be integrated into an ASIC.

DESCRIPTION OF SYMBOLS 1... Camera, 10... Control part, 10a... Shooting control part, 10b... Image processing part, 10c... View angle information part, 10d... Sound collection control and processing part, 10e... Distance and arrival time difference determination part, 10f... Occurrence timing Estimating unit, 11... Imaging unit, 11a... Optical system, 12... ST sound collecting unit, 13... Image feature extracting unit, 14... Recording unit, 14a... Linkage information unit, 16... Display unit, 17... Position and direction sensor unit , 18... communication unit, 2... recorder, 20... control unit, 20a... sound collection control unit, 22... ST sound collection unit, 25... recording unit, 25a... cooperation information unit, 3... playback device, 30... control unit, 30a... Time-axis shift addition processing section, 30b... Target speech extraction section, 30c... In-phase component extraction processing section.

Claims (17)

A first microphone that is arranged at a first distance from the sound collection target and that collects the sound emitted from the sound collection target to obtain the first sound;
A second microphone arranged apart from the sound collecting target by a second distance different from the first distance to collect a sound emitted from the sound collecting target and obtain a second sound;
A voice acquisition device, comprising: a voice enhancement unit that performs enhancement processing for enhancing a component based on a distance difference between the first distance and the second distance of the first and second voices. An image acquisition unit that acquires the image of the sound collection target,
By the image analysis of the acquired image, a generation timing estimation unit that estimates the generation timing of the sound emitted from the sound collection target,
The voice acquisition device according to claim 1, wherein the voice enhancement unit adjusts the enhancement processing based on a result of estimation of a timing at which the sound is generated. The voice enhancement unit obtains a time difference in which the voice emitted from the sound collection target reaches the first microphone and the second microphone according to the distance difference between the first distance and the second distance. The voice acquisition according to claim 1, wherein the enhancement processing is performed by delaying and adding at least one of the first voice and the second voice with a delay time based on the obtained arrival time difference. apparatus. The voice acquisition device according to claim 3, wherein the voice enhancement unit obtains the arrival time difference based on a distance between the first microphone and the second microphone. The voice acquisition device according to claim 3, further comprising a distance and arrival time difference determination unit that obtains the arrival time difference by obtaining the distance difference between the first distance and the second distance. The voice emphasizing unit delays at least one of the first voice and the second voice with a delay time based on the arrival time difference, and adds the first and the second voices from a synthesized voice obtained by a time-axis shift addition process. The voice acquisition apparatus according to claim 3, further comprising a target voice extraction unit that extracts components to be emphasized by the time-axis shift addition processing for the second voice. The first microphone is a stereo microphone that picks up a left first sound and a right first sound,
It further comprises an in-phase component extraction processing unit for extracting, as a third voice, a component in which the left and right first voices picked up by the first microphone have the same phase.
The voice emphasizing unit performs an emphasizing process for emphasizing a component based on a distance difference between the first distance and the second distance in the third sound and the second sound instead of the first sound. The voice acquisition device according to claim 1. An image acquisition unit for acquiring the captured image of the sound collection target,
The voice acquisition according to claim 7, wherein the same-phase component extraction processing unit extracts the third voice in a state in which the image acquisition unit positions the sound collection target at the center of the captured image. apparatus. The voice emphasizing unit is based on a time difference in which a voice emitted from the sound collection target reaches the first microphone and the second microphone according to a distance difference between the first distance and the second distance. A time axis shift addition processing unit that delays and adds at least one of the first voice and the second voice with a delay time to obtain a synthesized voice;
A playback unit for playing back the above synthetic sound,
The voice acquisition device according to claim 1, further comprising a shift amount adjustment unit that adjusts the delay time according to an operation signal based on a user operation. A first microphone arranged at a first distance from the sound collecting target collects a sound emitted from the sound collecting target to obtain a first sound,
A second microphone arranged apart from the sound collecting target by a second distance different from the first distance collects a sound emitted from the sound collecting target to obtain a second sound,
A voice acquisition method characterized by performing an emphasis process for emphasizing a component based on a distance difference between the first distance and the second distance among the first and second voices. The time difference in which the sound emitted from the sound pickup target reaches the first microphone and the second microphone is calculated according to the distance difference between the first distance and the second distance, and the calculated arrival time difference is obtained. 11. The voice acquisition method according to claim 10, wherein the enhancement processing is performed by delaying and adding at least one of the first voice and the second voice with a delay time based on the delay time. The first microphone picks up a left first sound and a right first sound,
A component in which the phases of the left and right first sounds picked up by the first microphone are the same is extracted as the third sound,
11. The emphasizing process for emphasizing a component based on a distance difference between the first distance and the second distance of the third voice and the second voice instead of the first voice is performed. The voice acquisition method described. The delay time based on the time difference between the sound emitted from the sound collection target and the first microphone and the second microphone according to the distance difference between the first distance and the second distance. At least one of the voice and the second voice is delayed and added to obtain a synthetic sound,
Play the above synthetic sound,
The voice acquisition method according to claim 10, wherein the delay time is adjusted by an operation signal based on a user operation. On the computer,
A first microphone arranged at a first distance from the sound collecting target collects a sound emitted from the sound collecting target to obtain a first sound,
A second microphone arranged apart from the sound collecting target by a second distance different from the first distance collects a sound emitted from the sound collecting target to obtain a second sound,
A voice acquisition program for executing a procedure of performing an emphasis process for emphasizing a component based on a distance difference between the first distance and the second distance among the first and second voices. The time difference in which the sound emitted from the sound pickup target reaches the first microphone and the second microphone is calculated according to the distance difference between the first distance and the second distance, and the calculated arrival time difference is calculated. 15. The voice acquisition program according to claim 14, which executes a procedure of performing the emphasizing process by delaying and adding at least one of the first voice and the second voice with a delay time based on the delay time. The first microphone picks up a left first sound and a right first sound,
A component in which the phases of the left and right first sounds picked up by the first microphone are the same is extracted as the third sound,
15. A procedure for executing an emphasizing process for emphasizing a component based on a distance difference between the first distance and the second distance of the third sound and the second sound instead of the first sound. The voice acquisition program described in. The delay time based on the time difference between the sound emitted from the sound collection target and the first microphone and the second microphone according to the distance difference between the first distance and the second distance. At least one of the voice and the second voice is delayed and added to obtain a synthetic sound,
Play the above synthetic sound,
15. The voice acquisition program according to claim 14, which executes a procedure of adjusting the delay time by an operation signal based on a user operation.

Images