JP2011035708A - Acoustic signal processor, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide: an acoustic signal processor capable of recording or reproducing a speaker sound by appropriately controlling a sound image direction of the speaker sound; or an imaging apparatus which records the speaker sound by appropriately controlling the sound image direction of the speaker sound when an acoustic signal of sound to be generated from the speaker is detected in the acoustic signal when an image signal and the acoustic signal are recorded by being associated with each other or when the image signal and the acoustic signal recorded by being associated with each other are reproduced. <P>SOLUTION: A speaker sound control section 10 separates and extracts sound from a plurality of sound source collected by microphones 5L and 5R for every sound source, and determines whether or not the separated and extracted sound from each sound source is the speaker sound. When it is determined that the sound is the speaker sound, the sound image direction is controlled so that a sound image from the sound source substantially matches to a photographing direction by the imaging apparatus 1. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an acoustic signal processing apparatus that processes an acoustic signal, and more particularly, to an acoustic signal processing apparatus or an imaging apparatus that controls a sound image direction of an acoustic signal emitted from a speaker.

  In places where people gather, such as lectures and various events, speakers often use a microphone to speak. The speaker's voice input to the microphone is amplified by an amplifier to which the microphone is connected, and is output at a loud volume from a speaker connected to the amplifier. Therefore, most of the speaker's voice comes from the speaker.

  A plurality of speakers are connected to the amplifier, and these speakers are arranged asymmetrically around the position of the speaker, or there is only one speaker, but this speaker is completely different from the position of the speaker. In such a case, the speaker's voice is heard from a position different from the position where the speaker exists.

  For example, an image and sound of such a scene are recorded with a video camera having two stereo microphones so that the speaker is positioned near the middle of the shooting area, and the recorded image signal and sound signal are reproduced and viewed. Think about the case. For example, when the playback device performs stereo playback using two speakers for the L channel and the R channel, a scene where the speaker is speaking is shown near the center of the monitor of the playback device. The speaker's voice can only be heard from one of the speakers, or can be heard biased to either one. Such reproduction of images and sounds is a problem because it makes viewers who are watching the image and sound feel very uncomfortable.

  Also, in sports events, BGM may only flow from one speaker. When recording and viewing images and sounds of such scenes, BGM can only be heard from one of the speakers, or Since it can be heard in either direction, it lacks a sense of power and is a problem.

  As a conventional technique for controlling the entire sound image in accordance with an image, there is Patent Document 1 below. In the following Patent Document 1, when reproducing an image signal and a sound signal acquired at the time of shooting, the directivity of a sound signal to be reproduced is controlled in accordance with the angle of view of the image signal to be reproduced and the reproduction sound field is corrected. Yes, it does not solve the above problem.

JP2006-287544

  The present invention has been made in view of the above problems, and when an image signal and an audio signal are recorded in association with each other, or when an image signal and an audio signal recorded in association with each other are reproduced, When the sound signal of the sound emitted from the speaker is detected, the sound signal processing device capable of appropriately recording and reproducing the sound image direction of the speaker sound, or the sound image direction of the speaker sound appropriately An object of the present invention is to provide an imaging apparatus that performs control and recording.

  A first acoustic signal processing apparatus according to the present invention includes a sound collecting means for collecting a sound arriving at the time of photographing to acquire an acoustic signal of the sound, and a speaker for detecting a speaker sound signal from the acoustic signal. Sound detection means, sound image direction control means for performing sound signal processing on the sound signal so that the sound image direction of the speaker sound signal coincides with the shooting direction when the speaker sound signal is detected, and the sound signal processing And acoustic signal recording means for recording the acoustic signal subjected to.

  A second acoustic signal processing apparatus according to the present invention includes a sound collecting means for collecting an incoming sound at the time of shooting to acquire an acoustic signal of the sound, and a speaker for detecting a speaker sound signal from the acoustic signal. Sound detection means, sound image direction control means for applying sound signal processing to the sound signal so that the sound image direction of the sound signal coincides with the shooting direction when the speaker sound signal is detected, and the sound signal processing. And an acoustic signal recording means for recording the applied acoustic signal.

  An imaging device according to the present invention includes the first or second acoustic signal processing device, and obtains an image signal of the imaging target by imaging the imaging target, and the imaging unit acquired by the imaging unit And image signal recording means for recording the image signal in association with the acoustic signal subjected to the acoustic signal processing by the sound image direction control means provided in the acoustic signal processing device.

  The imaging apparatus according to the present invention further includes face detection means for detecting a human face image signal from the image signal, and microphone detection means for detecting a microphone image signal from the image signal, and the acoustic signal processing apparatus The speaker sound detection means included in the sound collecting device provided in the acoustic signal processing apparatus when a face image signal of a person is detected by the face detection means and a microphone image signal is detected by the microphone detection means. A speaker sound signal is detected from an acoustic signal acquired by the means.

The image pickup apparatus according to the present invention includes an image pickup unit that acquires an image signal of the shooting target by shooting the shooting target, and a first sound of the sound by collecting the sound that arrives at the time of shooting. A sound collecting means for acquiring a signal, a first recording means for recording the first acoustic signal in association with the image signal acquired by the imaging means, and a second acoustic signal based on the first acoustic signal. A second recording means for recording the second acoustic signal in association with the image signal acquired by the imaging means; and a switching means for switching between the first recording means and the second recording means. (2) recording means for detecting a speaker sound signal from the acoustic signal; and when the speaker sound signal is detected, the recording means is arranged so that the sound image direction of the speaker sound signal coincides with the shooting direction. A sound image direction control means for generating the second audio signal by performing audio signal processing over mosquito sound signal,
It is provided with.

Furthermore, the imaging apparatus according to the present invention includes an imaging unit that acquires an image signal of the shooting target by shooting the shooting target, and a first of the sounds by collecting the sound that arrives at the time of shooting. Sound collecting means for acquiring an acoustic signal, first recording means for recording the first acoustic signal in association with the image signal acquired by the imaging means, and generating a second acoustic signal based on the first acoustic signal And a second recording means for recording the second acoustic signal in association with the image signal acquired by the imaging means, and a switching means for switching between the first recording means and the second recording means,
The second recording means includes speaker sound detection means for detecting a speaker sound signal from the sound signal, and when the speaker sound signal is detected, the sound signal direction of the sound signal matches the shooting direction. And a sound image direction control means for generating the second acoustic signal by performing acoustic signal processing on the signal.

  In the third acoustic signal processing apparatus according to the present invention, the image signal acquired by photographing by the photographing means and the acoustic signal of the sound that arrives at the time of photographing obtained by the sound collecting means are recorded in association with each other. Acquisition means for acquiring the acoustic signal from the recording means, speaker sound detection means for detecting a speaker sound signal from the acoustic signal acquired by the acquisition means, and when the speaker sound signal is detected, the speaker Sound image direction control means for performing acoustic signal processing on the acoustic signal so that a sound image direction of the sound signal matches a photographing direction by the photographing means, and reproducing means for reproducing the acoustic signal subjected to the acoustic signal processing. , Provided.

  In the fourth acoustic signal processing apparatus according to the present invention, the image signal acquired by the photographing by the photographing means and the acoustic signal of the sound arriving at the photographing obtained by the sound collecting means are recorded in association with each other. Acquisition means for acquiring the acoustic signal from the recording means, speaker sound detection means for detecting a speaker sound signal from the acoustic signal acquired by the acquisition means, and when the speaker sound signal is detected, the sound Sound image direction control means for performing acoustic signal processing on the acoustic signal such that a sound image direction of the signal matches a photographing direction by the photographing means, and reproducing means for reproducing the acoustic signal subjected to the acoustic signal processing; It is provided with.

  According to the present invention, when an image signal and an audio signal are recorded in association with each other, or when an image signal and an audio signal recorded in association with each other are reproduced, an audio signal of a sound emitted from a speaker is detected in the audio signal. In such a case, an acoustic signal processing device that appropriately records and reproduces the sound image direction of the speaker sound, or an imaging device that appropriately records and reproduces the sound image direction of the speaker sound is provided. be able to.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. Absent.

1 is an overall configuration diagram of an imaging apparatus according to an embodiment of the present invention. It is a figure explaining the outline | summary of the processing content of the speaker sound control part. 3 is a block diagram illustrating an outline of an internal configuration of a speaker sound control unit 100. FIG. It is a figure for demonstrating the method the direction determination part 102 calculates the direction from which an acoustic signal comes. It is a block diagram which shows the outline of the internal structure of the speaker sound control part 200 with which the acoustic process part 7 is provided. It is a figure for demonstrating the method to determine whether the acoustic signal of the nth frame has periodicity. It is a figure which shows the relationship between the autocorrelation value S (P) of the acoustic signal of the nth frame, and the variable P.

  Hereinafter, an embodiment in which an acoustic signal processing device according to the present invention is implemented in an imaging device will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating an outline of an internal configuration of an imaging apparatus according to an embodiment of the present invention. As shown in FIG. 1, an imaging device 1 includes an image sensor 2 including a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complimentary Metal Oxide Semiconductor) sensor that converts an incident optical image into an electrical signal. And a lens unit 3 that forms an optical image of a subject on the image sensor 2 and adjusts the amount of light. The lens unit 3 and the image sensor 2 constitute an imaging unit, and an analog image signal is generated by the imaging unit. The lens unit 3 includes various lenses (not shown) such as a zoom lens and a focus lens, and a diaphragm (not shown) that adjusts the amount of light input to the image sensor 2.

  Furthermore, the imaging apparatus 1 converts an image signal that is an analog signal output from the image sensor 2 into a digital image signal (hereinafter, the digital image signal may be simply referred to as an image signal) and adjusts the gain. AFE (Analog Front End) 4 to be performed, microphones (hereinafter simply referred to as microphones) 5L and 5R for converting input sound into electric signals and collecting sound, and analog sound output from the microphones 5L and 5R ADC (Analog to Digital Converter) 6L and 6R for converting a signal into a digital sound signal (hereinafter, digital sound signal may be simply referred to as a sound signal), and sound signals output from ADC 6L and 6R The sound processing unit 7 that performs various sound signal processing and outputs the image signal, and the image signal that is output from the AFE 4 performs various image signal processing and outputs the image signal. An image processing unit 8 is provided.

  Here, the sound processing unit 7 describes the sound collected by the microphones 5L and 5R from the speaker (hereinafter, the sound emitted from the speaker is the speaker sound, and the sound signal of the speaker sound is described as the speaker sound signal. .) Is detected, and when a speaker sound is detected, the sound image direction of the speaker sound is controlled, or the sound image direction of the entire sound collected by the microphones 5L and 5R is controlled. A sound control unit is provided. Details of the speaker sound control unit will be described later.

  Further, the imaging apparatus 1 performs compression encoding processing such as MPEG (Moving Picture Experts Group) compression method on the image signal output from the image processing unit 8 and the sound signal output from the sound processing unit 7. The processing unit 9, an external memory 11 that records the compressed encoded signal compressed and encoded by the compression processing unit 9, a driver unit 10 that records and reads the compressed encoded signal in the external memory 11, and the driver unit 10 And a decompression processing unit 12 that decompresses and decodes the compressed encoded signal read from the external memory 11.

  In addition, the imaging apparatus 1 includes an image signal output unit 13 that converts the image signal decoded by the expansion processing unit 12 into a signal that can be displayed on the display unit 21 such as a monitor, and the sound decoded by the expansion processing unit 12. And an acoustic signal output unit 14 that converts the signal into a signal in a format that can be output by the speaker unit 22.

  The imaging apparatus 1 also stores a CPU (Central Processing Unit) 15 that controls the overall operation of the imaging apparatus 1 and a memory 16 that stores each program for performing each process and temporarily stores a signal when the program is executed. A timing generator (TG) that outputs a timing control signal for matching the operation timing of the operation unit 17 to which an instruction from a photographer such as a button for starting shooting and a button for determining various settings is input. ) Unit 18, a bus 19 for exchanging signals between the CPU 15 and each unit, and a bus 20 for exchanging signals between the memory 16 and each unit.

  The external memory 11 may be anything as long as it can record image signals and sound signals. For example, a semiconductor memory such as an SD (Secure Digital) card, an optical disk such as a DVD, a magnetic disk such as a hard disk, or the like can be used as the external memory 11. Further, the external memory 11 may be detachable from the imaging device 1.

  Next, the basic operation of the imaging apparatus 1 will be described with reference to FIG. First, the imaging apparatus 1 generates an analog image signal that is an electrical signal by photoelectrically converting light incident from the lens unit 3 in the image sensor 2. In synchronization with the timing control signal input from the TG unit 18, the image sensor 2 sequentially outputs analog image signals generated to the AFE 4 at a predetermined frame period (for example, 1/30 second). Then, the image signal converted from the analog signal to the digital signal by the AFE 4 is input to the image processing unit 8. In the image processing unit 8, the image signal is converted into a signal using YUV, and various image signal processes such as gradation correction and contour enhancement are performed. The memory 16 operates as a frame memory, and temporarily holds an image signal when the image processing unit 8 performs processing.

  The microphones 5L and 5R collect sound, convert it into an analog acoustic signal that is an electrical signal, and output it. Analog audio signals output from the microphones 5L and 5R are input to the ADCs 6L and 6R and converted into digital audio signals. Furthermore, the acoustic signals from the ADCs 6L and 6R are input to the acoustic processing unit 7 and subjected to various acoustic signal processing such as noise removal and speaker sound control by the speaker sound control unit.

  Both the image signal output from the image processing unit 8 and the acoustic signal output from the acoustic processing unit 7 are input to the compression processing unit 9 and compressed by the compression processing unit 9 using a predetermined compression method. At this time, the image signal and the sound signal are temporally associated (paired), and are configured so that the image and the sound are not shifted during reproduction. The compressed image signal and sound signal are recorded in the external memory 11 via the driver unit 10.

  The compressed image signal and sound signal recorded in the external memory 11 are read out to the decompression processing unit 12 based on the reproduction instruction of the photographer input via the operation unit 17. The decompression processing unit 12 decompresses the compressed image signal and sound signal read out for reproduction, the image signal for reproduction to the image signal output unit 13, and the sound signal for reproduction to the sound signal output unit 14, respectively. Output. The image signal output unit 13 converts the image signal for reproduction into a signal in a format that can be displayed on the display unit 21, and the sound signal output unit 14 can output the sound signal for reproduction by the speaker unit 22. Convert to a format signal and output each. As a result, an image for reproduction is displayed on the display unit 21, and a sound for reproduction is output from the speaker unit 22.

  In addition, the imaging apparatus 1 according to the present embodiment displays the captured image on the display unit 21 before recording of the captured image is started or when a moving image is recorded. At this time, the image processing unit 8 generates a display image signal and outputs it to the image signal output unit 13 via the bus 20. Then, the image signal output unit 13 converts the image signal for display into a signal in a format that can be displayed on the display unit 21 and outputs the signal. The photographer can recognize the angle of view of the image to be recorded or currently recorded by checking the image displayed on the display unit 21.

  The display unit 21 and the speaker unit 22 may be integrated with the imaging device 1 or may be separate from each other and connected to a terminal provided in the imaging device 1 using a cable or the like. It does n’t matter. In addition, the microphones 5L and 5R may include a digital microphone that outputs a digital acoustic signal, and the ADC 6 may not be included.

<< First Example >>
Hereinafter, a first embodiment of the speaker sound control unit included in the acoustic processing unit 7 of the imaging apparatus 1 will be described. In the following description of the first embodiment, the speaker sound control unit is assigned the number 100.

  FIG. 2 is a diagram for explaining the outline of the processing contents of the speaker sound control unit 100. In FIG. 2, the photographer is photographing a speaker with a microphone using the imaging device 1. When the speaker speaks into the microphone, the voice is emitted from a speaker arranged at a position different from that of the speaker. That is, in FIG. 2, the speaker that emits the speaker's voice is the sound source P. On the other hand, it is assumed that there is also a sound source Q that emits sound other than speaker sound.

  The speaker sound control unit 100 separates and extracts sounds from a plurality of sound sources (two sound sources of the sound source P and the sound source Q in FIG. 2) collected by the microphones 5L and 5R for each sound source. And it is determined whether the sound from each sound source separated and extracted is a speaker sound. When it is determined that the sound is a speaker sound, the direction of the sound image is controlled so that the sound image from the sound source substantially coincides with the shooting direction of the image pickup apparatus 1. Here, the shooting direction refers to a direction in which the lens unit 3 of the imaging device 1 faces when shooting with the imaging device 1. In FIG. 2, there is a speaker with a microphone in the shooting direction, but there is no sound source P (speaker) and sound source Q.

  When the speaker sound control unit 100 determines that the sound from the sound source P is the speaker sound among the sounds from the sound source P and the sound source Q, it is as if the speaker sound is coming from the shooting direction of the imaging device 1. The sound signal processing is performed on the sound signal of the speaker sound. As a result of such processing, when the image signal and the sound signal acquired by shooting are reproduced and viewed, the viewer can hear the speaker's voice from the shooting direction, that is, the direction in which the speaker is present. I don't feel anymore.

  FIG. 3 is a block diagram illustrating an outline of the internal configuration of the speaker sound control unit 100. In FIG. 3, the acoustic signals output from the ADCs 6L and 6R are signals in the time domain, and when the elapsed time from a certain reference time is t (t is an integer), the acoustic signal is expressed as a function of t. it can. Hereinafter, the acoustic signals output from the ADCs 6L and 6R are referred to as an original signal Li (t) and an original signal Ri (t), respectively.

  FFT (Fast Fourier Transform) sections 101L and 101R respectively perform discrete Fourier transform on the original signals Li (t) and Ri (t) to generate a frequency spectrum. The frequency spectrum output from each of the FFT units 101L and 101R is obtained by converting acoustic signals output as signals on the time domain from the ADCs 6L and 6R into signals on the frequency domain. Therefore, the frequency spectrum can be expressed as a function of the frequency f (f is a positive integer). Hereinafter, the frequency spectra output from the FFT units 101L and 101R are referred to as frequency spectra L (f) and R (f), respectively.

  In the imaging apparatus 1 according to the present embodiment, the ADCs 6L and 6R each convert an analog acoustic signal into a digital acoustic signal at a sampling frequency of 48 kHz (kilohertz), for example. Then, the imaging apparatus 1 uses the generated acoustic signal 1024 samples, that is, approximately 21.3 msec (1024 × 1/48 kHz) as one frame, and performs acoustic signal processing on the acoustic signal in units of this frame.

  The FFT units 101L and 101R perform discrete Fourier transform on the acoustic signal in units of one frame. At this time, the frequency band of the acoustic signal is subdivided into M (M is an integer of 2 or more) with a sampling interval of Δf, and a frequency spectrum is calculated for each subdivided frequency band. Hereinafter, the subdivided frequency band is referred to as a subdivided band. For example, if the total frequency band of the acoustic signal is ΔF, the number M of subdivided bands is M = ΔF / Δf. Here, ideally, by narrowing the sample interval Δf, each of the subdivided bands can include only the component of the acoustic signal from one sound source. That is, the acoustic signal included in each subdivided band can be considered as a component of the acoustic signal of the sound emitted from any one of a plurality of sound sources.

  Assuming that the plurality of subdivided bands are f0, f1, f2,..., Fm-1 (m is an integer of 1 or more), the frequency spectra L (f) and R (f) are subdivided bands f0, .., fm-1 (m is an integer of 1 or more). Hereinafter, the frequency spectrums of the subdivided bands constituting the frequency spectra L (f) and R (f) are denoted by L (f0), L (f1), L (f2),... L (fm-1), respectively. And R (f0), R (f1), R (f2),... R (fm-1).

  The direction determination unit 102 calculates the phase difference when the acoustic signal included in each subband reaches the microphones 5L and 5R from the frequency spectrum of each subband output from each of the FFT units 101L and 101R. Based on this phase difference, the arrival direction of the acoustic signal included in each subdivided band is determined.

  FIG. 4 is a diagram for explaining a method by which the direction determination unit 102 calculates the direction in which the acoustic signal arrives. Now, a two-dimensional coordinate plane is assumed with the X axis and Y axis orthogonal to each other as coordinate axes. The X axis and the Y axis are orthogonal at the origin O. With the origin O as a reference, the X-axis positive direction is the right side, the negative direction is the left side, the Y-axis positive direction is the front, and the negative direction is the rear. It is assumed that the microphones 5L and 5R are arranged at different positions on the X axis and are symmetrical with respect to the Y axis, and the distance between the two microphones is D. The interval D is, for example, about several mm.

  Now, for example, an acoustic signal included in the subdivided band of f0 Hz is emitted from the sound source P, and the incident angle when the acoustic signal arrives at the origin O is assumed to be positive with a counterclockwise rotation centered on the origin. (Rad) (radian). At this time, the incident angle to the microphones 5L and 5R can also be approximated to θ (rad). Assuming that the phase difference when the acoustic signal reaches the microphones 5L and 5R is Δφ (rad), Δφ is calculated from the frequency spectra L (f0) and R (f0) output from the FFT units 101L and 101R, respectively. be able to.

  Specifically, if the real part of the frequency spectrum L (f0) calculated by the discrete Fourier transform is L_r (f0) and the imaginary part is L_i (f0), the phase φl of L (f0) is

Can be calculated.

  Similarly, if the real part of the frequency spectrum R (f0) is R_r (f0) and the imaginary part is R_i (f0), the phase φr of R (f0) is

Can be calculated.

  Here, since the phase difference Δφ can be calculated as Δφ = φr−φl, it can be calculated by the following equation (1).

  Further, if the sound velocity is C (mm / sec) and the distance between the microphones 5L and 5R is D (mm), Δφ can also be calculated from the following equation (2).

  Therefore, the incident angle θ can be calculated from the following equation (3) from the above equations (1) and (2).

  As described above, the incident angle θ that is the direction in which the acoustic signal included in the subdivided band f0 Hz arrives can be calculated. In this way, the direction determination unit 102 calculates the incident angles of the acoustic signals included in all the subdivided bands. Hereinafter, the incident angle of the acoustic signal included in the subdivided band is simply referred to as the incident angle of the subdivided band.

  In the present embodiment, the direction determination unit 102 outputs all the sets to the speaker sound determination unit 103 by setting the frequency spectrum of the subdivided band of the frequency spectrum L (f) and the incident angle of the subdivided band as one set. Note that the frequency spectrum of the subband of the frequency spectrum R (f) and the incident angle of the subband may be output as a set.

  For example, the incident angles of the subdivided bands f0, f1, f2, f3, f4, f5, f6, f7, f8, and f9 are θ0, θ1, θ0, θ0, θ1, θ1, θ1, θ2, θ2, and θ2, respectively. Assuming that (rad), the direction determination unit 102 determines that (L (f0), θ0), (L (f1), θ1), (L (f2), θ0), (L (f3), θ0), (L (f4), θ1), (L (f5), θ1), (L (f6), θ1), (L (f7), θ2), (L (f8), θ2), (L (f9) , Θ2) is output to the speaker sound determination unit 103.

  The speaker sound determination unit 103 extracts the frequency spectrum of the subdivided band for each incident angle from the set of the frequency spectrum of the subdivided band output from the direction determination unit 102 and the incident angle of the subdivided band and combines them. And a synthesized frequency spectrum is generated. That is, a synthesized frequency spectrum for each direction of arrival of the acoustic signal is generated. Hereinafter, the synthesized frequency spectrum generated by synthesizing the frequency spectrum of the subdivided band coming from the incident angle θ will be referred to as L (θ).

  For example, the speaker sound determination unit 103 receives (L (f0), θ0), (L (f1), θ1), (L (f2), θ0), (L (f3), θ0) from the direction determination unit 102. , (L (f4), θ1), (L (f5), θ1), (L (f6), θ1), (L (f7), θ2), (L (f8), θ2), (L (f9) ), Θ2), the frequency spectrum L (f0), L (f2), L (f3) of the subdivided band with the incident angle θ0 is extracted and synthesized to generate a synthesized frequency spectrum L (θ0). To do.

  Similarly, L (f1), L (f4), L (f5), and L (f6) are extracted and combined to obtain a combined frequency spectrum L (θ1) from the incident angle θ1. Further, L (f7), L (f8), and L (f9) are extracted and combined to obtain a combined frequency spectrum L (θ2) from the incident angle θ2.

  The speaker sound determination unit 103 determines whether the digital sound signal from each direction of arrival is a speaker sound signal from the characteristics of the combined frequency spectrum of each direction of arrival calculated in this way.

  Generally, the frequency band of an acoustic signal that can be reproduced by a speaker used in an event such as a seminar or athletic meet is often in the range of about 300 Hz to 6 kHz. Therefore, if the frequency spectrum of the acoustic signal in each direction of arrival is within a range of approximately 300 Hz to 6 kHz, it is determined that the possibility of speaker sound is high.

  Further, the frequency spectrum of the human voice is generally concentrated in the range of about 100 Hz to 4 kHz. The voiced sound has a harmonic structure composed of harmonic components thereof with a pitch frequency in a relatively low frequency band. Here, the pitch frequency is a fundamental frequency of a human voice emitted by vocal fold vibration, and usually exists in a range of about 100 Hz to 300 Hz. Therefore, if the pitch frequency is fp, the frequency spectrum of the human voice shows a characteristic that takes a maximum value at fp, 2fp, 3fp,... NfpHz (n is a positive integer).

  On the other hand, as described above, since the reproducible frequency band of the speaker used in events such as workshops and athletic meet is about 300 Hz to 6 kHz, if the speaker sound contains a human voice, its frequency spectrum Does not include the spectrum of the pitch frequency and shows frequency characteristics having a harmonic structure.

  For example, the speaker sound determination unit 103 performs autocorrelation on the synthesized frequency spectrum and determines whether or not the synthesized frequency spectrum includes a pitch frequency spectrum and whether or not the synthesized frequency spectrum has a harmonic structure. It is determined whether or not a voice speaker sound signal is included.

  Specifically, the speaker sound determination unit 103 first performs autocorrelation on the combined frequency spectrum L (θ) and detects a plurality of maximum values. Assume that the synthesized frequency spectrum L (θ) does not include a spectrum in the range of about 100 Hz to 300 Hz, and the frequencies at which the maximum values are obtained are, for example, fm1 = 300 Hz, fm2 = 450 Hz, fm3 = 600 Hz,.

  Here, assuming that the first frequency fm1 having the maximum value is twice the pitch frequency fp, that is, 300 Hz = 2 fp, the pitch frequency fp is fp = 150 Hz. Furthermore, if the synthesized frequency spectrum L (θ) includes the frequency spectrum of an audio signal having a pitch frequency fp = 150 Hz, L (θ) should have a maximum value at 2fp = 300 Hz, 3fp = 450 Hz, 4fp = 600 Hz. It is.

  Now, since the relationship of fm1 = 2fp, fm2 = 3fp, fm3 = 4fp is satisfied, in this case, the speaker sound determination unit 103 determines that L (θ) is a speaker sound signal of an audio signal whose pitch frequency is 150 Hz. It is determined that the frequency spectrum is included.

  In this way, the speaker sound determination unit 103 determines whether or not they include the frequency spectrum of the speaker sound signal of the sound signal for all the synthesized frequency spectra L (θ).

  In this embodiment, the frequency band of the acoustic signal for each direction of arrival is included in the reproducible band of speaker sound (in the range of about 300 Hz to 6 kHz) used in events such as workshops and athletic meet, and the audio signal Is included, it is determined that the sound signal is a speaker sound signal. Otherwise, the speaker sound signal is not determined even if the frequency band is 300 Hz to 6 kHz. Thereby, it is possible to accurately detect a speaker sound including a human voice and control the direction of the sound image.

  The speaker sound determination unit 103 notifies the gain adjustment unit 104 of the subband frequency including the speaker sound signal.

  The gain adjustment unit 104 adjusts the frequency spectrum corresponding to the frequency of the subdivided band including the speaker sound signal notified from the speaker sound determination unit 103 so that L (f) and R (f) have the same level.

  Specifically, for example, when it is notified that the subdivided band f0 Hz includes a speaker sound signal, L (f0) = VL, R (f0) = VR, and VL> VR, the gain adjustment The unit 104 adjusts the gain so that L (f0) = VL and R (f0) = VL. That is, the level of the frequency spectrum of L (f0) and R (f0) is adjusted so as to coincide with the higher one of the levels.

  IFFT (Inverse Fast Fourier Transform) sections 105L and 105R perform inverse Fourier transform on the gain-adjusted frequency spectra L (f) and R (f), respectively, to convert them into signals in the time domain, respectively Lo (t ) And Ro (t).

  As described above, the speaker sound control unit 100 according to the present embodiment detects whether or not a human voice speaker sound is included in sounds coming from a plurality of sound sources. When a speaker sound of a human voice is detected, control is performed so that only the speaker sound has a sound image that substantially coincides with the shooting direction of the imaging apparatus 1.

<< Second Example >>
Hereinafter, a second embodiment of the speaker sound control unit included in the acoustic processing unit 7 of the imaging apparatus 1 will be described. In the following description of the second embodiment, the number 200 is assigned to the speaker sound control unit.

  Generally, the voice of a person who is input from a microphone and is emitted from a speaker after being amplified by an amplifier is larger than the voice of a person who is directly emitted. In some cases, the voice of a person uttered directly is louder than the voice of a person uttered from the speaker, but the voice of the person uttered directly and the voice of the person uttered from the speaker differ even if they are the same person. To do. Usually, the voice of a person uttered from a speaker has a higher response than the voice of a person uttered directly. When a person utters a voice through a microphone, the microphone collects the voice of the person uttered from the speaker in addition to the voice directly uttered by the person. Because it will be.

  For this reason, it is considered that the voice of a person whose volume is high and whose response is large is a speaker sound of a person's voice. Here, a large echo means that there is a certain periodicity.

  In general, music acoustic signals are wideband signals and have a certain periodicity. As described above, many speakers that are normally used in events such as classes and athletic meet have a frequency band of sound signals that can be reproduced in the range of about 300 Hz to 6 kHz. Therefore, when an audio signal based on music is emitted from such a speaker, the frequency band is narrower than that of an audio signal based on direct music, but the volume is large and constant because of the amplification process. It will have the periodicity of.

  From the above, an acoustic signal that satisfies all the requirements of (A) the frequency band being included in the range of about 300 Hz to 6 kHz, (B) high volume, and (C) having a certain periodicity is It can be determined that the sound signal is a speaker sound. In other words, if any one of the requirements (A) to (C) is not satisfied, it can be determined that the signal is not a speaker sound signal.

  FIG. 5 is a block diagram illustrating an outline of an internal configuration of the speaker sound control unit 200 included in the acoustic processing unit 7. The speaker sound determination unit 201 determines whether or not Ri (t) output from the ADC 6R includes a speaker sound signal. If the speaker sound determination unit 201 determines that the speaker sound signal is included, the speaker sound determination unit 201 proceeds to the switching unit 202 described later. Outputs a switching signal.

  When the switching signal is output from the speaker sound determination unit 201, the switching unit 202 performs monaural processing on Li (t) and Ri (t) output from the ADCs 6L and 6R, respectively, and Lo (t) and Output as Ro (t). Here, monauralization processing is processing that sets Lo (t) = Ro (t). When the switching signal is not output from the speaker sound determination unit 201, Li (t) and Ri (t) are output as Lo (t) and Ro (t), respectively. Note that the speaker sound determination unit 201 determines whether or not Ri (t) is a speaker sound signal, but may also determine Li (t).

Hereinafter, specific processing of the speaker sound determination unit 201 will be described.
<Step 1: Whether the frequency band is included in the range of about 300 Hz to 6 kHz>
First, the speaker sound determination unit 201 calculates the frequency spectrum by performing FFT on each frame (1024 samples) of the acoustic signal Ri (t) output from the ADC 6R. It is determined whether or not the calculated frequency spectrum is included in a frequency band of about 300 Hz to 6 kHz.
<Step 2: Whether the volume is high>
Next, when the speaker sound determination unit 201 determines that the frequency spectrum of the acoustic signal Ri (t) is included in the frequency band of about 300 Hz to 6 kHz, the level of the acoustic signal Ri (t) ( It is determined whether the (power value) is equal to or greater than a predetermined threshold.

  Specifically, the speaker sound determination unit 201 determines whether or not the average value PRi (n) of the sound signal power of each frame calculated for Ri (t) by the following equation (4) is equal to or greater than a predetermined threshold. Here, consecutive frames in the time domain are described as first, second, third... Nth frame in order from the earliest time. n is a positive integer indicating a frame number.

<Step 3: Whether or not it has a certain periodicity>
Next, when the speaker sound determination unit 201 determines that the average power value PRi (n) of the sound signal of the nth frame (1024 samples) is equal to or greater than a predetermined threshold, the sound signal of the nth frame is a period. It is determined whether or not it has sex.

  FIG. 6 is a diagram for explaining a method of determining whether or not the acoustic signal of the nth frame has periodicity. In FIG. 6, among the acoustic signals Ri (t) of the nth frame, for example, the autocorrelation is calculated after using t = 1 to t0th Ri (t) as a reference block (t0 is 2 or more). integer). That is, an evaluation block consisting of t0 consecutive Ri (t) is defined for Ri (t) after t0, and the position of the evaluation block is sequentially shifted in the time direction between the reference block and the evaluation block. We will seek the correlation. In FIG. 6, P is a variable representing the shift width, in other words, the position of the evaluation block, and P> t0. Specifically, the autocorrelation value S (P) is calculated according to the following equation (5). The autocorrelation value S (P) is a function of the variable P that determines the position of the evaluation block.

  FIG. 7 shows the relationship between the calculated autocorrelation value S (P) and the variable P. In FIG. 7, the horizontal axis and the vertical axis represent the variable P and the autocorrelation value S (P), respectively. According to FIG. 7, the autocorrelation value S (P) takes a local maximum value that periodically exceeds a predetermined threshold with respect to the change of the variable P. In this case, the speaker sound determination unit 201 determines that the sound signal Ri (t) of the nth frame has periodicity and includes a speaker sound signal, and outputs a switching signal to the switching unit 202. The execution order of steps 1 to 3 may be changed.

When the switching signal is output from the speaker sound determination unit 201, the switching unit 202 switches from the stereo system to the monaural system. That is, the switching unit 202 outputs one of the acoustic signals Li (t) and Ri (t) as Lo (t) and Ro (t). As a result, if the collected sound includes speaker sound, it is recorded in monaural format.

  As described above, the speaker sound control unit 200 according to the present embodiment detects whether or not the speaker sound is included in the sound collected by the microphones 5L and 5R. When the speaker sound is detected, control is performed so that the sound image of the entire sound collected by the microphones 5L and 5R substantially coincides with the shooting direction by the imaging device 1.

<< Modification 1 >>
It is possible to combine the speaker sound control unit 100 of the first embodiment and the speaker sound control unit 200 of the second embodiment.

  For example, when the sound collected by the microphones 5L and 5R includes the speaker sound of a human voice, the direction of the sound image is controlled by the speaker sound control unit 100, and the speaker sound of the human voice is not included. When the speaker sound of music is included, it can be combined so that the sound image control by the speaker sound control unit 200 is performed.

<< Modification 2 >>
The imaging device 1 can be provided with a switch for switching between the normal recording mode and the speaker sound control recording mode.

  When the user sets the normal recording mode when shooting with the imaging device 1, the imaging device 1 does not control the sound image direction by the speaker sound control unit 100 or 200 when recording the sound. On the other hand, when the speaker sound control recording mode is set, the imaging apparatus 1 controls the sound image direction by the speaker sound control unit 100 or 200 with respect to the sounds collected by the microphones 5L and 5R.

  According to the imaging apparatus 1, the user can freely determine whether or not the sound image direction control by the speaker sound control unit 100 or 200 is necessary.

<< Modification 3 >>
In the first and second embodiments, the case where the sound image direction is controlled by the speaker sound control unit 100 or 200 when the image and sound are recorded by the imaging device 1 has been described. For example, the imaging device 1 or the playback device When reproducing the image signal and the sound signal recorded in the external recording medium (external memory 11 in the case of the imaging apparatus 1), the sound image direction may be controlled by the speaker sound control unit 100 or 200. According to such an imaging device 1 or a playback device, so-called raw image signals and acoustic signals that are not subjected to any acoustic signal processing for controlling the direction of the sound image can be acquired during photographing. Therefore, it is possible to avoid a situation where recording is performed by controlling the direction of the sound image unintended by the photographer.

<< Modification 4 >>
It is possible to detect a microphone image signal from the image signal (microphone image detection) using a known technique (face image detection) for detecting a human face image signal from an image signal acquired at the time of shooting. . As a known technique for detecting an image signal of a human face, for example, there is a technique described in Japanese Patent Application Laid-Open No. 2007-257358. It is possible to detect the image signal of the microphone from the image signal by replacing the weight table related to the face of the person to be referred to when detecting the face image signal in the technique with the weight table related to the microphone.

  In the imaging apparatus 1 of FIG. 1, the image processing unit 8 can be provided with a face image detection unit that detects a human face image signal from an image signal and a microphone image detection unit that detects a microphone image signal. Then, the CPU 15 causes the face image detection unit to perform face detection processing on the image signal output from the AFE 4, and when a human face image signal is detected, the microphone image detection unit performs microphone image detection processing. Can be performed. The microphone image detection unit includes a region in which the image signal of the human face is detected from the image signals output from the AFE 4 in order to detect whether or not the microphone is in the vicinity of the human face. The image signal of the microphone is detected for a predetermined large area.

  When the face image detection and microphone image detection detect a human face and microphone image signal in the image signal output from the AFE 4, there is a person in the shooting area, and the person uses the microphone. It can be determined that the scene is a voice.

  In such a case, the CPU 15 determines that the sound collected by the microphones 5L and 5R may include speaker sound, and causes the speaker sound control unit 100 or 200 to control the sound image direction. it can. On the other hand, when the face image signal is not detected in the image signal output from the AFE 4 or when the face image signal is detected but the microphone image signal is not detected, the CPU 15 controls the speaker sound control unit 100. Alternatively, 200 is not allowed to control the sound image direction.

  The face image detection process and the microphone image detection process, and the control of the sound image direction by the speaker sound control unit 100 or 200 when the face image signal and the microphone image signal are detected are recorded in the external memory 11. It may be executed when reproducing the existing image signal and sound signal. As a result, it is possible to appropriately determine a scene that requires control of the sound image direction, and control the sound image direction by the speaker sound control unit 100 or 200.

<< Modification 5 >>
In the first and second embodiments, the imaging apparatus 1 having two LR channel microphones for stereo recording and the imaging apparatus 1 or the playback apparatus having two LR channel speakers for performing stereo reproduction will be described. However, the number of microphones and speakers and the method for recording and reproducing the acoustic signal are not limited to these. For example, the present invention can be realized even with a 5.1 channel recording and playback method using six microphones and speakers.

5L microphone 5R microphone 6L ADC
6R ADC
DESCRIPTION OF SYMBOLS 100 Speaker sound control part 101L FFT part 101R FFT part 102 Direction determination part 103 Speaker sound determination part 104 Gain adjustment part 105L IFFT part 105R IFFT part 200 Speaker sound control part 201 Speaker sound determination part 202 Switching part

Claims (8)

Sound collecting means for collecting sound that arrives at the time of shooting to acquire an acoustic signal of the sound;
Speaker sound detection means for detecting a speaker sound signal from the acoustic signal;
Sound image direction control means for performing sound signal processing on the sound signal so that the sound image direction of the speaker sound signal matches the shooting direction when the speaker sound signal is detected;
Acoustic signal recording means for recording the acoustic signal subjected to the acoustic signal processing;
An acoustic signal processing device comprising: Sound collecting means for collecting sound that arrives at the time of shooting to acquire an acoustic signal of the sound;
Speaker sound detection means for detecting a speaker sound signal from the acoustic signal;
Sound image direction control means for performing sound signal processing on the sound signal so that the sound image direction of the sound signal matches the shooting direction when the speaker sound signal is detected;
Acoustic signal recording means for recording the acoustic signal subjected to the acoustic signal processing;
An acoustic signal processing device comprising: The acoustic signal processing device according to claim 1 or 2,
An imaging means for acquiring an image signal of the shooting target by shooting the shooting target;
Image signal recording means for recording the image signal acquired by the imaging means in association with an acoustic signal subjected to acoustic signal processing by a sound image direction control means included in the acoustic signal processing device;
An imaging apparatus, further comprising: Face detection means for detecting a human face image signal from the image signal;
A microphone detection means for detecting an image signal of a microphone from the image signal,
The speaker sound detection means provided in the acoustic signal processing device is configured to perform the acoustic signal processing when a face image signal of a person is detected by the face detection means and a microphone image signal is detected by the microphone detection means. The imaging apparatus according to claim 3, wherein a speaker sound signal is detected from an acoustic signal acquired by a sound collecting unit included in the apparatus. An imaging means for acquiring an image signal of the shooting target by shooting the shooting target;
Sound collecting means for acquiring a first acoustic signal of the sound by collecting the sound arriving at the time of shooting;
First recording means for recording the first acoustic signal in association with the image signal acquired by the imaging means;
Second recording means for generating a second acoustic signal based on the first acoustic signal and recording the second acoustic signal in association with the image signal acquired by the imaging means;
Switching means for switching between the first recording means and the second recording means;
With
The second recording means includes
Speaker sound detection means for detecting a speaker sound signal from the acoustic signal;
When the speaker sound signal is detected, sound image direction control means for generating the second sound signal by performing sound signal processing on the speaker sound signal so that the sound image direction of the speaker sound signal matches the shooting direction When,
An imaging apparatus comprising: An imaging means for acquiring an image signal of the shooting target by shooting the shooting target;
Sound collecting means for acquiring a first acoustic signal of the sound by collecting the sound arriving at the time of shooting;
First recording means for recording the first acoustic signal in association with the image signal acquired by the imaging means;
Second recording means for generating a second acoustic signal based on the first acoustic signal and recording the second acoustic signal in association with the image signal acquired by the imaging means;
Switching means for switching between the first recording means and the second recording means;
With
The second recording means includes
Speaker sound detection means for detecting a speaker sound signal from the acoustic signal;
Sound image direction control means for generating the second sound signal by performing sound signal processing on the sound signal so that a sound image direction of the sound signal matches a shooting direction when the speaker sound signal is detected;
An imaging apparatus comprising: An acquisition means for acquiring the acoustic signal from a recording means in which an image signal acquired by imaging by the imaging means and an acoustic signal of sound arriving at the time of imaging acquired by the sound collection means are recorded in association with each other. When,
Speaker sound detection means for detecting a speaker sound signal from the acoustic signal acquired by the acquisition means;
Sound image direction control means for performing sound signal processing on the sound signal so that a sound image direction of the speaker sound signal matches a shooting direction by the shooting means when the speaker sound signal is detected;
Reproduction means for reproducing the acoustic signal subjected to the acoustic signal processing;
An acoustic signal processing device comprising: An acquisition means for acquiring the acoustic signal from a recording means in which an image signal acquired by imaging by the imaging means and an acoustic signal of sound arriving at the time of imaging acquired by the sound collection means are recorded in association with each other. When,
Speaker sound detection means for detecting a speaker sound signal from the acoustic signal acquired by the acquisition means;
Sound image direction control means for performing sound signal processing on the sound signal so that a sound image direction of the sound signal matches a shooting direction by the shooting means when the speaker sound signal is detected;
Reproduction means for reproducing the acoustic signal subjected to the acoustic signal processing;
An acoustic signal processing device comprising: