JP2022054315A - Sound processor, control method, and program - Google Patents

Abstract

To effectively reduce noise.SOLUTION: A sound processor comprises: imaging means; a first microphone; a second microphone; means for performing Fourier transform on sound signals from the first microphone to generate first sound signals; means for performing Fourier transform on sound signals from the second microphone to generate second sound signals; means for generating noise data using the second sound signals and a first parameter related to noise of a noise source; subtraction means for subtracting the noise data from the first sound signals; transform means for performing inverse Fourier transform on sound signals from the subtraction means; record means for recording, on a recording medium, moving image data generated by the imaging means and sound signals from the transform means, as moving image data with sounds; and update means for updating the first parameter using a parameter generated with the first sound signals and the second sound signals in a state in which, recording of the moving image data with sounds is not executed by the record means.SELECTED DRAWING: Figure 13

Description

The present invention relates to a voice processing device capable of reducing noise contained in voice data.

When recording moving image data, a digital camera, which is an example of an audio processing device, can also record ambient audio. In addition, the digital camera has an autofocus function that focuses on the subject during recording of moving image data by driving an optical lens. In addition, the digital camera has a function of driving an optical lens to perform zooming while recording a moving image.

As described above, when the optical lens is driven during the recording of the moving image, the driving sound of the optical lens may be included as noise in the sound recorded together with the moving image. Therefore, conventionally, when a digital camera collects a sliding sound or the like generated when an optical lens is driven as noise, the noise can be reduced and the surrounding sound can be recorded. Patent Document 1 discloses a digital camera that reduces noise by a spectral subtraction method.

Japanese Unexamined Patent Publication No. 2011-205527

However, in Patent Document 1, since the digital camera creates a noise pattern from the noise collected by the microphone that records the surrounding sound, an accurate noise pattern is acquired from the sliding noise generated in the housing of the optical lens. It may not be possible. In this case, the digital camera may not be able to effectively reduce the noise contained in the collected sound.

Therefore, an object of the present invention is to effectively reduce noise.

The voice processing device Fouriers the device image pickup means, the first microphone for acquiring the environmental sound, the second microphone for acquiring the sound from the noise source, and the voice signal from the first microphone. A first conversion means for converting and generating a first audio signal, a second conversion means for Fourier-converting an audio signal from the second microphone to generate a second audio signal, and the second conversion means. From the generation means for generating noise data using the voice signal of the above and the first parameter related to the noise of the noise source, the subtraction means for subtracting the noise data from the first voice signal, and the subtraction means. A third conversion means for inverse Fourier transforming the audio signal of the above, a recording means for recording the moving image data generated by the imaging means and the audio signal from the third conversion means on a recording medium as moving image data with sound. And, in a state where the moving image data with sound is not recorded by the recording means, the parameter related to the noise of the noise source is generated by using the first sound signal and the second sound signal. It is characterized by having an update means for updating the first parameter using the generated parameter.

The voice processing apparatus of the present invention can effectively reduce noise.

It is a perspective view of the image pickup apparatus in 1st Example. It is a block diagram which shows the structure of the image pickup apparatus in 1st Example. It is a block diagram which shows the structure of the audio input part of the image pickup apparatus in the 1st Example. It is a figure which shows the arrangement of the microphone in the audio input part of the image pickup apparatus in the 1st Example. It is a figure which shows the noise parameter in 1st Example. It is a figure which shows the frequency spectrum of voice, and the frequency spectrum of a noise parameter in the case where the driving sound is generated in the situation which can be considered that there is no environmental sound in 1st Example. It is a figure which shows the frequency spectrum of the voice in the case where the driving sound is generated in the situation which there is an environmental sound in the 1st Example. It is a block diagram which shows the structure of the noise parameter selection part in 1st Example. It is a timing chart which concerns on voice noise reduction processing in 1st Example. It is a figure which shows the frequency spectrum of the voice in the case where the driving sound is generated in the situation which there is an environmental sound in the 1st Example. It is a block diagram which shows the structure of the noise parameter selection part in 1st Example. It is a timing chart which concerns on the update method of a noise parameter in 1st Example. It is a flowchart which concerns on the update process of a noise parameter in 1st Example. It is a timing chart which concerns on the update process of a noise parameter in 1st Example. It is a flowchart which concerns on the update process of a noise parameter in 2nd Example. It is a timing chart which concerns on the update process of a noise parameter in the 2nd Example.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Example]
<External view of the image pickup device 100>
1A and 1B show an example of an external view of an image pickup apparatus 100 as an example of an audio processing apparatus to which the present invention can be applied. FIG. 1A is an example of a front perspective view of the image pickup apparatus 100. FIG. 1B is an example of a rear perspective view of the image pickup apparatus 100. In FIG. 1, an optical lens (not shown) is attached to the lens mount 301.

The display unit 107 displays image data, character information, and the like. The display unit 107 is provided on the back surface of the image pickup apparatus 100. The display unit 43 outside the viewfinder is a display unit provided on the upper surface of the image pickup apparatus 100. The out-of-finder display unit 43 displays the set values of the image pickup apparatus 100 such as the shutter speed and the aperture value. The eyepiece finder 16 is a peep-type finder. The user can confirm the focus and composition of the optical image of the subject by observing the focusing screen in the eyepiece finder 16.

The release switch 61 is an operating member for the user to give a shooting instruction. The mode changeover switch 60 is an operation member for the user to switch between various modes. The main electronic dial 71 is a rotation operation member. The user can change the setting values of the image pickup apparatus 100 such as the shutter speed and the aperture value by turning the main electronic dial 71. The release switch 61, the mode changeover switch 60, and the main electronic dial 71 are included in the operation unit 112.

The power switch 72 is an operating member that switches the power of the image pickup apparatus 100 on and off. The sub electronic dial 73 is a rotation operation member. The user can move the selection frame displayed on the display unit 107 by the sub-electronic dial 73, advance the image in the reproduction mode, and the like. The cross key 74 is a cross key (four-way key) capable of pushing up, down, left, and right portions, respectively. The image pickup apparatus 100 executes processing according to the pressed portion (direction) of the cross key 74. The power switch 72, the sub electronic dial 73, and the cross key 74 are included in the operation unit 112.

The SET button 75 is a push button. The SET button 75 is mainly used for the user to determine a selection item displayed on the display unit 107. The LV button 76 is a button used to switch the live view (hereinafter referred to as LV) on and off. The LV button 76 is used to instruct to start and stop moving image recording (recording) in the moving image recording mode. The magnifying button 77 is a push button for turning on and off the magnifying mode and changing the magnifying ratio during the magnifying mode in the live view display of the shooting mode. The SET button 75, the LV button 76, and the enlargement button 77 are included in the operation unit 112.

The enlargement button 77 functions as a button for increasing the enlargement ratio of the image data displayed on the display unit 107 in the reproduction mode. The reduction button 78 is a button for reducing the enlargement ratio of the image data enlarged and displayed on the display unit 107. The play button 79 is an operation button for switching between a shooting mode and a play mode. When the user presses the play button 79 during the shooting mode, the image pickup device 100 shifts to the play mode and displays the image data recorded on the recording medium 110 on the display unit 107. The reduction button 78 and the play button 79 are included in the operation unit 112.

The quick return mirror 12 (hereinafter referred to as a mirror 12) is a mirror for switching the light beam incident from the optical lens mounted on the image pickup apparatus 100 so as to be incident on either the eyepiece finder 16 side or the image pickup unit 101 side. The mirror 12 is moved up and down by controlling an actuator (not shown) by the control unit 111 during exposure, live view shooting, and moving image shooting. The mirror 12 is normally arranged so that a light flux is incident on the eyepiece finder 16. In the case of shooting and live view display, the mirror 12 jumps upward so that a light flux is incident on the image pickup unit 101 (mirror lockup). Further, the central portion of the mirror 12 is a half mirror. A part of the light flux transmitted through the central portion of the mirror 12 is incident on a focal detection unit (not shown) for performing focus detection.

The communication terminal 10 is a communication terminal for communicating between the optical lens 300 mounted on the image pickup apparatus 100 and the image pickup apparatus 100. The terminal cover 40 is a cover that protects a connector (not shown) such as a connection cable for connecting a connection cable to an external device and the image pickup device 100. The lid 41 is a lid of a slot in which the recording medium 110 is stored. The lens mount 301 is a mounting portion to which an optical lens 300 (not shown) can be mounted.

The L microphone 201a and the R microphone 201b are microphones for collecting environmental sounds such as user's voice. When viewed from the back surface of the image pickup apparatus 100, the L microphone 201a is arranged on the left side and the R microphone 201b is arranged on the right side.

<Structure of image pickup device 100>
FIG. 2 is a block diagram showing an example of the configuration of the image pickup apparatus 100 in this embodiment.

The optical lens 300 is a lens unit that can be attached to and detached from the image pickup device 100. For example, the optical lens 300 is a zoom lens or a varifocal lens. The optical lens 300 has an optical lens, a motor for driving the optical lens, and a communication unit that communicates with the lens control unit 102 of the image pickup apparatus 100 described later. The optical lens 300 can focus and zoom on the subject and correct camera shake by moving the optical lens by a motor based on the control signal received by the communication unit.

The image pickup unit 101 generates image data or moving image data from an image pickup element for converting an optical image of a subject imaged on an image pickup surface through an optical lens 300 into an electric signal, and an electric signal generated by the image pickup element. It has an image processing unit to output the image. The image pickup device is, for example, a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor). In this embodiment, a series of processes in which the image pickup unit 101 generates image data including still image data and moving image data and outputs the image data from the image pickup unit 101 is referred to as “shooting”. In the image pickup apparatus 100, the image data is recorded on the recording medium 110 described later in accordance with the DCF (Design rule for Camera File system) standard.

The lens control unit 102 transmits a control signal to the optical lens 300 via the communication terminal 10 based on the data output from the image pickup unit 101 and the control signal output from the control unit 111 described later, and causes the optical lens 300 to operate. Control. Further, the lens control unit 102 receives lens information from the optical lens 300 mounted on the image pickup apparatus 100. The lens information is, for example, the type of lens, the model number of the lens, the zoom magnification, the type of noise source, and the like.

The information acquisition unit 103 detects the tilt of the image pickup device 100, the temperature inside the housing of the image pickup device 100, and the like. For example, the information acquisition unit 103 detects the inclination of the image pickup device 100 by an acceleration sensor or a gyro sensor. Further, for example, the information acquisition unit 103 detects the temperature inside the housing of the image pickup apparatus 100 by a temperature sensor.

The voice input unit 104 generates voice data from the voice acquired by the microphone. The voice input unit 104 acquires voice around the image pickup device 100 by a microphone, performs analog-digital conversion (A / D conversion) and various voice processing on the acquired voice, and generates voice data. In this embodiment, the voice input unit 104 has a microphone. A detailed configuration example of the voice input unit 104 will be described later.

The volatile memory 105 temporarily records the image data generated by the imaging unit 101 and the audio data generated by the audio input unit 104. The volatile memory 105 is also used as a temporary recording area for image data displayed on the display unit 107, a work area for the control unit 111, and the like.

The display control unit 106 controls to display the image data output from the image pickup unit 101, characters for interactive operation, a menu screen, and the like on the display unit 107. Further, the display control unit 106 controls the display unit 107 to sequentially display the digital data output from the image pickup unit 101 on the display unit 107 during still image shooting and moving image shooting, so that the display unit 107 functions as an electronic viewfinder. Can be done. For example, the display unit 107 is a liquid crystal display or an organic EL display. Further, the display control unit 106 displays image data and moving image data output from the image pickup unit 101, characters for interactive operation, a menu screen, and the like on an external display via an external output unit 115, which will be described later. It can also be controlled to cause.

The coding processing unit 108 can encode the image data and the audio data temporarily recorded in the volatile memory 105, respectively. For example, the coding processing unit 108 can generate moving image data in which image data is encoded and data-compressed according to a JPEG standard or a RAW image format. For example, the coding processing unit 108 uses the MPEG2 standard or H.M. It is possible to generate video data encoded and data-compressed according to the 264 / MPEG4-AVC standard. Further, for example, the coding processing unit 108 can generate voice data in which voice data is coded and data-compressed according to the AC3AAC standard, ATRAC standard, or ADPCM method. Further, the coding processing unit 108 may encode the voice data so as not to compress the voice data according to, for example, the linear PCM method.

The recording control unit 109 can record data on the recording medium 110 and read the data from the recording medium 110. For example, the recording control unit 109 can record the still image data, the moving image data, and the audio data generated by the coding processing unit 108 on the recording medium 110 and read them from the recording medium 110. The recording medium 110 is, for example, an SD card, a CF card, an XQD memory card, an HDD (magnetic disk), an optical disk, and a semiconductor memory. The recording medium 110 may be configured to be detachable from the image pickup device 100, or may be built in the image pickup device 100. That is, the recording control unit 109 may have at least a means for accessing the recording medium 110.

The control unit 111 controls each component of the image pickup apparatus 100 via the data bus 116 according to the input signal and the program described later. The control unit 111 has a CPU, a ROM, and a RAM for executing various controls. Instead of the control unit 111 controlling the entire image pickup device 100, a plurality of hardware may share control of the entire image pickup device 100. The ROM included in the control unit 111 stores a program for controlling each component. Further, the RAM included in the control unit 111 is a volatile memory used for arithmetic processing and the like.

The operation unit 112 is a user interface for receiving an instruction to the image pickup apparatus 100 from the user. The operation unit 112 includes, for example, a power switch 72 for turning on or off the power of the image pickup apparatus 100, a release switch 61 for instructing shooting, a play button for instructing playback of image data or moving image data, and the like. And has a mode changeover switch 60 and the like.

The operation unit 112 outputs a control signal to the control unit 111 according to the operation of the user. Further, the touch panel formed on the display unit 107 can also be included in the operation unit 112. The release switch 61 has SW1 and SW2. When the release switch 61 is in the so-called half-pressed state, SW1 is turned on. As a result, it receives preparation instructions for performing preparatory operations for imaging such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing. Further, when the release switch 61 is in the so-called fully pressed state, SW2 is turned on. By such a user operation, an imaging instruction for performing an imaging operation is received. Further, the operation unit 112 includes an operation member (for example, a button) capable of adjusting the volume of audio data reproduced from the speaker 114, which will be described later.

The audio output unit 113 can output audio data to the speaker 114 and the external output unit 115. The audio data input to the audio output unit 113 is audio data read from the recording medium 110 by the recording control unit 109, audio data output from the non-volatile memory 117, and audio data output from the coding processing unit. Is. The speaker 114 is an electroacoustic converter capable of reproducing audio data.

The external output unit 115 can output image data, moving image data, audio data, and the like to an external device. The external output unit 115 is composed of, for example, a video terminal, a microphone terminal, a headphone terminal, and the like.

The data bus 116 is a data bus for transmitting various data such as audio data, moving image data, and image data, and various control signals to each block of the image pickup apparatus 100.

The non-volatile memory 117 is a non-volatile memory, and stores a program or the like described later executed by the control unit 111. In addition, voice data is recorded in the non-volatile memory 117. This audio data includes, for example, a focusing sound output when the subject is in focus, an electronic shutter sound output when shooting is instructed, an operation sound output when the image pickup device 100 is operated, and the like. It is voice data of electronic sound.

<Operation of image pickup device 100>
From now on, the operation of the image pickup apparatus 100 of this Example will be described.

The image pickup apparatus 100 of this embodiment supplies electric power to each component of the image pickup apparatus from a power source (not shown) in response to the user operating the power switch 72 to turn on the power. For example, the power source is a battery such as a lithium ion battery or an alkaline manganese dry battery.

The control unit 111 determines, for example, which mode of the shooting mode and the reproduction mode operates based on the state of the mode changeover switch 60 according to the power supply. In the moving image recording mode, the control unit 111 records the moving image data output from the imaging unit 101 and the audio data output from the audio input unit 104 as one moving image data with audio. In the reproduction mode, the control unit 111 controls the recording control unit 109 to read out the image data or the moving image data recorded on the recording medium 110 and display the image data or the moving image data on the display unit 107.

First, the moving image recording mode will be described. In the moving image recording mode, first, the control unit 111 transmits a control signal to each component of the image pickup device 100 so as to shift the image pickup device 100 to the shooting standby state. For example, the control unit 111 controls the image pickup unit 101 and the voice input unit 104 to perform the following operations.

The image pickup unit 101 converts the optical image of the subject imaged on the image pickup surface through the optical lens 300 into an electric signal, and generates moving image data from the electric signal generated by the image pickup element. Then, the image pickup unit 101 transmits the moving image data to the display control unit 106, and the display unit 107 displays the moving image data. The user can prepare for shooting while viewing the moving image data displayed on the display unit 107.

The audio input unit 104 A / D-converts analog audio signals input from a plurality of microphones to generate a plurality of digital audio signals. Then, the voice input unit 104 generates voice data of a plurality of channels from the plurality of digital voice signals. The voice input unit 104 transmits the generated voice data to the voice output unit 113, and reproduces the voice data from the speaker 114. The user can adjust the volume of the audio data recorded in the moving image data with audio by the operation unit 112 while listening to the audio data reproduced from the speaker 114.

Next, in response to the user pressing the LV button 76, the control unit 111 transmits an instruction signal for starting shooting to each component of the image pickup apparatus 100. For example, the control unit 111 controls the image pickup unit 101, the voice input unit 104, the coding processing unit 108, and the recording control unit 109 to perform the following operations.

The image pickup unit 101 converts the optical image of the subject imaged on the image pickup surface through the optical lens 300 into an electric signal, and generates moving image data from the electric signal generated by the image pickup element. Then, the image pickup unit 101 transmits the moving image data to the display control unit 106, and the display unit 107 displays the moving image data. Further, the image pickup unit 101 also transmits the generated moving image data to the volatile memory 105.

The audio input unit 104 A / D-converts analog audio signals input from a plurality of microphones to generate a plurality of digital audio signals. Then, the voice input unit 104 generates multi-channel voice data from the plurality of digital voice signals. Then, the voice input unit 104 transmits the generated voice data to the volatile memory 105.

The coding processing unit 108 reads out the moving image data and the audio data temporarily recorded in the volatile memory 105 and encodes them respectively. The control unit 111 generates a data stream from the moving image data and the audio data encoded by the coding processing unit 108, and outputs the data stream to the recording control unit 109. The recording control unit 109 records the input data stream as moving image data with audio on the recording medium 110 according to a file system such as UDF or FAT.

Each component of the image pickup apparatus 100 continues the above operation during movie shooting.

Then, in response to the user pressing the LV button 76, the control unit 111 transmits an instruction signal for the end of shooting to each component of the image pickup apparatus 100. For example, the control unit 111 controls the image pickup unit 101, the voice input unit 104, the coding processing unit 108, and the recording control unit 109 to perform the following operations.

The image pickup unit 101 stops the generation of moving image data. The voice input unit 104 stops the generation of voice data.

The coding processing unit 108 reads out and encodes the remaining moving image data and audio data recorded in the volatile memory 105. The control unit 111 generates a data stream from the moving image data and the audio data encoded by the coding processing unit 108, and outputs the data stream to the recording control unit 109.

The recording control unit 109 records the data stream on the recording medium 110 as a file of moving image data with audio according to a file system such as UDF or FAT. Then, the recording control unit 109 completes the moving image data with audio in response to the stoppage of the input of the data stream. When the moving image data with sound is completed, the recording operation of the image pickup apparatus 100 is stopped.

The control unit 111 transmits a control signal to each component of the image pickup apparatus 100 so as to shift to the shooting standby state when the recording operation is stopped. As a result, the control unit 111 controls the image pickup apparatus 100 to return to the shooting standby state.

Next, the reproduction mode will be described. In the reproduction mode, the control unit 111 transmits a control signal to each component of the image pickup apparatus 100 so as to shift to the reproduction state. For example, the control unit 111 controls the coding processing unit 108, the recording control unit 109, the display control unit 106, and the voice output unit 113 to perform the following operations.

The recording control unit 109 reads out the moving image data with sound recorded on the recording medium 110 and transmits the read moving image data with sound to the coding processing unit 108.

The coding processing unit 108 decodes the image data and the audio data from the moving image data with audio. The coding processing unit 108 transmits the decoded moving image data to the display control unit 106 and the decoded audio data to the audio output unit 113, respectively.

The display control unit 106 displays the decoded image data by the display unit 107. The voice output unit 113 reproduces the decoded voice data by the speaker 114.

As described above, the image pickup apparatus 100 of this embodiment can record and reproduce image data and audio data.

In this embodiment, the voice input unit 104 executes voice processing such as adjustment processing of the level of the voice signal input from the microphone. In this embodiment, the voice input unit 104 executes this voice processing in response to the start of video recording. Note that this audio processing may be executed after the power of the image pickup apparatus 100 is turned on. Further, this audio processing may be executed depending on the shooting mode selected. Further, this voice processing may be executed depending on the selection of a mode related to voice recording such as a moving image recording mode and a voice memo function. Further, this voice processing may be executed in response to the start of recording of the voice signal.

<Structure of voice input unit 104>
FIG. 3 is a block diagram showing an example of a detailed configuration of the voice input unit 104 in this embodiment.

In this embodiment, the voice input unit 104 has three microphones, an L microphone 201a, an R microphone 201b, and a noise microphone 201c. The L microphone 201a and the R microphone 201b are examples of the first microphone, respectively. In this embodiment, the image pickup apparatus 100 picks up the ambient sound by the L microphone 201a and the R microphone 201b, and records the audio signals input from the L microphone 201a and the R microphone 201b in a stereo system. For example, the environmental sound is a user's voice, an animal's bark, a rain sound, and a sound generated outside the housing of the image pickup device 100 and the housing of the optical lens 300 such as music.

The noise microphone 201c is an example of the second microphone. The noise microphone 201c is a microphone for collecting noise such as drive sound from a predetermined noise source (noise source) generated in the housing of the image pickup apparatus 100 and the housing of the optical lens 300. .. The noise source is, for example, a motor such as an ultrasonic motor (USM) and a stepper motor (STM). Noise (noise) is, for example, vibration noise generated by driving a motor such as USM and STM. For example, the motor is driven in an AF process for focusing on the subject. The image pickup device 100 acquires noise such as driving sound generated in the housing of the image pickup device 100 and the housing of the optical lens 300 by the noise microphone 201c, and uses the acquired noise voice data to generate noise described later. Generate parameters. In this embodiment, the L microphone 201a, the R microphone 201b, and the noise microphone 201c are omnidirectional microphones. An arrangement example of the L microphone 201a, the R microphone 201b, and the noise microphone 201c in this embodiment will be described later with reference to FIG.

The L microphone 201a, the R microphone 201b, and the noise microphone 201c each generate an analog audio signal from the acquired voice and input it to the A / D conversion unit 202. Here, the audio signal input from the L microphone 201a is referred to as Lch, the audio signal input from the R microphone 201b is referred to as Rch, and the audio signal input from the noise microphone 201c is referred to as Nch.

The A / D conversion unit 202 converts the analog audio signal input from the L microphone 201a, the R microphone 201b, and the noise microphone 201c into a digital audio signal. The A / D conversion unit 202 outputs the converted digital audio signal to the FFT unit 203. In this embodiment, the A / D conversion unit 202 converts an analog audio signal into a digital audio signal by executing sampling processing with a sampling frequency of 48 kHz and a bit depth of 16 bits.

The FFT unit 203 performs a fast Fourier transform process on the digital audio signal in the time domain input from the A / D conversion unit 202, and converts it into a digital audio signal in the frequency domain. In this embodiment, the digital audio signal in the frequency domain has a frequency spectrum of 1024 points in the frequency band from 0 Hz to 48 kHz. Further, the digital audio signal in the frequency domain has a frequency spectrum of 513 points in the frequency band from 0 Hz to 24 kHz, which is the Nyquist frequency. In this embodiment, the image pickup apparatus 100 performs noise reduction processing by using the frequency spectrum of 513 points from 0 Hz to 24 kHz in the audio data output from the FFT unit 203.

Here, the frequency spectrum of the Lch subjected to the fast Fourier transform is represented by the array data of 513 points from Lch_Before [0] to Lch_Before [512]. When these sequence data are generically referred to, they are described as Lch_Before. Further, the frequency spectrum of the Rch subjected to the fast Fourier transform is represented by the array data of 513 points from Rch_Before [0] to Rch_Before [512]. When these sequence data are generically referred to, they are described as Rch_Before. Here, in this embodiment, Lch_Before [0] is the frequency spectrum of the voice of 0 Hz, and Lch_Before [512] is the frequency spectrum of the voice of 24 kHz. Note that Lch_Before and Rch_Before are examples of the first frequency spectrum data, respectively.

Further, the frequency spectrum of the Nch subjected to the fast Fourier transform is represented by the array data of 513 points from Nch_Before [0] to Nch_Before [512]. When these sequence data are generically referred to, they are described as Nch_Before. Note that Nch_Before is an example of the second frequency spectrum data.

The noise data generation unit 204 generates data for reducing noise contained in Lch_Before and Rch_Before based on Nch_Before. In this embodiment, the noise data generation unit 204 generates array data of NL [0] to NL [512] for reducing noise contained in Lch_Before [0] to Lch_Before [512] by using noise parameters. do. Further, the noise data generation unit 204 generates array data of NR [0] to NR [512] for reducing noise contained in Rch_Before [0] to Rch_Before [512], respectively. The frequency points in the sequence data of NL [0] to NL [512] are the same as the frequency points in the sequence data of Lch_Before [0] to Lch_Before [512]. Further, the frequency points in the sequence data of NR [0] to NR [512] are the same as the frequency points in the sequence data of Rch_Before [0] to Rch_Before [512].

In addition, when the sequence data of NL [0] to NL [512] is generically referred to, it is described as NL. When NR [0] to NR [512] are generically referred to, they are described as NR. NL and NR are examples of the third frequency spectrum data, respectively.

The noise parameter recording unit 205 records noise parameters for the noise data generation unit 204 to generate NL and NR from Nch_Before. The noise parameter recording unit 205 records a plurality of types of noise parameters according to the type of noise for each lens. When the noise parameters for generating NL from Nch_Before are generically referred to as PLx. When the noise parameters for generating NR from Nch_Before are generically referred to as PRx.

PLx and PRx have the same number of sequences as NL and NR, respectively. For example, PL1 is array data of PL1 [0] to PL1 [512]. Further, the frequency point of PL1 is the same as the frequency point of Lch_Before. Further, for example, PR1 is sequence data of PR1 [0] to PR1 [512]. The frequency point of PR1 is the same frequency point as Rch_Before. The noise parameters will be described later with reference to FIG.

The noise parameter selection unit 206 determines the noise parameter used in the noise data generation unit 204 from the noise parameter recorded in the noise parameter recording unit 205. The noise parameter selection unit 206 determines the noise parameter used in the noise data generation unit 204 based on data such as lens information received from Lch_Before, Rch_Before, Nch_Before, and the lens control unit 102. The operation of the noise parameter selection unit 206 will be described in detail later with reference to FIG.

In this embodiment, the noise parameter recording unit 205 records all the coefficients for each frequency spectrum of 513 points as noise parameters. However, it is sufficient that at least the coefficient of the frequency point necessary for reducing noise is recorded instead of the coefficient for all frequencies of 513 points. For example, the noise parameter recording unit 205 may record coefficients for each of the frequency spectra of 20 Hz to 20 kHz, which are considered to be typical audible frequencies, as noise parameters, and may not record coefficients of other frequency spectra. Further, for example, as a noise parameter, the coefficient for the frequency spectrum in which the value of the coefficient is zero may not be recorded in the noise parameter recording unit 205.

The subtraction processing unit 207 subtracts NL and NR from Lch_Before and Rch_Before, respectively. For example, the subtraction processing unit 207 has an L subtractor 207a that subtracts NL from Lch_Before, and an R subtractor 207b that subtracts NR from Rch_Before. The L subtractor 207a subtracts NL from Lch_Before and outputs 513 point sequence data of Lch_After [0] to Lch_After [512]. The R subtractor 207b subtracts NR from Rch_Before and outputs 513 point sequence data of Rch_After [0] to Rch_After [512]. In this embodiment, the subtraction processing unit 207 executes the subtraction processing by the spectral subtraction method.

The iFFT unit 208 converts the digital audio signal in the frequency domain input from the subtraction processing unit 207 into a digital audio signal in the time domain by performing an inverse fast Fourier transform (inverse Fourier transform).

The audio processing unit 209 executes audio processing for a digital audio signal in the time domain, such as an equalizer, an auto-level controller, and a stereo feeling enhancement process. The voice processing unit 209 outputs the voice-processed voice data to the volatile memory 105.

The noise parameter updating unit 216 updates the noise parameter recorded in the noise parameter recording unit 205. The noise parameter updating unit 216 generates noise parameters based on data such as lens information received from Lch_Before, Rch_Before, Nch_Before, and the lens control unit 102. Then, the noise parameter updating unit 216 updates the noise parameter recorded in the noise parameter recording unit 205 by using the generated noise parameter. The operation of the noise parameter updating unit 216 will be described later with reference to FIG.

In this embodiment, the image pickup apparatus 100 has two microphones as the first microphone, but the image pickup apparatus 100 may use the first microphone as one microphone or three or more microphones. For example, when the image pickup apparatus 100 has one microphone as the first microphone in the voice input unit 104, the voice data acquired by one microphone is recorded in a monaural manner. Further, for example, when the image pickup apparatus 100 has three or more microphones as the first microphone in the voice input unit 104, the sound data acquired by the three or more microphones is recorded in a surround system.

In this embodiment, the L microphone 201a, the R microphone 201b, and the noise microphone 201c are omnidirectional microphones, but these microphones may be directional microphones.

<Arrangement of microphones in voice input unit 104>
Here, an example of arranging the microphone of the voice input unit 104 of this embodiment will be described. FIG. 4 shows an arrangement example of the L microphone 201a, the R microphone 201b, and the noise microphone 201c.

FIG. 4 is an example of a cross-sectional view of a portion of the image pickup apparatus 100 to which the L microphone 201a, the R microphone 201b, and the noise microphone 201c are attached. The portion of the image pickup apparatus 100 is composed of an exterior portion 302, a microphone bush 303, and a fixed portion 304.

The exterior portion 302 has a hole (hereinafter referred to as a microphone hole) for inputting an environmental sound into the microphone. In this embodiment, the microphone hole is formed above the L microphone 201a and the R microphone 201b. On the other hand, the noise microphone 201c is provided to acquire the driving sound generated in the housing of the image pickup apparatus 100 and the housing of the optical lens 300, and it is not necessary to acquire the environmental sound. Therefore, in this embodiment, the microphone hole is not formed above the noise microphone 201c in the exterior portion 302.

The drive sound generated in the housing of the image pickup apparatus 100 and the housing of the optical lens 300 is acquired by the L microphone 201a and the R microphone 201b through the microphone holes. When a driving sound or the like is generated in the housing of the image pickup apparatus 100 and the optical lens 300 in a state where the environmental sound is small, the sound acquired by each microphone is mainly the driving sound. Therefore, the sound level from the noise microphone 201c is higher than the sound level from the L microphone 201a and the R microphone 201b. That is, in this case, the relationship between the levels of the audio signals output from each microphone is as follows.
Lch ≒ Rch <Nch

Further, when the environmental sound becomes louder, the sound level of the environmental sound from the L microphone 201a and the R microphone 201b becomes louder than the sound level of the drive sound generated by the image pickup device 100 or the optical lens 300 from the microphone 201c. .. Therefore, in this case, the relationship between the levels of the audio signals output from each microphone is as follows.
Lch ≒ Rch> Nch

In this embodiment, the shape of the microphone hole formed in the exterior portion 302 is elliptical, but it may be another shape such as a circular shape or a square shape. Further, the shape of the microphone hole on the microphone 201a and the shape of the microphone hole on the microphone 201b may be different from each other.

In this embodiment, the noise microphone 201c is arranged so as to be close to the L microphone 201a and the R microphone 201b. Further, in this embodiment, the noise microphone 201c is arranged between the L microphone 201a and the R microphone 201b. As a result, the audio signal generated by the noise microphone 201c from the drive sound or the like generated in the housing of the image pickup apparatus 100 and the housing of the optical lens 300 is generated by the L microphone 201a and the R microphone 201b from the drive sound or the like. It becomes a signal similar to an audio signal.

The microphone bush 303 is a member for fixing the L microphone 201a, the R microphone 201b, and the noise microphone 201c. The fixing portion 304 is a member that fixes the microphone bush 303 to the exterior portion 302.

In this embodiment, the exterior portion 302 and the fixing portion 304 are made of a mold member such as a PC material. Further, the exterior portion 302 and the fixing portion 304 may be made of a metal member such as aluminum or stainless steel. Further, in this embodiment, the Mike Busch 303 is made of a rubber material such as ethylene propylene diene rubber.

<Noise parameter>
FIG. 5 is an example of noise parameters recorded in the noise parameter recording unit 205. The noise parameter shown in FIG. 5 is a noise parameter for noise generated from a certain lens. The noise parameter is a parameter for correcting the audio signal generated by the noise microphone 201c acquiring the driving sound generated in the housing of the image pickup apparatus 100 and the housing of the optical lens 300. As shown in FIG. 5, in this embodiment, PLx and PRx are recorded in the noise parameter recording unit 205. In this embodiment, the source of the driving sound will be described as being inside the housing of the optical lens 300. The drive sound generated in the housing of the optical lens 300 is transmitted to the housing of the image pickup apparatus 100 via the lens mount 301, and is acquired by the L microphone 201a, the R microphone 201b, and the noise microphone 201c.

The frequency of the drive sound differs depending on the type of drive sound. Therefore, in this embodiment, the image pickup apparatus 100 records a plurality of noise parameters corresponding to the types of drive sounds (noise). Then, noise data is generated using any one of these plurality of noise parameters. In this embodiment, the image pickup apparatus 100 records a noise parameter with respect to white noise as constant noise. Further, the image pickup apparatus 100 records noise parameters for short-term noise generated by, for example, meshing of gears in the optical lens 300. Further, the image pickup apparatus 100 records noise parameters for sliding noise in the housing of the lens 300, for example, as long-term noise. In addition, the image pickup apparatus 100 may record noise parameters for each type of the optical lens 300, and for each temperature inside the housing of the image pickup apparatus 100 and the inclination of the image pickup apparatus 100 detected by the information acquisition unit 103.

<How to generate noise data>
The noise data generation process in the noise data generation unit 204 will be described with reference to FIGS. 6 and 7. Here, the noise data generation process for the Lch data will be described, but the noise data generation method for the Rch data is also the same.

First, a process of generating a noise parameter will be described in a situation where it can be considered that there is no environmental sound. FIG. 6A is an example of the frequency spectrum of Lch_Before when a driving sound is generated in the housing of the optical lens 300 in a situation where it can be considered that there is no environmental sound. FIG. 6B is an example of the frequency spectrum of Nch_Before when a driving sound is generated in the housing of the optical lens 300 in a situation where it can be considered that there is no environmental sound. The horizontal axis is the axis showing the frequency from the 0th point to the 512th point, and the vertical axis is the axis showing the amplitude of the frequency spectrum.

In Lch_Before and Nch_Before, the amplitude of the frequency spectrum in the same frequency band becomes large because the situation can be regarded as having no environmental sound. Further, since the driving sound is generated in the housing of the optical lens 300, the amplitude of each frequency spectrum with respect to the same driving sound tends to be larger in Nch_Before than in Lch_Before.

FIG. 6C is an example of PLx in this embodiment. In this embodiment, PLx is a coefficient of each frequency spectrum calculated by dividing the amplitude of each frequency spectrum of Lch_Before by the amplitude of each frequency spectrum of Nch_Before in a situation where the environmental sound is small. The result of this division is described as Lch_Before / Nch_Before. That is, PLx is the ratio of the amplitudes of Lch_Before and Nch_Before. The noise parameter recording unit 205 records the value of Lch_Before / Nch_Before as the noise parameter PLx. As described above, since the amplitude of the frequency spectrum for the same drive sound tends to be larger in Nch_Before than in Lch_Before, the value of each coefficient of the noise parameter PLx tends to be smaller than 1. However, when the value of Nch_Before [n] is smaller than a predetermined threshold value, the noise parameter recording unit 205 records the noise parameter PLx with PLx [n] = 0.

Next, the process of applying the generated noise parameter to Nch_Before will be described. FIG. 7A is an example of the frequency spectrum of Lch_Before when the driving sound is generated in the housing of the optical lens 300 in the presence of the environmental sound. FIG. 7B is an example of the frequency spectrum of Nch_Before when the driving sound is generated in the housing of the optical lens 300 in the presence of the environmental sound. The horizontal axis is the axis showing the frequency from the 0th point to the 512th point, and the vertical axis is the axis showing the amplitude of the frequency spectrum.

FIG. 7C is an example of NL in the case where the driving sound is generated in the housing of the optical lens 300 in the presence of the environmental sound. The noise data generation unit 204 multiplies each frequency spectrum of Nch_Before by each coefficient of PLx to generate NL. NL is the frequency spectrum thus generated.

FIG. 7D is an example of Lch_After when a driving sound is generated in the housing of the optical lens 300 in a situation where an environmental sound is present. The subtraction processing unit 207 subtracts NL from Lch_Before to generate Lch_After. Lch_After is a frequency spectrum thus generated.

As a result, the image pickup apparatus 100 can reduce the noise caused by the driving sound in the housing of the optical lens 300 and record the environmental sound with less noise.

<Explanation of noise parameter selection unit 206>
FIG. 8 is a block diagram showing an example of a detailed configuration of the noise parameter selection unit 206.

Lch_Before, Rch_Before, Nch_Before, lens information, and a lens control signal are input to the noise parameter selection unit 206.

The Nch noise detection unit 2061 detects noise due to the drive sound generated in the housing of the optical lens 300 from Nch_Before. In this embodiment, the Nch noise detection unit 2061 detects noise by using the data of 513 points of Nch_Before.

The environmental sound detection unit 2062 detects the level of the environmental sound from Lch_Before and Rch_Before.

The noise determination unit 2063 determines the noise parameters used by the noise data generation unit 204 based on the lens information and the lens control signal, the data input from the Nch noise detection unit 2061, and the data input from the environmental sound detection unit. The noise determination unit 2063 outputs data indicating the determined type of noise parameter to the noise data generation unit 204 and the noise parameter update unit 216.

The Nch differential unit 2064 executes the differential process on Nch_Before. The Nch differentiation unit 2064 outputs data indicating the result of differential processing of Nch_Before to the short-term noise detection unit 2065. The short-term noise detection unit 2065 detects whether or not short-term noise is generated based on the data input from the Nch differentiation unit 2064. The short-term noise detection unit 2065 outputs data indicating whether or not short-term noise is generated to the noise determination unit 2063. The Nch differentiation unit 2064 and the short-term noise detection unit 2065 are included in the Nch noise detection unit 2061.

The Nch integration unit 2066 executes an integration process on Nch_Before. The Nch integration unit 2066 outputs data indicating the result of differential processing of Nch_Before to the long-term noise detection unit 2067. The long-term noise detection unit 2067 detects whether or not long-term noise is generated based on the data input from the Nch integration unit 2066. The long-term noise detection unit 2067 outputs data indicating whether or not long-term noise is generated to the noise determination unit 2063. The Nch integration unit 2066 and the long-term noise detection unit 2067 are included in the Nch noise detection unit 2061.

The environmental sound extraction unit 2068 extracts the environmental sound. In this embodiment, the environmental sound extraction unit 2068 extracts frequency data that is less affected by noise based on the noise meter. For example, the environmental sound extraction unit 2068 extracts frequency data having a noise parameter of a predetermined value or less. Then, the environmental sound extraction unit 2068 outputs data indicating the magnitude of the environmental sound based on the extracted frequency data. The environmental sound extraction unit 2068 is included in the environmental sound detection unit 2062.

The environmental sound determination unit 2069 determines the loudness of the environmental sound. The environmental sound determination unit 2069 inputs data indicating the loudness of the determined environmental sound to the Nch noise detection unit 2061 and the noise determination unit 2063. The Nch noise detection unit 2061 changes the first threshold value and the second threshold value, which will be described later, based on the data indicating the loudness of the environmental sound input from the environmental sound determination unit 2069. The environmental sound determination unit 2069 is included in the environmental sound detection unit 2062.

<Timing chart of noise reduction processing>
The noise reduction processing in this embodiment will be described with reference to FIG.

9 (a) to 9 (i) are examples of timing charts of voice processing in the noise data generation unit 204, the noise parameter selection unit 206, and the subtraction processing unit 207. In this embodiment, the Lch voice processing will be described for the sake of simplification of the description, but the same applies to the Rch voice processing. The horizontal axes of the graphs in FIGS. 9 (a) to 9 (i) are all time axes.

FIG. 9A shows an example of a lens control signal. The lens control signal is a signal instructing the lens control unit 102 to drive the optical lens 300. In this embodiment, the level of the lens control signal is represented by two values, High and Low. When the level of the lens control signal is High, the lens control unit 102 is instructing the optical lens 300 to drive the lens 300. When the level of the lens control signal is Low, the lens control unit 102 is in a state in which the optical lens 300 is not instructed to drive.

FIG. 9B is a graph showing an example of the value of Lch_Before [n]. The vertical axis is an axis indicating the value of Lch_Before [n]. In this embodiment, Lch_Before [n] is a signal at the nth frequency point in which the signal indicating the driving sound of the optical lens 300 is characteristically displayed among the Lch_Before output from the FFT unit 203. In this embodiment, the signal at the nth frequency point will be described, but voice processing is similarly executed for other frequencies. Further, the signals represented by the signal X and the signal Y are signals containing noise. In this embodiment, the signal X indicates a signal including short-term noise. The signal Y indicates a noise signal including long-term noise.

FIG. 9C is a graph showing an example of the magnitude of the environmental sound extracted by the environmental sound extraction unit 2068. The vertical axis shows the level of the audio signal generated from the acquired environmental sound. The threshold value Th1 and the threshold value Th2 are two threshold values used in the environmental sound determination unit 2069.

FIG. 9D is a graph showing an example of the value of Nch_Before [n]. Nch_Before [n] is a signal at the nth frequency point in which the signal indicating the driving sound of the optical lens 300 is characteristically displayed among the Nch_Before output from the FFT unit 203. The vertical axis is an axis indicating the value of Nch_Before [n]. In Nch_Before [n], the noise signal shown by the signal X and the signal Y in FIG. 9B appears more characteristically than in Lch_Before.

FIG. 9 (e) is a graph showing an example of the value of Ndiff [n]. Ndiff [n] indicates the value of the signal at the nth frequency point in the Ndiff output from the Nch differential unit 2064. The vertical axis is an axis indicating the value of Ndiff [n]. When the amount of change in the value of Nch_Before [n] per predetermined time is large, the value of Ndiff [n] becomes large. The short-term noise detection unit 2065 has a threshold value Th_Ndiff [n], which is a first threshold value, in order to detect short-term noise. The threshold value Th_Ndiff [n] changes between levels 1 to 3 based on the data indicating the magnitude of the environmental sound input from the environmental sound determination unit 2069 and the lens control signal. The initial value of the threshold value Th_Ndiff [n] is level 2. The level of the threshold value Th_Ndiff [n] is represented by a horizontal broken line.

FIG. 9 (f) is a graph showing an example of the value of Nint [n]. In this embodiment, Nint [n] indicates the value of the signal at the nth frequency point of the Nint output from the Nch integrating unit 2066. The vertical axis is an axis indicating the value of Nint [n]. When Nch_Before [n] is continuously large, the value of Nint [n] becomes large. The long-term noise detection unit 2067 has a threshold value Th_Nint [n], which is a second threshold value, in order to detect long-term noise. The threshold value Th_Nint [n] changes between levels 1 to 3 based on the data indicating the loudness of the environmental sound input from the environmental sound determination unit 2069 and the lens control signal. The initial value of the threshold value Th_Nint [n] is level 2. The level of the threshold value Th_Nint [n] is represented by a horizontal broken line.

FIG. 9 (g) shows an example of the noise parameter selected by the noise parameter selection unit 206. In this embodiment, the plain portion indicates that only the noise parameter of PL1 is selected. The shaded area indicates that the noise parameters of PL1 and PL2 are selected. The plaid portion indicates that the noise parameters of PL1 and PL3 are selected.

FIG. 9 (h) is a graph showing an example of the value of NL [n]. In this embodiment, NL [n] indicates the value of the signal at the nth frequency point among the NLs generated by the noise data generation unit 204. The vertical axis is an axis indicating the value of NL [n].

FIG. 9 (i) is a graph showing an example of the value of Lch_After [n]. In this embodiment, Lch_After [n] indicates the value of the signal at the nth frequency point of the Lch_After output from the subtraction processing unit 207. The vertical axis is an axis indicating the value of Lch_After [n].

Next, the timing of each operation will be described using the times t701 to t709.

At time t701, the lens control unit 102 outputs a High signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (FIG. 9A). Since there is a high possibility that a driving sound is generated in the housing of the optical lens 300 at time t701, the short-term noise detection unit 2065 lowers the threshold value Th_Ndiff [n] to level 1 (FIG. 9 (e)). Further, at time t701, since there is a high possibility that a driving sound is generated in the housing of the optical lens 300, the long-term noise detection unit 2067 lowers the threshold value Th_Nint [n] to level 1 (FIG. 9 (f)).

At time t702, the optical lens 300 is driven, and a short-term driving sound such as a gear meshing sound is generated. The value of Ndiff [n] exceeds the threshold value Th_Ndiff [n] due to the noise microphone 201c picking up the short-term drive sound (FIG. 9 (e)). In response to this, the noise parameter selection unit 206 selects the noise parameters PL1 and PL2 (FIG. 9 (g)). The noise data generation unit 204 generates NL [n] based on Nch_Before [n] and the noise parameters PL1 and PL2 (FIG. 9 (h)). The subtraction processing unit 207 subtracts NL [n] from Lch_Before [n] and outputs Lch_After [n] (FIG. 9 (i)). In this case, Lch_After [n] becomes an audio signal in which constant noise and short-term noise are reduced.

At time t703, the optical lens 300 starts continuous driving, and a long-term driving sound such as a sliding sound is generated in the housing of the optical lens 300. The value of Nint [n] exceeds the threshold value Th_Nint [n] due to the noise microphone 201c picking up the long-term driving sound (FIG. 9 (f)). In response to this, the noise parameter selection unit 206 selects the noise parameters PL1 and PL3 (FIG. 9 (g)). The noise data generation unit 204 generates NL [n] based on Nch_Before [n] and the noise parameters PL1 and PL3 (FIG. 9 (h)). The subtraction processing unit 207 subtracts NL [n] from Lch_Before [n] and outputs Lch_After [n] (FIG. 9 (i)). In this case, Lch_After [n] becomes an audio signal in which constant noise and long-term noise are reduced.

At time t704, the optical lens 300 stops continuous driving. Since the noise microphone 201c does not collect the long-term driving sound, the value of Nint [n] becomes equal to or less than the threshold value Th_Nint [n] (FIG. 9 (f)). In response to this, the noise parameter selection unit 206 selects the noise parameter PL1 (FIG. 9 (g)). The noise data generation unit 204 generates NL [n] based on Nch_Before [n] and the noise parameter PL1 (FIG. 9 (h)). The subtraction processing unit 207 subtracts NL [n] from Lch_Before [n] and outputs Lch_After [n] (FIG. 9 (i)). In this case, Lch_After [n] becomes an audio signal with constant noise reduced.

At time t705, the lens control unit 102 outputs a Low signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (FIG. 9A). In this case, since the possibility that a driving sound is generated in the housing of the optical lens 300 is low, the short-term noise detection unit 2065 raises the threshold value Th_Ndiff [n] to level 2 (FIG. 9 (e)). Further, in this case, since the possibility that the driving sound is generated in the housing of the optical lens 300 is low, the long-term noise detection unit 2067 raises the threshold value Th_Nint [n] to level 2 (FIG. 9 (f)).

At time t706, the loudness of the environmental sound extracted by the environmental sound extraction unit 2068 exceeds the threshold value Th1. When the ambient sound is loud, the noise contained in the audio signal is less likely to be perceived by the user, so the short-term noise detection unit 2065 raises the threshold value Th_Ndiff [n] to level 3 (FIG. 9 (e)). Further, when the environmental sound is loud, the noise contained in the audio signal is less likely to be perceived by the user, so that the long-term noise detection unit 2067 raises the threshold value Th_Nint [n] to level 3 (FIG. 9 (f)).

At time t707, the lens control unit 102 outputs a High signal as a lens control signal to the optical lens 300 and the noise parameter selection unit 206 (FIG. 9A). In this case, since there is a high possibility that a driving sound is generated in the housing of the optical lens 300, the short-term noise detection unit 2065 lowers the threshold value Th_Ndiff [n] to level 2 (FIG. 9 (e)). Further, in this case, since there is a high possibility that a driving sound is generated in the housing of the optical lens 300, the long-term noise detection unit 2067 lowers the threshold value Th_Nint [n] to level 2 (FIG. 9 (f)).

At time t708, the loudness of the environmental sound extracted by the environmental sound extraction unit 2068 exceeds the threshold value Th2. When the ambient sound is louder, the user hardly feels the noise contained in the voice signal. Therefore, the noise parameter selection unit 206 selects only the noise parameter PL1 regardless of the data input from the Nch noise detection unit 2061.

As described above, the image pickup apparatus 100 can record the environmental sound with reduced noise by executing the noise reduction processing using the noise microphone 201c which is the second microphone.

In this embodiment, the image pickup device 100 reduces the drive sound generated in the housing of the optical lens 300, but the drive sound generated in the image pickup device 100 may be reduced. The driving sound generated in the image pickup apparatus 100 is, for example, the noise of the substrate and the radio wave noise. The squeal of the substrate is, for example, a sound generated by a squeak of the substrate generated when a voltage is applied to a capacitor on the substrate.

The threshold values Th1 and Th2 of the environmental sound determination unit 2069, the threshold value Th_Ndiff [n] of the short-term noise detection unit 2065, and the threshold value Th_Nint [n] of the long-term noise detection unit 2067 are based on the generated driving sound and the environmental sound. Will be decided. Therefore, the image pickup apparatus 100 may change these threshold values depending on the type of the optical lens 300, the inclination of the image pickup apparatus 100, and the like.

<Noise parameter for each lens>
First, when the user replaces the optical lens 300, the case where the same noise parameter is applied to each lens will be described with reference to FIGS. 7 and 10. In FIGS. 7 and 10, it is assumed that the image pickup apparatus 100 is equipped with different lenses. In addition, it is assumed that the same environmental sound is generated in FIGS. 7 and 10.

7 (a) and 10 (a) show the amplitude of each frequency spectrum of Lch_Before. 7 (b) and 10 (b) show the amplitude of each frequency spectrum of Nch_Before. Here, when comparing the respective frequency spectra, it can be read that they are different from each other in at least a part. This is because the noise acquired by each of the L microphone 201a, the R microphone 201b, and the noise microphone 201c differs depending on the shape and structure of the lens and the number and position of noise sources such as the drive body included in the lens. be.

7 (c) and 10 (c) show the amplitude of each frequency spectrum of NL. Here, the same noise parameter is used to generate the NL regardless of the lens mounted on the image pickup apparatus 100.

7 (d) and 10 (d) show the amplitude of each frequency spectrum of Lch_After. Since Lch_After is an audio signal recorded as an environmental sound, ideally, the frequency spectra shown in FIGS. 7 (d) and 10 (d) are the same. However, when the frequency spectra of FIGS. 7 (d) and 10 (d) are compared, it can be read that they are different from each other in at least a part. This is because the frequency spectrum of NL subtracted from Lch_Before is different from the frequency spectrum of the noise generated in each lens. The reason why the frequency spectrum of NL is different from the frequency spectrum of noise is that the noise parameter is not a parameter created for the noise generated from the lens, but a general-purpose parameter for a certain kind of noise. That is, when the same noise parameter is applied to the noise generated from various lenses, the audio data recorded as the environmental sound on the recording medium 110 becomes different audio data for each lens mounted on the image pickup apparatus 100. It ends up.

Therefore, in the present embodiment, the image pickup apparatus 100 effectively reduces the noise and records the environmental sound more accurately by updating the noise parameter according to the replacement of the lens.

<Update of noise parameters>
Here, a process of updating the noise parameter by the noise parameter updating unit 216 will be described. In this embodiment, the process of updating the noise parameter of PLx by the image pickup apparatus 100 will be described. The process of updating the noise parameter of PRx is the same as the process of updating the noise parameter of PLx.

FIG. 11 is an example of a block diagram of the noise parameter update unit 216.

The noise parameter calculation unit 2161 generates PLx from the input Lch_Before and Nch_Before.

The comparator 2162 compares the PLx generated in the noise parameter calculation unit 2161 with the PLx recorded in the noise parameter recording unit 205. Here, the comparator 2162 reads the PLx recorded in the noise parameter recording unit 205 based on the data indicating the type of the noise parameter input from the noise parameter selection unit 206. For example, when the type of the noise parameter is long-term noise, the comparator 2162 reads the noise parameter for the long-term noise from the noise parameter recording unit 205. The comparator 2162 overwrites and records a coefficient having a smaller value than the coefficient of each frequency spectrum of PLx recorded in the noise parameter recording unit 205 among the frequency spectrum coefficients of PLx generated by the noise parameter calculation unit 2161. .. The reason for updating in this way is as follows.

For example, let S be the amplitude of the frequency spectrum of the environmental sound picked up by the L microphone 201a at a certain frequency, and let Nfl be the amplitude of the frequency spectrum of the noise sound from the lens. Further, at that frequency, the amplitude of the frequency spectrum of the noise sound from the lens collected by the noise microphone 201c is defined as Nfn. In this embodiment, the amplitude of the environmental sound picked up by the noise microphone 201c is assumed to be sufficiently smaller than that of Nfn. In this case, PLx can be expressed by the following equation.
PLx = (S + Nfl) / Nfn

In order to effectively reduce noise, it is desirable that the noise parameter is PLx (≈Nfl / Nfn) when S is close to 0. That is, the value of the desirable noise parameter PLx is small. Further, the case where S is close to 0 is the case where the environmental sound is small. Therefore, in this embodiment, the comparator 2162 sets the coefficient of the frequency spectrum, which is smaller than the PLx value recorded in the noise parameter recording unit 205, among the PLx values generated by the noise parameter calculation unit 2161, into new noise. Record as a parameter. In this case, the amplitude of the environmental sound when the PLx is generated in the noise parameter calculation unit 2161 is smaller than the amplitude of the environmental sound when the PLx recorded in the noise parameter recording unit 205 is generated.

The comparator 2162 compares PLx for each frequency spectrum recorded in the noise parameter. Further, the comparator 2162 does not update the coefficient of the frequency spectrum having the PLx value or more recorded in the noise parameter recording unit 205 among the PLx values generated by the noise parameter calculation unit 2161.

The counter 2163 counts the number of times the noise parameters have been compared in the comparator 2162. In this embodiment, the initial value of the number of times the noise parameters are compared is 0. The counter 2163 resets the number of times the noise parameters were compared when the lens was replaced. In this embodiment, the counter 2163 resets the number of times the noise parameters are compared in response to receiving the lens information from the lens control unit 102.

From now on, the noise parameter update process by the noise parameter update unit 216 will be described with reference to the timing chart of FIG. In this timing chart, the timing at which the user turns on the power of the image pickup apparatus 100 is set to time t0. In this embodiment, the noise parameter PLx for one frequency spectrum will be described. In this embodiment, the initial value of this noise parameter PLx is 1.4.

In this embodiment, after the time t0, the image pickup apparatus 100 starts the live view operation and performs AF processing on the subject. When noise generated by driving the motor of the optical lens 300 or the like is detected along with this AF processing, the noise parameter is updated. During the noise parameter update process, the noise parameter selection unit 206 performs noise type discrimination process.

At time t1, the noise parameter calculation unit 2161 calculates the noise parameter PLx according to the start of driving the lens. For example, the lens is driven in response to the acceptance of the operation for focusing on the subject by the image pickup apparatus 100. Here, for example, when the lens is driven from the time t1 to the time t2, the lens control signal becomes High during that time. The noise parameter calculation unit 2161 calculates PLx based on the data indicating the type of noise parameter from the noise parameter selection unit 206 while the lens control signal is High. Further, the noise parameter calculation unit 2161 ends the calculation of PLx according to the fact that the lens control signal becomes Low at time t2. Here, the value of PLx generated by the noise parameter calculation unit 2161 is 1.3. For example, when a plurality of types of noise are generated in the optical lens 300 between the time t1 and the time t2, the noise parameter calculation unit 2161 generates PLx for each noise. The noise parameter calculation unit 2161 outputs data indicating the type of noise parameter and the generated PLx to the comparator 2162.

At time t2, the comparator 2162 compares the PLx generated by the noise parameter calculation unit 2161 with the PLx recorded in the noise parameter recording unit 205, and determines a smaller value as the value of the noise parameter PLx. At time t2, the PLx (= 1.3) newly generated by the noise parameter calculation unit 2161 is smaller than the PLx recorded in the noise parameter recording unit 205. Therefore, the comparator 2162 updates the PLx by recording 1.3 as the value of the noise parameter PLx in the noise parameter recording unit 205.

The comparator 2162 determines the type of noise parameter to be updated based on the data indicating the type of noise parameter and PLx output from the noise parameter calculation unit 2161. For example, when PLx for short-term noise is output from the noise parameter calculation unit 2161, the comparator 2162 compares the PLx output from the noise parameter calculation unit 2161 with the PL2 recorded in the noise parameter recording unit 205. do.

When a plurality of PLxs are output for the same type of noise from the noise parameter calculation unit 2161, the comparator 2162 is generated by combining the coefficients having the lowest values for each frequency spectrum among the plurality of PLxs. Use the noise parameters. This is because it is considered that the coefficient with a lower value has a smaller amplitude of the environmental sound when it is generated. In this case, the comparator 2162 compares the noise parameter generated in combination with the PLx recorded in the noise parameter recording unit 205.

Similarly, when the lens is driven from time t3 to time t4, PLx calculated by the noise parameter calculation unit 2161 at time t4 is 1.1. At time t4, the comparator 2162 compares the PLx calculated by the noise parameter calculation unit 2161 with the PLx recorded in the noise parameter recording unit 205, and determines a smaller value as the noise parameter PLx. At time t4, the PLx (= 1.1) calculated by the noise parameter calculation unit 2161 is smaller, so the comparator 2162 records 1.1 as the value of the noise parameter PLx in the noise parameter recording unit 205. , PLx is updated.

On the other hand, when the lens is driven from time t5 to time t6, PLx calculated by the noise parameter calculation unit 2161 at time t6 is 1.6. At time t6, the comparator 2162 compares the PLx calculated by the noise parameter calculation unit 2161 with the PLx recorded in the noise parameter recording unit 205, and determines a smaller value as the noise parameter PLx. At time t6, since the PLx (= 1.1) recorded in the noise parameter recording unit 205 is smaller, the comparator 2162 does not update the PLx.

After that, the noise parameter updating unit 216 executes the noise parameter PLx updating process in response to the driving of the optical lens 300 in the same manner.

In addition, even when the user performs an operation that causes the image pickup device 100 to perform an operation accompanied by noise before starting the moving image shooting, the noise parameter update process by the image pickup device 100 is performed. It will be done. For example, in response to the user pressing the release switch 61 halfway, the image pickup apparatus 100 drives the motor of the optical lens 300 to focus on the subject. Using the noise generated at this time, the image pickup apparatus 100 updates the noise parameter.

Even when the noise parameter updating unit 216 updates the noise parameter, the noise parameter recording unit 205 holds the initial value of the noise parameter. This is because the noise parameter updating unit 216 changes the noise parameter to the initial value when the optical lens 300 mounted on the image pickup apparatus 100 is replaced. As described above, when the optical lens 300 mounted on the image pickup apparatus 100 is replaced, the noise acquired by the L microphone 201a, the R microphone 201b, and the noise microphone 201c changes. In this embodiment, the noise parameter update unit 216 updates the noise parameter from a general-purpose initial value in order to cope with the change. By updating the noise parameter from a general-purpose initial value each time the lens is replaced, the image pickup apparatus 100 can perform noise reduction processing using the noise parameter adapted to the mounted lens.

Next, the noise parameter update process of the image pickup apparatus 100 will be described with reference to the flowchart of FIG. 13 and the timing chart of FIG. The processing of the image pickup apparatus 100 is started, for example, when the power switch 72 is operated by the user and the power is turned on. Further, for example, the process of the image pickup apparatus 100 is started when the mode dial is operated by the user and the mode of the image pickup apparatus 100 is set to the shooting mode.

Here, in this embodiment, the image pickup apparatus 100 displays the image captured by the image pickup unit 101 in a live view after the power is turned on.

In step S1300, the lens control unit 102 receives lens information from the optical lens 300. The lens information is, for example, the type of lens, the model number of the lens, the type of noise source, and the like. The lens control unit 102 outputs the received lens information to the control unit 111 and the counter 2163. The control unit 111 records the received lens information in the non-volatile memory 117.

In step S1301, the lens control unit 102 determines whether or not the lens has been replaced. For example, the lens control unit 102 compares the lens information recorded in the non-volatile memory 117 before the process of step S1301 with the lens information received in step S1300. When it is determined that the two lens information match, the lens control unit 102 determines that the lenses have not been exchanged. If it is determined that the two lens information do not match, the lens control unit 102 determines that the lenses have been exchanged. Further, for example, when the lens information is not recorded in the non-volatile memory 117 before the process of step S1301, the lens control unit 102 determines that the lens has been replaced. When the lens control unit 102 determines that the lens has been replaced, the process of step S1302 is executed. If the lens control unit 102 determines that the lens has not been replaced, the process of step S1305 is executed.

In step S1302, the noise parameter updating unit 216 changes the value of the noise parameter recorded in the noise parameter recording unit 205 to the initial value.

In step S1303, the counter 2163 resets the number of times the noise parameters have been compared. For example, the counter 2163 sets the number of times the noise parameters are compared to 0.

In step S1304, the control unit 111 turns on the recording path to the noise parameter update unit 216. For example, the control unit 111 controls the voice input unit 104 so as to start voice processing. In this step, the noise parameter selection unit 206 also starts processing.

In step S1305, the control unit 111 determines whether or not the number of times the noise parameters are compared is equal to or less than a predetermined number of times. For example, the control unit 111 determines whether or not the number of times the noise parameters are compared is 4 times or less. If it is determined that the number of times the noise parameters are compared is less than or equal to the predetermined number of times, the process of step S1306 is executed. If it is determined that the number of times the noise parameters are compared is greater than the predetermined number of times, the process of step S1312 is executed.

In step S1306, the control unit 111 determines whether or not the optical lens 300 has started driving. For example, the control unit 111 determines whether the level of the lens control signal output from the lens control unit 102 is High or Low. When the level of the lens control signal is High, the control unit 111 determines that the optical lens has started driving. When the level of the lens control signal is Low, the control unit 111 determines that the optical lens is not driven. If it is determined that the optical lens 300 has started driving, the process of step S1307 is executed. If it is determined that the optical lens 300 is not driven, the process of step S1310 is executed.

In step S1307, the noise parameter update unit 216 generates the noise parameter PLx. In this step, the noise parameter update unit 216 generates a noise parameter for the noise each time the noise is generated.

In step S1308, the control unit 111 determines whether or not the optical lens 300 has finished driving. For example, the control unit 111 determines whether the level of the lens control signal output from the lens control unit 102 is High or Low. When the level of the lens control signal is High, the control unit 111 determines that the optical lens is being driven. When the level of the lens control signal is Low, the control unit 111 determines that the optical lens has finished driving. If it is determined that the optical lens 300 is being driven, the process of step S1307 is executed. If it is determined that the optical lens 300 has finished driving, the process of step S1309 is executed.

In step S1309, the noise parameter update unit 216 increases the number of times the noise parameters are compared by one time.

In step S1310, the noise parameter updating unit 216 compares the noise parameter PLx generated in step S1307 with the PLx recorded in the noise parameter recording unit 205. If it is determined that the noise parameter PLx generated in step S1307 is smaller, the process of step S1311 is executed. If it is determined that the PLx recorded in the noise parameter recording unit 205 is smaller, the process of step S1313 is executed.

In step S1311, the noise parameter update unit 216 updates PLx. For example, the noise parameter updating unit 216 overwrites and records the noise parameter PLx generated in step S1307 on the PLx recorded in the noise parameter recording unit 205. As a result, the image pickup apparatus 100 can utilize the desirable noise parameters in the moving image recording described later. Further, the image pickup apparatus 100 can effectively reduce noise by using desirable noise parameters. The noise parameter recording unit 205 holds the initial value of the noise parameter even when the process of this step is executed.

Up to this point, the case where the number of comparisons of noise parameters is less than or equal to a predetermined number has been described. In the timing chart of FIG. 14, it corresponds to the period from time t1 to t8. Next, in the process of step S1305, the process when it is determined that the number of times of comparison of the noise parameters is more than the predetermined number of times will be described. In the timing chart of FIG. 14, it corresponds to the period after the time t8.

In step S1312, the control unit 111 turns off the recording path to the noise parameter update unit 216. For example, the control unit 111 controls the voice input unit 104 so as to stop the voice processing. In this step, the noise parameter selection unit 206 also stops processing.

In step S1313, the control unit 111 determines whether or not to start video recording. For example, the control unit 111 determines that the moving image recording is started in response to the pressing of the release switch 61. On the contrary, the control unit 111 determines that the moving image recording is not started when the release switch 61 is not pressed. If the control unit 111 determines to start video recording, the process of step S1314 is executed. If the control unit 111 determines that the moving image recording is not started, the process of step S1300 is executed.

In step S1314, the control unit 111 turns on the recording path to the voice input unit 104. For example, the control unit 111 controls the voice input unit 104 so as to start voice processing.

In step S1315, the control unit 111 records a moving image. For example, as described above, the control unit 111 records the moving image data with audio on the recording medium 110. In the voice processing during the moving image recording, the noise parameter recorded in the noise parameter recording unit 205 is used.

In step S1316, the control unit 111 determines whether or not to end the moving image recording. For example, the control unit 111 determines that the moving image recording is finished in response to the pressing of the release switch 61. On the contrary, the control unit 111 determines that the moving image recording is not finished when the release switch 61 is not pressed. When the control unit 111 determines that the moving image recording is finished, the process of step S1317 is executed. If the control unit 111 determines that the moving image recording is not completed, the process of step S1315 is executed.

In step S1317, the control unit 111 turns off the recording path to the noise parameter update unit 216. For example, the control unit 111 controls the voice input unit 104 so as to stop the voice processing.

The process of updating the noise parameter of the image pickup apparatus 100 has been described above. In this way, a new noise parameter is generated according to the drive of the lens before the start of the moving image recording process. Then, when the newly generated noise parameter can reduce the noise more appropriately than the already generated noise parameter, the newly generated noise parameter is used to update the recorded noise parameter. By using the updated noise parameters to reduce the noise contained in the sound during the moving image recording, the image pickup apparatus 100 can record the moving image data with sound having good sound quality.

The noise parameter recording unit 205 retains the updated noise parameters even after the power of the image pickup apparatus 100 is turned off. As a result, when the power of the image pickup device 100 is turned on next time, the image pickup device 100 can record the moving image data with sound using the updated noise parameter if the lens is not replaced. can.

In this embodiment, the noise parameter updating unit 216 does not execute the process of updating the noise parameter during the moving image recording. The image pickup apparatus 100 can record sound while maintaining sound quality by keeping the noise parameter constant during video recording.

In this embodiment, the noise parameter update unit 216 increases the number of times the noise parameters are compared in the step before the noise parameters are compared, but the noise parameters are compared after the noise parameters are compared. The number of times may be increased.

The noise parameter updating unit 216 may determine whether or not to update the noise parameter based on the inclination of the image pickup apparatus 100 acquired by the information acquisition unit 103. For example, when it is assumed that the noise parameter is applied when the image pickup device 100 is horizontal, the noise parameter update unit 216 may not be able to correctly calculate the noise parameter when the image pickup device 100 is facing upward. .. Therefore, in this case, the noise parameter update unit 216 executes an update process for updating the noise parameter when it is determined that the image pickup apparatus 100 is horizontal. On the contrary, when the noise parameter update unit 216 determines that the image pickup apparatus 100 is not horizontal, the noise parameter update unit 216 does not execute the update process for updating the noise parameter. For example, the range of inclination that the image pickup apparatus 100 can be regarded as horizontal is the range in which the inclination of the image pickup apparatus 100 is within ± 30 degrees with respect to the horizontal direction.

When the optical lens 300 is a zoom lens, the noise parameter updating unit 216 may determine whether or not to update the noise parameter based on the zoom magnification of the optical lens 300 acquired by the lens control unit 102. good. For example, if it is assumed that the noise parameter is applied when the optical lens 300 is at the wide end, the noise parameter update unit 216 may not be able to correctly calculate the noise parameter when the optical lens 300 is at the telephoto end. There is. Therefore, in this case, the noise parameter update unit 216 executes an update process for updating the noise parameter when it is determined that the optical lens 300 is at the wide end. On the contrary, when the noise parameter update unit 216 determines that the optical lens 300 is not at the wide end, the noise parameter update unit 216 does not execute the update process for updating the noise parameter. For example, the range of the zoom magnification that can be regarded as the wide end of the optical lens 300 is the range in which the zoom magnification of the optical lens 300 is 1 to 2 times.

The noise parameter recording unit 205 may record noise parameters for each lens. This is because the noise is reduced in response to the fact that the noise picked up by the microphone of the voice input unit 104 is different for each lens. In this case, the image pickup apparatus 100 executes an update process for updating the noise parameter with respect to the noise parameter for each lens.

[Second Example]
In the second embodiment, the noise parameter update process different from that of the first embodiment will be described. Here, the configuration of the image pickup apparatus 100 is the same as that of the first embodiment. Although the PLx noise parameter update process will be described in this embodiment as well, the PRx noise parameter update process is the same as the PLx noise parameter update process.

From now on, the noise parameter update process of the image pickup apparatus 100 will be described with reference to the flowchart of FIG. 15 and the timing chart of FIG. The processing of the image pickup apparatus 100 is started, for example, when the power switch 72 is operated by the user and the power is turned on.

Since the processes of steps S1500 to S1510 are the same as the processes of steps S1300 to S1310 of FIG. 13, the description thereof will be omitted.

In step S1511, the noise parameter updating unit 216 overwrites and records the noise parameter PLx generated in step S1307 on the PLx recorded in the noise parameter recording unit 205. After the process of step S1510, the process of step S1511 is executed.

In step S1512, the control unit 111 turns off the recording path to the noise parameter update unit 216. For example, the control unit 111 controls the voice input unit 104 so as to stop the voice processing.

Since the processes of steps S1513 to S1515 are the same as the processes of steps S1313 to S1315 of FIG. 13, the description thereof will be omitted.

In step S1516, the control unit 111 determines whether or not the optical lens 300 has been driven. For example, the control unit 111 determines whether the level of the lens control signal output from the lens control unit 102 is High or Low. When the level of the lens control signal is High, the control unit 111 determines that the optical lens has been driven. When the level of the lens control signal is Low, the control unit 111 determines that the optical lens is not driven. If it is determined that the optical lens 300 has been driven, the process of step S1517 is executed. If it is determined that the optical lens 300 is not driven, the process of step S1519 is executed.

In step S1517, the noise parameter update unit 216 generates the noise parameter PLx.

In step S1518, the noise parameter update unit 216 increases the number of times the noise parameters are compared by one time.

In step S1519, the noise parameter updating unit 216 compares the noise parameter PLx generated in step S1517 with the PLx recorded in the noise parameter recording unit 205. If it is determined that the noise parameter PLx generated in step S1517 is smaller, the process of step S1520 is executed. If it is determined that the PLx recorded in the noise parameter recording unit 205 is smaller, the process of step S1521 is executed.

In step S1520, the noise parameter update unit 216 updates PLx. For example, the noise parameter updating unit 216 overwrites and records the noise parameter PLx generated in step S1517 on the PLx recorded in the noise parameter recording unit 205. The noise parameter recording unit 205 holds the initial value of the noise parameter even when the process of this step is executed.

In step S1521, the control unit 111 determines whether or not to end the moving image recording. For example, the control unit 111 determines that the moving image recording is finished in response to the pressing of the release switch 61. On the contrary, the control unit 111 determines that the moving image recording is not finished when the release switch 61 is not pressed. When the control unit 111 determines that the moving image recording is finished, the process of step S1522 is executed. If the control unit 111 determines that the moving image recording is not completed, the process of step S1515 is executed.

In step S1521, the control unit 111 turns off the recording path to the noise parameter update unit 216. For example, the control unit 111 controls the voice input unit 104 so as to stop the voice processing.

As described above, in this embodiment, the image pickup apparatus 100 turns off the recording path in response to updating the noise parameter before recording the moving image. As a result, the image pickup apparatus 100 can turn off the recording path by driving the lens once at the earliest, so that the power consumption in the audio input unit 104 can be suppressed. Further, in this embodiment, the image pickup apparatus 100 updates the noise parameter even during video recording. As a result, it is possible to efficiently reduce noise and record voice even during video recording.

[Other embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the present invention is not limited to the above embodiment as it is, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments. In addition, components across different embodiments may be combined as appropriate.

Claims (20)

Imaging means and
With the first microphone to get the ambient sound,
With a second microphone to get the sound from the noise source,
The first conversion means for generating the first audio signal by Fourier transforming the audio signal from the first microphone,
A second conversion means that Fourier transforms the audio signal from the second microphone to generate a second audio signal,
A generation means for generating noise data using the second audio signal and the first parameter related to noise of the noise source.
A subtraction means for subtracting the noise data from the first audio signal,
A third transforming means for inverse Fourier transforming the audio signal from the subtracting means,
A recording means for recording the moving image data generated by the imaging means and the audio signal from the third conversion means on a recording medium as moving image data with audio.
In a state where the video data with audio is not recorded by the recording means, the first audio signal and the second audio signal are used to generate parameters related to noise of the noise source, and the generation is performed. An update means for updating the first parameter using the above-mentioned parameters, and
A voice processing device characterized by having. When the number of times the parameter relating to the noise of the noise source is generated exceeds a predetermined number of times in a state where the moving image data with audio is not recorded by the recording means, the updating means causes the noise of the noise source. The voice processing apparatus according to claim 1, wherein the process of generating the parameter is terminated. The update means according to claim 1 or 2, wherein when it is determined that the lens attached to the image pickup means has been replaced, the updating means resets the number of times the parameter related to the noise of the noise source is generated. Voice processing device. When the first parameter is updated in a state where the video data with audio is not recorded by the recording means, the updating means ends the process of generating the parameter related to the noise of the noise source. The voice processing apparatus according to any one of claims 1 to 3. The update means is any one of claims 1 to 4, wherein the updating means performs a process of updating the first parameter even when the recording means is recording the moving image data with audio. The audio processing device described in. Claims 1 to 5, wherein the updating means changes the first parameter updated by the updating means to an initial value when it is determined that the lens attached to the imaging means has been replaced. The voice processing device according to any one of the following items. When the first noise parameter is updated by the updating means, the generation means uses the first noise parameter updated by the updating means and the second voice signal to obtain the noise data. The voice processing apparatus according to any one of claims 1 to 6, wherein the voice processing apparatus is generated. When the updating means determines that the value of the parameter generated by using the first audio signal and the second audio signal is smaller than the value of the first parameter, the first audio signal. The voice processing apparatus according to any one of claims 1 to 7, wherein the first parameter is updated by using a parameter generated by using the second voice signal and the second voice signal. In the updating means, the amplitude of the environmental sound when the parameter generated by using the first audio signal and the second audio signal is generated is the amplitude of the environmental sound when the first parameter is generated. When the amplitude is smaller than the amplitude of, any one of claims 1 to 8, wherein the first parameter is updated by using the parameter generated from the first voice signal and the second voice signal. The audio processing device according to item 1. From claim 1, the updating means updates the first parameter for each frequency spectrum based on the parameters generated by using the first voice signal and the second voice signal. 9. The audio processing device according to any one of 9. The lens has a driving means as the noise source.
The updating means is characterized in that, while the driving means is being driven, the first voice signal and the second voice signal are used to generate a parameter related to noise of the noise source. The voice processing device according to any one of claims 1 to 10. When a plurality of parameters are generated while the driving means is being driven, the updating means uses the parameter having a smaller amplitude of the environmental sound when the plurality of parameters are generated, and the first one. The voice processing apparatus according to claim 11, wherein a process for updating a parameter is executed. The generation means obtains at least one of the plurality of the first parameters and the second voice signal, including a parameter corresponding to the first kind of noise and a parameter corresponding to the second kind of noise. The voice processing apparatus according to any one of claims 1 to 12, wherein the noise data is generated by using the voice processing apparatus. The generation means generates the noise data by using the parameter corresponding to the type of noise included in the second voice signal among the plurality of first parameters and the second voice signal. 13. The voice processing device according to claim 13. Further, it has a determination means for determining the type of noise contained in the second audio signal, and has a determination means.
The updating means is characterized in that the parameter corresponding to the type of noise included in the second voice signal among the plurality of first parameters is updated based on the type of noise determined by the determination means. The voice processing device according to claim 13 or 14. The first parameter according to any one of claims 1 to 15, further comprising means for holding the first parameter updated by the updating means when the power of the voice processing device is turned off. Voice processing device. The voice processing apparatus according to any one of claims 1 to 16, wherein the noise includes at least one of constant noise, short-term noise, and long-term noise. The voice processing apparatus according to any one of claims 1 to 17, wherein the first parameter is a ratio of the amplitudes of the first voice signal and the second voice signal. It is a control method of a voice processing device having an image pickup means, a first microphone for acquiring environmental sound, and a second microphone for acquiring sound from a noise source.
The step of Fourier transforming the audio signal from the first microphone to generate the first audio signal,
The step of Fourier transforming the audio signal from the second microphone to generate the second audio signal,
A generation step of generating noise data using the second audio signal and the first parameter related to noise of the noise source.
A subtraction step of subtracting the noise data from the first audio signal,
A step of inverse Fourier transforming the audio signal generated by the subtraction step,
A recording step of recording the moving image data generated by the imaging means and the inverse Fourier transformed voice signal as moving image data with sound on a recording medium.
In a state where the video data with audio is not recorded, the first audio signal and the second audio signal are used to generate a parameter related to the noise of the noise source, and the generated parameter is used. A control method comprising an update step for updating the first parameter. A computer-readable program for operating a computer as each means of the voice processing device according to any one of claims 1 to 18.

Images