JP2011234139A - Three-dimensional audio signal generating device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional audio signal that is linked with a position and depth capable of enhancing realistic sensation in accordance with the position and the depth state of an object (particularly, a person) inside a video signal, because even if a two-dimensional video signal becomes three-dimensional, since voice was as usual, a conventional audio signal was too innocuous for viewers to enjoy reality and presence inherent to a moving video signal and a powerful image.SOLUTION: Depth information of a screen is obtained from information of conversion processing (extraction of a motion vector and a high frequency component) from a two-dimensional image to a three-dimensional image, picture elements are moved so as to become the three-dimensional image based on the information thereof, and the object position information and depth information can be extracted. By using these object position information and depth information, a direction, a phase and a volume of a sound are controlled, thereby generating a three-dimensional audio signal from a two-dimensional audio signal.

Description

  The present invention extracts information obtained from a process of generating a 3D video from a 2D video, or person position and depth information in the video from the 3D video information, and 3D from a 2D audio signal based on the depth information. The present invention relates to an apparatus for generating a three-dimensional audio signal.

  In recent years, with the development of video technology, the spread of 3D video devices that display content produced as 3D video and programs produced as 2D video as 3D video full of realism is expected. The 3D video is created separately for the right-eye image and the left-eye image based on the binocular parallax technique. In the 3D video device, the right-eye image and the left-eye image are alternately switched and displayed, and the user can enjoy the 3D video by viewing the image synchronized with the switching timing using the 3D video viewing glasses. Can do.

  As a method for converting a 2D video signal into a 3D video signal, for example, Patent Document 1 discloses the following technique. For each of a plurality of parallax calculation areas set in each field screen of the 2D video signal, an image feature amount related to the perspective of the video is extracted, and based on this image feature amount, parallax information is obtained for each unit area. The image signal for right eye and left eye which generate | occur | produces and has the horizontal phase difference according to the parallax information corresponding to a predetermined unit area | region is produced | generated, and it displays as a three-dimensional video signal.

  Further, in Patent Document 2, information related to the position of a sound source or a subject is added to audio information, image information, and the like, and the information is used to give audio information with a sense of reality or a stereoscopic effect to the viewer. An information recording apparatus technology that can be used is disclosed.

JP-A-10-51812 JP 2004-32726 A

  FIG. 12 is a block diagram showing the overall configuration of the 2D / 3D video conversion device described in Patent Document 1. As shown in FIG. In the apparatus, an image feature amount (high-frequency component integrated value, luminance contrast, luminance integrated value, and saturation integrated value) related to the perspective of the video is calculated by each circuit from the input two-dimensional video signal. Parallax information is extracted based on the feature amount. An image signal for the right eye and an image signal for the left eye are created from the extracted parallax information, converted into a three-dimensional video signal, and output to the screen. However, even if the two-dimensional video signal is visually three-dimensional, the sound is the same as before, so the viewer can enjoy the original reality, presence, and powerful video of the three-dimensional video signal. Had the problem of lacking power.

  FIG. 13 is a block diagram showing the configuration of the information recording apparatus described in Patent Document 2. This information recording device is a method of measuring the position information of each object with a sensor or the like when recording a video and recording it with a plurality of microphones, which causes a work load of adding position information to audio information, image information, etc. Since expensive equipment such as a microphone is used, it is expensive. Further, when trying to convert a two-dimensional moving image signal into a three-dimensional moving image signal, there is a problem that a three-dimensional audio signal cannot be provided because position information is not added to the original signal.

  The present invention has been made to solve such a problem, and has the following configuration.

  A three-dimensional audio signal generation device according to the present invention includes a signal separation unit that separates an input moving image signal into a video signal and an audio signal, and object orientation / depth information extraction that extracts object orientation information and depth information from the video signal. And an object azimuth addition / gain adjustment unit for adding azimuth information to the audio signal or adjusting gain based on the audio signal, the object azimuth information, and the depth information.

  In the three-dimensional audio signal generation device according to the present invention, when the input video signal is a two-dimensional video signal, the object orientation / depth information extraction unit includes at least one of a contour detection unit and a motion vector detection unit, It is characterized by comprising a combination of at least one of a high frequency component detector and a luminance component detector.

  In the three-dimensional audio signal generation device according to the present invention, when the input video signal is a three-dimensional video signal, the object orientation / depth information extraction unit includes a contour detection unit and a pixel parallax detection unit. It is characterized by.

  In the three-dimensional audio signal generation device according to the present invention, the contour detection unit detects a contour of a person based on a correlation with an existing contour of a person, and the screen has a leading role when the contour has an area equal to or larger than a threshold value. It is characterized by.

  In the three-dimensional audio signal generation device according to the present invention, when the contour detection unit detects the contours of a plurality of persons, the depth information is set closer to the viewer as the contour area is larger. And

  A 3D audio signal generation apparatus according to the present invention includes a signal separation unit that separates an input moving image signal into a video signal and an audio signal, and a 2D that detects whether the video signal is a 2D video signal or a 3D video signal. / 3D video detection unit, a first object orientation / depth information extraction unit that extracts object orientation information and depth information when the video signal is a 2D video signal, and the video signal is a 3D video In the case of a signal, a second object orientation / depth information extraction unit that extracts object orientation information and depth information, and the audio signal and the object orientation extracted by the first or second object orientation / depth information extraction unit An object orientation addition / gain adjustment unit for adding orientation information or adjusting gain to the audio signal based on information and depth information is provided.

  The first object orientation / depth information extraction unit of the three-dimensional audio signal generation device according to the present invention includes at least one of a contour detection unit and a motion vector detection unit, and at least one of a high frequency component detection unit and a luminance component detection unit. It is characterized by comprising.

  The second object orientation / depth information extraction unit of the three-dimensional audio signal generation device according to the present invention includes an outline detection unit and a pixel parallax detection unit.

  The contour detection unit of the three-dimensional audio signal generation device according to the present invention detects the contour of a person based on a correlation with an existing contour of a person, and takes the case where the contour has an area equal to or larger than a threshold as the main role of the screen. Features.

  In the contour detection unit of the three-dimensional audio signal generation device according to the present invention, when the contour detection unit detects the contours of a plurality of persons, the depth information is displayed on the side closer to the viewer as the contour area is larger. It is characterized by setting.

  The object orientation addition / gain adjustment unit of the 3D audio signal generation device according to the present invention can provide a 3D audio signal according to a user's viewing mode by referring to a spatial transfer function database. .

  The object orientation addition / gain adjustment unit of the 3D audio signal generation device according to the present invention generates a 3D audio signal on the premise that the object is attached to a frame of 3D video viewing glasses and viewed with a speaker. To do.

  The object orientation addition / gain adjustment unit of the three-dimensional audio signal generation device according to the present invention generates a three-dimensional audio signal on the premise that the object is attached to a frame of 3D video viewing glasses and viewed with an inner speaker or headphones. It is characterized by.

  The object orientation addition / gain adjustment unit of the three-dimensional audio signal generation device according to the present invention generates a three-dimensional audio signal on the premise of viewing with a surround speaker.

  According to the present invention, it is possible to convert a two-dimensional video signal and a two-dimensional audio signal (stereo broadcast) of a general stereo broadcast such as a digital broadcast or a game into a three-dimensional video signal and a three-dimensional audio signal. It is possible to provide a viewer with a powerful video signal and audio signal in the sense of reality and presence.

  If the input video signal is already a 3D video signal, the binocular parallax information of the video signal is extracted to obtain the approximate position of each object in the image and the depth information for each object. Therefore, it is possible to control the azimuth, phase, and sound volume of the input audio signal in accordance with the video signal.

1 is a functional block diagram illustrating a main configuration of a three-dimensional audio signal generation device according to Embodiment 1. FIG. 3 is a functional block diagram illustrating a main configuration of an object orientation / depth information extraction unit that extracts object orientation information and depth information from a two-dimensional video signal according to Embodiment 1. FIG. It is a figure which shows the parallax calculation area | region which divided the image signal of 1 frame which concerns on Example 1 into m area | region in a horizontal direction, and n area | region in a vertical component. It is a figure which shows the glasses for 3D video viewing with a speaker for seeing the 3D video signal based on Example 1. FIG. It is a figure which shows an example in the case of recording in advance the spatial transfer function including a head-related transfer function using the dummy head which concerns on Example 1. FIG. FIG. 6 is a functional block diagram illustrating a main configuration of a three-dimensional audio signal generation device according to a second embodiment. FIG. 10 is a functional block diagram illustrating a main configuration of an object orientation / depth information extraction unit that extracts object orientation information and depth information from a 3D video signal according to Embodiment 2; It is a figure which shows the extraction method of the depth information which concerns on Example 2. FIG. FIG. 10 is a functional block diagram illustrating a main configuration of a three-dimensional audio signal generation device according to a third embodiment. It is explanatory drawing at the time of outputting the three-dimensional audio | voice signal which concerns on Example 4 to a surround speaker. It is explanatory drawing at the time of carrying out the outline detection of the person's face by the outline detection part on the parallax calculation area | region which concerns on Example 4. FIG. It is a block diagram which shows the whole structure of the 2D / 3D video conversion apparatus of the conventional display apparatus. It is a block diagram which shows the structure of the information recording device for recording the conventional three-dimensional sound.

  Hereinafter, the present invention will be described in detail with reference to the drawings illustrating embodiments thereof.

  FIG. 1 is a functional block diagram illustrating the main configuration of the three-dimensional audio signal generation device according to Embodiment 1 of the present invention. The three-dimensional audio signal generation apparatus of the present invention can be used by being incorporated in a device that handles video signals and audio signals, such as a television, a recording / playback device, or a personal computer (PC), or can be used as an external unit. .

A two-dimensional moving image signal S ₀₀ , which is a mixed signal of the video signal and the audio signal in FIG. 1, is input to the video / audio processing unit 100, and then separated into a video signal and an audio signal. A 3D video signal S ₂₁ and a 3D audio signal S ₂₂ are generated from the audio signal and output. The video / audio processing unit 100 includes a signal separation unit 101, an object orientation / depth information extraction unit 102, an object orientation addition / gain adjustment unit 103, a 3D video generation unit 104, and a synchronization adjustment unit 105. . Signal separator 101, decodes the video signal _{S 00,} separates the video signal _{S 11} and the audio signal _{S 12.} In the object orientation, the depth information extraction unit 102 extracts the object orientation information and depth information from the video signal S ₁₁ which is separated by the signal separation unit 101. In the 3D image generation unit 104, based on each object orientation information and depth information extracted by the object orientation / depth information extraction unit 102, the object orientation addition / gain adjustment unit 103, the object orientation / depth information extraction unit based on each object orientation information and the depth information extracted at 102, the add the object orientation information to the audio signal S ₁₂ which is separated by the signal separation unit 101, or to adjust the gain. The synchronization adjustment unit 105 synchronizes the video signal output from the 3D video generation unit 104 and the audio signal output from the object orientation addition / gain adjustment unit 103 to output the video / audio processing unit 100. A video signal S ₂₁ and an audio signal S ₂₂ are output as signals.

FIG. 2 is a functional block diagram showing a main configuration of the object orientation / depth information extraction unit 102 shown in FIG. The object orientation / depth information extraction unit 102 includes a contour detection unit 110, a motion vector detection unit 111, a high frequency component detection unit 112, and a luminance component detection unit 113. The luminance component detection unit 113 includes at least one of luminance value integration means and luminance contrast calculation means. First, decoded by the signal separation unit 101, attention is paid to two-dimensional video signal S ₁₁ separated. Two-dimensional image signal S ₁₁ which is separated in the signal separation unit 101 is input to the object orientation, the depth information extraction unit 102, as shown in FIG. 2, each for converting the 2D image signal to the 3D image signal Information is detected. The object orientation information is acquired from the contour information detected from the contour detection unit 110 and the motion vector information detected from the motion vector detection unit 111. The depth information may be processed from the contour information detected from the contour detection unit 110, the motion vector information detected from the motion vector detection unit 111, and any one information of the high frequency component detection unit 112 and the luminance component detection unit 113. good. Alternatively, in order to increase accuracy, depth information may be extracted by processing a plurality of pieces of information in an arbitrary combination. Here, the high-frequency component and the luminance component are described to extract the depth information, but instead of these, a saturation component may be used, or a combination of these may be processed.

  In general, an image has a subject in front and a background in back, and the focus is often on the subject. For this reason, it is considered that an object closer to the camera to be photographed has higher high-frequency components, contrast, brightness, and saturation.

FIG. 3 shows a parallax calculation area obtained by dividing an image signal of one frame into m areas in the horizontal direction and n areas in the vertical component. The corner regions of the upper left portion as A _{1, 1,} represent each region a region of the lower right corner A _n, as _m. In the apparatus of the present invention, object orientation information and depth information are assigned to each divided area. However, a region with a relatively high frequency component value or a threshold value for luminance difference is set, and a region that occupies a certain amount (not necessarily 100%) within the threshold range is considered as one group, and the unit You may divide by.

  By using the contour detection unit 110 and the motion vector detection unit 111 shown in FIG. 2 and looking at the correlation with the contour of an existing person, it is possible to obtain the orientation information of the person in the video signal. If the area of the contour information of the person detected by the contour detection unit 110 is a contour having an area equal to or larger than the threshold, the extracted person is regarded as the main person on the screen and the contour is detected. The area is recognized as the image closest to the viewer. Therefore, when the TV screen is set to the center of the depth information, the viewer side is set to the front side, and the reverse direction is set to the back side, the depth information is set to the front side. When the contour information of a plurality of persons is detected, the depth information is set to the front for a person with a larger area. However, the range setting on the back side and the front side can be arbitrarily determined.

  When the motion vector detection unit 111 shown in FIG. 2 is used, it can be considered that an object exists in the region where the motion vector is detected. Therefore, it can be considered that an object having a large motion is reflected in the foreground, and therefore, an object having a large motion vector in the region detected by the motion vector detecting unit 111, that is, a region in which an object having a large motion is displayed is moved forward. Depth information is set on the back side of the area where the small object is displayed.

  The high frequency component detection unit 112 shown in FIG. 2 detects a region with a high frequency component based on the result of integrating the high frequency components (threshold value or more) of each region divided in FIG. In general, an image often focuses on a subject. Therefore, the subject can be determined as a high-frequency component because the luminance difference between adjacent pixels naturally increases. The region where the high frequency component value is relatively high is set in front.

  Similarly to the high frequency component detection unit 112, the luminance component detection unit 113 performs integration of luminance values or calculation of luminance contrast for each region divided in FIG. Here, the depth information is determined using the luminance value of the region detected by the luminance component detection unit 113. Since the depth information is considered to be in the foreground as the detected region has a higher luminance value, the calculated luminance value indicates the depth information with the higher luminance value in front and the lower luminance value in the back. Set. When setting the depth information, the determination may be made using the luminance contrast value instead of the luminance value, or may be made by adding both the luminance value and the luminance contrast value.

Next, attention is paid to the sound signal S ₁₂ which is separated from the two-dimensional video signal S ₀₀ is a mixed signal of the video signal and audio signal. In many cases, the sound of a broadcast signal is generally a stereo signal of two left and right channels. In this embodiment, the azimuth information is convoluted with respect to the two-channel audio signal by the following method.

First, the two-channel audio signals S _12, against performs division in the frequency domain. In the embodiment, it is divided into three, a low frequency band from 0 to 300 Hz, a voice band of 0.3 to 3.4 kHz, and a high frequency band above 3.4 kHz. In general, since humans are insensitive to the direction of low-frequency sound, the direction information is added only to the voice band and the high-frequency band.

Next, an example of processing for an audio band signal will be described. When the contour detection unit 110 in FIG. 2 detects the contour of a person, there is a high possibility that a voice is being emitted. However, in order to ensure the presence of speech, the presence or absence of speech is determined using the Kurtosis value for speech signals x _L (t) and x _R (t) in the speech band.

Kurtosis is a dimensionless quantity that represents the relative sharpness and flatness of the amplitude distribution based on the Gaussian distribution, and generally has the following tendencies.
・ The audio signal has a relatively high value.
-Signals close to Gaussian, such as music signals, take values close to zero.
・ Sub-Gaussian signals such as uniformly distributed noise take negative values.

Kurtosis values K (x (t)) and K (x (t)) for audio signals x _L (t) and x _R (t) are as follows.

From this, threshold values are determined for the kurtosis (Kurtosis) values K (x (t)) and K (x (t)), and the presence or absence of speech is determined.

  In addition, when there are a plurality of contours of a person detected by the contour detection unit 110, it recognizes the face image, sees if the movement of the mouth and the audio signal are synchronized, and who is talking It is also possible to detect and specify whether or not there is. Furthermore, when the contour of a person is such that the gender can be discriminated, the frequency spectrum of the audio signal can be analyzed to identify and associate a male voice or a female voice.

  Next, the addition of azimuth information to the audio signal will be described. 3D video is produced based on binocular parallax technology, using 3D video viewing glasses that display only the right and left eye images by polarizing or active shuttering the right and left eye images. You can enjoy it as a 3D image.

  FIG. 4 is a diagram showing 3D video viewing glasses with speakers. FIG. 4A shows an example in which speakers are attached to both sides of a frame of 3D video viewing glasses. FIG. 4B shows an example in which inner speakers are attached to both sides of a frame of 3D video viewing glasses.

  In order to view 3D audio using a speaker attached to the frame of the 3D video viewing glasses of FIG. 4A, an inverse filter is applied to a normal stereo signal based on a spatial transfer function measured in advance. It is an indispensable condition to generate an audio signal that can be heard from outside rather than an audio signal that can be heard inside the head. In the case of using the inner speaker or the headphones of FIG. 4B, since it becomes an audio signal that can be heard inside the head, only the head-related transfer function needs to be considered.

  FIG. 5 shows an example in which a spatial transfer function including a head-related transfer function is recorded in advance using a dummy head. Also, image methods (methods that assume that you are in a room of a specific size and generate artificial reverberation by computation, such as JBAllenandD.A.Berkley, "Image Method for Efficiently Simulating Smallroom Acoustics", J.Acoust .Soc.Am., Vol.65, No.4, pp.943-950, 1979), etc., even if the head-related transfer function is convoluted, a pseudo impulse response ( Spatial transfer function). The spatial transfer function database is created in advance for each parallax calculation area divided in FIG. The database is held in the object azimuth addition / gain adjustment unit 103 in FIG. 1, and the optimum spatial transfer function is obtained from the spatial transfer function database using the azimuth information obtained from the object azimuth / depth information extraction unit 102. Select and use.

For example, the spatial transfer functions selected as the orientation of the contour detected by the contour detection unit 110 in FIG. 2 are A _L (k, n, m) and A _R (k, n, m). Here, L and R are transmitted to the left and right ears, respectively, k is the frequency number of the discrete spectrum, the spatial transfer function database m is the horizontal arrival direction number shown in FIG. 5, and n is the vertical arrival Represents the bearing number. M and n obtained here correspond to m and n in the parallax calculation areas An and m of FIG. 3, and a spatial transfer function is calculated for each parallax calculation area and stored in the spatial transfer function database.

The following outputs can be obtained for the signals X _L (k) and X _R (k) obtained by converting the audio band signals x _L (t) and x _R (t) into the frequency domain. (When returning to the time domain, y _L (t), y _R (t))

Next, gain adjustment will be described. In general, since the gain of low-frequency sound tends to be high, attention is paid to a signal in a sound frequency band and a signal in a high-frequency sound band without considering gain adjustment related to the low-frequency sound.

  In the case of a signal in the audio frequency band, it is confirmed how much the detected outline of the person occupies the divided area in FIG. 3, and the weighting coefficient is changed according to the size of the area occupied by the outline of the person. Since it is determined that the area occupied by the outline of the person is larger in front of the screen, the larger the area occupied by the outline of the person is, the larger the sound is set. Regarding the signal in the high frequency sound band, the object detected by the high frequency component detection unit 112 and the luminance component detection unit 113 in FIG. 2 is regarded as a high frequency sound, and the weighting coefficient is changed depending on the size of the detection region. That is, the weighting coefficient is changed according to the depth information extracted by the object orientation / depth information extraction unit 102, and the gain adjustment of the signal in the voice frequency band of the person is performed.

Then, the video signal S ₁₁ is to be calculated by the object orientation-depth detecting unit 102 of FIG. 1, a delay occurs. Similarly, because it is calculated by the audio signal S ₁₂ is also the object orientation gain adjustment unit 103, a delay occurs. Therefore, the output of the object orientation-depth detecting unit 102 and the object orientation gain adjustment unit 103 is output to the three-dimensional video signal S ₂₁ to synchronize adjusted by the synchronization adjustment section 105 as a three-dimensional audio signal S _22.

  Since a 2D video signal can be generated as a 3D video signal and a 3D audio signal, it can be provided as an adapter-type device that can be directly attached to the output terminal of an existing 2D video signal device, making viewers easy In addition, it is possible to enjoy the original reality, presence, and powerful images of a three-dimensional video signal. Further, since it is possible to generate a three-dimensional audio signal from depth information obtained from one or a combination of object orientation information, contour information, motion vector, high frequency component, and luminance extracted from the contour detection and motion vector of the video signal, Since the position information is not added to the original signal, the viewer can provide the 3D audio signal even when the 3D audio signal cannot be provided. Furthermore, since it can be heard using a speaker attached to the frame of 3D video viewing glasses, a comfortable 3D audio signal can be heard even when there is a large amount of external noise, and the sound field changes depending on the location. And multiple people can enjoy the same sound field environment.

FIG. 6 is a functional block diagram showing the main configuration of the three-dimensional audio signal generation device according to Embodiment 2 of the present invention. A three-dimensional moving image signal S ₀₁ , which is a mixed signal of a video signal and an audio signal, is input to the video / audio processing unit 120, and after various processing, the three-dimensional video signal S ₂₁ and the three-dimensional audio signal S ₂₂ are converted into a three-dimensional video signal S _21. Generate and output. The video / audio processing unit 120 includes a signal separation unit 101, an object orientation / depth information extraction unit 121, an object orientation addition / gain adjustment unit 103, and a synchronization adjustment unit 105. The signal separation unit 101 decodes the moving image signal S ₀₁ and separates it into a video signal S ′ ₁₁ and an audio signal S ₁₂ . The object orientation / depth information extraction unit 121 extracts object orientation information and depth information from the 3D video signal S ′ ₁₁ separated by the signal separation unit 101. The object azimuth addition / gain adjustment unit 103 converts the object azimuth into the audio signal S ₁₂ separated by the signal separation unit 101 based on each object azimuth information and depth information extracted by the object azimuth / depth information extraction unit 121. Add information or adjust gain. The synchronization adjustment unit 105 synchronizes the 3D video signal S ′ ₁₁ and the audio signal output from the object orientation addition / gain adjustment unit 103, and outputs the 3D video signal as an output signal of the video / audio processing unit 100. S ₂₁ and a three-dimensional audio signal S ₂₂ are output.

  FIG. 7 is a functional block diagram illustrating the main configuration of the object orientation / depth information extraction unit 121 that extracts object orientation information and depth information from the 3D video signal according to the second embodiment. The object orientation / depth information extraction unit 121 includes a contour detection unit 122 and a pixel parallax detection unit 123. The contour detection unit 122 is similar to the contour detection unit 110 in FIG. 2 and detects the contour of a person from the input video signal. Next, the pixel parallax detection unit 123 uses the configuration of the three-dimensional signal to extract depth information.

  Here, a side-by-side method or a frame packing method can be cited as a configuration of a 3D video signal that is currently mainstream. Although there is a slight difference between the two systems, the decoded video signal must output left and right images sequentially. That is, since left and right images can be extracted from the previous and subsequent images, depth information can be extracted from the pixel difference.

  FIG. 8 is a diagram illustrating a method for extracting depth information. In this case, the depth information a can be calculated by the following formula.

b is the distance from the eye position of the viewer H to the output panel P, parallax c is the distance between the left eye and right eye of the viewer H, binocular parallax d is the right-eye image R that is a specific pixel, and the left-eye image corresponding thereto The interval on the panel P of L is represented. In the case of b, a distance sensor is provided on the display device side of the panel P, the distance between the 3D video viewing glasses and the panel is measured, and the distance from the 3D video viewing glasses to the eyes of the viewer H (about approx. (1.5 to 2 cm) may be added and calculated. Alternatively, the distance of the viewing position from the panel may be designated in advance and matched with that position.

  Here, when a is larger than 0 (a> 0), it is understood that the pixel is displayed on the back side of the screen of the panel P. Conversely, when a is smaller than 0 (a <0), it is displayed in front of the screen of panel P. When a is 0 (a = 0), it is displayed on the screen of the panel P.

  Therefore, if the depth information can be extracted from all of the 1920 × 1080 pixels, even if the parallax calculation area is obtained by dividing the video signal of one frame shown in FIG. 3 into m areas in the horizontal direction and n areas in the vertical component, Or even if it is a group by pixels with similar depth information, three-dimensional sound can be generated by performing azimuth information and gain adjustment for each region. At that time, the voice band signal can be assigned to the contour region of the person detected by the contour detection unit 122 of FIG. However, even in this case, erroneous orientation assignment can be reduced by determining the presence or absence of speech using speech detection based on kurtosis and assigning using the result. The audio signal assigned to the contour region of the person is assigned as a three-dimensional audio signal according to the depth information a, and a sound with a sense of depth can be provided to the viewer.

  In addition, since only the audio signal of the three-dimensional video signal can be generated as the three-dimensional audio signal, it can be provided as an adapter-type device that can be directly attached to the output terminal of an existing three-dimensional video signal device, making it easy for the viewer In addition, it is possible to enjoy the original reality, presence, and powerful images of a three-dimensional video signal.

FIG. 9 is a functional block diagram of the main configuration of the three-dimensional audio signal generation device according to the third embodiment. The moving image signal S _02, which is a mixed signal of the video signal and the audio signal, is input to the video / audio processing unit 130, and after various processing, the 3D video signal S ₂₁ and the 3D audio signal S ₂₂ are generated and output. To do. The video / audio processing unit 130 includes a signal separation unit 101, a 2D / 3D video detection unit 131, an object orientation / depth information extraction unit 102 that extracts object orientation information and depth information from a 2D video signal, Object orientation / depth information extraction unit 121 that extracts object orientation information and depth information from a 3D video signal, object orientation addition / gain adjustment unit 103, and 3D video generation that generates a 3D video signal from the 2D video signal Unit 104 and synchronization adjustment unit 105. Signal separator 101, decodes the video signal _{S 02,} separates the video signal S and the voice signal _{S 12.} The video signal S is automatically detected by the 2D / 3D video detection unit 131 as to whether it is a 2D video signal or a 3D video signal. In the case of the 2D video signal S ₁₁ , the object orientation / depth information extraction unit 102 extracts the object orientation information and depth information, and the 3D video generation unit 104 generates a 3D video signal. In the case of the three-dimensional video signal S ′ ₁₁ , the object orientation / depth information extraction unit 121 extracts object orientation information and depth information. Based on each object orientation information and depth information extracted by either the object orientation / depth information extraction unit 102 or the object orientation / depth information extraction unit 121, the object orientation addition / gain adjustment unit 103 uses the signal separation unit Add the object orientation information to the audio signal S ₁₂ separated by 101, or to adjust the gain. In the synchronization adjustment unit 105, either the video signal output from the 3D video generation unit 104 or the video signal of the 3D video signal S ′ ₁₁ and the audio output from the object orientation addition / gain adjustment unit 103. The video signal S ₂₁ and the audio signal S ₂₂ are output as output signals of the video / audio processing unit 130 in synchronization with the signals. The configuration of the object orientation / depth information extraction unit 102 in FIG. 9 is the same as that in FIG. 2, and the configuration of the object orientation / depth information extraction unit 121 is the same as that in FIG.

  The video signal S separated by the signal separation unit 101 of FIG. 9 is automatically determined by the 2D / 3D video detection unit 131 as a 2D video signal or a 3D video signal, and for each signal, This is a device that generates a three-dimensional sound, and can be used as a three-dimensional sound signal generating device that is very convenient because it can be used without any restrictions on the input signal.

  In the three-dimensional audio signal generation device according to the first embodiment of the present invention, in the object orientation / gain adjustment unit 103 in FIG. 1, a speaker is attached to glasses for viewing three-dimensional images, and an inverse filter is convoluted based on the head-related transfer function. In this example, the direction information and the depth information are added to generate the three-dimensional sound. However, an example of generating a three-dimensional audio signal in the case where a surround speaker that is currently mainstream is used will be described as a fourth embodiment.

  FIG. 10 is an explanatory diagram when a three-dimensional audio signal according to the fourth embodiment of the present invention is output to a surround speaker. As shown in FIG. 10, the surround speaker has a configuration in which a plurality of speakers emit sound from all directions with a viewer at the center. If you listen to the sound with a surround speaker while watching the 3D video signal, the power increases, but generally the audio signal for the 3D video signal is often a two-channel stereo signal. Therefore, the object orientation information and depth information obtained by the object orientation / depth information extraction unit 102 in FIG. 1 are utilized to convert the information into a multi-channel signal and generate a three-dimensional audio signal.

  First, in the same manner as in the first embodiment, it is divided into three, a low frequency band from 0 to 300 Hz, a voice band from 0.3 to 3.4 kHz, and a high frequency band above 3.4 kHz. The low-frequency sound is output from the subwoofer (SW) in FIG. 10, and the signals in other bands are output with the same gain from each speaker of the LR channel by default.

FIG. 11 is an explanatory diagram when the contour of a human face is detected by the contour detection unit (110 or 122) on the parallax calculation area of FIG. For example, as shown in FIG. 11, when there is contour detection on the left side of the center and the kurtosis value is equal to or greater than a threshold value, FL, voice band signals x _L (t), x _R (t) X _L (t) output from SL and RL is multiplied by a weighting factor k of 1.2. Conversely, FR, SR, and RR for which x _R (t) is output are multiplied by a weighting factor k of 0.8. The weighting factor k described above is an example, and can be changed according to the preference of the viewer and set in another combination.

Further, when the depth information is in the foreground, the weighting coefficient k multiplied by x _L (t) and x _R (t) output from the speaker C is set larger than 1. On the other hand, if it is the back side, the weight coefficient k is made smaller than 1. In addition, since the left and right information and the depth information can be obtained from the pixel parallax detection unit 123 even for a signal in a high frequency sound band, a three-dimensional audio signal is generated by converting the signal into a multichannel signal in conjunction with the weighting factor k. Can be output.

  Therefore, it is possible to listen to a powerful 3D audio signal with a surround speaker while watching the 3D video signal.

  As described above, the apparatus for generating a three-dimensional audio signal according to the present invention is not limited to the contents described in the first to fourth embodiments, and is used by being incorporated in a television, a recording / reproducing device, a personal computer (PC), or the like. And can be used as an external unit. The present invention can be widely applied to devices that handle video signals and audio signals, such as set-top boxes such as CATV, car navigation systems, mobile phones, electronic dictionaries, and electronic books.

  The three-dimensional audio signal generation device according to the present invention extracts information obtained from a process of generating a three-dimensional image from a two-dimensional image, or person position and depth information in the image from the three-dimensional image information, and the depth information is extracted. The present invention relates to an apparatus for generating a three-dimensional audio signal from a two-dimensional audio signal.

100, 120, 130 Video / Audio Processing Unit 101 Signal Separation Unit 102, 121 Object Direction / Depth Information Extraction Unit 103 Object Direction Addition / Gain Adjustment Unit 104 3D Video Generation Unit 105 Synchronization Adjustment Unit 110, 122 Contour Detection Unit 111 Motion Vector detection unit 112 High frequency component detection unit 113 Luminance component detection unit 123 Pixel parallax detection unit 131 2D / 3D video detection unit S ₀₀ , S ₀₁ , S ₀₂ Video signal S, S ₁₁ , S ′ ₁₁ , S ₂₁ video signal _{S 12,} S ₂₂ voice signal

Claims (14)

A signal separation unit for separating the input video signal into a video signal and an audio signal;
An object orientation / depth information extraction unit that extracts object orientation information and depth information from the video signal;
3. A three-dimensional audio signal generation apparatus comprising: an object direction addition / gain adjustment unit that adds direction information to the audio signal or adjusts gain based on the audio signal, the object direction information, and depth information.   When the input video signal is a two-dimensional video signal, the object orientation / depth information extraction unit includes at least one of a contour detection unit and a motion vector detection unit, and at least one of a high frequency component detection unit and a luminance component detection unit. The three-dimensional audio signal generation device according to claim 1, wherein the three-dimensional audio signal generation device is configured by a combination of the two.   The three-dimensional audio according to claim 1, wherein when the input video signal is a three-dimensional video signal, the object orientation / depth information extraction unit includes a contour detection unit and a pixel parallax detection unit. Signal generator.   4. The outline detection unit according to claim 2, wherein the outline detection unit detects an outline of a person based on a correlation with an outline of an existing person, and the main part of the screen is a case where the outline has an area equal to or larger than a threshold value. Dimensional audio signal generator.   5. The three-dimensional audio signal generation according to claim 4, wherein when the contour detection unit detects the contours of a plurality of persons, the depth information is set closer to the viewer as the person having a larger area of the contours. apparatus. A signal separation unit for separating the input video signal into a video signal and an audio signal;
A 2D / 3D video detector for detecting whether the video signal is a 2D video signal or a 3D video signal;
A first object orientation / depth information extraction unit for extracting object orientation information and depth information when the video signal is a two-dimensional video signal;
A second object azimuth / depth information extraction unit for extracting object azimuth information and depth information when the video signal is a three-dimensional video signal;
Object orientation addition / gain for adding orientation information or adjusting gain to the audio signal based on the audio signal and the object orientation information and depth information extracted by the first or second object orientation / depth information extraction unit. A three-dimensional audio signal generation device comprising an adjustment unit.   The first object orientation / depth information extraction unit is configured by a combination of at least one of a contour detection unit and a motion vector detection unit, and at least one of a high frequency component detection unit and a luminance component detection unit. The three-dimensional audio | voice signal production | generation apparatus of Claim 6.   The three-dimensional audio signal generation device according to claim 6, wherein the second object orientation / depth information extraction unit includes an outline detection unit and a pixel parallax detection unit.   The said outline detection part detects the outline of a person by correlation with the outline of the existing person, and the case where the outline is an area more than a threshold value plays a leading role of a screen, 3 Dimensional audio signal generator.   The three-dimensional audio signal generation according to claim 9, wherein when the contour detection unit detects the contours of a plurality of persons, depth information is set closer to the viewer as the contour area is larger. apparatus.   11. The three-dimensional audio signal generation device according to claim 1, wherein the object orientation addition / gain adjustment unit can provide a three-dimensional audio signal according to a user's viewing mode with reference to a spatial transfer function database. A three-dimensional audio signal generation device characterized in that: 12. The three-dimensional audio signal generation device according to claim 11, wherein the object orientation addition / gain adjustment unit generates a three-dimensional audio signal on the premise that the object is attached to a frame of 3D video viewing glasses and viewed with a speaker. A three-dimensional audio signal generation device characterized by the above. 11. The three-dimensional audio signal generation device according to claim 1, wherein the object orientation addition / gain adjustment unit is attached to a frame of glasses for viewing three-dimensional images and is viewed on an inner speaker or headphones. An apparatus for generating a three-dimensional audio signal, characterized by generating an audio signal. 11. The three-dimensional audio signal generation device according to claim 1, wherein the object orientation addition / gain adjustment unit generates a three-dimensional audio signal on the premise that the object is viewed with a surround speaker. Audio signal generator.