JP2012169822A - Image processing method and image processing device - Google Patents

Abstract

An image processing method and an image processing apparatus capable of providing a sense of reality even when the distance between a user and a display changes.
When a user brings his / her face close to the display device, the left-eye image and the right-eye image are displayed in a 3D image with a larger horizontal parallax, and when the user removes his / her face from the display device, the left-eye image and The 3D image is displayed by reducing the parallax in the horizontal direction of the right-eye image.
[Selection] Figure 1

Description

  The present invention relates to an image processing method and an image processing apparatus for processing a 3D image.

2. Description of the Related Art In recent years, with the improvement of image processing technology, image processing apparatuses that record and reproduce stereoscopic images (hereinafter referred to as 3D images) have been developed.
In the stereoscopic view, the user is made to recognize the stereoscopic by adding parallax between the left-eye image and the right-eye image. The parallax will be described with reference to FIGS. 4 (a) and 4 (b).

4A and 4B are explanatory diagrams of parallax.
A case where the user is viewing the objects 3R and 3L displayed on the display 1 will be described.
For images of the same objects 3R and 3L that are displayed on the display 1 and have different display angles for stereoscopic viewing, the horizontal coordinate of the object 3L in the left-eye image is X, and the horizontal coordinate of the object 3R in the right-eye image Is Xr.
For the difference ΔX = Xr−X, when the difference ΔX> 0, the user's eyes 2L and 2R appear to have the object 4a in the back (the back side of the display 1) as shown in FIG.
On the other hand, when the difference ΔX <0, the user's eyes 2L and 2R appear to have the object 4b in front (the front side of the display 1) as shown in FIG. 4B. The larger the absolute value of the difference ΔX, the closer it appears to the user or in the back, but if the absolute value of the difference ΔX is too large, stereoscopic viewing may not be possible.

  In addition, since the display is constant regardless of the position of the user, for example, the appearance that it looks large or small like an actual three-dimensional object even if the user moves his face close to or away from the display 1 Because there is no change, there is room for improvement in the sense of reality.

Patent Documents 1 and 2 can be cited as techniques related to an image processing apparatus for 3D.
The invention described in Patent Document 1 relates to a “stereoscopic TV apparatus”. Specifically, “the binocular parallax is calculated from the size data of the image display unit and the left and right images, and the maximum value or the minimum value thereof. A parallax calculation unit that calculates the viewing distance of the observer, a resolution determination unit that detects the synchronization frequency of the input image signal and determines the type of the input image signal, and the size of the display screen The parallax for calculating the binocular parallax size of the displayed image using the output of the parallax calculation unit, the viewing distance of the observer, and the output of the resolution discrimination unit, and entering the binocular fusion range of the observer A stereoscopic image that has a proper parallax determination unit that calculates a change amount and a parallax control unit that translates the left and right images horizontally according to the proper parallax determination unit, and is displayed even when the synchronization frequency of the input video signal changes Left so that is within the binocular fusion range of the observer It controls the image "is intended.
That is, the invention described in Patent Document 1 adjusts the amount of parallax so that stereoscopic viewing can be performed from the display size.

The invention described in Patent Document 2 relates to a “display device”. Specifically, “a video signal including information on a parallax amount between a left-eye video and a right-eye video is converted into a video and the video is converted into a video. An image display unit to be displayed on the screen, a projection distance detection unit to detect a projection distance that is a distance between the image display unit and the screen, and the projection distance detected by the projection distance detection unit to the screen. A display size calculation unit that calculates a display size of the video to be displayed; a parallax amount detection unit that detects information on the parallax amount from the video signal; and a display size and the parallax amount of the video displayed on the screen And a parallax amount adjustment unit that adjusts the parallax amount using the information on the parallax amount detected by the detection unit.
That is, the invention described in Patent Document 2 detects the size of the projection screen, generates a left-eye image and a right-eye image, and eliminates a parallax shift.

Japanese Patent No. 2848291 JP 2010-98479 A

  However, the inventions described in Patent Documents 1 and 2 are intended to be adjusted so as to be possible when stereoscopic viewing is not possible as it is, but the relationship between the distance from the user's display is not considered. (If the user moves his face closer or away in front of the display, there will be no change in appearance such that it looks larger or smaller like an actual three-dimensional object.) There is no effect of putting out.

  Therefore, an object of the present invention is to provide an image processing method and an image processing apparatus that can provide a sense of reality even when the distance between the user and the display changes.

  In order to solve the above-mentioned problem, according to the first aspect of the present invention, when the user brings his / her face close to the display device, the parallax in the horizontal direction of the left-eye image and the right-eye image is increased to display a 3D image. When the face is removed from the display device, a 3D image is displayed by reducing a horizontal parallax between the left-eye image and the right-eye image.

  According to a second aspect of the present invention, in the first aspect of the present invention, the parallax is changed before and after the user's face approaches or separates from the display device.

  According to a third aspect of the present invention, an image analysis unit that detects a position difference in which the same object in the left-eye image and the right-eye image is shown in each of the left and right images, and the amount of parallax of the image can be changed from the position difference. A stereoscopic range detection unit that detects a range, a user position detection unit that detects a distance between the user and the display device, and a range in which the user position and the parallax amount can be changed. The apparatus includes a parallax adjustment unit that determines a parallax amount, and a display device that displays a 3D image based on the determined parallax amount.

  According to a fourth aspect of the present invention, in the third aspect of the invention, the parallax adjustment unit increases the parallax in the horizontal direction of the left-eye image and the right-eye image when the user brings a face close to the display device. The parallax in the horizontal direction of the image for the left eye and the image for the right eye is reduced when the user removes his / her face from the display device.

  ADVANTAGE OF THE INVENTION According to this invention, provision of the image processing method and image processing apparatus which can obtain a realistic feeling even if the distance between a user and a display changes is realizable.

It is an example of the flowchart which shows one Embodiment of the image processing method which concerns on this invention. 1 is an example of a block diagram of an image forming apparatus according to the present invention. It is explanatory drawing of a block matching method. (A), (b) is explanatory drawing of a parallax.

<Configuration>
FIG. 2 is an example of a block diagram of the image forming apparatus according to the present invention.
The image forming apparatus illustrated in FIG. 2 includes a storage unit 10, an image analysis unit 11, a stereoscopic range detection unit 12, a display device (hereinafter referred to as a display) 13, a user position detection unit 14, and a parallax adjustment unit 15. And an image display unit 16. In the specification of the present application, only the image processing is mentioned, and the sound processing function is omitted.

The storage unit 10 is a circuit that temporarily stores an input image input from the outside. The storage unit 10 is a still image / moving image taken by a 3D-compatible digital camera or a 3D-compatible digital video camera, a DVD, a terrestrial digital television broadcast, a satellite. It has a function of storing 3D image data by digital television broadcasting, Internet broadcasting, or the like.
The storage unit 10 is realized by, for example, an HDD (Hard Disk Drive). The image processing unit 17 including the image analysis unit 11, the stereoscopic viewing range detection unit 12, the parallax adjustment unit 15, and the image display unit 16 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random). (Access Memory).
The display device 13 has a function of displaying a 3D image, and is realized by, for example, a liquid crystal panel. The size of the display device 13 is not particularly limited, and the display device 13 can be applied to a 50-inch or larger display from a portable terminal.

  The image analysis unit 11 reads the input image for the left eye and the input image for the right eye from the storage unit 10, and the position difference (parallax) of the same object shown in each of the input image for the left eye and the input image for the right eye It has a function to detect. Examples of this detection method include a block matching method as disclosed in Japanese Patent No. 2848291. Alternatively, a method of detecting a face area of the same person from each image by using a face recognition technique and obtaining a position difference from the coordinates of the area may be considered.

Here, the block matching method will be described with reference to FIG.
FIG. 3 is an explanatory diagram of the block matching method.
In FIG. 3, the input image IL for the left eye is described in the upper part, and the input image IR for the right eye is described in the lower part.
The left-eye input image IL and the right-eye input image IR include the same persons PL and PR having different display angles for stereoscopic viewing.
Here, in the input image IL for the left eye, a block BL1 in the image (here, the position of the face of the person PL) and the blocks BR1, BR2, BR3,... Of the same size in the input image IR for the right eye. (BRi) is compared. In BL1 and BRi, by calculating the sum of the absolute values of the difference in luminance between the corresponding dots, the similarity between BL1 and BRi as an image can be calculated. Then, the block BRj having the smallest sum among BR1, BR2, BR3,... Is determined as a block corresponding to BL1, and the deviation of the position in the X direction between the blocks BRj and BL1 becomes the parallax ΔX. .
The parallax ΔX is obtained for all or a part of the blocks BL1, BL2, BL3,... In the image, and the smallest ΔX is obtained as ΔXmin and the largest ΔX is obtained as ΔXmax.

The stereoscopic range detection unit 12 has a function of detecting a range in which the parallax amount of the image can be changed from the position difference. Regarding the parallax that can be stereoscopically viewed, for example, since the distance between both eyes of a human is about 6 to 7 cm, the adjustable range is set to −6 cm to +6 cm in actual size on the display.
This width may be configured so that the setting can be changed by an operation unit (not shown) of the image processing apparatus. The stereoscopic range detection unit 12 has a function of calculating how much the image can be shifted in the X direction (right direction or left direction which is the −X direction) so as to be within this range.
When the number of dots per 1 cm on the display is d [dot / cm] and the amount of shifting the object (in this case, a person) in the left and right input images is α [dot], Equations (1) and (2) The value of α can be set within a range that satisfies the above.
−6d ≦ ΔXmin + α ≦ + 6d (1)
−6d ≦ ΔXmax + α ≦ + 6d (2)
From Equations (1) and (2), Equation (3) is obtained.
−6d−ΔXmin ≦ α ≦ + 6d−ΔXmax (3)

The user position detection unit 14 has a function of detecting the position of the user (distance from the display).
As a method for detecting the user's position, for example, two cameras are installed on the outer frame of the display and the user is photographed. From the distance between the two cameras and the direction in which the user's face is reflected in each camera, triangulation is performed. The distance to the user's face (eye position) is measured in a manner, or when the user wears stereoscopic glasses, the distance is measured with an infrared LED (Light Emitted Diode) or the like. A well-known method is mentioned.

  The parallax adjustment unit 15 determines the parallax amount to be set for the current user position from the range in which the user position and the parallax amount can be changed. The value of α is minimum (−6−ΔXmin) when the user is closest (for example, within 10 cm) within the range satisfying the mathematical formula (3), and α is when the user is farthest (for example, 5 m or more). Is set to a maximum (6-ΔXmax). It is preferable to set the value of α so that the value of α increases monotonously (or preferably increases by a linear function) according to the distance between the approaching and separating distances.

  The image display unit 16 has a function of displaying an image with a parallax amount determined by the parallax adjustment unit 15 based on a program stored in a ROM (not shown). As this method, the input image for the left eye and the input image for the right eye are shifted by the value of α (in addition to shifting one of them, each image may be shifted in the opposite direction by α / 2). Conceivable.

<Operation>
Next, the operation of the image processing apparatus will be described with reference to FIG.
FIG. 1 is an example of a flowchart showing an embodiment of an image processing method according to the present invention.
The subject of operation is a CPU (not shown).
When the operation starts, the image analysis unit 11 reads the input image for the left eye and the input image for the right eye, and examines the parallax of the image to be displayed as described in the paragraphs “0019” to “0020” (step S1). ).
Next, the stereoscopic range detection unit 12 detects a range in which the image can be stereoscopically viewed as described in the paragraph 0021 (step S2).
Next, the user position detection unit 14 detects the position of the user as described in the paragraph 0022 (step S3).
Next, the parallax adjustment unit 15 calculates the parallax amount from the position of the user and the stereoscopically viewable range as described in the paragraph 0023 (step S4).
Next, the image display unit 16 displays an image with the calculated amount of parallax as described in paragraph 0024 above (step S5).
Next, the user position detection unit 14 detects the position of the user again (step S6).
It is determined whether or not the position of the user detected again is changed (step S7). When it is determined that there is a change (step S7 / Yes), the process returns to step S4, and when it is determined that there is no change (step S7). / No), the process proceeds to step S8.
When it is determined in step S8 whether or not the display of the image has ended, and when it is determined that the display of the image has not ended (step S8 / No), the process returns to step S6 and the display of the image is determined to have ended (Step S8 / No), it ends.

  Note that stereoscopic viewing is possible by changing the parallax even before and after the user's face approaches and separates from the display.

<Effect>
As described above, according to the present embodiment, when the user brings his / her face close to the display, the disparity between the left and right images is increased and displayed, thereby increasing the amount of parallax and displaying the image closer to the front. . When the user moves his / her face away from the display, the amount of parallax is reduced by displaying the right and left images with a small positional shift so that the image is displayed deeper. However, these adjustments are performed within a range where the image can be stereoscopically viewed.

  To do this, it detects the position difference between the same objects in the left and right images, calculates the range of stereoscopic viewing, calculates the distance from the user's display, and within the range of stereoscopic viewing from the user's display. Adjust the parallax amount according to the distance. The amount of parallax is changed when the distance from the user's display changes. Stereoscopic viewing is possible so that the parallax amount is the largest when the user is closest to the display (for example, set to 10 cm) and the parallax amount is the smallest when the user is the farthest from the display (for example, set to 5 m). The parallax amount is adjusted in accordance with the position of the user within a wide range. Thereby, even if the distance between the user and the display changes, a sense of reality can be obtained.

<Program>
The image processing apparatus according to the present invention described above is realized by a program that causes a computer to execute processing. Examples of the computer include general-purpose computers such as personal computers and workstations, but the present invention is not limited to this. Therefore, as an example, a case where the present invention is realized by a program will be described below.

For example, in the computer of the image processing apparatus,
A procedure in which the image analysis unit detects a position difference in which the same object in the image for the left eye and the image for the right eye is shown in each of the left and right images;
A procedure in which the stereoscopic range detection unit detects a range in which the parallax amount of the image can be changed from the position difference;
A procedure in which the user position detection unit detects a distance between the user and the display device;
A procedure in which a parallax adjustment unit determines a parallax amount with respect to a current user position from a range in which the user position and the parallax amount can be changed;
A procedure in which the image display unit displays a 3D image based on the determined amount of parallax;
A program that executes
Is mentioned.

  Thus, the apparatus according to the present invention can be realized anywhere as long as there is a computer environment capable of executing the program.

<Storage medium>
Such a program may be stored in a computer-readable storage medium.
Here, examples of the storage medium include a computer-readable storage medium such as a CD-ROM (Compact Disk-ROM), a flexible disk (FD), and a CD-R (CD-Recordable), a flash memory, a RAM, and a ROM. And semiconductor memories such as FeRAM (ferroelectric memory) and HDDs.

  The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.

DESCRIPTION OF SYMBOLS 10 Memory | storage part 11 Image analysis part 12 Stereoscopic range detection part 13 Display apparatus (display)
14 User position detection unit 15 Parallax adjustment unit 16 Image display unit 17 Image processing unit

Claims (4)

  When the user brings the face close to the display device, the parallax in the horizontal direction of the left-eye image and the right-eye image is increased to display a 3D image, and when the user moves the face away from the display device, the left-eye image and the left-eye image An image processing method comprising: displaying a 3D image while reducing a horizontal parallax of an image for the right eye.   The image processing method according to claim 1, wherein the parallax is changed even before and after the user's face approaches and separates from the display device. An image analysis unit for detecting a position difference in which the same object in the image for the left eye and the image for the right eye is shown in each of the left and right images;
A stereoscopic range detection unit for detecting a range in which the parallax amount of the image can be changed from the position difference;
A user position detecting unit for detecting a distance between the user and the display device;
A parallax adjustment unit that determines a parallax amount with respect to a current user position from the user position and a range in which the parallax amount can be changed;
A display device that displays a 3D image based on the determined amount of parallax;
An image processing apparatus comprising: The parallax adjustment unit increases the horizontal parallax of the left-eye image and the right-eye image when the user brings the face close to the display device, and the left-eye image and the right-eye image when the user moves the face away from the display device. The image processing apparatus according to claim 3, wherein the parallax in the horizontal direction of the right-eye image is reduced.

Images