JP6727917B2 - Projection device, electronic device, and image processing method - Google Patents

Description

The present invention relates to a projection device, an electronic device, and an image processing method for displaying an image.

2. Description of the Related Art Conventionally, in a projector that projects an image, a technique of performing correction for removing the influence of ambient light is known. Patent Document 1 discloses a technique for reducing the influence of ambient light by correcting the pixel value of the input image based on the result of measuring the intensity of ambient light applied to the screen.

JP, 2007-281893, A

If ambient light is blocked by obstacles such as eaves, pillars, and window frames, and only part of the image display surface is exposed to ambient light, the ambient light and the ambient light will be exposed. A boundary line due to the abrupt change in the intensity of the ambient light is visually recognized between the area and the non-open area. When such a boundary line is visually recognized, there arises a problem that it is difficult for the user to see the displayed image. When only the pixel value of the area affected by the ambient light is corrected using the conventional technology, the chromaticity changes between the area exposed to the ambient light and the area not exposed to the ambient light. There was a problem that the borderline was conspicuous.

Therefore, the present invention has been made in view of the above-mentioned points, and an object of the present invention is to provide a projection device, an electronic device, and an image processing method capable of making a boundary line generated on a display surface under the influence of ambient light less noticeable. And

In order to solve the above problems, a projection device of the present invention is based on an image acquisition unit that acquires a captured image of a projection surface onto which a projection image generated based on an input image is projected, and the captured image based on the captured image. Identifying a step region that is a region in which the rate of change in brightness due to ambient light on the projection surface is larger than a predetermined threshold value, and correcting the input image so that the rate of change in brightness in the specified step region is small. And a projection unit configured to project the projection image generated by the correction image generation unit onto the projection surface.

The electronic device of the present invention includes an image acquisition unit that acquires a captured image of a projection surface onto which a projection image generated based on an input image is projected, and brightness of ambient light on the projection surface based on the captured image. A correction for generating the projection image by specifying a step area that is an area whose change rate is larger than a predetermined threshold value and correcting the input image so that the change rate of the luminance in the specified step area is small. It is characterized by comprising image generation means and output means for outputting the projection image generated by the corrected image generation means.

The image processing method of the present invention includes a step of acquiring a captured image of a projection surface onto which a projection image generated based on an input image is executed, which is executed by a computer, and based on the captured image, on the projection surface. The step of identifying a step area where the rate of change in brightness due to ambient light is larger than a predetermined threshold value, and the projection image by correcting the input image so that the rate of change in brightness in the step area becomes small. And a step of outputting the generated projection image.

According to the present invention, there is an effect that the boundary line generated on the display surface due to the influence of ambient light can be made inconspicuous.

It is a figure for explaining the outline of this embodiment. FIG. 6 is a diagram for explaining the outline of the operation of the projector 100. FIG. 3 is a diagram showing a configuration of a projector 100. FIG. 6 is a diagram showing a configuration of an image processing unit 440. 7 is an operation flowchart of an ambient light level difference correction process in the projector 100. It is a figure for demonstrating the calculation method of the ambient light intensity distribution after correction|amendment. It is a figure for demonstrating the method of correct|amending the ambient light level|step difference in a curved shape. It is a figure for explaining a correction value calculation method when a correction value used on the bright part gradation side is attenuated. It is a figure for demonstrating the detection method of edge continuity.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to the following embodiments. Note that the following embodiments do not limit the invention according to the claims, and all combinations of the features described in the embodiments are not necessarily essential to the solution means of the invention.

Further, each functional block described in the present embodiment does not necessarily have to be individual hardware. That is, for example, the functions of some functional blocks may be executed by one piece of hardware. Further, the function of one functional block or the function of a plurality of functional blocks may be executed by the cooperative operation of some hardware. Further, the function of each functional block may be executed by a computer program loaded on the memory by the CPU.

[Outline of this embodiment]
FIG. 1 is a diagram for explaining the outline of the present embodiment. FIG. 1 shows a state in which an image is projected from a projector 100, which is a projection device, onto a projection surface 101 (for example, a screen). Sunlight is incident as ambient light from the window 103, and the area on the right side of the projection surface 101 is exposed to sunlight, so that the image on the right side of the projection area 102 is displayed brighter than the image on the left side area. Has been done. As a result, in the projection area 102, a boundary line between the bright area and the dark area is visible. In such a state, the projector 100 specifies the position of the boundary line based on the captured image obtained by shooting the projection surface 101, and corrects the image around the specified boundary line to thereby detect the boundary line. You can make it less noticeable.

FIG. 2 is a diagram for explaining the outline of the operation of the projector 100 according to the present embodiment. FIG. 2A is a diagram showing a state in which the projector 100 is displaying an image without correcting the image. In FIG. 2A, the environment light is shown as illuminating the right area, and in the projection area 102, the image in the left area is visually recognized as dark, the image in the right area is visually recognized as bright, and the left area is visually recognized. The boundary line is clearly visible between the area and the area on the right.

FIG. 2B is a diagram showing a state where the projector 100 is displaying an image with the image corrected. The projector 100 generates a corrected image in which the pixel value is corrected so that the brightness of the pixel in the region near the boundary in the left region is high, and displays the corrected image. As a result, as compared with FIG. 2A, the boundary line between the left side region and the right side region is not clearly visible, and the image quality of the visually recognized image is improved.
The configuration and operation of the projector 100 will be described in detail below.

[Configuration of Projector 100]
FIG. 3 is a diagram showing the configuration of the projector 100.
The control unit 410 is, for example, a CPU, and controls each unit of the projector 100 by executing a program stored in the ROM 412.

The RAM 411 is a volatile memory that temporarily stores a control program and data as a work memory. The ROM 412 is a non-volatile memory that stores data such as a program used by the control unit 410, a look-up table used by a gamma correction unit (not shown), setting parameters of each operation block, and factory adjustment values.

The operation unit 413 includes operation buttons and generates an operation signal indicating the operation content of the projector 100 by the user. The operation unit 413 notifies the control unit 410 of the generated operation signal.

The image input unit 420 acquires an image input to the projector 100. The image input unit 420 reads image data stored in the recording medium 421, for example. The image input unit 420 also acquires image information such as resolution and frame rate together with the image data.

The recording medium 421 is a medium capable of recording image data such as still image data and moving image data, and control data necessary for the projector 100 to project the image data. The recording medium 421 is, for example, a magnetic disk, an optical disk, or a semiconductor memory, and may be any type of recording medium capable of recording data. The recording medium 421 may be a recording medium detachable from the projector 100 or may be a recording medium built in the projector 100.

The recording/reproducing unit 422 reproduces the still image data and the moving image data acquired by the image input unit 420. The recording/reproducing unit 422 also records the still image data or the moving image data input from the control unit 410 on the recording medium 421.

The light source control unit 430 controls the amount of light output from the light source 460 based on the control of the control unit 410.
The image processing unit 440 generates a projection image by performing various kinds of image processing on the image data acquired by the image input unit 420. Although details will be described later, for example, the image processing unit 440 corrects the pixel values of the pixels near the boundary line so that the boundary line caused by the influence of the ambient light is less visible.

The light modulation element control unit 450 controls the intensity of light output from the light modulation element 470 by applying a voltage to each of the red (R), green (G), and blue (B) light modulation elements 470. Specifically, the light modulation element control unit 450 controls the intensity of light output from the light modulation element 470 by changing the modulation intensity of each light modulation element 470.

The light source 460 supplies light to the light modulation element 470.
The light modulation element 470 can change the intensity of the output light by modulating the light emitted from the light source 460. The red (R) light modulator 470R modulates red (R) light, the green (G) light modulator 470G modulates green (G) light, and the blue (B) light modulator. 470B modulates blue (B) light.

The color combining unit 480 combines the red (R), green (G), and blue (B) lights output from the light modulation elements 470R, 470G, and 470B. The color composition unit 480 has, for example, a dichroic mirror or a prism.

The display optical system 481 displays an optical image obtained by the light supplied from the light source 460 being modulated by the light modulator 470 corresponding to red (R), green (G), and blue (B) on the projection surface. To do. The display optical system 481 has an actuator that changes the position of the lens, and lens shift and zoom can be performed by controlling the actuator.

The image capturing unit 491 is provided at a position and an orientation capable of capturing the same direction as the projection direction of the display optical system 481, captures a region including the projection region of the display optical system 481, and captures an image of the captured region. To generate. The image capturing unit 491 acquires a captured image of the projection surface onto which the projection image generated based on the input image is projected. The imaging unit 491 can change optical parameters such as a shutter speed and an aperture based on an instruction from the control unit 410.

The captured image processing unit 492 generates projection region image data by cutting out an image of the projection region included in the captured image generated by the imaging unit 491. The captured image processing unit 492 outputs the generated projection area image data to the image processing unit 440.
The details of the configuration and operation of the image processing unit 440 will be described below.

[Configuration of Image Processing Unit 440]
FIG. 4 is a diagram showing the configuration of the image processing unit 440.
The intensity distribution identifying unit 441 identifies the ambient light intensity distribution indicating the distribution of brightness due to ambient light in the projection region, based on the projection region image data generated by the captured image processing unit 492. The intensity distribution specifying unit 441, for example, subtracts the value corresponding to the brightness value of each pixel of the projection image projected by the display optical system 481 from the brightness value of each pixel of the projection region image data, thereby changing the ambient light. The intensity of ambient light is calculated by calculating the magnitude of the luminance. The intensity distribution identifying unit 441 may identify the brightness distribution of the captured image generated by the image capturing unit 491 in a state where the projection image is not projected, as the ambient light intensity distribution.

The level difference processing unit 442 has a level difference because the rate of change in luminance due to ambient light on the projection surface is larger than a predetermined threshold value based on the ambient light intensity distribution identified by the intensity distribution identifying unit 441 based on the captured image. A step region that is a region is specified. The rate of change in brightness is the amount of change in brightness per unit width on the projection surface.

The level difference processing unit 442 is an intensity distribution when the area around the level difference is corrected so that the level difference of the brightness due to the ambient light in the ambient light intensity distribution acquired by the intensity distribution specifying unit 441 is corrected, Identify the intensity distribution. Specifically, the step processing unit 442 sets the first length at the position of the step where the ambient light intensity in the ambient light intensity distribution changes by a predetermined magnitude or more within a predetermined first length range. The corrected ambient light intensity distribution in the case where the ambient light intensity is corrected to change by the same amount within the range of the second length larger than the range is specified. In other words, the level difference processing unit 442 identifies the corrected ambient light intensity distribution in which the slope of the change in the ambient light intensity with respect to the change in the position of the projection region becomes gentle.

The correction value intensity distribution calculation unit 443 detects the amount of light that needs to be added in order to correct the pre-correction ambient light intensity distribution acquired by the intensity distribution identification unit 441 to the post-correction ambient light intensity distribution identified by the step processing unit 442. A correction value intensity distribution, which is an intensity distribution, is calculated. Specifically, the correction value intensity distribution calculation unit 443 calculates the distribution in the projection region of the difference between the pre-correction ambient light intensity distribution and the post-correction ambient light intensity distribution as the correction value intensity distribution.

The correction value generation unit 444 generates a correction value to be added to the input image in order to correct the level difference of the ambient light. The correction value is a value including at least one of a gain value by which the pixel value of a pixel included in the input image is multiplied and an offset value by which the pixel value is added.

The correction unit 445 includes a multiplier that multiplies the input image by the calculated correction value and an adder that adds the correction value. The correction unit 445 adds the correction value corresponding to the pixel included in the step area to the pixel value of the corresponding pixel in the input image to reduce the change rate of the brightness of the input image in the step area due to the ambient light. The image is corrected and a projection image to be projected on the projection surface from the display optical system 481 is generated.

The correction unit 445 generates a projection image by correcting the input image in a predetermined range from the step area, for example. The predetermined range from the step region is a range larger than the width of the step region and smaller than the width of the projection region, for example, a range equal to the width of the step region. Further, when the predetermined range is larger than the width of the step region, the correction unit 445 may generate the projection image by correcting the input image such that the correction intensity becomes weaker as the distance from the step region increases. .. By doing so, the correction unit 445 can make the brightness level difference inconspicuous and prevent the image quality of the area other than the level difference area from changing as much as possible from the image quality before correction.

The test pattern adding unit 446 superimposes an OSD (On Screen Display) image such as a message presented to the user and a correction test pattern other than the pattern for measuring the color of the projection surface on the projection image.

[Operation Flowchart of Projector 100]
FIG. 5 is an operation flowchart of the ambient light level difference correction processing in the projector 100.
The ambient light level difference correction processing is executed based on an instruction from the control unit 410 when the user instructs the operation unit 413 to correct the level difference due to ambient light via a user interface such as a remote controller.

(Compensation method for ambient light level difference)
First, the control unit 410 causes the image processing unit 440 to generate a test pattern image for detecting a projection region, and instructs the imaging unit 491 to capture an image of the region on which the test pattern is projected ( S101). The image processing unit 440 uses, for example, a white solid image and a black solid image as the test pattern image for detecting the projection area. The imaging unit 491 generates a white solid projection captured image while projecting a white solid image, and generates a black solid projection captured image while projecting a black solid image.

Then, the image processing unit 440 generates a difference image between the white solid projection imaged image and the black solid projection imaged image. In this difference image, the ambient light component is removed. The image processing unit 440 binarizes the difference image to generate a mask image used for cutting out the projection region from the captured image. The image processing unit 440 cuts out a projection area from the captured image by detecting edges in the mask image (S102).

Specifically, the image processing unit 440 detects the four corner coordinates of the projection area in the captured image by detecting the edge in the mask image. The image processing unit 440 can detect the projection area from the captured image by calculating the projective transformation matrix from the coordinates of the four corners. The image processing unit 440 uses the projective transformation matrix to generate an image in which the projection area is cut out from each of the white solid projection imaged image and the black solid projection imaged image.

Subsequently, the image processing unit 440 calculates the intensity distribution of ambient light based on the cut-out image of the projection area in the intensity distribution specifying unit 441 (S103). The intensity distribution specifying unit 441 calculates the ratio of the ambient light intensity to the output brightness of the projector 100 for each area for each of the white solid projection-time projected area cutout image and the black solid projection-time projected area cutout image. Get the light intensity distribution.

Next, the step processor 442 calculates the step width in the ambient light intensity distribution (S104). Specifically, the step processing unit 442 analyzes the amount of change in pixel value in a predetermined number of pixels in the horizontal and vertical directions in the ambient light distribution, and detects a region where the amount of change is equal to or more than a certain amount as an ambient light step. .. The step processor 442 calculates the horizontal and vertical lengths of the area where the ambient light step is detected. Subsequently, the step processing unit 442 calculates the corrected ambient light intensity distribution obtained by the blurring process for making the ambient light step inconspicuous, based on the length of the region where the ambient light step is detected ( S105).

FIG. 6 is a diagram for explaining a method of calculating the corrected ambient light intensity distribution. The horizontal axis of FIG. 6 represents the horizontal pixel position of the light modulation element 470, and the vertical axis represents the ambient light intensity. Although FIG. 6 shows an example of correcting the ambient light intensity distribution in the horizontal direction, the step processing unit 442 similarly calculates the corrected ambient light intensity distribution in the vertical direction as well.

FIG. 6A shows the ambient light intensity distribution before the correction of the ambient light step. FIG. 6B is an example of the corrected ambient light intensity distribution that is the result of performing the blurring process on the step area corresponding to the step width of the ambient light. The step processor 442 linearly extends the step region in the direction of the side not exposed to the ambient light, thereby relaxing the inclination of the ambient light intensity and making the boundary less visible. The step processor 442 extends the step region by an amount equal to or greater than a predetermined threshold. The step processing unit 442 may determine the amount by which the step is extended by multiplying the width of the ambient light step before correction by a predetermined safety factor.

Next, the correction value intensity distribution calculation unit 443 calculates a correction value intensity distribution to be added to the input image from the ambient light intensity distribution and the corrected ambient light intensity distribution (S106). Specifically, the correction value intensity distribution calculation unit 443 calculates the correction value intensity distribution by subtracting the uncorrected ambient light intensity shown in FIG. 6A from the corrected ambient light intensity.

FIG. 6C shows the correction value intensity distribution. As shown in FIG. 6C, the intensity of the correction value is large around the rising position of the step in the uncorrected ambient light intensity distribution shown in FIG. 6A.

Next, the correction value generation unit 444 calculates a correction value for correcting the input image based on the correction value intensity distribution and the input image (S107). Since the correction value intensity shown in FIG. 6C is the ratio of the correction value intensity to the brightness of the projector 100, the correction value generation unit 444 applies the correction value intensity to the input image in order to correct the input image. Convert to a value. Specifically, the correction value generation unit 444 adds the brightness value obtained by linearly converting the brightness of the input image and the brightness value calculated from the correction value intensity distribution in order to calculate the brightness value given to the input image. To do. Hereinafter, ITU-R BT. The HDTV coefficient of H.709 will be described as an example.

First, the conversion formula from the (RGB) input value to the (YUV) input value is shown in Expression (1).
Yinput = 0.299Rinput + 0.587Ginput + 0.114Binput
Uinput = -0.169Rinput-0.331Ginput + 0.500Binput
Vinput = 0.500Rinput-0.419Ginput-0.081Binput Formula (1)
Here, R, G, and B are gradation values of each pixel of the image.

Further, the formula for calculating the brightness on the corrected projection plane is shown in formula (2). The brightness on the corrected projection surface can be calculated as the brightness Ygoal which is the sum of the input image brightness value Yinput and the brightness value calculated from the correction value intensity distribution in the brightness linear space.
Ygoal = Yinput ^γ + (Yoffset / 100 * 255) / (1-Yoffset / 100) Formula (2)
Here, γ represents the γ value of the display device, and is converted to the luminance linear space by multiplying the luminance value of the input image by the γ power. Yoffset is the value of the correction value intensity distribution.

The correction value generation unit 444 degamma (1/gamma) the calculated luminance Ygoal on the projection plane after correction, and subtracts the luminance Yinput of the input image from the degamma-corrected luminance to obtain an input in the ambient light level difference correction. A correction value Ycor to be added to the image is calculated.

Subsequently, the correction unit 445 calculates the corrected brightness Youtput by adding the correction value Ycor calculated by the correction value generation unit 444 to the brightness Yinput of the input image (S108).
An equation for converting (YUV) output into (RGB) output is shown by the following equation (3).
Uoutput = Uinput
Voutput = Vinput
Routput = 1.000Youtput + 0.114Voutput
Goutput = 1.000Youtput-0.344Uoutput-0.714Voutput
Boutput = 1.000Youtput + 1.772U Expression (3)
The correction unit 445 uses the ITU-R BT. The same calculation can be performed on the signals of colorimetry other than 709 by changing the coefficient.

(Variation of correction method for ambient light level difference)
Through the above procedure, the projector 100 can correct the input image so that the step of the ambient light is not noticeable. Hereinafter, a process for further improving the correction effect will be described.

(1) Utilization of Difference in Intensity of Ambient Light In step S105 of FIG. 5, the step processing unit 442 corrects the step width of the ambient light to be equal to or larger than a predetermined threshold. However, in reality, not only the step width of the ambient light but also the intensity difference is an important factor, and when the intensity difference of the ambient light at the step is large and the step width is narrow and steep, the ambient light step is easily visible to the user. turn into. Therefore, the step processing unit 442 may determine the step width after correction based on the difference between the maximum value and the minimum value of the ambient light intensity. Then, the correction unit 445 generates the projection image by correcting the input image using the correction value based on the step width determined based on the difference between the maximum value and the minimum value of the intensity of the ambient light step. ..

For example, the step processing unit 442 determines the step width after correction so that the step width is determined based on the difference in ambient light intensity. Specifically, the step processing unit 442 calculates a coefficient n (n is a positive real number) that is determined based on the difference in ambient light intensity, and the corrected step width after correction is multiplied by the coefficient n. Determine the step width. Further, the step processing unit 442 may correct the inclination of the ambient light step difference to be equal to or less than a predetermined threshold value.

By doing so, the correction unit 445 corrects the input image such that the larger the difference between the maximum value and the minimum value of the ambient light intensity is, the smaller the change amount of the luminance per unit width of the step region is. By doing so, it is possible to generate a projection image in which the steps are less noticeable. The correction unit 445 can also generate the projection image by correcting the input image such that the ratio of the amount of change in luminance to the width of the step region is equal to or less than the threshold value. The level difference processing unit 442 sets the inclination of the level difference in brightness to a degree that is not visible to the user, so that the projector 100 can project an image in which the level difference is inconspicuous.

(2) Correcting the step into a curved shape Further, the step processing unit 442 makes the step inconspicuous by linearly extending the change in the ambient light step in the horizontal direction in step S105 of FIG. The processing unit 442 may correct the ambient light level difference so as to change in a curved shape.
FIG. 7 is a diagram for explaining a method of correcting the ambient light step difference in a curved shape. In FIG. 7, as in FIG. 6, the horizontal axis indicates the horizontal pixel position of the light modulation element 470, and the vertical axis indicates the ambient light intensity.

FIG. 7A is an example of the ambient light intensity distribution before correction and is similar to FIG. 6A.
FIG. 7B shows a result of calculating the expanded light intensity by performing the expansion process on the ambient light intensity shown in FIG. 7A by the step processing unit 442. The expansion process is a process of changing the center of gravity and the width of the brightness step as necessary, assuming the corrected ambient light intensity distribution.

FIG. 7C shows the result of the step processing unit 442 calculating the S-shaped corrected ambient light intensity distribution by multiplying the spatial average of the expanded light intensity shown in FIG. 7B. In this way, when the step processing unit 442 calculates the corrected ambient light intensity distribution so that the step region in the corrected environmental light intensity distribution on the projection surface becomes S-shaped, the slopes of the start point and the end point of the step region are tilted. Becomes loose. In the linear correction method shown in FIG. 6B, since the amount of change between the start point and the end point of the step region in the corrected ambient light intensity distribution is large, there is a possibility that a brightness step due to a visual characteristic called the Mach effect may be seen. However, if the stepped region in the corrected ambient light intensity distribution is S-shaped as shown in FIG. 7C, it is possible to prevent image quality interference due to the Mach effect. FIG. 7D is a correction value intensity distribution corresponding to FIG. 6C, and is obtained by subtracting the uncorrected ambient light intensity from the corrected ambient light intensity.

(3) Utilization of Statistics In the above description, the correction value generation unit 444 calculated the correction value for correcting the input image based on the correction value intensity distribution and the input image. The correction value generation unit 444 may calculate the correction value based on the statistical amount of the pixel value of the input image such as the in-plane brightness average value instead of using the luminance Yinput for each pixel of the input image. Specifically, the correction value generation unit 444 calculates the in-screen luminance average value of the input image and uses it instead of Yinput in Expression (2). The correction value generation unit 444 calculates the correction value using the statistic in this way, for example, calculates the luminance average value of the entire moving image that comes in advance, and generates the correction value based on this. Thus, it is possible to reduce the process of calculating the luminance offset value to be added as a correction value to the input image for each frame.

(4) Prevention of Gradation Degradation In the above description, the correction unit 445 adds the correction value to the input image as it is. However, if the correction unit 445 adds the correction value to the input image as it is, the brightness on the bright part gradation side may stick to the upper limit, and the gradation may deteriorate. Therefore, the correction unit 445 may decrease the correction value as the brightness value of the pixel of the input image increases. The correction unit 445 applies the correction value only to the dark side, for example, and corrects the correction data to 0 on the bright side of the input image.

FIG. 8 is a diagram for explaining a correction value calculation method when the correction value used on the bright part gradation side is attenuated. The horizontal axis of the graph in FIG. 8 is the luminance before correction of the input image, and the vertical axis is the luminance after correction. As shown in FIG. 8, the correction unit 445 does not apply the correction value in order to maintain the gradation in the area of the threshold brightness A to 255 of the brightness before correction. Further, the correction unit 445 corrects the input image by the correction value Ycor at 0 in the area of uncorrected brightness 0 to A, decreases the correction amount as the brightness increases, and the correction value at the threshold A becomes 0. Correct so that By doing so, it is possible to prevent the gradation property of the bright portion from being impaired due to the luminance step correction.

(5) Correction Based on Change in Chromaticity of Ambient Light In the above description, the image processing unit 440 performs correction so that the brightness change of the ambient light intensity on the projection surface becomes gentle. However, some ambient light has a low color temperature like a light bulb. Therefore, the image processing unit 440 may correct not only the change in the luminance but also the change in the chromaticity of the ambient light to be gentle. In this case, the image processing unit 440 uses, for example, the respective intensities of the RGB values acquired by the imaging unit 491 instead of the ambient light intensities based on the luminance shown in FIG.

For example, the level difference processing unit 442 specifies the level difference region based on the amount of change in chromaticity in the captured image, and the correction unit 445 corrects the input image so that the amount of change in chromaticity in the level difference region becomes small. Then, a projection image is generated. Specifically, the correction unit 445 converts the RGB values of the projector 100 into YUV values, and instead of converting the RGB values of the projector 100 into RGB camera values of the imaging unit 491, the RGB values acquired by the imaging unit 491. Can be corrected using the respective intensities of. As the conversion method, a color matrix as shown in the above equations (1) and (3) may be used, or a 1D-LUT (one-dimensional lookup table) may be used. However, it is necessary to perform calibration so that the image capturing unit 491 and the RGB values of the projector 100 can be associated with each other.
By the above method, not only the change in brightness due to ambient light but also the change in color can be corrected.

(6) Timing of Step Correction In the above description, the timing at which the correction value intensity distribution calculation unit 443 calculates the correction value intensity distribution was not taken into consideration. However, the correction unit 445 uses the step processing unit 442 on the projection surface. The input image may be corrected in accordance with the fact that the step area is present in the distribution of the ambient light. In this case, the correction value intensity distribution calculation unit 443 calculates the correction value intensity distribution based on the captured image on condition that the step processing unit 442 has changed the distribution of the ambient light on the projection surface.

In addition, the control unit 410 uses a timer to cause the test pattern adding unit 446 to display the ambient light measurement test pattern each time a predetermined time has elapsed since the imaging unit 491 captured the projection area last time, and the correction unit 445 may change the correction value of the input image every predetermined time. Further, the control unit 410 detects that the projector 100 has moved using an acceleration sensor as a motion detection unit that detects the motion of the projector 100, and when detecting that the projector 100 has moved, the test pattern addition unit 446 is performed again. The ambient light measurement test pattern may be displayed on the screen. Then, the correction unit 445 corrects the input image using the correction value updated according to the movement of the projector 100.

(7) Method of suppressing erroneous correction In the above description, it has not been considered that random noise is superimposed on the captured image. However, when random noise is superimposed on the captured image, the projector 100 causes the noise to be changed to the environment. It is assumed that it will be detected as an edge of light and erroneously corrected. Therefore, the step processing unit 442 may perform a process of distinguishing random noise from the ambient light step difference in the captured image in order to suppress erroneous correction in the ambient light step correction. The step processing unit 442 can use the continuity of the edge of the step region (the portion where the change in the ambient light intensity is large) in order to distinguish the random noise from the ambient light step.

FIG. 9 is a diagram for explaining a method of detecting edge continuity.
The step processing unit 442 uses the amount of change in the ambient light intensity when detecting the ambient light step in step S104 of FIG. The step processor 442 detects the amount of change in the ambient light intensity in the horizontal direction and the vertical direction of the ambient light intensity distribution, and determines that the location where the amount of change exceeds the threshold is an edge. At this time, the step processing unit 442 distinguishes the random noise from the ambient light step by determining that a continuous pixel has a change amount exceeding the threshold as an edge caused by the ambient light step.

Lattice points in FIG. 9 represent pixels of the light modulation element 470, and gray pixels represent locations where the step processing unit 442 determines that they are edges. The step processing unit 442 specifies the step area on the condition that a predetermined number or more of pixels in which the amount of change in the intensity of ambient light with respect to an adjacent pixel is larger than a predetermined threshold are continuous. The step processor 442 uses the search region to detect the continuity of the edges. The search area is an area range for checking whether or not there is another edge pixel around the edge pixel of interest. FIG. 9 shows an example in which the search region is one pixel around the pixel of interest. In the search area 901, the pixel of interest in the center is an edge, and the other surrounding pixels include the edge. Therefore, the step processing unit 442 considers that the edge included in the search area 901 is the ambient light step. ..

On the other hand, in the case of the search region 902, the center pixel of interest is an edge, but other peripheral pixels are not edges, and the center pixel of interest exists as an edge by itself. Therefore, the level difference processing unit 442 removes the edge included in the search region 902 from noise as the target for performing the ambient light level difference correction. By doing so, the step processing unit 442 can prevent erroneous detection of the presence of a step due to the influence of noise.

Note that the step processing unit 442 may specify the step region based on the intensity distribution after removing the high frequency component in the intensity distribution of the ambient light specified based on the captured image. For example, the step processing unit 442 may detect the ambient light level difference after passing the ambient light intensity distribution through a noise removal filter such as a low-pass filter.

Further, the step processing unit 442 identifies a dark gradation area in which the intensity of the ambient light in the captured image is equal to or less than a predetermined threshold value, and does not erroneously perform the ambient light step correction on the area partially darkened by noise. You may do it. For example, the step processing unit 442 calculates the area of the dark gradation region based on the ambient light intensity distribution binarized by the threshold brightness, and the condition that the area of the dark gradation region is equal to or larger than a predetermined threshold value is set. It may be determined to correct the ambient light level difference. The correction unit 445 generates the projection image by correcting the input image on the condition that the area of the dark gradation region is equal to or larger than a predetermined value.

(8) Acquisition Timing of Captured Image In the above description, an example in which the imaging unit 491 images the projection surface in the state where the projector 100 is projecting the projection image has been described. , But not limited to this. The imaging unit 491 images the projection surface in a state where the projector 100 does not project the projection image, and the intensity distribution specifying unit 441 converts the projection image in a state where the projector 100 does not project the projection image into a captured image. The ambient light intensity distribution may be specified based on this.

(9) Application to electronic device other than projector 100 In the above description, the process of correcting a step in the projector 100 has been described, but the present invention can be applied to other electronic devices.
For example, even in a direct-viewing type display device, it is possible to perform the luminance level difference correction by the same method. In this case, the image capturing unit 491 is installed so as to capture an image of the display area of the direct-view display device.

Further, the computer may execute the program stored in the storage medium to generate an image in which the difference in luminance is corrected. Specifically, a computer that inputs a projection image to the projector 100 may acquire a captured image of the projection surface and correct the projection image based on the acquired captured image so that the step becomes less noticeable. When the computer inputs the corrected projection image to the projector 100, the projector 100 can project an image in which a step due to ambient light is inconspicuous.

[Effects of this embodiment]
As described above, according to the present embodiment, it is possible to make the step inconspicuous by blurring the step in luminance or color caused by the influence of ambient light. As a result, as shown in FIG. 2B, it is possible to display an image so that the steps are not noticeable.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist thereof. is there.
For example, in the above description, the example in which the image is corrected based on the captured image generated by the image capturing unit 491 incorporated in the projector 100 has been described, but the captured image is generated by another imaging device. Good.

100 Projector 442 Step processing unit 445 Correction unit 481 Display optical system 491 Imaging unit

Claims (19)

Image acquisition means for acquiring a captured image of a projection surface onto which the projection image generated based on the input image is projected;
Based on the captured image, a step area, which is an area in which the rate of change in brightness due to ambient light on the projection surface is larger than a predetermined threshold value, is specified, and the change rate in brightness in the specified step area is reduced. A correction image generation unit that generates the projection image by correcting the input image;
A projection unit configured to project the projection image generated by the correction image generation unit onto the projection surface. The correction image generation means corrects the input image based on a magnitude of a difference between a maximum value and a minimum value of the intensity of the ambient light.
The projection device according to claim 1. The correction image generation unit corrects the input image such that the larger the difference is, the smaller the variation amount of the luminance in the step region is.
The projection device according to claim 2. The correction image generation means corrects the input image so that a ratio of a luminance change amount to a width of the step region is equal to or less than a threshold value.
The projection device according to any one of claims 1 to 3. The correction image generation means corrects the input image by adding a correction value corresponding to a pixel included in the step area to a pixel value of a corresponding pixel in the input image.
The projection device according to claim 1. The correction image generation means determines the correction value based on a statistical amount of pixel values of the input image.
The projection device according to claim 5. The correction image generation unit decreases the correction value as the luminance value of a pixel of the input image increases.
The projection device according to claim 5. The corrected image generating means, and determines the correction value according to the specific means has identified the stepped region,
The projection device according to any one of claims 5 to 7. The correction image generation means determines the correction value based on the captured image generated every predetermined time,
The projection device according to claim 5. Further comprising motion detection means for detecting the motion of the projection device,
The corrected image generation means determines the correction value in response to the movement of the projection device being detected by the movement detection means.
The projection device according to claim 5. The specifying unit specifies the step region based on the amount of change in chromaticity in the captured image,
The corrected image generating means corrects the input image so that a change amount of the chromaticity in the step area becomes small.
The projection device according to claim 1. The specifying unit specifies the step region on the condition that a predetermined number or more of pixels in which the amount of change in the intensity of ambient light with respect to an adjacent pixel is larger than a predetermined threshold are continuous.
The projection device according to any one of claims 1 to 11. The identifying unit identifies the step region based on an intensity distribution after removing a high frequency component in the intensity distribution of the ambient light identified based on the captured image.
The projection device according to any one of claims 1 to 11. The specifying unit specifies a dark gradation region in which the intensity of the ambient light in the captured image is equal to or less than a predetermined threshold,
The correction image generation means corrects the input image on condition that the area of the dark gradation region is equal to or larger than a predetermined value.
The projection device according to any one of claims 1 to 13. The correction image generating means may generate the projection image by correcting the input image in a predetermined range from the step area.
The projection device according to any one of claims 1 to 14. The correction image generation means may generate the projection image by correcting the input image such that the correction intensity becomes weaker as the distance from the step region increases.
The projection device according to any one of claims 1 to 15. Image acquisition means for acquiring a captured image of a projection surface onto which the projection image generated based on the input image is projected;
Based on the captured image, a step area, which is an area in which the rate of change in brightness due to ambient light on the projection surface is larger than a predetermined threshold value, is specified, and the change rate in brightness in the specified step area is reduced. A correction image generation unit that generates the projection image by correcting the input image;
Output means for outputting the projection image generated by the corrected image generation means,
An electronic device comprising: Computer running,
Acquiring a captured image of a projection surface onto which a projection image generated based on the input image is projected;
Specifying a step area that is an area in which the rate of change in brightness due to ambient light on the projection surface is larger than a predetermined threshold value based on the captured image;
Generating the projection image by correcting the input image so that the rate of change in luminance in the step region becomes small,
Outputting the generated projection image,
An image processing method comprising: On the computer,
Acquiring a captured image of a projection surface onto which a projection image generated based on the input image is projected;
Specifying a step area that is an area in which the rate of change in brightness due to ambient light on the projection surface is larger than a predetermined threshold value based on the captured image;
Generating the projection image by correcting the input image so that the rate of change in luminance in the step region becomes small,
Outputting the generated projection image,
A computer-readable recording medium in which a program for executing is recorded.