CN107770453A - Brightness Compensation Method and System for Perspective Device - Google Patents

Abstract

The invention discloses a brightness compensation method of a perspective device, which comprises the steps of firstly providing a brightness model, wherein the total response of the brightness model is equal to the sum of the response of device light rays from the perspective device and the response of scene light rays from a scene. The calibration phase is performed in the transform domain, which is only related to the characteristics of the projector and the image capture device of the see-through apparatus. And executing a compensation stage, namely obtaining the response of the original image in the darkroom, and determining the response of the compensated image according to the response of the original image and the response of the scene light. Generating a compensated image according to the response of the compensated image.

Description

Brightness Compensation Method and System for Perspective Device

technical field

The present invention relates to photometric compensation, in particular to a method and system for photometric compensation of a see-through device.

Background technique

See-through smart glasses (smart glasses), as a tool for augmented reality, allow users to receive additional information about the surrounding world through images projected by embedded projectors. Users can see the projected image and the real world scene. Because the augmented visual information is digitally manipulated, it can generate interesting and interactive user experiences.

The tiny projectors in most smart glasses are far less powerful than traditional projectors. When the projected image mixes with the scene and illuminates the user's retina, if the irradiance of the projector is the same as or smaller than that of the scene light, it is easy to cause photometric distortion. This brightness distortion is a major determinant of the quality of smart glasses.

Although the brightness distortion is caused by the scene light, to eliminate the brightness distortion, we must consider the characteristics of the scene light, the reflectance of the projector and the smart glasses, which can be achieved by using a camera and a set of calibration patterns. to achieve. A projector projects an amplified or calibrated image to the user's eyes, while a camera captures an image of the scene.

Whenever the scene of the field of view of the smart glasses changes or the user moves, brightness calibration must be performed again, thus interrupting the user interaction. Another determinant of the quality of smart glasses is efficiency. Projecting and processing the calibration pattern takes time, usually from several seconds to tens of seconds, so it is not acceptable for real time applications. Therefore, it is urgent to propose a novel mechanism to overcome the lack of traditional methods.

Contents of the invention

In view of the above, one purpose of the embodiments of the present invention is to provide a brightness compensation method and system for a see-through device. In one embodiment, an algorithm for brightness compensation based on distorted images is proposed, which requires only one brightness calibration. Each subsequent compensation operation is performed based on the distorted image captured at each time point without recalibration, thereby achieving real-time brightness compensation.

The purpose of the present invention is achieved by adopting the following technical solutions.

The invention discloses a brightness compensation method of a see-through device, which comprises: providing a brightness model whose total response is equal to the sum of the response of the device light from the see-through device and the response of the scene light from the scene; performing a calibration phase in the transformation domain, It is only related to the characteristics of the projector and the image capture device of the see-through device; the compensation stage is performed, the response of the original image is obtained in the darkroom, and the response of the compensated image is determined according to the response of the original image and the response of the scene light; and A compensation image is generated according to the response of the compensation image.

The object of the present invention can also be realized by adopting the following technical measures.

The brightness compensation method of the above-mentioned see-through device, wherein the light response of the device is equal to the product of the channel mismatch of the projector and the image capture device and the gamma function of the projector.

In the brightness compensation method of the above-mentioned see-through device, the calibration stage is performed in a dark room to block scene light so as to uniquely obtain the light response of the device.

The brightness compensation method of the above-mentioned see-through device, wherein the light response of the device is equal to the product of the decoupling transformation and the scale gamma function.

In the brightness compensation method of the see-through device, the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light.

The object of the present invention can also be realized by adopting the following technical solutions.

The invention discloses a brightness compensation system of a see-through device, which comprises: a calibration device, which performs a calibration phase in the transformation domain, which is only related to the characteristics of the projector and the image capture device of the see-through device, and provides a brightness model whose total response It is equal to the sum of the response of the device light from the perspective device and the response of the scene light from the scene; and the compensation device performs the compensation stage to obtain the response of the original image in the darkroom, and then according to the response of the original image and the response of the scene light to determine the response of the compensated image; wherein the compensated image is generated according to the response of the compensated image.

The object of the present invention can also be realized by adopting the following technical measures.

The brightness compensation system of the see-through device above, wherein the light response of the device is equal to the product of the channel mismatch of the projector and the image capture device and the gamma function of the projector.

In the above brightness compensation system for a see-through device, the calibration stage is performed in a dark room to block scene light so as to uniquely obtain the light response of the device.

The brightness compensation system of the above-mentioned see-through device, wherein the light response of the device is equal to the product of the decoupling transformation and the scale gamma function.

In the brightness compensation system of the above-mentioned see-through device, the calibration stage only needs to be executed once, no matter the image or the scene projected to the see-through device changes dynamically.

In the brightness compensation system of the see-through device, the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light.

The brightness compensation system of the above-mentioned see-through device, wherein the compensation device includes: a brightness generation unit, which generates the response of the original image; a scene generation unit, after the calibration stage, generates the response of the scene light; a compensation determination unit, according to the original image and the response of the scene light to determine the response of the compensation image; and the compensation image generating unit generates the compensation image according to the response of the compensation image.

The brightness compensation system of the above-mentioned see-through device is characterized in that the see-through device includes smart glasses.

The object of the present invention can also be realized by adopting the following technical solutions.

The invention discloses a see-through device, which comprises: at least one lens; a projector, which projects an image onto the lens; an image capture device, which captures light from the lens, and captures scene light from a scene; a calibration device, which is executed in a transformation domain a calibration phase, which relates only to the characteristics of the projector and the image capture device, and provides a luminance model whose total response is equal to the sum of the response of the device light and the response of the scene light; and compensation means, which perform the compensation phase, in The darkroom obtains the response of the original image, and then determines the response of the compensation image according to the response of the original image and the response of the scene light; wherein the compensation image is generated according to the response of the compensation image.

The object of the present invention can also be realized by adopting the following technical measures.

The see-through device above, wherein the light response of the device is equal to the product of the channel mismatch between the projector and the image capture device and the gamma function of the projector.

In the see-through device described above, the calibration stage is performed in a darkroom to block scene light so as to uniquely obtain the light response of the device.

The see-through device above, wherein the light response of the device is equal to the product of the decoupling transformation and the scale gamma function.

In the above perspective device, the calibration stage only needs to be executed once, no matter the image or the scene changes dynamically.

The perspective device above, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light.

The above-mentioned see-through device, wherein the compensating device includes: a luminance generation unit, which generates the response of the original image; a scene generation unit, after the calibration stage, generates the response of the scene light; a compensation determination unit, according to the response of the original image and the The response of scene light is used to determine the response of the compensation image; and the compensation image generating unit is used to generate the compensation image according to the response of the compensation image.

By virtue of the above technical solutions, the present invention has at least the following advantages and beneficial effects:

According to an embodiment of the invention, a luminance model is provided whose total response is equal to the sum of the response to device light from the see-through device and the response to scene light from the scene. The calibration phase is performed in the transformation domain, which is only related to the characteristics of the projector and the image capture device of the see-through device. In the compensation stage, the response of the original image is obtained in the darkroom, and then the response of the compensated image is determined according to the response of the original image and the response of the scene light. The compensation image is generated according to the response of the compensation image.

Description of drawings

FIG. 1 shows a system block diagram of a brightness compensation system of a see-through device according to an embodiment of the present invention.

FIG. 2 shows a flowchart of a brightness compensation method of a see-through device according to an embodiment of the present invention.

FIG. 3 shows a schematic diagram of a device for implementing the brightness compensation system ( FIG. 1 ) and the brightness compensation method ( FIG. 2 ) of the present embodiment.

4A and 4B respectively illustrate the spectral sensitivity of the projector and the spectral sensitivity of the camera.

[Description of main component symbols]

100: brightness compensation system

11: Projector

12: Image capture device/camera

13: Calibration device

131: Channel decoupling unit

14: Compensation device

141: Luminance generating unit

142: Scene generation unit

143: Compensation decision unit

144: compensation image generation unit

200: brightness compensation method

21: Projecting Image

22: Capture device light and scene light

23: Using Decoupled Transforms for Calibration

24: Determine the response C(I _O ) of the original image in the darkroom

25: Determine the response C(S) of the scene light

26: Determine the response C(I _C ) of the compensated image

27: Generate Compensation Image I _C

300: device

31: Smart glasses

32: Prism

33: Installation light

34:Scene light

Detailed ways

FIG. 1 shows a system block diagram of a brightness compensation system 100 for a see-through device according to an embodiment of the present invention, and FIG. 2 shows a flowchart of a brightness compensation method 200 for a see-through device according to an embodiment of the present invention. The blocks in FIG. 1 and the steps in FIG. 2 can be implemented by hardware, software or a combination thereof, and can be executed by a processor (such as a digital image processor). The see-through device may be a wearable see-through device, such as smart glasses, but is not limited thereto.

FIG. 3 shows a schematic diagram of an apparatus 300 for executing the brightness compensation system 100 ( FIG. 1 ) and the brightness compensation method 200 ( FIG. 2 ) of the present embodiment. The device 300 includes a projector 11 (such as a micro-projector), and projects an image to the smart glasses 31 through a prism 32 (step 21 ). The image capture device 12 (such as a camera) captures the device light 33 from the smart glasses 31 . Camera 12 also captures scene rays 34 from the scene (step 22). The purpose of this embodiment is to cancel the influence of the scene light 34, so that the color of the projected image can be preserved.

In this embodiment, a luminance model is provided first. Traditional brightness models assume that scene light is constant or negligible relative to device light. However, the luminance model of this embodiment considers both device light and scene light. The brightness model of this embodiment can be expressed as the following vector form:

T(I,S)=C(I)+C(S)=MG(I)+C(S) (1)

Where T(I,S) is the total camera response (response), C(I) is the camera response of the device light, C(S) is the camera response of the scene light, and M is the channel mismatch between the projector 11 and the camera 12 (mismatch), G(·) is the gamma function of the projector 11 .

4A and 4B respectively illustrate the spectral sensitivity of the projector 11 and the spectral sensitivity of the camera 12 , showing the channel mismatch between the projector 11 and the camera 12 .

In step 23, a calibration phase is performed in a darkroom using the calibration device 13 to block the scene light and thus directly obtain the camera response of the device light. Thus, formula (1) becomes:

T(I,S)=C(I)=MG(I) (2)

In general, since M and G(·) are coupled unknowns, it is difficult to solve them directly. According to one of the features of this embodiment, the calibration phase is performed in the transformed domain using the channel decoupling unit 131, so equation (2) can be expressed as follows:

in For a decoupled transformation, it only depends on the characteristics of the projector 11 and the camera 12, and V(·) is a scaled gamma function.

Since the solution problem of M and G(·) is transformed into and V(·), so no matter how the scene or image changes dynamically, it only needs to be calculated once. To speed up the calibration procedure, a look up table can be used to construct V(·).

In detail, the decoupled camera response for each channel X can be written as:

where X ∈ {R, G, B}, V _X ( ) is defined as the scale gamma function.

In this way, solve And V(·) is equivalent to solving M and G(·). The details of the solution can be referred to Grossberg (MDGrossberg) et al. in the Institute of Electrical and Electronics Engineers - Computer Vision and Pattern Recognition (Proc.IEEE Computer Vision and Pattern Recognition (CVPR)) 2004, Volume 1, pages 452–459 Published "Making One Object LookLike Another: Controlling Appearance Using a Projector-Camera System."

Next, a brightness compensation stage is performed using the compensation means 14 . The total camera response of the raw image is expressed as:

T(I _O ,S)=C(I _O )+C(S) (5)

The total camera response of the compensated image is expressed as:

T(I _C ,S)＝C(I _C )+C(S) (6)

During brightness compensation, the total camera response T(I _C ,S) of the compensated image must be equal to the camera response C(I _O ) of the original image in the darkroom, that is:

T(I _C ,S)=C(I _C )+C(S)=C(I _O ) (7)

In order to get C(I _C ), we must first get C(I _O ) and C(S). In step 24, the luminance generation unit 141 is used to obtain C(I _O ):

On the other hand, in step 25, C(S) can be obtained by using the scene generating unit 142 according to formula (5), because T(I _O , S) and C(I _O ) are known. Therefore, in step 26, the camera response C(I _C ) of the compensated image can be determined by using the compensation determination unit 143 and according to C(I _O ) and C(S).

when get Afterwards, in step 27, use the compensation image generation unit 144 to obtain I _C :

in

According to the above, this embodiment proposes a method for compensating the brightness distortion of see-through smart glasses. Since the brightness compensation process only uses distorted images, the method of this embodiment does not require recalibration and thus does not interrupt user interaction. In this way, the method of this embodiment can achieve the real-time performance of the augmented reality application of the smart glasses. The method of this embodiment is applicable to the situation where the intensity of the scene light is the same as that of the device light. When the scene light is smaller than the device light, the luminance distortion can be ignored. On the other hand, when the scene light is greater than the device light, it is difficult to restore the image through brightness compensation. In this case, the user can use a principle similar to sunglasses to reduce the scene light, or increase the power of the projector of the smart glasses.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention; all other equivalent changes or modifications that do not deviate from the spirit disclosed by the invention should be included in the claims within the patent scope of the book.

Claims (20)

1. A brightness compensation method of a see-through device, characterized in that it comprises: provide a luminance model whose total response is equal to the sum of the response to the device light from the see-through device and the response to the scene light from the scene; Perform a calibration phase in the transformation domain, which is only related to the characteristics of the projector and the image capture device of the see-through device; Execute the compensation stage, obtain the response of the original image in the dark room, and then determine the response of the compensated image according to the response of the original image and the response of the scene light; and A compensation image is generated according to the response of the compensation image. 2. The brightness compensation method of a see-through device according to claim 1, wherein the light response of the device is equal to the product of the channel mismatch between the projector and the image capture device and the gamma function of the projector. 3 . The brightness compensation method of a see-through device according to claim 2 , wherein the calibration stage is performed in a dark room to block scene light so as to uniquely obtain the light response of the device. 4 . 4. The brightness compensation method of a see-through device according to claim 3, wherein the light response of the device is equal to the product of the decoupling transformation and the scale gamma function. 5. The brightness compensation method of a see-through device according to claim 1, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light. 6. A brightness compensation system of a see-through device, characterized in that it comprises: Calibration device, performing a calibration phase in the transformation domain, which is only related to the characteristics of the projector and the image capture device of the see-through device, and provides a luminance model whose total response is equal to the response of the device light from the see-through device and the scene from the scene the sum of the light responses; and The compensation device performs the compensation stage, obtains the response of the original image in the darkroom, and then determines the response of the compensated image according to the response of the original image and the response of the scene light; Wherein the compensation image is generated according to the response of the compensation image. 7. The brightness compensation system of a see-through device according to claim 6, wherein the light response of the device is equal to the product of the channel mismatch of the projector and the image capture device and the gamma function of the projector. 8 . The brightness compensation system of a see-through device according to claim 7 , wherein the calibration stage is performed in a dark room to block scene light so as to uniquely obtain the light response of the device. 9. The brightness compensation system of a see-through device according to claim 8, wherein the light response of the device is equal to the product of the decoupling transformation and the scale gamma function. 10 . The brightness compensation system of a see-through device according to claim 9 , wherein the calibration stage only needs to be executed once, no matter the image or scene projected to the see-through device changes dynamically. 11 . 11. The brightness compensation system of a see-through device according to claim 6, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light. 12. The brightness compensation system of the see-through device according to claim 6, wherein the compensation device comprises: a luminance generation unit that generates a response to the original image; A scene generation unit, after the calibration phase, generates the response of the scene light; a compensation determining unit, which determines the response of the compensated image according to the response of the original image and the response of the scene light; and The compensation image generating unit generates the compensation image according to the response of the compensation image. 13. The brightness compensation system of a see-through device according to claim 6, wherein the see-through device comprises smart glasses. 14. A see-through device, characterized in that it comprises: at least one lens; a projector for projecting an image onto the lens; an image capture device that captures device light from the lens and captures scene light from the scene; a calibration device, which performs a calibration phase in the transformation domain, which is only related to the characteristics of the projector and the image capture device, and provides a luminance model whose total response is equal to the sum of the response of the device light and the response of the scene light; and The compensation device performs the compensation stage, obtains the response of the original image in the darkroom, and then determines the response of the compensated image according to the response of the original image and the response of the scene light; Wherein the compensation image is generated according to the response of the compensation image. 15. The see-through device according to claim 14, wherein the light response of the device is equal to the product of the channel mismatch of the projector and the image capture device and the gamma function of the projector. 16. The see-through device according to claim 15, wherein the calibration stage is performed in a dark room to block scene light so as to uniquely obtain the light response of the device. 17. The see-through device of claim 16, wherein the light response of the device is equal to the product of the decoupling transformation and the scale-gamma function. 18. The fluoroscopy device according to claim 17, wherein the calibration stage only needs to be performed once, no matter the image or the scene is dynamically changed. 19. The see-through device according to claim 14, wherein the response of the compensated image is equal to the response of the original image of the darkroom minus the response of the scene light. 20. The fluoroscopy device according to claim 14, wherein the compensating device comprises: a luminance generation unit that generates a response to the original image; A scene generation unit, after the calibration phase, generates the response of the scene light; a compensation determining unit, which determines the response of the compensated image according to the response of the original image and the response of the scene light; and The compensation image generating unit generates the compensation image according to the response of the compensation image.