WO2016129404A1

WO2016129404A1 - Image processing device, image processing method, and program

Info

Publication number: WO2016129404A1
Application number: PCT/JP2016/052580
Authority: WO
Inventors: 源吾森年; 星野　和弘
Original assignee: ソニー株式会社
Priority date: 2015-02-09
Filing date: 2016-01-29
Publication date: 2016-08-18

Abstract

This technology relates to an image processing device, an image processing method, and a program which make it possible to increase the flexibility of placement of an AR marker without decreasing the detection accuracy of the AR marker. An image processing unit is provided with: a separation unit for separating an invisible light component from an input image including a visible light component and the invisible light component; a noise elimination unit for eliminating noise by decreasing the pixel value of a first image based on the invisible light component of the input image; and a marker detection unit for detecting an augmented reality (AR) marker indicating a predetermined pattern by invisible light in the first image from which the noise is eliminated. This technology is applicable, for example, to a camera that uses AR technology.

Description

Image processing apparatus, image processing method, and program

The present technology relates to an image processing device, an image processing method, and a program, and more particularly, to an image processing device, an image processing method, and a program suitable for use in detecting an AR (Augmented Reality) marker.

As one of AR (Augmented Reality) technologies, there is a marker type AR that superimposes and displays additional information at a position where an AR marker of a predetermined pattern is detected (see, for example, Patent Document 1).

JP 2009-266096 A

By the way, when the AR marker is arranged in a real space, the position of the AR marker may be limited in order to damage the scenery and the design.

The present technology has been made in view of such a situation, and is intended to increase the degree of freedom of arrangement of the AR marker without degrading the detection accuracy of the AR marker.

An image processing apparatus according to an aspect of the present technology includes a separation unit that separates an invisible light component from an input image including a visible light component and an invisible light component, and a pixel value of a first image based on the invisible light component of the input image. A noise removal unit that removes noise by reducing, and a marker detection unit that detects an AR (Augmented Reality) marker indicating a predetermined pattern by invisible light in the first image from which noise has been removed.

The separation unit further separates a visible light component from the input image, and the noise removal unit includes a luminance around the AR marker in the second image based on the visible light component of the input image, or the Based on at least one of the ambient temperatures of the image processing apparatus, it is possible to set a subtraction amount that is an amount by which the pixel value of the first image is reduced.

The AR marker reflects the invisible light to show the pattern, and at least one of the brightness around the AR marker, the subtraction amount, and the detection result of the AR marker in the second image. Based on the above, it is possible to further provide an irradiation control unit for controlling the intensity of the invisible light irradiated to the AR marker.

An imaging device that includes a first pixel that detects a visible light component and an invisible light component, and a second pixel that detects only an invisible light component, and captures the input image can be further provided.

An imaging element that includes a plurality of types of pixels that detect invisible light components and visible light components having different wavelength bands and that captures the input image can be further provided.

The separation unit further includes an additional information superimposing unit that further separates a visible light component from the input image and superimposes additional information corresponding to the AR marker on a second image based on the visible light component of the input image. Can be provided.

An image processing method according to an aspect of the present technology includes a separation step of separating an invisible light component from an input image including a visible light component and an invisible light component, and a pixel value of a first image based on the invisible light component of the input image. A noise removal step of removing noise by reducing, and a marker detection step of detecting an AR (Augmented Reality) marker indicating a predetermined pattern by invisible light in the first image from which the noise has been removed.

A program according to an aspect of the present technology includes a separation step of separating an invisible light component from an input image including a visible light component and an invisible light component, and reducing a pixel value of the first image based on the invisible light component of the input image. The computer executes a process including a noise removing step for removing noise and a marker detecting step for detecting an AR (Augmented Reality) marker indicating a predetermined pattern with invisible light in the first image from which the noise has been removed. Let

In one aspect of the present technology, the invisible light component is separated from the input image including the visible light component and the invisible light component, and noise is removed by reducing the pixel value of the first image based on the invisible light component of the input image. In the first image from which noise is removed, an AR (Augmented Reality) marker indicating a predetermined pattern is detected by invisible light.

According to one aspect of the present technology, it is possible to increase the degree of freedom of arrangement of the AR marker without reducing the detection accuracy of the AR marker.

It should be noted that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a block diagram showing a 1st embodiment of a camera system to which this art is applied. It is a figure which shows the example of arrangement | positioning of the pixel of an image sensor. It is a flowchart for demonstrating AR process. It is a figure which shows the example of AR marker. It is a figure which shows the example of the area | region which detects a brightness | luminance. It is an example of the graph for calculating the subtraction amount of the pixel value with respect to surrounding luminance. It is an example of the graph for calculating the subtraction amount of the pixel value with respect to ambient temperature. It is a block diagram which shows 2nd Embodiment of the camera system to which this technique is applied. It is an example of the graph for calculating the subtraction amount of the pixel value with respect to ambient luminance or ambient temperature. It is a block diagram which shows the structural example of a computer.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. 1st Embodiment (when irradiating IR light to AR marker)
2. Second embodiment (when the AR marker emits IR light itself)
3. Modified example

<1. First Embodiment>
First, a first embodiment of the present technology will be described with reference to FIGS. In the first embodiment, a marker-type AR using a colorless and transparent AR marker that reflects IR light (infrared light) is realized.

{Configuration example of camera system 1}
FIG. 1 shows a configuration example of a camera system 1 that is the first embodiment of the present technology.

The camera system 1 is a system that can superimpose additional information on a captured image using AR technology. That is, the camera system 1 detects the AR marker by irradiating the colorless and transparent AR marker that reflects IR light with the IR light, and displays the additional information superimposed on the captured image based on the detected AR marker. .

Note that the content of the additional information is not particularly limited. For example, the additional information includes information related to the photographed subject and location.

The camera system 1 is configured to include a bandpass filter 11, an image sensor 12, a temperature sensor 13, an image processing unit 14, and an IR light irradiation device 15.

The band-pass filter 11 transmits light in a predetermined wavelength band including a visible light region and an IR light region, and blocks light in other wavelength bands among the light from the subject.

The image sensor 12 receives light from the subject that has passed through the band-pass filter 11 and generates an image (hereinafter referred to as an input image) composed of pixel signals indicating the intensity of incident light on each pixel. The image sensor 12 supplies the generated input image to the separation unit 21 of the image processing unit 14.

FIG. 2 shows an arrangement example of the pixels of the image sensor 12. In the image sensor 12, each pixel is arranged in a grid pattern with a pattern of 2 vertical pixels × 2 horizontal pixels as one unit. In this example, the pixels are arranged in a ratio of G pixel: R pixel: B pixel: IR pixel = 1: 1: 1: 1. Compared to a conventional Bayer array image sensor, IR pixels are arranged in place of G pixels.

The G pixel mainly receives a component in the G (green) wavelength band (hereinafter referred to as G component) and a component in the IR light wavelength band (hereinafter referred to as IR light component), and performs photoelectric conversion. Is a possible pixel. That is, the G pixel is a pixel that can detect the G component and the IR light component. The R pixel is a pixel capable of receiving and photoelectrically converting a component in the R (red) wavelength band (hereinafter referred to as an R component) and an IR light component. That is, the R pixel is a pixel that can detect the R component and the IR light component. The B pixel is a pixel capable of receiving and photoelectrically converting a component in the B (blue) wavelength band (hereinafter referred to as a B component) and an IR light component. That is, the B pixel is a pixel that can detect the B component and the IR light component. The IR pixel is a pixel that can receive an IR light component and perform photoelectric conversion. That is, the IR pixel is a pixel that can detect only the IR light component.

The temperature sensor 13 measures the ambient temperature of the camera system 1 and supplies the measurement result to the coring unit 24 of the image processing unit 14. For example, the temperature sensor 13 is disposed in the vicinity of the image sensor 12.

The image processing unit 14 performs image processing such as AR processing on the input image supplied from the image sensor 12. The image processing unit 14 includes a separation unit 21, a visible light image processing unit 22, a luminance calculation unit 23, a coring unit (noise removal unit) 24, a threshold determination unit 25, a marker detection unit 26, an additional information superimposition unit 27, IR light. The control unit 28 and the buffer 29 are included.

The separation unit 21 separates the visible light component and the IR light component of each pixel of the input image. Then, the separation unit 21 supplies a mosaic image based on the visible light component to the visible light image processing unit 22 and causes the buffer 29 to store an image based on the IR light component (hereinafter referred to as an IR light image).

The visible light image processing unit 22 performs predetermined image processing such as demosaic processing on the mosaic image, thereby generating an RGB image in which each pixel has pixel values of R component, G component, and B component. The visible light image processing unit 22 supplies the generated RGB image to the luminance calculation unit 23 and the additional information superimposing unit 27.

The luminance calculation unit 23 calculates the luminance around the AR marker in the RGB image (hereinafter referred to as peripheral luminance), and stores the calculation result in the buffer 29.

The coring unit 24 reads the peripheral luminance calculation result and the IR light image from the buffer 29. The coring unit 24 removes noise from the IR light image by performing a coring process on the IR light image based on the ambient brightness and the ambient temperature of the camera system 1. The coring unit 24 stores the IR light image after the coring process in the buffer 29.

The threshold determination unit 25 compares the pixel value of each pixel of the IR light image after the coring process stored in the buffer 29 with a predetermined threshold. Then, the threshold determination unit 25 sets the pixel value of the pixel determined to be greater than or equal to the predetermined threshold to 1, and sets the pixel value of the pixel determined to be less than the predetermined threshold to 0. A binary image is generated and stored in the buffer 29.

The marker detection unit 26 detects the AR marker using the binarized image stored in the buffer 29. The marker detection unit 26 stores the detection result of the AR marker in the buffer 29 and supplies it to the IR light control unit 28.

The additional information superimposing unit 27 superimposes additional information corresponding to the AR marker on the RGB image based on the detection result of the AR marker stored in the buffer 29. The additional information superimposing unit 27 outputs the RGB image on which the additional information is superimposed to a subsequent device or the like.

The IR light control unit 28 controls the amount of IR light emitted from the IR light irradiation device 15 based on the detection result of the AR marker and the calculation result of the peripheral luminance stored in the buffer 29.

The buffer 29 is configured by a memory such as a RAM, for example.

The IR light irradiation device 15 irradiates a region including the imaging range of the camera system 1 with IR light in a predetermined wavelength band under the control of the IR light control unit 28.

{Shooting process}
Next, the AR process executed by the camera system 1 will be described with reference to the flowchart of FIG.

In step S1, the camera system 1 performs shooting. Specifically, the IR light irradiation device 15 irradiates IR light in the shooting direction of the camera system 1 under the control of the IR light control unit 28. On the other hand, light from the subject enters the bandpass filter 11. The incident light includes reflected light obtained by reflecting the IR light from the IR light irradiation device 15 by the subject.

The band-pass filter 11 transmits light in a predetermined wavelength band including a visible light region and an IR light region among incident light from the subject, and blocks light in other wavelength bands. Light that has passed through the bandpass filter 11, that is, light that includes only the visible light component and the IR light component among the light from the subject, enters the image sensor 12 and forms an image on the light receiving surface of the image sensor 12.

The image sensor 12 photoelectrically converts the light imaged on the light receiving surface into an electrical signal corresponding to the intensity of the light for each pixel, and further generates an input image composed of pixel signals obtained by AD conversion of the electrical signal of each pixel. The image sensor 12 supplies the generated input image to the separation unit 21.

As described above with reference to FIG. 2, in the image sensor 12, R pixels, G pixels, B pixels, and IR pixels are arranged in a grid pattern. The signal output from the R pixel includes the R component and the IR light component, the pixel signal output from the G pixel includes the G component and the IR light component, and the pixel signal output from the B pixel includes the B component and The pixel signal that includes the IR light component and is output from the IR pixel includes only the IR light component.

In the following, it is assumed that the AR marker 101 shown in FIG. 4 is arranged within the shooting range of the camera system 1. The AR marker 101 is divided into a reflective portion 101A filled with black and a non-reflective portion 101B filled with white in the drawing. The reflective portion 101A is, for example, a portion to which a colorless and transparent paint having a high reflectivity with respect to IR light is applied. The non-reflective portion 101B is a portion where a colorless and transparent paint is not applied.

Therefore, when the AR marker 101 is irradiated with IR light, the IR light is reflected only by the reflecting portion 101A and enters the camera system 1. Therefore, as will be described later, in the camera system 1, the shapes of the reflective portion 101A and the non-reflective portion 101B are recognized as the pattern of the AR marker 101.

Also, the AR marker 101 is transparent and colorless in both the reflection part 101A and the non-reflection part 101B, and cannot be seen by human eyes. Therefore, even if the AR marker 101 is arranged in various places, the scenery and the design are not impaired.

In step S2, the separation unit 21 separates the visible light component and the IR light component. That is, the separation unit 21 separates each pixel signal component of the input image into a visible light component and an IR light component.

For example, the separation unit 21 extracts the R component by subtracting the IR light component included in the pixel signal of the neighboring IR pixel from the component of the pixel signal of the R pixel. Further, the separation unit 21 extracts the IR light component by subtracting the extracted R component from the component of the pixel signal of the R pixel. Thereby, the R component (visible light component) and the IR light component of the pixel signal of the R pixel are separated.

Similarly, the G component (visible light component) and the IR light component of the pixel signal of the G pixel are separated, and the B component (visible light component) and the IR light component of the pixel signal of the B pixel are separated.

The separation unit 21 corresponds to the intensity of the IR light component of the pixel signal of the R pixel, the IR light component of the pixel signal of the G pixel, the IR light component of the pixel signal of the B pixel, and the IR light component of the IR pixel. An IR light image having pixel values is generated and stored in the buffer 29. The separation unit 21 also includes an R component of the pixel signal of the R pixel, a G component of the pixel signal of the G pixel, a B component of the pixel signal of the B pixel, and a visible light component of the pixel signal of the IR pixel (in practice, A mosaic image having a pixel value corresponding to the intensity of 0) is generated and supplied to the visible light image processing unit 22.

In step S3, the visible light image processing unit 22 performs visible light image processing. For example, the visible light image processing unit 22 performs demosaic processing on the mosaic image, and generates an RGB image by complementing the R component, G component, and B component pixel values of each pixel. In addition, the visible light image processing unit 22 performs processing on a normal RGB image such as white balance processing and noise removal processing on the generated RGB image, for example. Then, the visible light image processing unit 22 supplies the RGB image to the luminance calculation unit 23 and the additional information superimposing unit 27.

In step S4, the luminance calculation unit 23 calculates the luminance around the AR marker 101. Specifically, the luminance calculation unit 23 calculates the luminance component (Y component) of each pixel from the pixel value of each RGB component of each pixel of the RGB image using a predetermined conversion formula. Then, the luminance calculation unit 23 calculates the peripheral luminance around the AR marker 101 based on the luminance component of each pixel of the RGB image.

For example, the luminance calculation unit 23 calculates an average value of luminance components of each pixel of the entire RGB image as the peripheral luminance (overall luminance).

Alternatively, for example, as shown in FIG. 5, the luminance calculation unit 23 calculates the average value of the luminance components of the pixels in the rectangular luminance detection region 112 around the AR marker 101 of the RGB image 111 as the peripheral luminance. This makes it possible to detect the brightness around the AR marker 101 more accurately.

The position of the luminance detection area 112 (for example, the coordinates (X, Y), width W, height H, etc. of the luminance detection area 112) may be set by the user, or the camera system 1 may It may be set automatically.

For example, when the user sets, since the AR marker 101 is not visible as described above, it is usually difficult to set the luminance detection region 112 so as to include the AR marker 101. However, for example, when the object or place where the AR marker 101 is arranged is known in advance, the brightness detection area 112 is reliably set by setting the brightness detection area 112 so as to include the object or place. It is possible to include the AR marker 101 inside.

Further, when the camera system 1 is set, for example, since the outer periphery of the AR marker 101 is rectangular, it is relatively easy to detect the outer periphery of the AR marker 101 using an edge component or the like and detect the position of the AR marker 101. is there. Therefore, before performing the detailed detection process of the AR marker 101 in step S8 described later, for example, the luminance calculation unit 23 performs only the position detection of the AR marker 101, and sets the area including the AR marker 101 as a luminance detection area. 112 may be set.

Alternatively, for example, the luminance calculation unit 23 may set the luminance detection region 112 based on the detection result of the AR marker 101 in the previous frame.

Then, the luminance calculation unit 23 stores the calculation result of the peripheral luminance in the buffer 29.

In step S5, the temperature sensor 13 measures the temperature around the camera system 1. The temperature sensor 13 supplies the measurement result to the coring unit 24.

In step S6, the coring unit 24 performs a coring process. Specifically, the coring unit 24 reads the IR light image and the calculation result of the peripheral luminance from the buffer 29. Then, the coring unit 24 corrects each pixel value of the IR light image based on the ambient brightness and the ambient temperature of the camera system 1.

Specifically, the coring unit 24 uses, for example, the graph shown in FIG. 6 to calculate the subtraction amount (hereinafter referred to as luminance-based subtraction amount) for the pixel value of each pixel of the IR light image based on the peripheral luminance. ) Is calculated. In this graph, the horizontal axis represents luminance (peripheral luminance), and the vertical axis represents subtraction amount (luminance-based subtraction amount).

In this example, the luminance base subtraction amount is fixed to the smallest SYL in the range where the peripheral luminance is less than YL. In the range where the peripheral luminance is YL or more and YH or less, the luminance base subtraction amount increases linearly from the minimum SYL to the maximum SYH in proportion to the peripheral luminance. In a range where the peripheral luminance exceeds YH, the luminance base subtraction amount is fixed to the maximum SYH. That is, since the noise component of the IR light image increases as the peripheral luminance increases, the luminance base subtraction amount is set to be large.

Further, the coring unit 24 uses, for example, a graph shown in FIG. 7 to calculate a subtraction amount (hereinafter referred to as a temperature-based subtraction amount) for the pixel value of each pixel of the IR light image based on the ambient temperature of the camera system 1. Is calculated). In this graph, the horizontal axis indicates the temperature around the camera system 1 and the vertical axis indicates the subtraction amount (temperature-based subtraction amount).

In this example, the temperature base subtraction amount is fixed to the minimum STL in the range where the ambient temperature is less than TL. In the range where the ambient temperature is not less than TL and not more than TH, the temperature base subtraction amount increases linearly from the minimum STL to the maximum STH in proportion to the ambient temperature. In the range where the ambient temperature exceeds TH, the temperature-based subtraction amount is fixed to the maximum STH. That is, since the noise component of the IR light image increases as the ambient temperature increases, the temperature base subtraction amount is set to be larger.

Next, the coring unit 24 calculates a final subtraction amount (hereinafter referred to as a total subtraction amount) based on the luminance base subtraction amount and the temperature base subtraction amount. For example, the coring unit 24 calculates the total subtraction amount by adding the luminance base subtraction amount and the temperature base subtraction amount. Alternatively, for example, the coring unit 24 calculates the total subtraction amount by weighted addition of the luminance base subtraction amount and the temperature base subtraction amount.

The coring unit 24 corrects the pixel value of each pixel to be reduced by subtracting the total subtraction amount from the pixel value of each pixel of the IR light image. Thus, the pixel value of each pixel is reduced according to the amount of noise component generated in the IR light image, and the noise component in the IR light image is appropriately reduced. The coring unit 24 stores the IR light image after the coring process in the buffer 29.

In step S7, the threshold determination unit 25 performs a threshold determination process. Specifically, the threshold determination unit 25 reads the IR light image after the coring process from the buffer 29. Next, the threshold determination unit 25 determines whether the pixel value of each pixel of the IR light image is equal to or greater than a predetermined threshold.

Then, the threshold value determination unit 25 sets the pixel value of a pixel whose pixel value is equal to or greater than a predetermined threshold, in other words, a pixel whose IR light component after the coring process is equal to or greater than a predetermined intensity to 1, for example. In addition, the threshold determination unit 25 sets, for example, the pixel value of a pixel having a pixel value less than a predetermined threshold, in other words, a pixel having an IR light component after coring processing less than a predetermined intensity to 0. Thereby, a binarized image is generated. The threshold determination unit 25 stores the generated binarized image in the buffer 29.

It should be noted that the threshold value used for this processing may be adjusted based on, for example, the ambient brightness, the ambient temperature of the camera system 1, and the like.

In step S8, the marker detection unit 26 performs a process of detecting the AR marker 101. Specifically, the marker detection unit 26 reads a binarized image from the buffer 29. Next, the marker detection unit 26 detects the AR marker 101 by searching the binarized image for an image having the same shape as the reflection unit 101A of the AR marker 101 by a predetermined method such as pattern matching. Do.

In step S9, the marker detection unit 26 determines whether or not the AR marker 101 has been detected based on the result of the process in step S8. If it is determined that the AR marker 101 has been detected, the process proceeds to step S10.

In step S10, the image processing unit 14 performs additional information superimposition processing. Specifically, the marker detection unit 26 stores the detection result of the AR marker 101 in the buffer 29. This detection result includes, for example, the coordinates of the position where the AR marker 101 is detected in the binarized image.

The additional information superimposing unit 27 reads the detection result of the AR marker 101 from the buffer 29. Next, the additional information superimposing unit 27 superimposes additional information corresponding to the AR marker 101 in the vicinity of the position where the AR marker 101 is detected in the RGB image. Then, the additional information superimposing unit 27 outputs an RGB image on which the additional information is superimposed, and supplies the RGB image to a subsequent device or the like. Thereby, for example, an RGB image on which additional information is superimposed is displayed in the subsequent device.

Thereafter, the process returns to step S1, and the processes after step S1 are executed. That is, the above-described AR processing is performed on each frame of the input image.

On the other hand, if it is determined in step S9 that the AR marker 101 cannot be detected, the process proceeds to step S11.

In step S11, the image processing unit 14 adjusts the irradiation amount of the IR light. Specifically, the marker detection unit 26 notifies the IR light control unit 28 that the AR marker 101 has not been detected.

The IR light control unit 28 reads out the calculation result of the peripheral luminance from the buffer 29. The IR light control unit 28 calculates an appropriate amount of IR light based on the peripheral luminance. For example, the IR light control unit 28 decreases the amount of IR light when the peripheral luminance is high, and increases the amount of IR light when the peripheral luminance is low. It should be noted that it is desirable not to increase the amount of IR light too much in order to suppress an increase in power consumption and noise generation due to irregular reflection of IR light. Then, the IR light control unit 28 controls the IR light irradiation device 15 so that the amount of IR light irradiated by the IR light irradiation device 15 becomes the calculated light amount.

Thereby, the amount of IR light is set to an appropriate value based on the peripheral luminance, and the detection accuracy of the AR marker 101 is improved.

The coring process in step S6 and the IR light irradiation amount adjusting process in step S11 may be limited in the number and time of execution so that hunting does not occur.

Thereafter, the process returns to step S1, and the processes after step S1 are executed.

As described above, the detection accuracy of the AR marker 101 can be improved. As a result, additional information can be superimposed on an appropriate position of the RGB image and presented to the user.

In addition, since the AR marker 101 is colorless and transparent, the AR marker 101 can be arranged at a desired position without deteriorating the scenery or the design. That is, the degree of freedom of arrangement of the AR marker is improved. Further, since the AR marker 101 is colorless and transparent, the confidentiality of information is improved.

Furthermore, it is not necessary to take an image only for detecting the AR marker, and the AR marker can be detected while performing normal shooting.

Also, compared to a markerless method that does not use an AR marker, the load and data required for processing can be reduced.

<2. Second Embodiment>
Next, a second embodiment of the present technology will be described with reference to FIG. In the second embodiment, a marker AR using an AR marker that emits IR light itself is realized.

{Configuration example of camera system 201}
FIG. 8 illustrates a configuration example of the camera system 201 according to the second embodiment of the present technology. In the figure, portions corresponding to those in FIG.

The camera system 201 is common to the camera system 1 in that it includes a bandpass filter 11, an image sensor 12, and a temperature sensor 13. On the other hand, the camera system 201 differs from the camera system 1 in that an image processing unit 214 is provided instead of the image processing unit 14 and that the IR light irradiation device 15 is not provided.

Compared with the image processing unit 14 of the camera system 1, the image processing unit 214 is common in that it includes a visible light image processing unit 22 and an additional information superimposing unit 27. On the other hand, the image processing unit 214 is different from the image processing unit 14 of the camera system 1 in place of the separation unit 21, the luminance calculation unit 23, the coring unit 24, the threshold determination unit 25, and the marker detection unit 26. The difference is that it includes a separation unit 221, a luminance calculation unit 223, a coring unit 224, a threshold value determination unit 225, and a marker detection unit 226, and that the IR light control unit 28 is not provided.

The separation unit 221, the luminance calculation unit 223, the coring unit 224, the threshold determination unit 225, and the marker detection unit 226 are the separation unit 21, the luminance calculation unit 23, the coring unit 24, and the threshold determination unit 25 of the image processing unit 14. And basically the same processing as the marker detection unit 26 is performed. However, the separation unit 221, the luminance calculation unit 223, the coring unit 224, the threshold value determination unit 225, and the marker detection unit 226 directly exchange data with each other without using a buffer.

Specifically, the separation unit 221 separates the visible light component and the IR light component of each pixel of the input image, like the separation unit 21. Then, the separation unit 221 supplies the mosaic image including the visible light component to the visible light image processing unit 22, and supplies the IR light image including the IR light component to the luminance calculation unit 223.

The luminance calculation unit 223 calculates the luminance around the AR marker in the RGB image supplied from the visible light image processing unit 22 (peripheral luminance), as with the luminance calculation unit 23. The luminance calculation unit 223 supplies the IR light image and the calculation result of the peripheral luminance to the coring unit 224.

The coring unit 224 performs the coring process on the IR light image based on the ambient brightness and the ambient temperature of the camera system 1 measured by the temperature sensor 13, similarly to the coring unit 24. The coring unit 24 supplies the IR light image after the coring process to the threshold value determination unit 225.

Like the threshold value determination unit 25, the threshold value determination unit 225 performs a threshold value determination process on the IR light image after the coring process. The threshold determination unit 225 supplies the binarized image generated by the threshold determination process to the marker detection unit 226.

The marker detection unit 226 detects an AR marker using a binarized image, as with the marker detection unit 26. The marker detection unit 226 supplies the AR marker detection result to the additional information superimposing unit 27.

Note that, unlike the camera system 1, the IR light irradiation device 15 is not provided in the camera system 201. Therefore, the camera system 201 detects an AR marker that emits IR light. For example, the AR marker is detected by replacing the reflective portion 101A of the AR marker 101 in FIG. 4 described above with a light emitting portion that emits IR light.

<3. Modification>
{Variation related to coring processing}
In the above description, an example in which a common total subtraction amount is set for each pixel has been described. However, a plurality of total subtraction amounts may be used for each region or pixel in the image. For example, the total amount of subtraction may be set for each region by dividing the IR light image into a plurality of regions, calculating the peripheral luminance for each region, and setting the luminance base subtraction amount.

Further, for example, the total subtraction amount may be set for each pixel by setting the luminance base subtraction amount for each pixel based on the luminance component of each pixel of the IR light image.

Further, for example, the subtraction amount may be expressed not by the absolute value as described above but by a ratio for decreasing the pixel value. For example, when the total subtraction amount is set to 10%, the pixel value of each pixel of the IR light image may be corrected by 10%.

Moreover, the graphs for calculating the subtraction amount in FIGS. 6 and 7 are examples thereof, and other graphs may be used. For example, instead of a linearly changing graph as shown in FIGS. 6 and 7, it is also possible to use a curved graph as shown in FIG. For such a curved graph, for example, a plurality of representative points are set, a pseudo curve connecting the representative points with a straight line, a curve obtained by neighborhood interpolation processing using the plurality of representative points, or the like is used. .

Further, for example, the subtraction amount may be set based on only one of the ambient brightness and the ambient temperature.

{Modification regarding control of the amount of IR light}
In the above description, an example in which the IR light control unit 28 controls the amount of IR light from the IR light irradiation device 15 based on the peripheral luminance has been shown. For example, based on the subtraction amount of the coring process, the IR light control unit 28 The amount of light may be controlled. For example, when the subtraction amount is large, the IR light control unit 28 assumes that the peripheral luminance is high and the noise amount is large. Therefore, when the IR light amount is decreased and the subtraction amount is small, the peripheral luminance is low and the noise amount is small. Therefore, the amount of IR light may be increased.

For example, the IR light control unit 28 may control the amount of IR light based on both the peripheral luminance and the subtraction amount of the coring process.

Furthermore, for example, when the AR marker is not detected, the IR light control unit 28 may increase or decrease the amount of IR light by a predetermined value without using the peripheral luminance and the subtraction amount of the coring process. Good.

For example, the IR light control unit 28 may control the amount of IR light based on at least one of the peripheral luminance and the subtraction amount of the coring process even when the AR marker is detected.

{Variation of image sensor 12}
In the above description, as shown in FIG. 2 described above, an example in which the image sensor 12 including IR pixels is used has been described. However, in the present technology, an image sensor that does not include IR pixels may be used. . For example, a conventional Bayer array type image sensor can be used.

When a conventional Bayer array type image sensor is used, pixels capable of receiving an IR light component and performing photoelectric conversion are used for both the R pixel, the G pixel, and the B pixel. Further, the IR light is set to enter the image sensor without blocking the IR light by an IR cut filter or the like as in a normal camera.

Further, when a Bayer array type image sensor is used, for example, frequency analysis is performed on each pixel signal of R pixel, G pixel, and B pixel by using a technique such as FFT (Fast Fourier Transform). Based on the result, it is possible to separate the visible light component and the IR light component.

Furthermore, for example, it is possible to use an image sensor that can detect invisible light (for example, ultraviolet light) having a wavelength band different from that of IR light. In this case, an AR marker that reflects or emits the invisible light is used.

Also, for example, it is possible to use an image sensor capable of detecting visible light in a wavelength band other than the R wavelength band, the G wavelength band, and the B wavelength band (for example, a yellow wavelength band).

{Other variations}
In the above description, the additional information is superimposed on the RGB image in the camera system 1 and then output. However, the detection result of the RGB image and the AR marker is output, and the additional information is superimposed on the RGB image in the subsequent apparatus. You may make it do.

Further, for example, the threshold determination unit 25 (or the threshold determination unit 225) determines whether or not the pixel value of the IR light image after the coring process satisfies a value that can detect the AR marker 101. May be. For example, the threshold determination unit 25 determines that the AR marker 101 can be detected when the average value of the pixel values of the IR light image after the coring process is equal to or greater than a predetermined threshold. 101 is determined to be undetectable.

In this case, the threshold determination unit 25 may use an average value of pixel values of the entire IR light image, or may use an average value of pixel values for each predetermined region. In the former case, it is determined whether or not the AR marker 101 can be detected in the entire IR light image. In the latter case, it is determined whether or not the AR marker 101 can be detected for each region of the IR light image. .

Further, for example, in the camera system 1 of FIG. 1, as in the camera system 201 of FIG. 8, each unit may directly exchange data without passing through the buffer 29. Further, for example, in the camera system 201 in FIG. 8, each unit may exchange data via a buffer as in the camera system 1 in FIG.

In addition, this technology can be applied to both moving images and still images.

{Example of computer configuration}
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

FIG. 10 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processing by a program.

In the computer, a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, and a RAM (Random Access Memory) 403 are connected to each other by a bus 404.

Further, an input / output interface 405 is connected to the bus 404. An input unit 406, an output unit 407, a storage unit 408, a communication unit 409, and a drive 410 are connected to the input / output interface 405.

The input unit 406 includes a keyboard, a mouse, a microphone, and the like. The output unit 407 includes a display, a speaker, and the like. The storage unit 408 includes a hard disk, a nonvolatile memory, and the like. The communication unit 409 includes a network interface. The drive 410 drives a removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 401 loads, for example, a program stored in the storage unit 408 to the RAM 403 via the input / output interface 405 and the bus 404 and executes the program, and the series described above. Is performed.

The program executed by the computer (CPU 401) can be provided by being recorded on a removable medium 411 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 408 via the input / output interface 405 by attaching the removable medium 411 to the drive 410. The program can be received by the communication unit 409 via a wired or wireless transmission medium and installed in the storage unit 408. In addition, the program can be installed in the ROM 402 or the storage unit 408 in advance.

The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

Furthermore, embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

For example, the present technology can take a cloud computing configuration in which one function is shared by a plurality of devices via a network and is jointly processed.

Further, each step described in the above flowchart can be executed by one device or can be shared by a plurality of devices.

Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

Further, the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

Furthermore, for example, the present technology can take the following configurations.

(1)
A separation unit that separates the invisible light component from the input image including the visible light component and the invisible light component;
A noise removing unit for removing noise by reducing a pixel value of the first image based on an invisible light component of the input image;
An image processing apparatus comprising: a marker detection unit configured to detect an AR (Augmented Reality) marker indicating a predetermined pattern with invisible light in the first image from which noise is removed.
(2)
The separation unit further separates a visible light component from the input image;
The noise removal unit is configured to determine the first image based on at least one of luminance around the AR marker in the second image based on a visible light component of the input image or temperature around the image processing device. The image processing apparatus according to (1), wherein a subtraction amount that is an amount by which the pixel value of the image is reduced is set.
(3)
The AR marker shows the pattern by reflecting invisible light;
Irradiation control for controlling the intensity of the invisible light irradiated to the AR marker based on at least one of the brightness around the AR marker in the second image, the subtraction amount, and the detection result of the AR marker. The image processing apparatus according to (2), further including a unit.
(4)
(1) to (3), further comprising: an imaging element that includes a first pixel that detects a visible light component and an invisible light component, and a second pixel that detects only the invisible light component, and captures the input image. An image processing apparatus according to any one of the above.
(5)
The imaging device according to any one of (1) to (3), further including an imaging element that includes a plurality of types of pixels that detect invisible light components and visible light components each having a different wavelength band, and that captures the input image. Image processing device.
(6)
The separation unit further separates a visible light component from the input image;
The image processing device according to any one of (1) to (5), further including an additional information superimposing unit that superimposes additional information corresponding to the AR marker on a second image based on a visible light component of the input image. .
(7)
A separation step of separating the invisible light component from the input image including the visible light component and the invisible light component;
A noise removal step of removing noise by reducing a pixel value of the first image based on an invisible light component of the input image;
A marker detection step of detecting an AR (Augmented Reality) marker that shows a predetermined pattern with invisible light in the first image from which noise has been removed.
(8)
A separation step of separating the invisible light component from the input image including the visible light component and the invisible light component;
A noise removal step of removing noise by reducing a pixel value of the first image based on an invisible light component of the input image;
A program for causing a computer to execute processing including: a marker detection step of detecting an AR (Augmented Reality) marker showing a predetermined pattern with invisible light in the first image from which noise has been removed.

1 camera system, 11 band pass filter, 12 image sensor, 13 temperature sensor, 14 image processing unit, 15 IR light irradiation device, 21 separation unit, 22 visible light image processing unit, 23 luminance calculation unit, 24 coring unit, 25 Threshold determination unit, 26 marker detection unit, 27 additional information superimposition unit, 28 IR light control unit, 101 AR marker, 101A reflection unit, 101B non-reflection unit, 201 camera system, 214 image processing unit, 221 separation unit, 223 luminance calculation Part, 224 coring part, 225 threshold judgment part, 226 marker detection part

Claims

A separation unit that separates the invisible light component from the input image including the visible light component and the invisible light component;
A noise removing unit for removing noise by reducing a pixel value of the first image based on an invisible light component of the input image;
An image processing apparatus comprising: a marker detection unit configured to detect an AR (Augmented Reality) marker indicating a predetermined pattern with invisible light in the first image from which noise is removed.
The separation unit further separates a visible light component from the input image;
The noise removal unit is configured to determine the first image based on at least one of luminance around the AR marker in the second image based on a visible light component of the input image or temperature around the image processing device. The image processing apparatus according to claim 1, wherein a subtraction amount, which is an amount by which the pixel value of is reduced, is set.
The AR marker shows the pattern by reflecting invisible light;
Irradiation control for controlling the intensity of the invisible light irradiated to the AR marker based on at least one of the brightness around the AR marker in the second image, the subtraction amount, and the detection result of the AR marker. The image processing apparatus according to claim 2, further comprising a unit.
The image processing according to claim 1, further comprising: an imaging element that includes a first pixel that detects a visible light component and an invisible light component, and a second pixel that detects only the invisible light component, and captures the input image. apparatus.
The image processing apparatus according to claim 1, further comprising: an imaging element that includes a plurality of types of pixels that detect invisible light components and visible light components having different wavelength bands, and that captures the input image.
The separation unit further separates a visible light component from the input image;
The image processing apparatus according to claim 1, further comprising an additional information superimposing unit that superimposes additional information corresponding to the AR marker on a second image based on a visible light component of the input image.
A separation step of separating the invisible light component from the input image including the visible light component and the invisible light component;
A noise removal step of removing noise by reducing a pixel value of the first image based on an invisible light component of the input image;
A marker detection step of detecting an AR (Augmented Reality) marker that shows a predetermined pattern with invisible light in the first image from which noise has been removed.
A separation step of separating the invisible light component from the input image including the visible light component and the invisible light component;
A noise removal step of removing noise by reducing a pixel value of the first image based on an invisible light component of the input image;
A program for causing a computer to execute processing including: a marker detection step of detecting an AR (Augmented Reality) marker showing a predetermined pattern with invisible light in the first image from which noise has been removed.