JP4962363B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  Conventionally, an image processing apparatus that detects an abnormal region using a surveillance camera or the like has been proposed. In the detection of an abnormal area, for example, in Patent Document 1, a change area is calculated by a difference process between monitoring images captured at different times in an already detected abnormal area, and a background image is detected outside the abnormal area. It is disclosed that a change area is calculated by a difference process with a monitoring image.

Here, in patent document 1, since environmental conditions at the time of image capturing are not taken into consideration, there is a possibility that an abnormal region cannot be accurately detected when images are taken outdoors or indoor lighting conditions are different. . Therefore, in Patent Document 2, in a traffic monitoring device, an image feature of an image region in an input image is extracted, an external environment is determined based on the extracted image feature, and a threshold based on a determination result is used to determine a difference between images. A technique for detecting an abnormal region from a difference image is disclosed.
JP 2007-180933 A JP 2002-83301 A

  However, in Patent Document 2, the threshold value used for determining the abnormal region is changed according to the environmental condition of the input image, but the relationship between the background image and the input image is not considered in the first place. That is, it is not possible to remove the partial influence due to the environmental condition only by changing the threshold according to the environmental condition of the input image. Such a case will be described with reference to FIG.

  FIG. 14 is a diagram illustrating a background image and an input image. For example, as shown in FIG. 14, when different shadows A appear in the background image and the input image, the difference image between the background image and the input image is caused by the luminance difference due to the shadow A and the abnormal region (object B). If the difference in brightness is the same, both may be determined to be abnormal areas.

  Therefore, simply changing the threshold value for determining the abnormal region does not mean that the environmental condition of the background image is taken into consideration, and the environmental condition between images for detecting the abnormal region cannot be matched. In addition, in the method of detecting an abnormal area using one background image, there is a problem that an abnormal area cannot be detected properly if there is no background image similar to the input image.

  The present invention has been made in view of the above problems, and provides an image processing apparatus capable of detecting an appropriate abnormal region using a plurality of background images while considering environmental conditions between images. Objective.

  A storage unit that stores a plurality of captured images in association with environmental condition specifying information that specifies an environmental condition when the image is captured; a first input unit that inputs a newly captured input image; , Second input means for inputting environmental condition specifying information for specifying environmental conditions when the input image is captured, each image stored in the storage means, and an input image input by the first input means Based on the generation means for generating each difference image, each environmental condition specifying information stored by the storage means, and the environmental condition specifying information input by the second input means Based on a result image obtained by adding weighted difference values at the same position to each weight difference image accumulated by the accumulation means Characterized in that it comprises a detection means for detecting an abnormal region of the input image.

  According to the present invention, it is possible to detect an appropriate abnormal region using a plurality of background images while considering environmental conditions between images.

  Embodiments of the present invention will be described below with reference to the drawings.

An image processing apparatus according to an embodiment of the present invention has a functional configuration as shown in FIG.
FIG. 1 is a schematic functional block diagram of an image processing apparatus according to an embodiment of the present invention.

  The image processing apparatus 100 according to the first embodiment includes a database 101, an image input unit 102, a difference image generation unit 103, an environmental condition specifying information input unit 104, a similarity calculation unit 105, a weight difference result image generation unit 106, an abnormal region. The detection means 109 is included.

  The database 101 stores an image captured by an imaging unit such as a small camera or a surveillance camera (hereinafter, an image stored in the database 101 is referred to as a background image). Further, the database 101 stores environmental condition specifying information in association with the stored background image.

  The environmental condition specifying information will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of environmental condition specifying information. As shown in FIG. 2, the environmental condition specifying information includes the time when the image is captured, the position, weather information, and the like. Here, the weather information at the time of imaging refers to all or part of the image when the sky is imaged by the imaging means, the feature amount extracted from the captured sky image, and the like.

  FIG. 3 is a diagram illustrating an example of a part of an image obtained by imaging the sky in typical weather such as sunny, cloudy, and rainy. FIG. 3A is an image of a sunny day, and a blue sky is captured. FIG. 3B is an image of a cloudy day, in which a dull cloud is captured. FIG.3 (c) is an image of a rainy day, and the dark rain cloud is imaged.

  The feature amount extracted from the captured image is an average RGB value or color variation amount of all or a part of the sky image as shown in FIG. Further, this feature amount may be manually extracted.

  The image input unit 102 inputs an image captured by the imaging unit (a newly captured image is referred to as an input image) and outputs the image to the difference image generation unit 103.

  The difference image generation unit 103 generates a difference image n (1 ≦ n ≦ N) from the input image acquired from the image input unit 102 for each of the N background images stored in the database 101. Further, the difference image generation unit 103 outputs the generated difference image n to the weight difference result image generation unit 106.

  Here, the difference image will be described with reference to FIGS. 4 and 5. FIG. 4 is a diagram showing an input image and a background image n (1 ≦ n ≦ N).

  FIG. 4A shows an input image captured at position A and time A. FIG. In the input image, the shadow A and the object A of the building are imaged. FIG. 4B shows the background image 1 captured at position A and time B. A shadow B is captured in the background image 1. FIG. 4C shows the background image 2 captured at position A and time C. In the background image 2, a shadow C is captured. FIG. 4D is a background image N captured at position B and time C. In the background image N, an object B (traffic light) and an object C (vehicle) are imaged.

At this time, the difference image generation unit 103 first calculates a difference value for each pixel between the input image and the background image n. Next, if the difference value is equal to or greater than the set threshold value, it is determined that the position is abnormal, and the difference value is set to 1. If the difference value is less than the set threshold value, it is determined that there is not much difference between the pixels. Set the difference value to 0. This process is performed for all the pixels, and an image in which each pixel is converted to 1 or 0 is a difference image.

  In this embodiment, the difference images all have values of 0 or 1, but the difference image itself or the absolute value of the difference value may be used as the difference image.

  FIG. 5 is a diagram showing a difference image n between the input image shown in FIG. 4 and the background image n. In the difference image 1, it is determined that the object A and the difference portion between the shadow A and the shadow B are abnormal positions, and the value is 1. When the same processing is performed, in the difference image 2, it is determined that the object A and the difference portion of the shadow A and the shadow C are abnormal positions. In the difference image N, the object A, the object B, and the object C are mainly It is determined that the position is abnormal.

  Returning to FIG. 1, the environmental condition specifying information input unit 104 inputs time, position, and weather information when the input image is captured, and outputs the environmental condition specifying information to the similarity calculation unit 105. More specifically, the time when the input image is captured is input by receiving it from a built-in clock or radio wave.

  As for the imaged position, when the image processing apparatus 100 is built in a moving body such as a vehicle, the GPS function is used to acquire the current position, or the image processing apparatus 100 is used as a stationary object such as a monitoring camera. If connected, the current position is directly input or set.

  Further, since the position does not change when connected to a stationary object, the position element in the environmental condition specifying information may be ignored. The captured weather information is input by an imaging unit that images the sky.

  The similarity calculating unit 105 calculates the similarity between the environmental condition specifying information acquired by the environmental condition specifying information input unit 104 and the environmental condition specifying information n (1 ≦ n ≦ N) read from the database 101. In addition, the similarity calculation unit 105 outputs the calculated similarity n (1 ≦ n ≦ N) to the weight difference result image generation unit 106.

  Here, the similarity will be described with reference to FIG. FIG. 6 is a diagram illustrating an example in which the environment information specifying information is arranged in a one-dimensional array. FIG. 6A is a diagram in which the elements of the environmental condition specifying information are arranged horizontally. FIG. 6B is a diagram in which specific values of the respective elements are arranged in a one-dimensional array. FIG. 6C is a diagram in which the specific values in FIG. 6B are expressed numerically for each element so that the similarity can be easily calculated. In this case, in FIG. 6C, RGB, north latitude, and east longitude are collectively digitized, but may be separately provided.

  Next, the similarity calculation unit 105 defines the input environmental condition specifying information as a one-dimensional array as shown in FIG. 6C and defines a vector, and uses the Euclidean distance between the vectors as the similarity. Here, a method of calculating the Euclidean distance will be described with reference to FIG. FIG. 7A is a diagram showing a primary array of environmental condition specifying information of a background image. FIG. 7B is a diagram showing a primary array of the environmental condition specifying information of the input image. FIG. 7C is a diagram showing an expression of the Euclidean distance. As shown in FIG. 7C, the Euclidean distance is calculated by taking a difference value for each element and square root of the sum of squares of the difference values.

  In addition to the Euclidean distance as the similarity, an angle between vectors may be used as the similarity. As for the angle formed between the vectors, the cosine value of the angle formed between the vectors is calculated by dividing the inner product between the vectors by the value obtained by integrating the magnitudes of the vectors.

  Returning to FIG. 1, the weight difference result image generation unit 106 includes an integration unit 107 and an addition unit 108. The accumulating unit 107 accumulates the weight coefficient n (1 ≦ n ≦ N) corresponding to the similarity n acquired from the similarity calculating unit 105 for each pixel with respect to the difference image n acquired from the difference image generating unit 103. The weight difference image n (1 ≦ n ≦ N) is generated.

  Here, the weighting coefficient n is a larger coefficient as the Euclidean distance as the similarity n is smaller. Note that the weighting coefficient is set to a value corresponding to the degree of similarity in advance through experiments or the like.

  Here, the weight coefficient is set to 0 if the Euclidean distance, which is the similarity, is equal to or greater than the threshold. As a result, the difference image corresponding to the environmental condition specifying information having a large Euclidean distance that is the similarity to the environmental condition specifying information of the input image is removed from the abnormal region detection process.

  Next, the adding means 108 adds the values at the same pixel position to all the weight difference images n to generate one weight difference result image. The process of generating the weight difference result image will be specifically described with reference to FIGS.

  FIG. 8 is a diagram illustrating an example in which the weighting coefficient n is integrated with respect to the difference image n. In the example of FIG. 8, the weighting difference image n is generated by integrating the weighting coefficient 15 for the difference image 1, the weighting coefficient 80 for the difference image 2, and the weighting coefficient 0 for the difference image N for each pixel.

  FIG. 9 is a diagram illustrating an example of a weight difference result image. In the example shown in FIG. 9, the weight difference result image is generated by adding the values of the same pixel position to all the weight difference images n.

  Here, it is noted that the values of the weight difference image N are all 0. As a result of calculating the similarity between the environmental condition specifying information N and the input image, it can be seen that the input image and the background image N are captured under different environmental conditions. Therefore, the weighting coefficient corresponding to the similarity N is set to 0, and the influence of the difference image N in the abnormal area detection is removed.

  Returning to FIG. 1, the weight difference result image generation unit 106 outputs the weight difference result image generated by the addition unit 108 to the abnormal region detection unit 109.

  When the abnormal area detection unit 109 acquires the weight difference result image from the weight difference result image generation unit 106, the abnormal area detection unit 109 determines whether or not each pixel of the weight difference result image is equal to or more than a threshold value. Further, the abnormal area detection unit 109 sets a position determined to be equal to or greater than the threshold as an abnormal position, and sets a set of abnormal positions as an abnormal area.

  FIG. 10 is a flowchart of the abnormal area detection process in the image processing apparatus according to the first embodiment. In step 1001, the image input unit 102 inputs an input image captured by the imaging unit and outputs the input image to the difference image generation unit 103.

  Progressing to step 1002 following step 1001, the difference image generation unit 103 generates a difference image n between the input image acquired from the image input unit 102 and the background image n stored in the database 101, and the difference image n Is output to the weight difference result image generation means 106.

  In step 1003, the environmental condition specifying information input unit 104 inputs the environmental condition specifying information corresponding to the input image and outputs it to the similarity calculation unit 105.

  Proceeding to step 1004 following step 1003, the similarity calculating unit 105 calculates the similarity n between the environmental condition specifying information acquired from the environmental condition specifying information input unit 104 and the environmental condition specifying information n stored in the database 101. And the similarity n is output to the weight difference result image generation means 106.

  Progressing to step 1005 following step 1002 and step 1004, the accumulating unit 107 integrates the weighting coefficient n corresponding to the similarity n acquired from the similarity calculating unit 105 to the difference image n acquired from the difference image generating unit 103. Then, a weight difference image n is generated.

  Further, the adding means 108 generates a weight difference result image by adding the values of the same pixel position to all the weight difference images n. Next, the weight difference result image generation unit 106 outputs the weight difference result image to the abnormal area detection unit 109.

  Proceeding to step 1006 following step 1005, the abnormal area detecting unit 109 determines whether or not each pixel of the weight difference result image acquired from the weight difference result image generating unit 106 is equal to or greater than the threshold value. The position described above is an abnormal position, and a set of abnormal positions is an abnormal region.

  As described above, according to the image processing apparatus according to the first embodiment, it is possible to detect an appropriate abnormal region using a plurality of background images while considering environmental conditions between images. That is, in detecting an abnormal area in an input image, even if there is no background image similar to the input image, an appropriate abnormal area can be detected by using a plurality of background images.

  Further, even when the number of background images stored in the database is small, the detection accuracy of the abnormal region can be increased by using a plurality of stored background images.

  Here, FIG. 11 shows an application example of the image processing apparatus according to the first embodiment described above. FIG. 11A is a diagram illustrating an example when the image processing apparatus according to the first embodiment is mounted on a vehicle. An imaging unit is provided in front of the vehicle and is connected to an image processing device mounted on the vehicle.

  In the case of FIG. 11 (a), it is assumed that the vehicle passes the same place at the same time, and when the image processing apparatus detects an abnormal area, a warning is issued, for example, to the driver as usual. Can tell you that there is something (abnormal).

  FIG. 11B is a diagram illustrating a case where the imaging unit is a monitoring camera or the like, and the image processing apparatus of the first embodiment is a personal computer or the like. In the case of FIG. 11B, it is possible to detect a true intruder or the like after removing the influence of the shadow of the object on the monitoring screen.

  Although the above two examples have been shown as application examples of the present invention, it goes without saying that the present invention is not limited to the above examples as long as the purpose is to detect an abnormal region of an input image using a plurality of background images. .

(Modification 1)
In the first modification of the image processing apparatus according to the first embodiment, the similarity is calculated by a method different from the similarity calculation method described above. In the first modification, when calculating the similarity, if there is an identifiable element among the elements of the environmental condition specifying information corresponding to the input image, an intermediate value of the similarity is calculated in advance for the identifiable element. .

  For example, in the case of FIG. 11A, since it is assumed that the same place is passed at the same time, the imaging time and the imaging position can be specified in advance in the environmental condition specifying information. After that, if the weather information at the time of capturing the input image is input, the similarity can be calculated.

  At this time, with respect to the imaging time and imaging position that can be specified in advance, an intermediate value of the similarity is calculated, and when the weather information is input, the similarity is finally calculated. Here, the intermediate value will be described with reference to FIG.

  FIG. 12 is a diagram illustrating an intermediate value when only weather information is not known. FIG. 12A shows the environmental condition specifying information of the background image. FIG. 12B shows environmental condition specifying information of the input image. FIG. 12C shows an intermediate value of similarity. As shown in FIG. 12B, only the weather information element of the environmental condition specifying information has not been specified yet. At this time, the calculation process similar to the similarity is performed using only the elements of the already specified environmental condition specifying information, and the obtained value is set as the intermediate value. After the weather information is input, the final similarity is calculated using the intermediate value.

  As described above, according to the image processing apparatus according to the first modification, it is possible to reduce the online processing load in the similarity calculation by calculating the intermediate value.

(Modification 2)
In the second modification of the image processing apparatus according to the first embodiment, the similarity is calculated by a method different from the similarity calculation method described above. In the second modification, when calculating the similarity, the variation amount (for example, variance) of each element of the environmental condition specifying information is measured, and the similarity is calculated after normalizing each element with the variation amount. To do.

  For example, when the weather information is represented by an empty RGB value and RGB is an element of the environmental condition specifying information, the respective RGB variation amounts are measured in advance. That is, for the sky color, if it is known that R varies most and G and B do not vary so much, normalization is performed in advance with each variation amount of R, G, and B.

  As described above, according to the image processing apparatus according to the modified example 2, it is possible to calculate the similarity considering the variation amount of each element of the environmental condition specifying information.

(Modification 3)
The modification 3 of the image processing apparatus according to the first embodiment calculates the similarity between the environmental condition specifying information before the generation of the difference image, and when the Euclidean distance that is the similarity is large, the corresponding difference image Is configured not to generate. This is because if the Euclidean distance, which is a similarity, is large, the weighting coefficient is 0, so that even if a difference image is generated, it is not taken into account for abnormal region detection.

  In this case, the similarity calculation unit 105 generates a difference image when the calculated similarity m (1 ≦ m ≦ N) is equal to or greater than a threshold value (when the weighting factor m corresponding to the similarity m is 0). The means 103 is instructed not to generate a difference image m between the corresponding background image m and the input image.

  When the difference image generation unit 103 is instructed not to generate the difference image m by the similarity calculation unit 105, the difference image generation unit 103 does not generate the difference image m.

  As described above, according to the image processing apparatus according to the third modification, even when a plurality of background images are used, an abnormal region can be detected in consideration of the processing load.

(Modification 4)
In the fourth modification of the image processing apparatus according to the first embodiment, the method for detecting an abnormal region is changed according to the number of background images stored in the database. If the number of background images stored in the database increases, the processing load increases. When the number of background images exceeds a threshold value, one background image is selected based on the similarity between the environmental condition specifying information. Like that.

  This is based on the premise that if the number of background images stored in the database increases, at least one background image similar to the input image can be found.

  More specifically, a background image number determination unit is newly provided, and when the background image number determination unit determines that the number of background images is less than the threshold value, the same processing as in the above-described embodiments and modifications is performed.

  If the background image number determination means determines that the number of background images is equal to or greater than the threshold value, the process shown in FIG. 13 is performed. FIG. 13 is a flowchart illustrating an abnormal area detection process when the number of background images is equal to or greater than a threshold value.

  In step 1301, the environmental condition specifying information input unit 104 inputs the environmental condition specifying information corresponding to the input image captured by the imaging unit.

  Proceeding to step 1302 following step 1301, the similarity calculating unit 105 calculates the similarity between the environmental condition specifying information acquired from the environmental information specifying information input unit 104 and the environmental condition specifying information stored in the database 101.

  Progressing to step 1303 following step 1302, the similarity calculation means 105 selects a background image corresponding to the environmental condition specifying information having the smallest Euclidean distance, which is the calculated similarity, from the database 101, and the selected background image. To the difference image generation means 103.

  Progressing to step 1304 following step 1303, the image input unit 102 inputs an image captured by the imaging unit and outputs the image to the difference image generation unit 103.

  In step 1305 following step 1304, the difference image generation unit 103 acquires the background image notified from the similarity calculation unit 105 from the database 101, and converts the acquired background image and the image input from the image input unit 102. On the other hand, a difference value is calculated for each pixel. Further, the difference image generation unit 103 outputs the calculated difference value to the abnormal area detection unit 109.

  Proceeding to step 1306 following step 1305, the abnormal area detection unit 109 determines whether or not the absolute value of the difference value acquired from the difference image generation unit 103 is equal to or greater than a set threshold value. If an abnormal position is detected, a set of detected abnormal positions is defined as an abnormal region.

  As described above, according to the image processing apparatus according to the modification example 4, an appropriate abnormal area detection process can be selected according to the number of background images stored in the database.

  Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and other than the above-described modifications within the scope of the gist of the present invention described in the claims. In addition, various modifications and changes can be made.

1 is a diagram illustrating schematic functional blocks of an image processing apparatus according to an embodiment of the present invention. The figure which shows an example of environmental condition specific information. The figure which shows an example of a part of image which imaged the sky. The figure which shows an example of an input image and a background image. The figure which shows an example of a difference image. The figure which shows the example which made environmental information specific information into the one-dimensional arrangement | sequence. The figure which shows how to obtain | require the Euclidean distance based on environmental condition specific information. The figure which shows the calculation process of a weight difference image. The figure which shows the calculation process of a weight difference result image. 3 is a flowchart illustrating abnormality detection processing of the image processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an application example of the image processing apparatus according to the first embodiment. The figure which shows an intermediate value. 10 is a flowchart showing an abnormal area detection process of an image processing apparatus according to a fourth modification. The figure which shows an example of a background image and an input image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Database 102 Image input means 103 Difference image generation means 104 Environmental condition specific information input means 105 Similarity calculation means 106 Weight difference result image generation means 107 Integration means 108 Addition means 109 Abnormal area detection means

Claims (4)

Storage means for storing a plurality of captured images in association with environmental condition specifying information for specifying environmental conditions when the images are captured;
First input means for inputting a newly captured input image;
A second input means for inputting environmental condition specifying information for specifying an environmental condition when the input image is captured;
Generating means for performing difference processing on each image stored in the storage means and the input image input by the first input means to generate each difference image;
Integration means for integrating the weighting factor based on each environmental condition specifying information stored by the storage means and the environmental condition specifying information input by the second input means for each pixel of each difference image;
Detecting means for detecting an abnormal region of the input image based on a result image obtained by adding weighted difference values at the same position to each weighted difference image accumulated by the accumulation unit;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the generation unit detects a difference between images for each pixel, and generates a difference image in which 1 is a pixel having a difference and 0 is a pixel having no difference.   The image processing apparatus according to claim 1, wherein the environmental condition specifying information includes an imaging time, an imaging position, or weather information at the time of imaging. The integrating means includes
Calculating means for calculating the similarity between each environmental condition specifying information stored by the storage means and the environmental condition specifying information input by the second input means;
The image processing apparatus according to any one of claims 1 to 3, wherein a weighting factor corresponding to the similarity is integrated for each pixel of each difference image.

Images