JP2017004295A - Traffic light recognition apparatus and traffic light recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately recognize a traffic light.SOLUTION: A traffic light recognition apparatus includes a storage part which stores a plurality of reference images indicating the traffic light, a captured image acquisition part which acquires a captured image captured by an imaging part, an extraction part which extracts a light emission area from the captured image, and a traffic light determination part which compares the image of the candidate area of the traffic light specified on the basis of the light emission area extracted by the extraction part among the captured images with the reference images and determines whether or not the image of the candidate area is the image of the traffic light.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a traffic signal recognition apparatus and a traffic signal recognition method.

  Signal recognition devices are used in various devices such as guide robots and driving support devices. Image recognition is performed for traffic signal recognition. Specifically, for example, Patent Document 1 describes a method of detecting a traffic signal in a range of color components of a traffic signal after adjusting exposure for a captured image.

  By the way, the color of the traffic lights is various. Therefore, in order to detect lights of various colors, it can be considered that the range of color components to be detected is widened to some extent. In this case, the probability of erroneous detection of an object other than a traffic light increases. Thus, for example, Patent Document 2 describes that the degree of separation is calculated for two or more types of regions extracted based on the range of color components, and a traffic light is determined.

  By the way, colors other than the lighting part of the traffic light are not characteristic. For this reason, in the above-described technique, there is a case where the traffic signal cannot be correctly recognized even when the separation degree of two types of regions is used, such as when there is a false detection in the extraction of the lighting part by the range of the color component. Therefore, for example, Non-Patent Document 1 describes that template detection is applied to the position of a traffic light estimated from an area extracted by a color range to reduce erroneous detection.

JP 2009-177411 A JP2013-242686A

Chunxiang WANG, Tao JIN, Ming YANG, Bing WANG, “Robust and Real-Time Traffic Lights Recognition in Complex Urban Environments”, International Journal of Computational Intelligence Systems, Vol. 4, No. 6 (December, 2011), 1383-1390

  However, the conventional technology cannot always recognize the traffic signal with high accuracy. Specifically, for example, since template matching has low robustness, matching accuracy may be reduced when the appearance of the traffic light changes due to a change in the position and direction of viewing the traffic light, ambient light, or the like.

  The present invention has been made in view of the above points, and provides a traffic signal recognition device and a traffic signal recognition method capable of accurately recognizing a traffic signal.

  (1) The present invention has been made to solve the above-described problems, and one aspect of the present invention acquires a storage unit that stores a plurality of reference images representing a traffic light and a captured image captured by the imaging unit. A captured image acquisition unit, an extraction unit that extracts a light emitting region from the captured image, an image of a candidate region of a traffic light that is identified based on the light emitting region extracted by the extraction unit among the captured images, and the reference A traffic light recognition apparatus comprising: a traffic light determination unit that compares an image with the image to determine whether the image of the candidate area is a traffic light image.

  (2) Further, according to one aspect of the present invention, in the traffic signal recognition apparatus according to (1), the plurality of reference images are images obtained based on a plurality of traffic signal images captured in different environments. It is characterized by being.

  (3) Further, according to one aspect of the present invention, in the traffic signal recognition device according to (1) or (2), the extraction unit is based on a relationship between parameters representing colors for each lighting color of the traffic light. The light emitting area is extracted.

  (4) Further, according to one aspect of the present invention, in the traffic light recognition device according to any one of (1) to (3), after the light emitting region extracted by the extraction unit is expanded, the expanded region An expansion / contraction part that performs a process of inflating again after expansion, and the traffic light determination unit is a candidate area of the traffic light that is identified based on the light-emitting area after processing by the expansion / contraction part And the reference image to determine whether the candidate area is an image area of a traffic light.

  (5) In addition, according to one aspect of the present invention, in the traffic signal recognition device according to any one of (1) to (4), the traffic signal determination unit may determine the size of the image of the candidate area and the reference image. It is characterized by comparing after aligning.

  (6) According to another aspect of the present invention, there is provided a traffic signal recognition method in a traffic signal recognition apparatus including a storage unit that stores a plurality of reference images representing traffic signals. The traffic signal recognition apparatus captures a captured image captured by an imaging unit. A captured image acquisition step to acquire, an extraction step in which the traffic light recognition device extracts a light emission region from the captured image, and a signal recognition device based on the light emission region extracted in the extraction step in the captured image. A traffic light determining step of comparing the image of the candidate area of the specified traffic light with the reference image to determine whether the image of the candidate area is an image of the traffic light; It is a recognition method.

  According to the present invention, it is possible to accurately recognize a traffic light.

It is a figure for demonstrating the outline | summary of the traffic signal recognition system which concerns on the 1st Embodiment of this invention. It is a figure which shows an example of the captured image which concerns on the same embodiment. It is a block diagram which shows schematic function structure of the traffic signal recognition apparatus 10 which concerns on the embodiment, and its peripheral device. It is a figure for demonstrating the production | generation process of the template image which concerns on the same embodiment. It is a figure which shows an example of the setting table of the color range which concerns on the same embodiment. It is a figure for demonstrating the setting of the threshold value of the saturation and the brightness of the imaging region of the light of the traffic light which concerns on the embodiment. It is a figure for demonstrating the expansion / contraction process by the traffic signal recognition apparatus which concerns on the same embodiment. It is a figure for demonstrating the labeling process by the traffic signal recognition apparatus which concerns on the same embodiment. It is a flowchart which shows an example of the operation | movement by the traffic signal recognition apparatus which concerns on the embodiment. It is a flowchart which shows an example of operation | movement of the template image generation apparatus which concerns on the same embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
[Outline of traffic light recognition system]
FIG. 1 is a diagram for explaining an outline of a traffic signal recognition system 1 according to the first embodiment of the present invention.
The traffic signal recognition system 1 according to the present embodiment includes a traffic signal recognition device 10 and an imaging device 30. The traffic signal recognition device 10 and the imaging device 30 are mounted on the vehicle 2. The signal recognition device 10 and the imaging device 30 can transmit and receive information to and from each other wirelessly or by wire.

The imaging device 30 is installed, for example, at the front end portion of the vehicle 2 and can image an environment in front of the vehicle 2. The imaging device 30 has a sufficient angle of view to image the traffic signal device 20 installed on a telephone pole beside the road, for example. Here, the traffic light 20 is a light-emitting sign for various traffic arrangements such as a traffic signal for pedestrians and a traffic signal for vehicles. Below, as an example, the case where the traffic signal 20 is a traffic signal device that includes three substantially circular lighting parts of blue, yellow, and red side by side will be described.
FIG. 2 is a diagram illustrating an example of a captured image according to the present embodiment.
In the example illustrated in FIG. 2, the traffic light 20, the sign 50, and the sky are captured in the captured image PT. Moreover, in the traffic light 20, the blue lighting part 21 is lit among the three lighting parts 21, 22, and 23 of blue, yellow, and red.

  The traffic signal recognition device 10 includes a storage unit 11 that stores a plurality of template images (reference images) representing the traffic signal 20. The template image will be described later. The traffic signal recognition device 10 acquires a captured image captured by the imaging device 30 and extracts a light emitting area from the captured image. The light emitting area is an area of a lighting part that is lit in the traffic light 20 captured in the captured image. Specifically, in the example illustrated in FIG. 2, the light emitting region is a region of the lighting unit 21.

  Next, the traffic signal recognition device 10 specifies a candidate area of the traffic signal 20 based on the extracted light emitting area in the captured image. Specifically, in the example illustrated in FIG. 2, the candidate area of the traffic light 20 is an area representing the traffic light 20 in the captured image. Then, the traffic signal recognition apparatus 10 compares the image of the identified candidate area with the template image, and determines whether the image of the candidate area is the image area of the traffic signal 20 by so-called template matching. As described above, the traffic signal recognition device 10 according to the present embodiment determines, for example, matching between a plurality of template images that differ in the position and direction in which the traffic light 20 is viewed, how the ambient light hits the traffic light 20, and captured images. Therefore, it is possible to accurately recognize the traffic light 20 in the captured image.

[Configuration of traffic light recognition system]
[Configuration of Signal Recognition System 1: Configuration of Imaging Device]
Next, the configuration of the traffic signal recognition device 10 and its peripheral devices will be described.
First, peripheral devices of the traffic signal recognition device 10 will be described.
FIG. 3 is a block diagram showing a schematic functional configuration of the traffic signal recognition device 10 and its peripheral devices according to the present embodiment.
The imaging device 30 includes an imaging unit 31 and outputs a captured image captured by the imaging unit 31 to the traffic signal recognition device 10.
The imaging unit 31 includes, for example, a pan focus camera module. For example, the imaging unit 31 acquires a captured image of the HSV color system.

[Configuration 2: Traffic Signal Recognition System: Configuration of Template Image Generation Device]
The template image generation device 40 includes a template image generation unit 41.
The template image generation unit 41 generates a template image. The template image generated by the template image generation unit 41 is stored in the storage unit 11 of the traffic signal recognition device 10.
Here, generation of a template image by the template image generation unit 41 will be described.
FIG. 4 is a view for explaining template image generation processing according to the present embodiment.
In the present embodiment, a plurality of template images are generated from a large number of sample images. The sample image is an image obtained by capturing the traffic light 20 in various environments. Here, the environment includes the installation position and orientation of the traffic light 20, the incident angle of light, the imaging angle, the date and time, and the like. In addition, the sample image may be an image of the traffic signal 20 of a different model. A sample image for generating the template image described above is prepared in advance. The sample image is prepared according to the color of the light that is lit in the traffic light 20.

  In generating a template image, the template image generating unit 41 classifies (clusters) a large number of sample images according to their characteristics. These many sample images are images having the same vertical and horizontal sizes (number of pixels). Here, as an example, a case where the template image generation unit 41 classifies sample images using the K-means method will be described, but any known method may be used as a clustering method. Here, as an example, a case where the template image generation unit 41 generates five template images will be described. In this case, the template image generation unit 41 classifies the sample images into five types. Here, in template matching, various factors such as the position and direction of viewing the traffic light 20 and how the ambient light strikes the traffic light 20 affect the accuracy. However, the recognition of the traffic light 20 is particularly affected by ambient light. Experience shows that it is big. The way the ambient light strikes the traffic light 20 is, for example, from the back (forward light), from the front (backlight), from the right (right light), from the left (left light), depending on the position of the light emission source, It can be classified into five (top light) from the top. Therefore, in the present embodiment, a case where the template images are classified into five will be described as an example in view of the influence on the matching accuracy of how the ambient light strikes the traffic light 20.

  Next, the template image generation unit 41 generates one template image from the sample image group classified into the same group. FIG. 4 shows a plurality of sample images PS11, PS12,... Classified into the same group. The template image PS2 is an image generated from a plurality of sample images PS11, PS12,. The hue, saturation, and brightness values of each pixel constituting the template image are the same for the pixels at the same position among the pixels constituting the sample image PS11, PS12,. It is the average of hue, saturation, and brightness. That is, the hue value of the upper left pixel PX2 of the template image PS2 is calculated as the average value of the hue value of the upper left pixel PX11 of the sample image PS11, the hue value of the upper left pixel PX12 of the sample image PS12, and so on. Is obtained. In this way, the template image generation unit 41 generates one template image by averaging the sample image groups classified into the same group.

[Configuration of signal recognition system 2: Configuration of signal recognition device]
Next, returning to FIG. 3, the description of the traffic light recognition device 10 will be continued.
The traffic signal recognition device 10 is an electronic device such as an in-vehicle terminal device, for example. The traffic light recognition device 10 may be an electronic device that can be moved from the vehicle 2 such as a personal computer, a mobile phone, a tablet, a smartphone, a PHS (Personal Handy-phone System), or a PDA (Personal Digital Assistant).

The traffic signal recognition apparatus 10 includes a storage unit 11, a captured image acquisition unit 12, a presentation unit 13, an extraction unit 14, and a traffic signal determination unit 15.
The storage unit 11 includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). In addition, the storage unit 11 may include an HDD (Hard Disc Drive), an EEPROM (Electrically Erasable Programmable Read Only Memory), a flash memory, and the like. The storage unit 11 stores various programs to be executed by a CPU (Central Processing Unit, not shown) included in the traffic signal recognition device 10, results of processing executed by the CPU, and the like. Specifically, the storage unit 11 stores, for example, a color range setting table that is referred to in the light emission region extraction processing by the extraction unit 14. The extraction unit 14 and the traffic light determination unit 15 are realized by the CPU executing a program stored in the storage unit 11.
In addition, the storage unit 11 includes a template image storage unit 111 that stores in advance a template image generated by the template image generation device 40. In the present embodiment, the template image storage unit 111 stores five template images for each of the three colors of lights.

The captured image acquisition unit 12 includes a communication IC and communicates with the imaging device 30. A captured image is acquired from the imaging device 30, and captured image information indicating the acquired captured image is output to the extraction unit 14.
The presentation unit 13 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The presentation unit 13 includes, for example, a speaker (audio output device).

The extraction unit 14 performs a process of extracting a pixel group (region) representing light emission from the captured image. That is, a color range extraction unit 141, an expansion / contraction unit 142, a labeling unit 143, and a shape determination unit 144 are provided.
When the captured image information is acquired from the captured image acquisition unit 12, the color range extraction unit 141 acquires the three colors based on the color range setting table that defines the hue, saturation, and brightness range for each color of the lighting from the captured image. Pixels representing light emission are extracted.

Here, pixel extraction processing by the color range extraction unit 141 will be described.
The color range extraction unit 141 performs extraction of pixels representing lighting in two stages. In the first extraction process, the color range extraction unit 141 extracts pixels representing lighting based on the hue of each pixel of the captured image. In the second extraction process, the color range extraction unit 141 narrows down the pixels representing lighting based on the saturation and brightness of each candidate pixel extracted by the first extraction process.
The extracted area has, for example, the same number of pixels in the vertical and horizontal directions as the captured image, and is represented by binary area information in which binary information of “1” or “0” is written in each pixel. . In this case, for example, the value of the pixel included in the extracted area is “1”, and the value of the pixel included in the unextracted area is “0”. That is, each extracted region is configured by a pixel group having a value “1” adjacent in at least one of the vertical direction and the horizontal direction.

First, the first extraction process will be described.
FIG. 5 is a diagram illustrating an example of a color range setting table according to the present embodiment.
The color range setting table according to the present embodiment defines a range of hue, saturation, and lightness according to the color of the light using the HSV color system. Specifically, in the case of a red light, the value of hue H is 0 or more and 10 or less (0 ≦ H ≦ 10), the value of saturation S is greater than 0.3 (S> 0.3), and lightness The value of V is greater than 120 (V> 120). In this embodiment, when the color range is set based on “Study on Road Traffic Signal Recognition Technology from Night Video” (Toshiya Umezawa, Kazuhiko Eguchi, Aichi Institute of Technology Research Report No. 43B 2008) However, the hue, saturation, and lightness ranges may be arbitrarily determined based on separate rules of thumb. The color range extraction unit 141 sequentially extracts pixels representing a blue light, a yellow light, and a red light from the same captured image based on the color range setting table.

Next, the second extraction process will be described.
FIG. 6 is a diagram for explaining setting of the threshold values of the saturation and lightness of the image capturing area of the traffic light 20 according to the present embodiment.
The vertical axis in FIG. 6 represents saturation. The horizontal axis represents the brightness. The plurality of points plotted in FIG. 6 represent the saturation and brightness of the pixel representing the light when the blue light of the traffic light 20 is imaged in various environments. As shown in FIG. 6, when the lighting of the traffic light 20 is imaged, the saturation and brightness of the pixels representing the lighting are distributed substantially linearly (for example, the region R1 in FIG. 6). Therefore, it can be considered that a pixel whose saturation and brightness are not included in the substantially linear region does not represent the lighting of the traffic light 20. In the case of the example illustrated in FIG. 6, the color range extraction unit 141 narrows down pixels that represent lighting based on, for example, the following equation (1) that represents the line L1 (FIG. 6).
minS = 0.73 / 120 × V−0.3 (1)
Here, the saturation S is a value between 0 and 1. Further, the brightness V is a value of 0 or more and 255 or less. minS indicates a lower limit value of the saturation S of a candidate to be extracted as a light emitting region. That is, the color range extraction unit 141 extracts pixels whose saturation S exceeds the lower limit value minS shown in the above expression as pixels representing lighting.

  Thereby, since the color range extraction part 141 can exclude the pixel which does not originate in the light of the traffic light 20, the recognition accuracy of the traffic light 20 by the traffic light recognition apparatus 10 can be improved. However, in the first extraction process and the second extraction process, it is not always possible to exclude all pixels that do not originate from the lighting of the traffic light 20. Specifically, for example, when a blue light pixel is extracted, a blue signboard (for example, the signboard 50 in FIG. 2), a vehicle, a blue sky (for example, the sky 51 in FIG. 2), or the like may be erroneously detected. is there. Therefore, the extraction unit 14 performs processing for further narrowing down the pixels to be extracted. Subsequent pixel extraction processing is performed for each color of light, and these processing is the same for each color. Therefore, in the following, only processing for pixels representing blue lights will be described.

  The expansion / contraction unit 142 performs a contraction process on the light emission region candidates (hereinafter referred to as “light candidate regions”), thereby excluding a minute region composed of about several pixels from the light candidate regions. . Such a minute region may appear innumerably in the background image of the captured image (for example, the sky 51 in FIG. 2). On the other hand, for example, when the amount of ambient light is small, the aperture of the camera of the imaging unit 31 is opened, and so-called halation may occur in which the lightness of the central portion of the light of the traffic light 20 is saturated. In this case, the lamp candidate region is detected as, for example, a region having a lack in the annular central portion. The expansion / contraction part 142 fills in the lack of such a region by performing expansion processing. The expansion / contraction unit 142 performs processing in the order of expansion processing, contraction processing, and expansion processing, and outputs information on the candidate area after processing to the labeling unit 143.

Here, the expansion process by the expansion / contraction part 142 and the contraction process will be described.
FIG. 7 is a diagram for explaining the expansion / contraction process by the traffic signal recognition apparatus 10 according to the embodiment.
In the example shown in FIG. 7A, the expansion / contraction unit 142 performs an expansion process for expanding one lamp for the lamp candidate region SR1 including one pixel. Thereby, the lamp candidate region SR1 becomes a region composed of nine pixels. Next, the expansion / contraction unit 142 performs a contraction process that contracts by two pixels, which is larger than the expansion amount of the previous expansion process. As a result, the lamp candidate region SR1 disappears and is excluded from the lamp candidates. Then, the expansion / contraction unit 142 performs an expansion process for one pixel that is excessively contracted on the remaining lamp candidate region.

  In the example shown in FIG. 7B, the expansion / contraction unit 142 has an expansion of one pixel with respect to the annular lamp candidate region SR2 having one pixel inside and composed of eight pixels. It is carried out. As a result, the lamp candidate region SR2 becomes a region composed of 25 pixels, and the internal omission is filled. Next, the expansion / contraction unit 142 performs a contraction process that contracts by two pixels, which is larger than the expansion amount of the previous expansion process, and performs an expansion process for one pixel that is excessively contracted. As a result, the expansion / contraction part 142 can fill the gap inside the annular lamp candidate region SR2 and make the size of the outer edge the same as that before the expansion / contraction process.

In this way, the expansion / contraction part 142 expands the lamp candidate region, and then contracts the region after the expansion more than before the expansion, and then expands again, thereby supplementing the lack of the region due to halation, It is possible to remove a minute region that may cause processing noise.
Hereinafter, as an example, the case where the expansion / contraction unit 142 performs the expansion process for two pixels, performs the contraction process for three pixels, and then performs the expansion process for one pixel will be described.

  The labeling unit 143 performs labeling on the lamp candidate area. Specifically, as described above, the labeling unit 143 identifies pixels constituting one region based on the continuity in the vertical direction and the horizontal direction of each pixel, and separates the regions. The labeling unit 143 gives (labels) unique identification information (label) to each separated region. The labeling unit 143 outputs label area information representing the candidate lamp areas after labeling to the shape determining unit 144.

Here, processing by the labeling unit 143 will be described.
FIG. 8 is a diagram for explaining labeling processing by the traffic signal recognition apparatus 10 according to the present embodiment.
In the binary area information, the labeling unit 143 gives the same label to an area where the value “1” is continuous in the vertical direction or the horizontal direction. First, when the labeling unit 143 scans from the edge of the image and finds a pixel having a value “1”, the labeling unit 143 gives a new label to the found pixel. Next, if the value of the adjacent pixel is “1”, the labeling unit 143 gives the same label as that of the adjacent pixel. By repeating this, a label is assigned to each block of pixels having a value of “1” to identify each region.

  The shape determination unit 144 extracts a lamp candidate region based on the shape of the lamp candidate region. In the present embodiment, the traffic light 20 has a substantially circular lighting part. Therefore, when the lamp candidate area is not substantially circular, it can be considered that the area does not originate from the lamp of the traffic light 20. The shape determination unit 144 determines the circularity of each region, and when the shape of the region is not substantially circular, excludes the region from the lamp candidate region. Moreover, the shape determination part 144 determines the circularity of each area | region, and when the shape of an area | region is substantially circular, the said area | region is extracted as a lamp candidate area | region. For example, in the example illustrated in FIG. 8, the region labeled “1” and the region labeled “2” are substantially circular, and thus are extracted as lamp candidate regions. On the other hand, the region labeled “3” and the region labeled “4” are not substantially circular, and are excluded from lamp candidates.

Specifically, the shape determination unit 144 determines the circularity of the lamp candidate region based on the following equation (2).
R = 4πM / L ² (2)
Here, the circularity R, the area M, and the peripheral length L are positive values. Further, the circumference ratio π is approximately “3.14”. The area M is the number of pixels constituting each region, for example. The perimeter L is, for example, the number of pixels located at the outer edge of each region. The shape determination unit 144 determines that the shape of the region is circular when the circularity R is equal to or greater than a predetermined threshold (for example, 0.7). In addition, the shape determining unit 144 determines that the shape of the region is not circular when the circularity R is less than a predetermined threshold (for example, 0.7). The shape determination unit 144 outputs the extracted label area information of the candidate lamp region to the traffic light determination unit 15.

  The traffic light determination unit 15 identifies the area of the traffic light 20 based on the acquired label area information. The casing of the traffic light 20 and the lighting unit are in a predetermined positional relationship. Therefore, the traffic light 20 estimates the area where the traffic light 20 is imaged from the color type of the light and the size of the light emitting area. Thereby, the traffic light determination unit 15 specifies an area (hereinafter referred to as “traffic light area”) in which the traffic light 20 is estimated to be captured in the captured image.

  Next, the traffic signal determination unit 15 compares the image of the specified traffic signal area with the template image. The traffic light determination unit 15 first reads a light color template image corresponding to the image of the traffic signal region to be compared from the template image storage unit 111. Next, the traffic signal determination unit 15 performs template matching between the image of the specified traffic signal area and the template image. Any known method may be used as the template matching method. Here, it is empirically known that the template matching is less accurate when the size of the comparison object is different. For example, the lighting part of the traffic light 20 includes an LED (light emitting diode) type and a light bulb type. In the case of the LED type, the image is brightly captured up to the outer edge of the lighting part, but in the case of the bulb type, the amount of light at the outer edge is reduced, so the light of the bulb type lighting part is detected smaller than in the case of the LED type. May be. In this case, the traffic signal area cannot be properly specified, and a difference occurs between the template image and the image size, and the matching accuracy may be reduced. Therefore, the traffic light determination unit 15 sets the size of the template image to “0.8” times, “0.9” times, “1.0” times, “ The template matching is performed a plurality of times by changing the value to 1.1 times or 1.2 times. And let the highest score in this multiple times of template matching be a matching score of the combination.

  The traffic signal determination unit 15 performs template matching with a plurality of template images for one traffic signal area image. Specifically, in this embodiment, template matching with five template images is performed. And when the highest matching score in all the performed template matching is more than a predetermined threshold value, it determines with the signal apparatus 20 being imaged by the signal apparatus area | region image. At this time, since the traffic signal area image is extracted for each color of the light, the color of the light, that is, the instruction content of the signal is recognized simultaneously. The traffic signal determination unit 15 notifies the user of predetermined information corresponding to the recognition result of the traffic signal 20 through display by the presentation unit 13, voice output, and the like. Specifically, the presentation unit 13 notifies the user of the presence, position, lighting color, signal instruction content, and the like of the traffic light 20.

[Operation of signal recognition device]
Next, the operation of the traffic signal recognition device 10 will be described.
FIG. 9 is a flowchart showing an example of the operation of the traffic signal recognition apparatus 10 according to the present embodiment.
(Step S <b> 101) The captured image acquisition unit 12 of the traffic signal recognition device 10 acquires a captured image captured by the imaging device 30. The captured image acquisition unit 12 outputs captured image information representing the acquired captured image to the color range extraction unit 141. Thereafter, the traffic signal recognition apparatus 10 advances the processing to step S102.

(Step S102) The color range extraction unit 141 of the traffic light recognition apparatus 10 extracts a lamp candidate region from the captured image using the color range setting table based on the hue of each pixel of the captured image. In addition, the color range extraction unit 141 narrows down the lamp candidate areas based on the saturation and the brightness from the extracted lamp candidate areas, for example, using Expression (1). The color range extraction unit 141 outputs binary region information representing the extracted lamp candidate region to the expansion / contraction unit 142. Thereafter, the traffic signal recognition apparatus 10 advances the processing to step S103.

(Step S103) When the expansion / contraction unit 142 of the traffic light recognition apparatus 10 acquires the binary region information from the color range extraction unit 141, the expansion / contraction unit 142 performs expansion processing, and fills in the lack of a region having a lack in the light candidate region. . Next, the expansion / contraction unit 142 contracts the region by an amount larger than the expansion amount of the expansion process, and excludes the minute region from the lamp candidates. Next, the expansion / contraction unit 142 performs an expansion process, and returns the size of the lamp candidate region to a level before the series of expansion / contraction processes. Next, the expansion / contraction unit 142 outputs binary region information representing each region after processing to the labeling unit 143. Thereafter, the traffic signal recognition apparatus 10 advances the processing to step S104.

(Step S104) When the labeling unit 143 of the traffic signal recognition apparatus 10 acquires the binary region information from the expansion / contraction unit 142, it separates each lamp candidate region and performs labeling. The labeling unit 143 outputs label area information representing each separated lamp candidate area to the shape determining unit 144. Thereafter, the traffic signal recognition apparatus 10 advances the processing to step S105.
(Step S <b> 105) When the shape determination unit 144 of the traffic signal recognition apparatus 10 acquires the label region information from the labeling unit 143, the shape determination unit 144 determines whether the shape of each lamp candidate region is substantially circular. If the shape of the lamp candidate area is not substantially circular, the shape determination unit 144 excludes the area from the lamp candidates. Based on the shape determination result, the shape determination unit 144 outputs label area information representing the extracted lamp candidate region to the traffic light determination unit 15. Thereafter, the traffic signal recognition apparatus 10 advances the processing to step S106.

(Step S106) When the traffic signal determination unit 15 of the traffic signal recognition device 10 acquires the label region information from the labeling unit 143, the traffic signal determination unit 15 specifies a region where the traffic signal 20 is captured based on the lamp candidate region, and acquires a traffic signal region image. Next, the traffic light determination unit 15 reads a plurality of template images for each lamp color from the template image storage unit 111. Next, the traffic signal determination unit 15 compares the plurality of read template images with the traffic signal region image by template matching. At this time, the traffic light determination unit 15 performs matching a plurality of times, for example, by changing the size of the template image to a predetermined magnification, that is, aligning the sizes of images to be matched. The traffic signal determination unit 15 outputs predetermined information corresponding to the determination result of each traffic signal area to the presentation unit 13. Thereafter, the traffic signal recognition apparatus 10 advances the processing to step S107.

(Step S107) The presentation unit 13 of the traffic signal recognition device 10 performs display, audio output, and the like based on the information acquired from the traffic signal determination unit 15, and notifies the user of the recognition result of the traffic signal 20, and the like. And the traffic signal recognition apparatus 10 complete | finishes the process shown in FIG.

[Operation of signal recognition device]
Hereinafter, the operation of the template image generation device 40 will be described.
FIG. 10 is a flowchart showing an example of the operation of the template image generating apparatus 40 according to the present embodiment.
(Step S201) The template image generation unit 41 of the template image generation device 40 acquires, for example, a large number of sample images stored in advance in the storage unit of the own device. Thereafter, the template image generating apparatus 40 proceeds with the process to step S202.

(Step S202) The template image generation unit 41 classifies the acquired many sample images into a predetermined number of groups by, for example, the K-means method. Thereafter, the template image generating unit 41 advances the processing to step S203.
(Step S203) The template image generation unit 41 calculates an average value of the hue, brightness, and saturation of each pixel from the grouped sample images, and generates a template image. Then, the template image generating apparatus 40 ends the process shown in FIG.

[Summary of First Embodiment]
As described above, the traffic signal recognition apparatus 10 according to the present embodiment acquires the template image storage unit 111 that stores a plurality of reference images (for example, template images) representing a traffic signal, and the captured image captured by the imaging unit 31. When a luminescent region is extracted from the captured image acquisition unit 12 and the captured image, an image of a candidate region of the traffic signal 20 identified based on the luminescent region extracted by the extraction unit 14 among the captured images (for example, a traffic signal region image) And a traffic light determination unit 15 that compares the reference image with the reference image to determine whether the image of the candidate area is an image of the traffic light 20.
Thereby, since the traffic signal recognition apparatus 10 performs template matching between the traffic signal area image and a plurality of template images, the traffic signal recognition device 10 recognizes the traffic signal 20 with higher accuracy than when performing template matching with one template image. Can do.

In addition, the plurality of reference images according to the present embodiment are images obtained based on the plurality of images of the traffic light 20 captured in different environments.
Thereby, the signal apparatus 20 can recognize the signal apparatus 20 imaged in various environments with high accuracy.

In addition, the color range extraction unit 141 of the extraction unit 14 extracts the light emitting region based on a relationship between parameters (for example, saturation and brightness) representing colors for each lighting color of the traffic light 20.
Thereby, the traffic light 20 can exclude the area | region which does not originate in the traffic light 20 from the extraction object from the area | region which belongs to the color range similar to a light color in a captured image. Accordingly, the traffic signal recognition apparatus 10 can accurately recognize the traffic signal 20 and can reduce the number of determination target areas, thereby reducing the processing load.

The expansion / contraction unit 142 performs a process of expanding the light emitting region extracted by the color range extraction unit 141, then contracting the expanded region more than before expansion, and then expanding it again.
Thereby, the expansion / contraction part 142 can suppress the omission of the area | region by halation, and can exclude the micro area | region which does not originate in the lighting of the traffic light 20 from extraction object. Therefore, the traffic light 20 can recognize the traffic light 20 with high accuracy and can reduce the number of determination target areas, thereby reducing the processing load.

Further, the traffic light determination unit 15 compares the traffic signal area image with the reference image after aligning the sizes thereof.
Thereby, since the traffic signal recognition apparatus 10 can prevent misrecognition of template matching based on the difference in image size, the traffic signal 20 can be recognized with high accuracy.

[Modification]
[First Modification: Pedestrian Traffic Light Recognition Device]
In the first modification, the traffic light recognition device 10 is an electronic device including an imaging unit such as a mobile phone, a tablet, a smartphone, a PHS, or a PDA that can be carried by a pedestrian. In using the traffic light 20, the user moves the imaging unit toward the traveling direction of the user. The traffic signal recognition device 10 recognizes a traffic signal from the image captured by the imaging unit of the device itself, and notifies the user of the presence or absence of the traffic signal 20, the lighting state thereof, and the like.
Thereby, even if it is a case where it is difficult to visually recognize the traffic light 20 for reasons such as weak visual acuity, the user can recognize the presence / absence of the traffic light 20, the lighting state, and the like.

[Second Modification: Selection of Reference Image According to Movement State]
In the second modification, the traffic light recognition apparatus 10 selects a reference image according to the movement state of the user. When the user, that is, the vehicle 2 is stopped, the user may be stopped by a red light. Therefore, the traffic signal recognition device 10 preferentially performs template matching corresponding to a blue light over a yellow or red light. Thereby, the user does not notice that the red signal is switched to the green signal, and can prevent other vehicles from being obstructed. On the other hand, when the user, that is, the vehicle 2 is moving, it is important in ensuring safety that the yellow or red stop signal is not overlooked. Therefore, the traffic light recognition device 10 preferentially performs template matching corresponding to yellow and red lights rather than blue lights. Thereby, the user can move safely. In the second modified example, the movement state of the user may be acquired from the vehicle 2, for example, or may be acquired by an acceleration sensor built in the traffic signal recognition device 10.

In each embodiment described above, the number of template images may be six or more for each color of lighting, or may be four or less. The traffic signal recognition device 10 can reflect the imaging angle of the traffic signal 20 and the like in the matching as the number of comparison target template images is increased. Therefore, the accuracy of template matching can be increased. However, if the number of template images to be compared increases, the processing load on the traffic signal recognition device 10 increases. Therefore, in the present embodiment, the case where five template images are prepared for each lighting color has been described.
Moreover, in each embodiment mentioned above, the shape of the lighting part of the traffic light 20 may not be circular. For example, in Japan, a traffic signal for pedestrians generally has two approximately square-shaped lighting parts of blue and red. In this case, the shape determination unit 144 determines whether or not the candidate region is substantially square based on, for example, the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the candidate region. As described above, the shape determination unit 144 may change the shape determination method according to the shape of the lighting unit of the signal device 20 to be determined.

  Moreover, in each embodiment mentioned above, the traffic signal 20 is not restricted to the traffic signal 20 mentioned above. The traffic light 20 may be, for example, a traffic light that informs the presence or absence of road construction, or a traffic light that informs the availability of a parking lot. As described above, the traffic light 20 may be a luminescent sign installed at an arbitrary place for an arbitrary purpose.

Moreover, in each embodiment mentioned above, the traffic signal recognition apparatus 10 may be used for an automatic driving control apparatus, for example, and may recognize the traffic signal 20 on the course of the vehicle 2 carrying the said automatic driving control apparatus.
In each embodiment described above, the color of the captured image may be expressed in a format other than the HSV color system. Specifically, for example, a color system such as RGB, CMYK, or HLS may be used. In this case, the extraction of the lamp candidate region using the parameters representing the hue, saturation, and lightness colors described above is performed based on the parameters in each color system.

  In addition, a part of the traffic signal recognition device 10, the imaging device 30, and the template image generation device 40 in each embodiment described above, for example, the storage unit 11, the captured image acquisition unit 12, the presentation unit 13, the extraction unit 14, and the traffic signal determination unit. 15. The template image generation unit 41 and the like may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. The “computer system” here is a computer system built in the traffic signal recognition device 10, the imaging device 30, and the template image generation device 40, and includes hardware such as an OS (Operating System) and peripheral devices. Shall be included.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In this case, a volatile memory inside a computer system that serves as a server or a client may be included that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  Moreover, you may implement | achieve part or all of the traffic signal recognition apparatus 10, the imaging device 30, and the template image generation apparatus 40 in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional unit of the traffic signal recognition device 10, the imaging device 30, and the template image generation device 40 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

  DESCRIPTION OF SYMBOLS 1 ... Traffic light recognition system, 2 ... Vehicle, 10 ... Traffic light recognition apparatus, 11 ... Memory | storage part, 111 ... Template image memory | storage part, 12 ... Captured image acquisition part, 13 ... Presentation part, 14 ... Extraction part, 141 ... Color range extraction , 142, expansion / contraction part, 143, labeling part, 144, shape determination part, 15 ... traffic light determination part, 20 ... traffic light, 30 ... imaging device, 31 ... imaging part, 40 ... template image generation device, 41 ... template image Generator

Claims (6)

A storage unit for storing a plurality of reference images representing traffic lights;
A captured image acquisition unit that acquires a captured image captured by the imaging unit;
An extraction unit for extracting a light emitting region from the captured image;
The image of the candidate area of the traffic light identified based on the light emitting area extracted by the extraction unit in the captured image is compared with the reference image, and whether the image of the candidate area is the image of the traffic light A traffic light determination unit for determining whether or not
A traffic light recognizing device comprising: The signal recognition apparatus according to claim 1, wherein the plurality of reference images are images obtained based on a plurality of images of a traffic signal captured in different environments. The traffic light recognition apparatus according to claim 1, wherein the extraction unit extracts the light emitting area based on a relationship between parameters representing colors for each lighting color of the traffic light. An expansion / contraction unit that performs a process of expanding the light emitting region extracted by the extraction unit, and then expanding the light-emitting region after the expansion after expanding the region after expansion.
Further comprising
The traffic light determination unit compares the candidate area of the traffic signal identified based on the light emitting area after processing by the expansion / contraction part with the reference image, and determines whether the candidate area is an image area of the traffic signal. The traffic light recognition device according to any one of claims 1 to 3, wherein: 5. The traffic signal recognizing device according to claim 1, wherein the traffic signal determination unit compares the candidate area image with the reference image after aligning the sizes thereof. A traffic signal recognition method in a traffic signal recognition apparatus including a storage unit that stores a plurality of reference images representing traffic signals,
A captured image acquisition step in which the traffic light recognition device acquires a captured image captured by the imaging unit;
An extraction step in which the signal recognition device extracts a light emitting region from the captured image;
The signal recognition device compares the image of the candidate area of the traffic signal identified based on the light emitting area extracted in the extraction step of the captured image with the reference image, and the image of the candidate area is A traffic light determining step for determining whether or not the image is,
A traffic light recognition method comprising:

Images