JP7363279B2 - Image recording device, image recording method, and image recording program - Google Patents

Description

The present invention relates to an image recording technique for recording images taken with a camera.

Vehicles are now equipped with drive recorders to capture and record images of the front of the vehicle while driving, which can be used to help understand the situation at the scene in the event of an accident. Since the storage memory of the drive recorder has a limited capacity, it is necessary to utilize the storage capacity effectively.

Patent Document 1 discloses a drive recorder that detects abnormal approach of a vehicle using information from a magnetic sensor that detects magnetism around the vehicle, and increases the frame rate of image data recorded in a recording memory.

JP2007-88541A

In order to secure recording time given the small storage capacity of the drive recorder's memory, it is necessary to set the frame rate low when recording captured images, but the visibility of recorded images deteriorates in stormy weather. This may lead to a lack of information about dangerous scenes.

The drive recorder described in Patent Document 1 cannot cope with bad weather such as heavy rain and keep the frame rate at an appropriate level during recording.

The present invention has been made in view of these circumstances, and its purpose is to provide an image recording technique for recording images taken with a camera at an appropriate frame rate depending on the weather.

In order to solve the above problems, an image recording device (200) according to an aspect of the present invention includes an imaging unit (10) that captures an image at an imaging frame rate, and a camera that uses the captured image to detect whether the weather is rough or not. a stormy weather determination processing unit (50) that determines whether or not the weather is stormy; In this case, a frame rate switching unit (30) that switches the recording frame rate to a second frame rate higher than the first frame rate, and a recording unit (60) that records the captured image at the recording frame rate. ).

Another aspect of the invention is an image recording method. This method includes an imaging step of capturing an image at an imaging frame rate, a stormy weather determination processing step of determining whether or not it is stormy weather using the captured image, and a step of recording the captured image. Setting the recording frame rate to a first frame rate lower than the imaging frame rate, and switching the recording frame rate to a second frame rate higher than the first frame rate when it is determined that the weather is stormy. The method includes a frame rate switching step and a recording step of recording the captured image at the recording frame rate.

Note that arbitrary combinations of the above-mentioned components and expressions of the present invention converted between methods, devices, systems, recording media, computer programs, etc. are also effective as aspects of the present invention.

According to the present invention, images taken with a camera can be recorded at an appropriate frame rate depending on the weather.

FIG. 1(a) is a configuration diagram of an image recording apparatus according to the underlying technology, and FIG. 1(b) is a configuration diagram of an image recording apparatus according to an embodiment of the present invention. 3 is a flowchart illustrating an image recording method by the image recording device according to the embodiment. FIG. 3 is a configuration diagram of a stormy weather determination processing section of the image recording apparatus according to the embodiment. FIG. 3 is a diagram illustrating a method for estimating a wiper operation cycle by a wiper operation cycle estimation unit of the image recording apparatus according to the embodiment. FIG. 3 is a diagram illustrating a method for estimating a wiper operation cycle by a wiper operation cycle estimation unit of the image recording apparatus according to the embodiment. 7 is a flowchart illustrating a stormy weather determination process performed by a stormy weather determination processing section of the image recording apparatus according to the embodiment. FIG. 3 is a diagram illustrating a method of specifying a specific period for increasing the recording frame rate.

FIG. 1(a) is a block diagram of an image recording apparatus 100 according to the base technology, and FIG. 1(b) is a block diagram of an image recording apparatus 200 according to an embodiment of the present invention. The image recording devices 100 and 200 are, for example, vehicle-mounted recording devices (drive recorders), but are not limited to this.

First, the configuration of an image recording apparatus 100 according to the underlying technology will be described with reference to FIG. 1(a). The image recording device 100 includes an imaging section 10, an image compression section 20, a frame rate switching section 30, a manual selection section 40, and a recording section 60.

The imaging unit 10 is installed in a vehicle, photographs the outside world in front of the vehicle at a predetermined frame rate, and supplies frames of the photographed video to the image compression unit 20. The imaging unit 10 may be installed so as to be able to photograph the range in which a wiper provided on the windshield of a vehicle operates. In other words, the imaging unit 10 may be installed to be able to photograph most of the windshield. The imaging unit 10 captures an image by focusing light from a subject on an image sensor (not shown). An example of the imaging unit 10 is a CCD (Charge Coupled Device) monocular camera, but the camera is not limited to this.

The frame rate of video captured by the imaging unit 10 is referred to as an "imaging frame rate." The imaging frame rate is, for example, 60 frames/second, 30 frames/second, 20 frames/second, 10 frames/second, or the like.

The image compression unit 20 encodes the frame to be encoded of the video using various encoding parameters according to standards such as MPEG, AVC, HEVC, etc., and supplies the encoded frame to the frame rate switching unit 30. Note that the image recording device 100 does not need to include the image compression section 20. In that case, the frames of the captured video captured by the imaging unit 10 may be supplied to the frame rate switching unit 30 without being encoded.

The manual selection section 40 allows the user to manually select a frame rate (referred to as a "recording frame rate") at which the image supplied from the image compression section 20 is recorded in the recording section 60. The manual selection section 40 instructs the selected recording frame rate to the frame rate switching section 30.

The recording frame rate is set to be lower than the imaging frame rate. When the imaging frame rate is 30 frames/second, the recording frame rate is set to, for example, 30 frames/second, 20 frames/second, or 10 frames/second. For example, the user may select a low frame rate such as 10 frames/second in good weather such as sunny or cloudy, and select a high frame rate such as 30 frames/second in case of stormy weather such as wind and rain, heavy rain, or snowfall. do. In addition to the weather, the user selects the frame rate based on, for example, the remaining storage capacity of the recording unit 60.

The frame rate switching unit 30 switches the recording frame rate according to instructions from the manual selection unit 40.

The recording unit 60 records the image supplied from the image compression unit 20 at the recording frame rate set by the frame rate switching unit 30. The recording unit 60 is, for example, a storage medium such as a memory card, but is not limited to this.

When the recording frame rate is lower than the imaging frame rate, the recording unit 60 thins out the captured frames and records them in the recording unit 60. For example, if the imaging frame rate is 30 frames/second and the recording frame rate is 10 frames/second, two of the three captured frames will be thinned out, and one of the three frames will be thinned out. By recording frames, the recording frame rate is reduced by 1/3.

The configuration of an image recording apparatus 200 according to an embodiment of the present invention will be described with reference to FIG. 1(b). The image recording device 200 includes an imaging section 10, an image compression section 20, a frame rate switching section 30, a stormy weather determination processing section 50, and a recording section 60. Description of the configuration and operation common to the image recording device 100 of the base technology shown in FIG. 1A will be omitted, and only the configuration and operation that are different from the image recording device 100 of the base technology will be described.

The stormy weather determination processing unit 50 acquires a photographed image from the imaging unit 10, determines whether or not it is stormy weather, and notifies the frame rate switching unit 30 of the determination result.

In the initial setting, the frame rate switching unit 30 sets the recording frame rate to a first frame rate lower than the imaging frame rate, and according to the determination result supplied from the stormy weather determination processing unit 50, when the weather is stormy, the recording frame rate is set to a first frame rate lower than the imaging frame rate. The frame rate is switched to a second frame rate higher than the first frame rate. The second frame rate may be an imaging frame rate.

For example, if the imaging frame rate is 30 frames/second, the recording frame rate is initially set to 10 frames/second as the first frame rate, and if it is determined that the weather is stormy, the recording frame rate is , the second frame rate is switched to 30 frames/second.

FIG. 2 is a flowchart illustrating an image recording method by the image recording apparatus 200 according to the embodiment.

The imaging unit 10 photographs an image in front of the vehicle at an imaging frame rate (S10). The frame rate switching unit 30 sets the recording frame rate to a low frame rate below the imaging frame rate (S20). Note that in the initial settings, the recording frame rate is set to a low frame rate.

The stormy weather determination processing unit 50 determines whether or not it is stormy weather using the photographed image (S30).

If the weather is stormy (Yes in S30), the frame rate switching unit 30 switches the recording frame rate from the low frame rate to the high frame rate (S40). If the weather is not stormy (No in S30), the process skips step S40 and proceeds to step S50. Incidentally, if the recording frame rate is a high frame rate in No in step S30, the frame rate switching unit 30 may switch the recording frame rate from the high frame rate to the low frame rate before proceeding to step S50.

The recording unit 60 records the captured image at the recording frame rate set by the frame rate switching unit 30 (S50). The photographed image is recorded at a low frame rate when the weather is not stormy, and at a high frame rate when the weather is stormy.

FIG. 3 is a configuration diagram of the stormy weather determination processing section 50 of FIG. 1(b). The stormy weather determination processing section 50 includes a wiper detection section 52 , a storage section 54 , a wiper operation cycle estimation section 56 , and a stormy weather determination section 58 .

The wiper detection unit 52 detects a wiper from the captured image acquired from the imaging unit 10, and extracts a rectangular area surrounding the wiper as a wiper area. The wiper area does not have to be rectangular. As for the method of detecting a specific object area such as a wiper area, an existing method of extracting a specific object from an image based on its features can be used. As a method for extracting objects, for example, an image recognition chip equipped with a deep learning model trained on images of various wipers as a data set can be used. The deep learning model can be realized using object detection algorithms such as SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once).

The wiper detection unit 52 supplies current wiper area data to the wiper operation cycle estimation unit 56 and stores the wiper area data in the storage unit 54. The storage unit 54 stores a plurality of past wiper area data. For example, wiper area data for the past 10 frames is stored. The storage unit 54 supplies past wiper area data to the wiper operation cycle estimation unit 56.

The wiper operating cycle estimating unit 56 estimates the wiper operating cycle by referring to the current wiper area data supplied from the wiper detecting unit 52 and a plurality of past wiper area data supplied from the storage unit 54. Specifically, the wiper operation cycle is estimated based on the point in time when the percentage of the wiper area included in the determination area of the photographed image becomes maximum. The wiper operating cycle estimating unit 56 supplies wiper operating cycle data to the stormy weather determining unit 58.

The stormy weather determination unit 58 determines that it is stormy weather when the wiper operation cycle is less than a predetermined threshold, and determines that it is not stormy weather when the wiper operation cycle is greater than or equal to the predetermined threshold, and uses the stormy weather determination result as a frame rate. It is supplied to the switching section 30. For example, if the predetermined threshold value is 2 seconds and the wiper operation cycle is less than 2 seconds, it is determined that the weather is stormy. The stormy weather determination result is "1" if the weather is stormy, and "0" if it is not stormy.

4 and 5 are diagrams illustrating a method of estimating the wiper operating cycle by the wiper operating cycle estimation unit 56.

In FIG. 4, past frames F8 to F1 and the current frame F0 of the photographed image are shown with the horizontal axis as the time axis, and an example of the reciprocating movement of the wiper is shown here. The reciprocating motion of the wiper includes a one-way "outward" motion from frames F8 to F4 and a one-way "return" motion from frames F4 to F0. A determination area 220 is set in the center of the photographed image 230. The wiper detection unit 52 extracts the wiper area 210 from each captured image 230 in each frame. Here, the time when the wiper is making a "forward" reciprocating motion is defined as the wiper going time, and the time when the wiper is making a "returning" reciprocating motion is defined as the wiper returning time.

In each frame, if the wiper region 210 is included in the determination region 220, the wiper operation cycle estimation unit 56 stores a determination result of "1", and if the wiper region 210 protrudes from the determination region 220, the determination result is "1". 0" is stored. More specifically, the determination result is set to "1" in a frame in which the proportion of the wiper region 210 included in the determination region 220 is maximum, and the determination result is set to "0" in other frames. For example, the determination result may be set to "1" in a frame in which the percentage of the wiper region 210 included in the determination region 220 is 100%, that is, the wiper region 210 is completely included. Among the frames F8 to F4 or frames F4 to F0 of the one-way wiper operation, the determination result of the frame in which the ratio of the wiper region 210 included in the determination region 220 is the highest may be set to "1".

Here, in frame F6, the wiper area 210 is completely included in the determination area 220, so the determination result is "1", and in frame F2, the wiper area 210 is completely included in the determination area 220, so the determination result is "1". 1" is the determination result. In other frames, the wiper area 210 protrudes from the determination area 220, so the determination result is "0". In frame F6, the wiper is in a "going" reciprocating motion, and in frame F2, the wiper is in a "returning" reciprocating motion, and arrows indicating this are illustrated for convenience.

In FIG. 5, the reciprocating operation of the wiper is shown in three patterns: clear weather, light rain, and heavy rain. In the case of clear weather, the determination result explained with reference to FIG. 4 is always "0", and the wiper operation cycle estimation unit 56 sets the maximum value as the initial value for the wiper operation cycle. The maximum value is a number that is sufficiently larger than a predetermined threshold value by which the stormy weather determination unit 58 determines whether or not it is stormy weather. For example, if the predetermined threshold value is 2 seconds, the maximum value is 30 seconds.

In the case of light rain, the wipers operate intermittently. The wiper operation cycle estimating unit 56 calculates the period of operation based on the time difference from the time when the determination result in FIG. 4 when the wiper returns is "1" to the time when the determination result in FIG. 4 when the wiper returns to the next time is "1" in the past frame. , estimate the wiper operating period. For example, the wiper operation cycle can be determined by multiplying the number of frames from the frame where the determination result when the wiper returns is "1" to the next frame where the determination result when the wiper returns is "1" by the reciprocal of the imaging frame rate. presume. In this example, the wiper operating period is estimated to be 3 seconds.

In the case of heavy rain, the wipers move back and forth at high speed. Similarly, the wiper operation cycle estimating unit 56 calculates the period from the time when the determination result of FIG. 4 when the wiper returns is "1" to the time when the determination result of FIG. 4 when the wiper returns is "1" in the past frame. The wiper operating cycle is estimated based on the time difference. In this example, the wiper operating period is estimated to be 1.5 seconds.

The stormy weather determination unit 58 determines that it is stormy weather when the wiper operation cycle is less than 2 seconds. In the case of light rain in Figure 5, the wiper operation cycle is 3 seconds, so the judgment result "0" is output indicating that it is not stormy weather, and in the case of heavy rain, the wiper operation cycle is 1.5 seconds, so it is determined that it is stormy weather. Outputs the determination result "1".

FIG. 6 is a flowchart illustrating the stormy weather determination process performed by the stormy weather determination processing section 50.

The stormy weather determination processing section 50 acquires the photographed image 230 from the imaging section 10 (S60). The wiper detection unit 52 extracts the wiper region 210 from the captured image 230, supplies it to the wiper operation cycle estimation unit 56, and stores it in the storage unit 54 (S70).

The wiper operation cycle estimating unit 56 determines whether the wiper operation period is next within the determination area 220 from the point in time when the determination result at the time of wiper return is "1" at which the ratio of the wiper area 210 being included in the determination area 220 in the past frame is the maximum. The wiper operation cycle is estimated based on the time difference until the time when the determination result at the time of wiper return is "1" at which the percentage of area 210 included is maximum (S80).

The stormy weather determination unit 58 compares the wiper operation cycle with a predetermined threshold (here, 2 seconds) (S90), and if the wiper operation cycle is less than 2 seconds (Yes at S90), it determines that the weather is stormy (S100). ), if the wiper operation cycle is 2 seconds or more (No in S90), it is determined that there is no stormy weather (S110).

Up to now, the stormy weather determination unit 58 has determined whether or not there is stormy weather, but it may determine three or more weather conditions. For example, the stormy weather determination unit 58 compares the wiper operation cycle with a plurality of predetermined threshold values to determine three or more weather conditions. The stormy weather determination unit 58 sets the first threshold to 2 seconds and the second threshold to 1 second, and determines that if the wiper operation cycle is 2 seconds or more, it is not rainy weather, and if the wiper operation cycle is 1 second or more and less than 2 seconds. If it is raining and the wiper operation cycle is less than 1 second, it is determined to be stormy weather. The frame rate switching unit 30 sets the recording frame rate to a high frame rate when it is determined that the weather is stormy, sets the recording frame rate to a medium frame rate when it is determined that it is raining, and sets the recording frame rate to a medium frame rate when it is determined that it is not raining. Set the rate to a low frame rate.

In the above embodiment, when it is determined that the weather is stormy, the recording frame rate is switched from the normal low frame rate to the high frame rate, but in another embodiment, when it is determined that the weather is stormy , specify a specific period during which the recording frame rate is increased during the wiper's reciprocating motion, and switch the recording frame rate to the high frame rate during that specific period, but you can record at the normal low frame rate during other periods. .

FIG. 7 is a diagram illustrating a method of specifying a specific period for increasing the recording frame rate. Similar to FIG. 4, an example of the reciprocating movement of the wiper is shown in frames F8 to F0 of the photographed image. A specific period determination area 240 is set in the center of the photographed image 230. The specific period determination area 240 is, for example, a wider area in the horizontal direction than the determination area 220. If it is determined that the weather is stormy and the wiper area 210 is included within the specific period determination area 240 of the photographed image 230, the recording frame rate is switched to a high frame rate and the wiper area 210 protrudes from the specific period determination area 240. If so, leave it at the normal low frame rate.

In this example, during the reciprocating motion of the wiper, the wiper region 210 is included in the specific period determination region 240 in frames F7, F6, and F5, so the frame rate is switched to a high frame rate during the periods of frames F7, F6, and F5. Furthermore, when the wiper reciprocates back and forth, the wiper area 210 is included in the specific period determination area 240 in frames F3, F2, and F1, so the frame rate is switched to a high frame rate during the periods of frames F3, F2, and F1. Note that during the periods of frames F8, F4, and F0, the normal low frame rate remains.

More specifically, when the stormy weather determination processing unit 50 determines that the weather is stormy, it determines that the period in which the wiper area 210 is included in the specific period determination area 240 of the photographed image 230 exceeds a predetermined threshold value as the specific period. do. When determining that the weather is stormy, the frame rate switching unit 30 switches the recording frame rate from a normal low frame rate to a high frame rate during a specific period. The predetermined threshold value is, for example, 80%. As a result, during stormy weather, images are recorded at a high frame rate during periods when raindrops and snow are removed from the windshield by the windshield wipers and visibility increases, and during other periods, images are recorded at a low frame rate, thereby improving the visibility of recorded images. It is possible to suppress the increase in recording capacity during stormy weather while increasing the storage capacity.

The various processes of the image recording device 200 described above can of course be realized as a device using hardware such as a CPU and memory, or can be stored in a ROM (read-only memory), flash memory, etc. It can also be realized by firmware, computer software, etc. The firmware program and software program may be recorded on a computer-readable recording medium and provided, or sent and received with a server through a wired or wireless network, or sent and received as data broadcasting on terrestrial or satellite digital broadcasting. is also possible.

As described above, according to the embodiment of the present invention, images taken by a camera are normally recorded at a low frame rate, and when stormy weather such as heavy rain is detected, the frame rate is automatically switched to a high frame rate. and record it. This makes it possible to minimize the recording capacity and prevent insufficient information about dangerous scenes. In addition, even if stormy weather is detected, instead of switching to a high frame rate all the time, the frame rate is switched to a high frame rate only during the period when raindrops are removed by the wipers and visibility increases, and the frame rate is kept at a low frame rate during other periods. By leaving it as is, it is possible to suppress the increase in storage capacity during stormy weather while increasing the visibility of recorded images.

The present invention has been described above based on the embodiments. Those skilled in the art will understand that the embodiments are merely illustrative, and that various modifications can be made to the combinations of their components and processing processes, and that such modifications are also within the scope of the present invention. .

In the above embodiment, the stormy weather determination processing unit 50 determines whether or not it is stormy weather based on the wiper operation cycle, but it also determines whether it is stormy weather based on vehicle information obtained from a CAN (Control Area Network), image analysis, various sensor information, etc. It may also be determined whether the weather is stormy or not. Further, the image recording device 200 may include both the manual selection section 40 and the stormy weather determination processing section 50.

10 imaging section, 20 image compression section, 30 frame rate switching section, 40 manual selection section, 50 stormy weather determination processing section, 52 wiper detection section, 54 storage section, 56 wiper operation cycle estimation section, 58 stormy weather determination section, 60 recording section , 100 image recording device, 200 image recording device, 210 wiper area, 220 determination area, 230 photographed image, 240 specific period determination area.

Claims (5)

an imaging unit that captures an image at an imaging frame rate;
a stormy weather determination processing unit that determines whether or not it is stormy weather using the captured image;
The recording frame rate when recording the captured image is set to a first frame rate lower than the imaging frame rate, and when it is determined that the weather is stormy, the recording frame rate is set to the first frame rate. a frame rate switching unit that switches to a second frame rate higher than the second frame rate;
a recording unit that records the captured image at the recording frame rate ,
The stormy weather determination processing unit specifies a specific period for increasing the recording frame rate in stormy weather,
The image recording apparatus is characterized in that the frame rate switching unit switches the recording frame rate to the second frame rate during the specific period when it is determined that the weather is stormy. The image recording device according to claim 1, wherein the second frame rate is the imaging frame rate. 3. The stormy weather determination processing unit specifies a period in which a wiper area is included in a specific period determination area of the captured image as the specific period. The image recording device described. an imaging step of imaging an image at an imaging frame rate;
a stormy weather determination processing step of determining whether or not it is stormy weather using the captured image;
The recording frame rate when recording the captured image is set to a first frame rate lower than the imaging frame rate, and when it is determined that the weather is stormy, the recording frame rate is set to the first frame rate. a frame rate switching step of switching to a second frame rate higher than the second frame rate;
a recording step of recording the captured image at the recording frame rate ,
The stormy weather determination processing step specifies a specific period for increasing the recording frame rate in stormy weather;
The image recording method is characterized in that the frame rate switching step switches the recording frame rate to the second frame rate during the specific period when it is determined that the weather is stormy. an imaging step of imaging an image at an imaging frame rate;
a stormy weather determination processing step of determining whether or not it is stormy weather using the captured image;
The recording frame rate when recording the captured image is set to a first frame rate lower than the imaging frame rate, and when it is determined that the weather is stormy, the recording frame rate is set to the first frame rate. a frame rate switching step of switching to a second frame rate higher than the second frame rate;
causing a computer to execute a recording step of recording the captured image at the recording frame rate ;
The stormy weather determination processing step specifies a specific period for increasing the recording frame rate in stormy weather;
The image recording program is characterized in that the frame rate switching step switches the recording frame rate to the second frame rate during the specific period when it is determined that the weather is stormy.

Images