JP7252775B2 - Video analysis support device and method - Google Patents

Description

The present invention relates generally to techniques for assisting video analytics processes, such as detection of objects to be sensed.

As a background art of this technical field, there is, for example, Japanese Patent Application Laid-Open No. 2018-022234 (Patent Document 1). In Patent Document 1, "The detection area selection unit 103 selects another vehicle as a detection target based on the detection result of the road surface area by the road surface detection unit 102 and the time series verification result by the time series verification unit 105. A vehicle detection region for detection is set in the input image, and the detection region selection unit 103 causes the road surface region interpolation unit 106 to interpolate the road surface region based on the results of the time-series verification by the time-series verification unit 105. In this case, the vehicle detection area including the interpolated road surface area is set based on the interpolation result."

JP 2018-022234 A

In video analysis that detects detection targets (objects such as people, packages, animals, ships, and vehicles) in images, objects that exist in places where there is no possibility of detection targets being detected are mistakenly detected as detection targets. In some cases, the mirror image of the detection target reflected on a wall, glass surface, mirror, or the like may be erroneously detected as the detection target.

In order to reduce such erroneous detection, it is necessary to determine the presence or absence of the detection target only in areas where the detection target may appear in the image (hereinafter referred to as the detection target area), or to determine the presence or absence of the detection result. If the position information on the image is not included in the detection target area, the detection result may be rejected. Through the above processing, it is possible to improve the accuracy of object detection in video analysis.

However, if the detection target area is inappropriate, the effect of improving detection accuracy (in other words, reducing false detections) will not be sufficiently exhibited.

Therefore, setting an appropriate detection target area is one of the important factors for improving the detection accuracy.

However, setting an appropriate detection target area is difficult for at least one of the following reasons.
・In an image captured by a camera, the actual size corresponding to one pixel may differ depending on the pixel position due to various shooting conditions such as camera posture, shooting magnification (angle of view) and position. Estimating the actual magnitude for each pixel location is difficult.
• If an object (a sensing object or other object) for which the actual size is based is visible, it is expected to estimate the corresponding actual size for each pixel in the image. However, it is not always possible to shoot an image including such an object. In order to obtain images of such objects, it is necessary to continue shooting for a certain period of time. This problem is considered to be particularly large in cases where relatively few objects are captured (for example, the camera is a surveillance camera).
- On-site camera installation work is usually carried out during hours when there are no objects to be detected by the camera. For this reason, there is a problem that setting of the detection target area can be started when the installation of the camera is finished and the time period when the detection target exists.

According to one aspect of the present invention, a video analysis support device estimates a relevant region and calculates camera parameters based on an input video in an image coordinate system. The device derives the relevant region of the world coordinate system by converting the coordinates of the relevant region from the image coordinates to the world coordinates using the calculated camera parameters, and based on the derived relevant region, the world coordinate system A detection target region is derived, and the coordinates of the derived detection target region are converted from world coordinates to image coordinates using the calculated camera parameters, thereby deriving the detection target region of the image coordinate system.

According to one aspect of the present invention, it is possible to derive an appropriate detection target region even if the video is a single still image and the detection target is not shown in the still image.

Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is an overall configuration diagram of a video analysis system according to Example 1 of the present invention; FIG. 1 is a hardware configuration diagram of a video analysis system according to Example 1 of the present invention; FIG. 1 is a sequence diagram of a video analysis system according to Example 1 of the present invention; FIG. FIG. 3 is a diagram showing an example of a shooting range of a world coordinate system of a video shooting device; 4B is a diagram showing an example of the XY plane of the world coordinate system shown in FIG. 4A; FIG. It is a figure which shows an example of the estimated pertinent area|region. It is a figure which shows an example of the corresponding area|region after transform|converting into world coordinates from image coordinates. It is a figure which shows an example of the detection object area|region of a world coordinate system. It is a figure which shows an example of the detection target area|region of an image coordinate system. FIG. 8 is a diagram showing an example of a result of adding an offset to the Z coordinate of the detection target area in the world coordinate system; FIG. 10 is a diagram showing an example of the result of adding a smaller offset to the Z coordinate of the detection target area in the world coordinate system; FIG. 10 is an explanatory diagram of an example of a setting screen according to the first embodiment of this invention; FIG. 5 is an explanatory diagram of an example of an area confirmation screen according to the first embodiment of this invention; FIG. 10 is a diagram showing an example of an imaging range in Example 2 of the present invention; FIG. 10 is a diagram showing an example of an imaging range in Example 2 of the present invention; It is a figure which shows an example of the applicable area|region estimated in Example 2 of this invention. It is a figure which shows an example of the setting rule in Example 2 of this invention. FIG. 10 is a diagram showing an example of a plurality of detection target areas in the image coordinate system according to Example 2 of the present invention; FIG. 10 is a diagram showing an example of a plurality of detection target areas in the image coordinate system according to Example 2 of the present invention;

In the following description, an "interface device" may be one or more interface devices. The one or more interface devices may be at least one of the following:
- One or more I/O (Input/Output) interface devices. An I/O (Input/Output) interface device is an interface device for at least one of an I/O device and a remote display computer. The I/O interface device to the display computer may be a communications interface device. The at least one I/O device may be any of a user interface device, eg, an input device such as a keyboard and pointing device, and an output device such as a display device.
- One or more communication interface devices. The one or more communication interface devices may be one or more of the same type of communication interface device (e.g., one or more NICs (Network Interface Cards)) or two or more different types of communication interface devices (e.g., NIC and It may be an HBA (Host Bus Adapter).

Also, in the following description, "memory" may be one or more memory devices, typically a main memory device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

Also, in the following description, a "persistent storage device" is one or more persistent storage devices. A permanent storage device is typically a non-volatile storage device (for example, an auxiliary storage device), and specifically, for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive).

Also, in the following description, "storage" may be at least memory of memory and persistent storage.

Also, in the following description, a "processor" is one or more processor devices. The at least one processor device is typically a microprocessor device such as a CPU (Central Processing Unit), but may be another type of processor device such as a GPU (Graphics Processing Unit). At least one processor device may be single-core or multi-core. At least one processor device may be a processor core. At least one processor device may be a broadly defined processor device such as a hardware circuit (for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)) that performs part or all of processing.

In the following description, the expression "xxx table" may be used to describe information that provides an output for an input. A learning model such as a neural network that generates Therefore, the "xxx table" can be called "xxx information". Also, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. may

Further, in the following explanation, the processing may be explained with the subject of "program", but the program is executed by the processor, so that the specified processing can be performed by the storage device and/or the interface device as appropriate. As it occurs while in use, the subject of processing may be a processor (or a device, such as a controller, having that processor). A program may be installed on a device, such as a computer, from a program source. The program source may be, for example, a program distribution server or a computer-readable (eg, non-temporary) recording medium. Also, in the following description, two or more programs may be implemented as one program, and one program may be implemented as two or more programs.

In addition, in the following description, the function may be described using the expression “kkk unit”, but the function may be realized by executing one or more computer programs by a processor, or may be realized by executing one or more computer programs. It may be realized by the above hardware circuits (FPGA or ASIC, for example). When a function is realized by executing a program by a processor, the defined processing is performed while appropriately using a storage device and/or an interface device, etc., so the function may be at least part of the processor. good. A process described with a function as the subject may be a process performed by a processor or a device having the processor. Programs may be installed from program sources. The program source may be, for example, a program distribution computer or a computer-readable recording medium (for example, a non-temporary recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

Also, in the following description, the “video analysis support device” may be configured with one or more computers. Specifically, for example, when a computer has a display device and displays information on its own display device, the computer may be the video analysis support device. Also, for example, when the first computer (eg server computer) transmits display information to a remote second computer (display computer (eg client computer)) and the display computer displays the information (the first computer When displaying information on the second computer), at least the first computer out of the first computer and the second computer may be the video analysis support device. In other words, the fact that the video analysis support device "displays the display information" may be displaying the display information on the display device of the device, or the device may display the display information on the display computer. (In the latter case, the information for display is displayed by a computer for display).

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is an overall configuration diagram of a video analysis system 100 of Example 1 of the present invention.

A video analysis system 100 of this embodiment comprises a video camera 101 , a video storage device 102 , an input device 103 , a display device 104 and a server computer 110 .

The image capturing device 101 is a device that captures an image, creates image data, and outputs the image data. The image capturing device 101 may be a video camera, a still camera, a high-sensitivity camera, a low-illuminance camera, a night-vision camera, a thermal camera, an X-ray camera, etc., or may be composed of a plurality of units including any of these. may be That is, the image capture device 101 may be a device having at least one camera (eg, one camera itself).

The video storage device 102 is a storage device that stores video data and outputs it in response to a request. can be configured using a storage system with

Images output from the image capturing device 101 or the image storage device 102 are both input to the image input unit 111 of the server computer 110 . The image analysis system 100 may include both the image capturing device 101 and the image storage device 102 as shown in FIG. 1, or may include only one of them. When the video analysis system 100 includes both the video shooting device 101 and the video storage device 102, even if the input source of the video data to the video input unit 111 is switched to the video shooting device 101 or the video storage device 102 as necessary. Alternatively, the image data output from the image capturing device 101 may be temporarily stored in the image storage device 102 and input to the image input unit 111 from there. In that case, the video storage device 102 may be, for example, a cache memory that temporarily holds video data continuously input from the video shooting device 101 .

Note that the image data stored in the image storage device 102 and the image data created by the image capturing device 101 may be data of any format. For example, the image capturing device 101 may be a video camera, and moving image data captured by it may be output as image data, or such image data may be stored in the image storage device 102 . Alternatively, the image capturing device 101 may be a still camera, and a series of still image data captured at predetermined intervals (at least intervals at which the captured object can be tracked) may be output as image data. Such video data may be stored in video storage device 102 .

The input device 103 is an input interface, such as a mouse, keyboard, or touch device, for transmitting user operations to the server computer 110 . The display device 104 is an output interface such as a liquid crystal display, and is used for displaying the video analysis results of the server computer 110 and interactive operations with the user. For example, the input device 103 and the display device 104 may be integrated by using a so-called touch panel or the like.

The server computer 110 is an example of a video analysis support device. The server computer 110 generates (derives) a mask region (detection target region in the image coordinate system) for limiting the region targeted for object detection processing in the video in the image coordinate system indicated by the input video data, and inputs It functions as a device that performs image analysis processing (processing including determining whether or not the object to be detected exists in the mask area) within the generated mask area of the generated image data. Note that the server computer 110 may function as a device that generates a mask area for limiting an area targeted for object detection processing in an image indicated by input image data but does not perform image analysis processing.

The images handled by the server computer 110 may be fixed-point observation images captured at one or more locations, or may be images installed on mobile objects such as in-vehicle cameras, drone-mounted cameras, wearable cameras, action cameras, and the like. It may also be a video shot by a camera that has been used.

The server computer 110 includes an image input unit 111, a camera parameter calculation unit 121, an area estimation unit 123, an area derivation unit 30 (for example, a coordinate conversion unit 122 and an area setting unit 124), an area store 125, an image analysis unit 131, and management information. A store 10 is provided.

The video input unit 111 receives video data shot by the video shooting device 101 or reads video data from the video storage device 102 and converts it into a data format used inside the server computer 110 . Specifically, for example, the video input unit 111 performs video decoding processing for decomposing a moving image (moving image data format) into frames (still image data format). The obtained frames are sent to camera parameter calculator 121 , region estimator 123 , and video analyzer 131 . That is, in this embodiment, the video input from the video input unit 111 is a still image.

The camera parameter calculator 121 estimates camera parameters based on the image received from the image input unit 111 . The coordinate transformation unit 122 receives camera parameters and coordinates from the area setting unit 124 and performs mutual transformation processing between image coordinates and world coordinates. The image coordinates are the coordinates on the image plane (coordinates of the image coordinate system), and the world coordinates are the coordinates of the space in which the subject in the image exists (image of the world coordinate system). Coordinate transformation unit 122 performs coordinate transformation processing on the coordinates received from area setting unit 124 , and the transformed coordinates are sent to area setting unit 124 . In the present embodiment, the camera parameter calculation unit 121 calculates camera parameters based on an image each time an image is received. However, in the present invention, at least one of the following may be employed.
- The camera parameter calculator 121 may be omitted. In this case, camera parameters may be determined in advance for each camera. For at least one camera, at least some of the camera parameters may be changed manually by a user, for example as described below.
・For some cameras, the camera parameters are calculated by the camera parameter calculation unit 121 based on the video each time the video is received. You can decide. At least some of the predetermined camera parameters may be manually changed by the user, for example, as described below.

Camera parameters are numerical data necessary for the coordinate transformation unit 122 to transform image coordinates and world coordinates, and are camera orientations (for example, camera position on world coordinates, camera pan angle, camera tilt angle , and roll angle of the camera), the angle of view (focal length) of the camera, the aspect ratio of the captured image, and so on. Image coordinates (u, v) and world coordinates (X, Y, Z) can be mutually converted by (Equation 1), and matrix C of (Equation 1) is a camera parameter. Any known technique may be used to derive the camera parameters. For example, a plurality of frames including a structure whose world coordinates are known may be received from the video input unit 111, and calculation may be performed using the bundle adjustment method based on a data set of corresponding points between world coordinates and image coordinates. .

Based on the image received from the image input unit 111, the area estimation unit 123 applies a machine learning technique to determine a corresponding area, which is a preset area or an area specified by the user using the input device 103. presume.

A machine learning method related to region estimation, for example, divides an image into partial regions, extracts image features from each partial region with a feature extractor, and classifies the partial region image in a desired region based on the image features with a classifier. A region estimation function that identifies whether there is a partial region image and presents a desired region in the original image by integrating the identification results of these partial region images. By automatically optimizing the parameters of the classifier and the classifier, it is a method that acquires an effective region estimation function even for unknown images that are not included in the training data.

Deep learning is known as a specific example of such a processing technique. As an example of a feature parameter extraction method using deep learning, a feature extraction method using a convolutional neural network is known. In a convolutional neural network, a large number of input/output functions (activation functions) called neuron units are combined for each small image area, and these are stacked in multiple layers to form a pyramid structure. According to this method, classifier parameters are extracted for each layer of the neuron unit so that the object can be identified step by step while changing the position and image size of the object to be detected, and finally, It is possible to obtain discriminator parameters that can discriminate the entire object. Region estimation methods that apply convolutional neural networks are generally called semantic region estimation (semantic segmentation), and specific network models include FCN, U-Net, SegNet, PSPNet, and Mask R-CNN. ing.

The area setting unit 124 applies a setting rule prepared in advance or a setting rule specified by the user through the input device 103 to the corresponding area estimated by the area estimation unit 123, and utilizes coordinate transformation by the coordinate transformation unit 122. to derive the detection target area. The detection target area derived by the area setting unit 124 is stored in the area store 125 . If the server computer 110 does not have the video analysis unit 131, the detection target area derived by the area setting unit 124 may be output as a file or sent to an external system. The details of the detection target area derivation procedure will be described later. Also, the area setting unit 124 may display the detection target area in the image coordinate system in the area store 125 on the display device 104 .

The video analysis unit 131 performs video analysis processing on the frame (video as a still image) received from the video input unit 111 within the detection target area acquired from the area store 125 . The video analysis processing performed by the video analysis unit 131 may be detection processing including determination of whether or not a detection target exists in the detection target area, or other processing (for example, object tracking processing, feature extraction processing, etc.). may include The objects referred to herein are not all common objects such as people, animals, vehicles, ships, and luggage, but any of them may be designated as a detection target. The video analysis result in the video analysis unit 131 is processed into information suitable for display, and then sent to the display device 104 .

Further, the image analysis unit 131 performs image analysis processing on the entire frame received from the image input unit 111, and if the result obtained by the image analysis processing does not fit in the detection target area, the information can be discarded. good. For example, when the video analysis unit 131 performs object detection processing on frames received from the video input unit 111, the video analysis unit 131 may perform filtering processing including the following.
- If the detection result area fits within the detection target area received from the area store 125, the detection result is output.
- If the acquired detection result area of any object does not fit within the detection target area received from the area store 125, the detection result is not output.

FIG. 2 is a hardware configuration diagram of the video analysis system 100 of Example 1 of the present invention.

The server computer 110 is, for example, a general computer having processors 201 and storage devices 202 interconnected. The processor 201 is configured by any processing device capable of computation of video analysis processing. For example, processor 201 may include any of a CPU, GPU, FPGA, and ASIC. Storage device 202 comprises any type of storage medium. For example, storage device 202 may include semiconductor memory and hard disk drives.

In this example, functional units such as the image input unit 111, the camera parameter calculation unit 121, the coordinate transformation unit 122, the area estimation unit 123, the area setting unit 124, and the image analysis unit 131 shown in FIG. It is realized by executing the processing program 203 stored in the . In other words, in this example, the processing executed by each of the above functional units is actually executed by processor 201 according to instructions written in processing program 203 . Also, each of the region store 125 and the management information store 10 may be a storage region included in the storage device 202 . The management information store 10 stores management information including information indicating setting rules. Management information may include, for example, camera parameters.

Server computer 110 further includes a network interface device (NIF) 204 connected to the processor. The image capturing device 101 is connected to the server computer 110 via the network interface device 204, for example. The video storage device 102 may be NAS or SAN connected to the server computer 110 via the network interface device 204 or may be included in the storage device 202 .

The input device 103 and display device 104 may be the input device and display device of the client computer connected to the server computer 110 .

FIG. 3 is a sequence diagram of the video analysis system 100 of Example 1 of the present invention.

The operation sequence of each component of the video analysis system 100 will be described with reference to FIG.

First, the image input unit 111 extracts frames from the image received from the image capturing device 101 or the image storage device 102 . The extracted frames are sent to the camera parameter calculator 121 (step S301) and the region estimator 123 (step S302). Note that when the video includes an object other than the background, the video input unit 111 selects a frame consisting only of the background, or generates a frame corresponding to the background image by learning video data. can be derived. In this case, in steps S301 and S302, the derived frame corresponding to the background image is sent to the camera parameter calculator 121 and the region estimator 123. FIG.

Next, the camera parameter calculator 121 calculates camera parameters based on the frames received from the video input unit 111 . The calculated camera parameters are sent to the area setting unit 124 (step S303). When the user's input information from the input device 103 is used in calculating the camera parameters, the user's input information from the input device 103 is sent to the camera parameter calculation unit 121 (step S304). Alternatively, the camera parameter calculation unit 121 may read information necessary for camera parameter calculation (for example, camera posture and angle of view) stored in advance in the storage device 202 to calculate camera parameters. Note that the camera parameter calculation unit 121 may read and use camera parameters stored in advance in the storage device 202 without calculating the camera parameters. In this case, it is assumed that the camera parameters corresponding to the frame received from the video input unit 111 are read.

Next, the area estimation unit 123 estimates a preset area for the frame received from the video input unit 111 . The area estimation unit 123 may estimate a single area, multiple areas of the same type, or multiple areas of different types. The estimated area is sent to the area setting unit 124 (step S305). Note that the area sent to the area setting unit 124 may be in any data format. For example, it may be an image showing the area, or a set of points showing the outline of the area.

The region setting unit 124 derives a detection target region based on the region received from the region estimation unit 123 and the camera parameters received from the camera parameter calculation unit 121 . If the derivation requires user input, the input device 103 accepts user input (step S306). In deriving the detection target region, the region setting unit 124 notifies the coordinates of the points forming the region and the camera parameters to the coordinate conversion unit 122 in order to convert the image coordinates of the points forming the region and the world coordinates. , the result of the transformation from image coordinates to world coordinates or from world coordinates to image coordinates is received (step S307). Note that the coordinates of the points forming the area notified from the area setting unit 124 to the coordinate conversion unit 122 may be a set of coordinates extracted at equal intervals from the coordinates included in the area, or may be a set of coordinates extracted at equal intervals from the coordinates included in the area. It may be a set of coordinates on a line. When using a set of contour line coordinates, the region setting unit 124 performs contour line extraction processing on the region received from the region estimation unit 123 . Step S307 is described once in FIG. 3, but may be performed multiple times. The derived detection target area is sent to the area store 125 (step S308). At the same time as step S308, the video analysis unit 131 may be notified that the detection target area in the area store 125 has been updated. Also, the detection target area may be sent to a system or device different from the video analysis system 100 via the NIF 204 .

Derivation and registration of the detection target area are completed by the above steps S301 to S308 (“R” in FIG. 3). Through the above processing, the video analysis system 100 can derive a detection target region, which is a region in which the detection target may appear from the video, even if the detection target is not shown in the video.

Next, the video analysis unit 131 reads the registered detection target area of the area store 125 (step S309). The video analysis unit 131 may present the detection target area to the user through the display device 104, and may receive correction of the detection target area by the user using the input device 103 (step S310).

Next, the video analysis unit 131 receives a frame from the video input unit 111 (step S311), and performs video analysis processing on the detection target area of the frame. The video analysis result is sent to the display device 104 (step S312). Note that the image analysis unit 131 may perform image analysis processing on the entire frame received from the image input unit 111, and then perform filter processing for selecting only the detection target area as the image analysis result. Also, the video analysis result of the video analysis unit 131 may be saved in the storage device 202 without being sent to the display device 104 or may be sent to a system different from the video analysis system 100 via the NIF 204 .

Through steps S301 to S312 described above, the derivation and registration of the detection target area and the video analysis processing using the detection target area are completed. Through the above processing, the video analysis system 100 can derive only the video analysis result within the detection target area.

Next, the camera parameter calculation unit 121 receives a frame from the video input unit 111 (step S321), calculates camera parameters, and sends the calculated parameters to the area setting unit 124 (step S322). If the region setting unit 124 detects a change in the camera parameters in the case of periodic or intermittent implementation, the detection target region in the image coordinate system is converted to world coordinates by the coordinate transformation unit 122 using the camera parameters before the change. (step S323), and further, the detection target area converted to the world coordinates is converted to image coordinates by the coordinate conversion unit 122 using the changed camera parameters (step S327). to correct (change) the detection target area. The corrected detection target area may be sent to the area store 125, and the detection target area held by the area store 125 may be updated (step 324). The processing of steps S321 to S324 has the effect that when the user corrects the detection target area in step S306, even if the posture or angle of view of the camera changes, it is possible to omit the correction by the user again. .

An example of the region setting procedure by the video analysis system 100 according to the first embodiment of the present invention will be described with reference to FIGS. 4A to 4H.

FIG. 4A shows an example of the shooting range of the world coordinate system of the video shooting device 101 . According to FIG. 4A, a video camera 101 is installed on the street so that a house, a sidewalk, and a roadway are subjects. The arrows shown in FIG. 4A indicate the world coordinate system in which the world coordinates (X, Y, Z) exist. The image coordinate system in which the image coordinates (u, v) exist is omitted from the drawing, but the origin is the upper left corner of the frame frame shown in FIG. 4A, and the u axis extends horizontally to the right from the origin. It is a coordinate system in which the vertically downward axis is the v-axis.

FIG. 4B shows the positional relationship of houses, sidewalks and roadways along the XY plane (Z=0) of the world coordinate system shown in FIG. 4A. A black area 81 in FIG. 4A is a person, and although it is not directly related to the area setting procedure, it is described for reference.

4C to 4H, an example of extracting a detection target region using the video analysis system 100 for a frame of a background image in the shooting range illustrated in FIG. 4A will be described.

FIG. 4C shows the result of estimating the roadway area in FIG. 4A by the area estimation unit 123 in step S305. In this example, an example in which the region estimation unit 123 estimates the roadway region will be described, but an object other than the roadway may also be the estimation target. However, when using a semantic region estimation method that applies deep learning as a region estimation method, it is easier to collect training data by targeting objects that are likely to have similar appearances regardless of location. Moreover, it can be expected to expand the application place by generalization. In general, compared to houses and sidewalks, roadways are considered to be less dependent on the location of their appearance. In cases other than the example in FIG. 4A , the area estimation target by the area estimation unit 123 is, for example, a cash register or entrance at a commercial facility, a sea surface at a wharf, a road, a tollgate or a service area entrance on a highway, a bridge pier near the sea, the sky or Horizon is fine.

The shaded area in FIG. 4D is the result of converting the estimated roadway area in FIG. 4C from image coordinates to world coordinates in step S307. The XY plane shown in FIG. 4D is a plane including the estimated roadway area in FIG. 4C, and is the XY plane (Z=0) shown in FIG. 4C. However, the method of setting the world coordinates for the image coordinates is arbitrary, and the XYZ coordinates may be determined so that the planes other than those shown in FIG. 4 are on the XY plane.

The hatched area in FIG. 4E is an area in which both ends are estimated as sidewalks by offset correction based on the estimated roadway area converted to the world coordinates in FIG. 4D by the area setting unit 124 . According to the example of FIG. 4E, an area that does not include the relevant area is derived as the detection target area of the world coordinate system by offset correction for the relevant area of the world coordinate system. The example of FIG. 4E shows an example in which an arbitrary range of distances at both ends of the circumference of the estimated roadway area is specified as a sidewalk. Any other area specification method may be used.

The shaded area in FIG. 4F is a frame obtained by transforming the estimated sidewalk area estimated by the area setting unit 124 into image coordinates in step S307. As described above, according to the processes described in the order of FIGS. 4C, 4D, 4E, and 4F, even when the area estimation unit 123 does not estimate the sidewalk, the sidewalk area can be estimated based on the roadway estimation result. becomes possible.

In FIG. 4E, furthermore, for the estimated sidewalk area on the XY plane of the world coordinates, the image coordinates corresponding to different Z values can be obtained to obtain the position in the sky image of the estimated sidewalk area. According to this method, it is possible to obtain an area on the image when an object of arbitrary height moves on the sidewalk estimation area. In this manner, the area setting unit 124 derives the detection target area using the sidewalk estimation area converted into image coordinates at a plurality of Z coordinates. 4G and 4H both show the appearance range on the frame when a person moves on the sidewalk estimation area. FIG. 4G shows the results when a person taller than that in FIG. 4H is analyzed. That is, FIG. 4G shows the result of deriving the detection target area using the image coordinates of the sidewalk estimation area at a larger Z than in the case of FIG. 4H. In this way, according to the processes described in the order of FIGS. Based on the obtained area, it is possible to set the detection target area in consideration of the size of the detection target.

FIG. 5 is a setting screen when the user sets a detection target area setting rule suitable for a camera and a detection target. The region setting unit 124 sets the detection target region according to the procedure described with reference to FIG. 4 based on the rule set through the setting screen. Also, the UI (user interface) in FIGS. 5 and 6 may be a GUI (Graphical User Interface) part. Therefore, the screens shown in FIGS. 5 and 6 may be GUI screens.

The screen shown in FIG. 5 includes UIs 501 to 504, for example.

First, the user selects a camera or video included in the video shooting device 101 or video storage device 102 for which a detection target rule is to be set from the dropdown list of the camera selection UI 501 . A camera selection UI 501 is a user interface that receives selection of a desired camera from one or more cameras.

Next, the user specifies a detection target area setting rule on the detection target selection UI 502 . Here, the user may enter a new rule name or select a candidate displayed in the drop-down list.

Next, the user inputs the detection target size on the size input UI 503 . Here, an example of inputting only the height as the detection target size is shown, but it is also possible to specify a complicated shape including width or depth. Suppose we have an input UI.

Next, the user inputs a setting rule for setting another area based on the area (black in the figure) estimated by the area estimation unit 123 on the world coordinates in the rule specifying UI 504 . The rule specification UI 504 in FIG. 5 is a total of one area with a specified width on the left end side of the area estimated by the area estimation unit 123 and one area with a specified width on the right end side of the area estimated by the area estimation unit 123. This is an example of inputting a rule for setting two areas (hatched areas in the figure). A three-dimensional image may be displayed on the rule designation UI 504 instead of the two-dimensional image. The detection target selection UI 502, the size input UI 503, and the rule designation UI 504 may be examples of setting UIs.

As described above, the setting screen allows the user to set the area setting rule for the area setting unit 124 to set the detection target area.

FIG. 6 is an area confirmation screen for the user to confirm whether the detection target area setting rule suitable for the camera and the detection target is set.

The screen shown in FIG. 6 includes UIs 501 to 503 and UIs 601 and 602, for example. The camera selection UI 501, detection target selection UI 502, and size input UI 503 are the same as in FIG.

The area UI 601 displays the detection target area in the image coordinate system derived in step S308 or step S324. The camera parameter UI 602 displays the camera parameters used for deriving the detection target area in step S308 or step S324. Note that the camera parameter UI 602 in FIG. 6 shows an example in which the matrix C is displayed as a camera parameter. A value converted to a parameter containing one or more may be displayed.

The user can confirm through the area UI 601 whether the detection target area derived by the area setting unit 124 is a desired area. Furthermore, when it is desired to modify either the detection target area or the camera parameters, the area displayed on the area UI 601 and the camera parameters displayed on the camera parameter UI 602 can be directly edited.

Next, the video analysis system 100 of Example 2 will be described. At that time, the points of difference from the first embodiment will be mainly described, and the explanations of the points in common with the first embodiment will be omitted or simplified.

In this embodiment, an example will be described in which different detection target regions are set depending on time on a roadway having a reversible lane (center line transition) and a bus-only lane. In the second embodiment, all parts other than the contents described below are common to the video analysis system 100 described in the first embodiment.

FIG. 7A is an image of a roadway with two diagonal lines on the left side and three lanes on the right side, and the one lane on the right side is a dedicated bus lane, taken using a camera located above the center of the roadway. FIG. 7B is an example of imaging the same location as in FIG. 7A, but the center line has moved due to the reversible lanes, resulting in three lanes on the left side and two lanes on the right side.

On a roadway with reversible lanes and dedicated bus lanes with designated days of the week and time slots associated with the reversible lanes, traffic classifications change depending on the time and day of the week, as shown in FIGS. 7A and 7B. In order to detect traveling direction violations and vehicle classification violations on such roads, it is necessary to set different detection target areas depending on the time and day of the week so as to be able to respond to changes in the traveling direction and vehicle classification.

An example of extracting a detection target region using the video analysis system 100 for the frame of the background image in FIG. 7A or 7B will be described with reference to FIGS. 7C to 7E. The background image may be either of Figures 7A and 7B.

The hatched area in FIG. 7C is the result of the area estimating unit 123 estimating the roadway based on FIG. 7A or 7B in step S305, and converting the estimated roadway area from image coordinates to world coordinates in step S307. The Y-axis shown in FIG. 7C is the traveling direction of the vehicle on the roadway of FIG. 7A or 7B, and the X-axis shown in FIG. 7C is the crossing direction of the roadway of FIG. 7A or 7B.

FIG. 7D shows a table of setting rules that the area setting unit 124 refers to when setting the detection target area. The table in FIG. 7D shows that the rules for area setting change depending on the time period, and the corresponding lanes and traffic divisions for each area.

In the case of the second embodiment, as shown in FIG. 7C, the region estimation unit 123 only estimates the roadway region, and the region setting unit 124 estimates the vehicle region from the vehicle region according to the time zone region setting rule shown in FIG. 7D. As shown in FIG. 7B, derivation of the detection target area corresponding to the location that changes depending on the time period is realized. FIG. 7D shows an example in which an area setting rule for each time zone is described. good. The setting rule as shown in FIG. 7D may be held by the area setting unit 124, but is stored in the management information store 10 in the storage device 202 in this embodiment. Furthermore, the data format and data format related to the setting rule may be of any type. For example, it may be text data or a database. Further, these data may be registered in the area setting unit 124 or the storage device 202 in advance, or may be input by the user through the input device 103 .

The three shaded areas in FIG. 7E are the left coordinates derived based on the row No. 1 of the area setting rules in FIG. 7D based on the estimated roadway area converted to the world coordinates in FIG. It is an area of 2 lanes, 2 right lanes, and 1 right lane. The area setting rule in FIG. 7D describes how to divide the estimated roadway area. Taking No. 1 as an example, the area 0 to 40% from the left end in the X-axis direction is the left lane area. A range of distances from 80% to 80% indicates a right lane, and a range of distances from 80% to 100% indicates a right lane and a dedicated bus lane. Similarly, the two hatched areas in FIG. 7F are derived based on the estimated roadway area converted to the world coordinates in FIG. It is an area with 3 lanes on the left and 2 lanes on the right.

By step S307, the regions obtained in FIGS. 7E and 7F are converted into image coordinates, and the region setting unit 124 derives a detection target region.

Thus, according to the above-described processing, it is possible to set a detection target area that changes depending on the time, day of the week, weather, temperature, etc., based on the area estimated by the area estimation unit 123 .

The description of Examples 1 and 2 above can be summarized, for example, as follows. It should be noted that the following summary may include matters (for example, modified examples) that are not included in the above description.

A server computer 110 , which is an example of a video analysis support device, has a video input unit 111 , an area estimation unit 123 , and an area derivation unit 30 . The image input unit 111 inputs an image in an image coordinate system captured by the image capturing device 101 (an example of a camera) from the image capturing device 101 (or the image storage device 102), for example. The area estimation unit 123 estimates the corresponding area in the image based on the input image in the image coordinate system. A region derivation unit 30 derives a relevant region in the world coordinate system by converting the estimated coordinates of the relevant region from image coordinates to world coordinates using camera parameters, and calculates a world coordinate based on the derived relevant region. A detection target area in the coordinate system is derived, and the coordinates of the set detection target area are converted from world coordinates to image coordinates using camera parameters, thereby deriving the detection target area in the image coordinate system. In this way, even if the video is a single still image and the object to be detected does not appear in the still image, the corresponding area in the world coordinate system is derived using the camera parameters, and the corresponding area in the world coordinate system is derived based on the corresponding area. is derived, and the same camera parameters are used to derive a detection target area in an image coordinate system that can be used as a mask area (or other application). This makes it possible to derive an appropriate detection target area even if the video is a single still image and the detection target is not shown in the still image.

Derivation of the detection target area in the world coordinate system is based on setting rules (e.g., rules including offsets applied to the world coordinates of the relevant area in the world coordinate system) that are rules for changing the world coordinates of the relevant area in the world coordinate system. can be applied. In this way, since the detection target area of the world coordinate system is derived based on the relevant area of the world coordinate system, it can be expected that the detection target area of the world coordinate system is appropriate. Derivation of the detection target area of the coordinate system can be expected.

The area derivation unit 30 specifies information that determines the changed world coordinates of the relevant area in the world coordinate system (for example, the offset applied to the world coordinates of the relevant area, or the changed world coordinates of the relevant area itself). A setting UI, which is a user interface that accepts The applied setting rule may be a setting rule according to information specified via a setting UI. As a result, it can be expected that the detection target area of the world coordinate system derived by applying the setting rule to the relevant area will be appropriate.

The area deriving unit 30 determines a detection target area in a world coordinate system set using a setting rule according to information specified via a setting UI and a detection target area in an image coordinate system based on the detection target area in the world coordinate system. A region UI 601, which is a user interface for outputting at least one of, may be provided. By looking at the detection target area output to the area UI 601, the user can determine whether the information specified via the setting UI is appropriate.

When the specified information is changed via the setting UI, the area derivation unit 30 may change the area displayed on the area UI based on the setting rule according to the changed information. This allows the user to determine whether or not the changed information is appropriate.

The region deriving unit 30 stores at least one of a detection target region in the world coordinate system and a detection target region in the image coordinate system, and camera parameters associated with the region and used for deriving the region in the storage device 202. good. The region derivation unit 30 may provide a camera parameter UI 602, which is a user interface that outputs camera parameters associated with the region output to the region UI 601 and receives changes to the camera parameters. This allows the user to know the camera parameters that have affected the detection target area.

In at least one of the case where the designated information is changed via the setting UI and the case where the camera parameters are changed via the camera parameter UI 602, the region deriving unit 30 outputs the information to the region UI 601 based on the change. You may change the area where This allows the user to determine whether the changed information or camera parameters are appropriate.

For each of the multiple time slots, there may be one or more configuration rules that are applied to respectively configure one or more world coordinate system sensing regions for that time slot (see, for example, FIG. 7D). . The region deriving unit 30 derives one or more detection target regions in the world coordinate system using one or more setting rules corresponding to the time zone belonging to the time when setting the detection target region in the world coordinate system. good. As a result, it can be expected to derive a detection target area in the world coordinate system that is appropriate for the environment related to the imaging range.

The region deriving unit 30 may provide a setting UI, which is a user interface for accepting specification of information related to the range captured by the video capturing device 101 . The applied setting rule may be a setting rule determined from information specified via the setting UI. If the user knows more about the information about the shooting range than about the details about the setting rule, the area derivation unit 30 generates an appropriate setting rule by specifying the information. As such, it may be highly convenient for the user. When this mechanism is employed, the storage device 202 of the server computer 110 may store conversion information, which is information indicating association between information relating to the range captured by the video capturing device 101 and a plurality of setting rules. Conversion information may be part of the management information. When information relating to the range to be captured by the image capturing device 101 is specified via the setting UI, the region derivation unit 30 identifies one or more setting rules corresponding to the specified information from the conversion information. , the identified one or more configuration rules may be configured or displayed in a configuration UI. Information specified via the setting UI (information on the shooting range) must be at least one of the following:
・ Shooting time,
・Shooting location,
・The weather at the shooting time and shooting location, and
- the size of the object to be detected, or object attributes that affect the size;
OK. Such information is considered to be information that is easy for the user to grasp, and therefore, it is expected that the convenience for the user will be improved.

A video analysis unit 131 may be further provided. The video analysis unit 131 uses the derived detection target area in the image coordinate system to perform video analysis processing including determination of whether or not the detection target object appears in the captured and input video in the image coordinate system. good. In this way, since the video analysis unit 131 and the process of deriving the detection target area in the image coordinate system used in the video analysis process performed by the video analysis unit 131 are performed by the same device, an improvement in convenience is expected. be done.

A camera parameter calculator 121 may be further provided. The camera parameter calculation unit 121 may calculate (estimate) camera parameters based on the video input by the video input unit 111 . The camera parameters calculated by the camera parameter calculation unit 121 may be used as the camera parameters used in each of the derivation of the relevant region in the world coordinate system and the derivation of the detection target region in the image coordinate system. As a result, even if the shooting conditions are changed, the camera parameters calculated based on the image after the shooting conditions (e.g. camera posture, shooting magnification (angle of view) and position) are changed are also changed. Deriving a suitable sensing target area can be maintained.

The region deriving unit 30 stores at least one of a detection target region in the world coordinate system and a detection target region in the image coordinate system, and camera parameters associated with the region and used for deriving the region in the storage device 202. good too. If the camera parameters associated with the detection target region in the image coordinate system are different from the calculated camera parameters, the region derivation unit 30 calculates the image coordinate system in the storage device based on the calculated camera parameters. The detection target area may be changed. In this way, it is possible to maintain the derivation of an appropriate detection target area even if the camera parameters are changed as the imaging conditions are changed.

Although several embodiments have been described above, these are examples for explaining the present invention, and are not meant to limit the scope of the present invention only to these embodiments. The invention can also be implemented in various other forms.

100: Video analysis system

Claims (12)

an image input unit for inputting an image in an image coordinate system captured by a camera;
a region estimating unit for estimating a corresponding region in the video based on the input video in the image coordinate system, using a network model trained using images having a plurality of correct information;
Deriving a corresponding region in a world coordinate system by converting the estimated coordinates of the corresponding region from image coordinates to world coordinates using camera parameters, and obtaining world coordinates different from the corresponding region estimated by the region estimating unit. By applying the setting rule for setting the above area to the derived corresponding area, the offset correction for the derived corresponding area can be performed in another world coordinate system that does not include the derived corresponding area . Deriving a detection target area in the world coordinate system, which is an area in which the detection target may appear, and converting the coordinates of the derived detection target area from the world coordinates to the image coordinates using the camera parameters. A video analysis support device, comprising: a region derivation unit that derives a detection target region in an image coordinate system by transformation. The area derivation unit provides a setting UI, which is a user interface for accepting specification of information that determines the changed world coordinates of the corresponding area of the world coordinate system,
The applicable setting rule is a setting rule according to information specified via the setting UI.
The video analysis support device according to claim 1 . The area derivation unit is configured to: detect a detection target area in a world coordinate system derived using a setting rule according to information specified via the setting UI; and a detection target area in an image coordinate system based on the detection target area in the world coordinate system. Provide a region UI that is a user interface that outputs at least one of
3. The video analysis support device according to claim 2 . When the specified information is changed via the setting UI, the area deriving unit changes the area displayed on the area UI based on a setting rule according to the changed information.
The video analysis support device according to claim 3 . The region derivation unit stores at least one of a detection target region in the world coordinate system and a detection target region in the image coordinate system, and camera parameters associated with the region and used for deriving the region in a storage device. ,
The area derivation unit provides a camera parameter UI, which is a user interface that outputs camera parameters associated with the area output to the area UI and receives changes to the camera parameters.
The video analysis support device according to claim 3 . In at least one of a case where the specified information is changed via the setting UI and a case where camera parameters are changed via the camera parameter UI, the region derivation unit calculates the change the region displayed in the region UI,
The video analysis support device according to claim 5 . for each of the plurality of time zones, there are one or more configuration rules applied to respectively derive one or more world coordinate system sensing regions of interest for that time zone;
The region derivation unit derives one or more detection target regions in the world coordinate system using one or more setting rules corresponding to the time zone belonging to the time when setting the detection target region in the world coordinate system,
The video analysis support device according to claim 1 . The region derivation unit provides a setting UI, which is a user interface for accepting designation of information related to the range captured by the camera,
The applicable setting rule is a setting rule determined from information specified via the setting UI.
The video analysis support device according to claim 1 . A video analysis unit that uses the derived detection target area of the image coordinate system to perform video analysis processing including determination of whether or not the detection target object appears in the captured and input video of the image coordinate system;
The video analysis support device according to claim 1, further comprising: further comprising a camera parameter calculation unit that calculates camera parameters based on the input video;
The camera parameters used in each of the derivation of the corresponding region in the world coordinate system and the derivation of the detection target region in the image coordinate system are the calculated camera parameters.
The video analysis support device according to claim 1. The region derivation unit stores at least one of a detection target region in the world coordinate system and a detection target region in the image coordinate system, and camera parameters associated with the region and used for deriving the region in a storage device. ,
If the camera parameters associated with the detection target region in the image coordinate system are different from the calculated camera parameters, the region derivation unit calculates the image coordinates in the storage device based on the calculated camera parameters. change the detection target region of the system,
The video analysis support device according to claim 10 . a computer inputting an image in an image coordinate system captured by a camera;
a step in which a computer estimates a corresponding region in the video based on the input video in the image coordinate system, using a network model trained using an image having a plurality of correct information;
a computer deriving the region of interest in the world coordinate system by transforming the estimated coordinates of the region of interest from image coordinates to world coordinates using camera parameters;
A computer applies a setting rule for setting a region on world coordinates different from the region estimated by the estimating step to the derived region, a step of deriving , by offset correction, a detection target area in a world coordinate system that is an area in another world coordinate system that does not include the derived relevant area and is an area in which a detection target may appear;
a step of a computer deriving a detection target area in an image coordinate system by transforming the derived coordinates of the detection target area from world coordinates to image coordinates using the camera parameters;
A video analysis support method having

Images