JP5198151B2 - Video search device and video search method - Google Patents

Description

  The present invention relates to a video search apparatus and a video search method, and is suitable for application to a video surveillance system, for example.

  In office buildings, hotels and department stores, video surveillance systems have been introduced for a long time, and surveillance cameras are installed throughout the facility. The video captured by these surveillance cameras is displayed in real time in a central control room or the like, or recorded on a recording medium such as a video tape or a hard disk device and stored for a certain period. The captured video recorded on the recording medium is used for situation verification when an incident or the like occurs.

  By the way, normally, a plurality of surveillance cameras are installed in such a facility, and photographing with these surveillance cameras is always performed regardless of day or night. For this reason, the amount of captured video recorded on the recording medium becomes enormous, and when a situation occurs in which it is necessary to detect a scene in which a specific person is captured from such captured video, a great deal of time is spent on the work. In addition, there is a problem of requiring labor.

As one method for solving such a problem, for example, in Patent Document 1, each subject has an identification information element such as an IC tag, and the identification information obtained from the identification information element is recorded on a recording medium. how to keep association with captured image of a subject who possesses the identification information element is disclosed.

  However, since the method disclosed in Patent Document 1 requires each subject to have an identification information element, there is a problem that is difficult to apply to a facility where an unspecified number of people enter and exit, such as hotels and department stores. there were.

  The present invention has been made in consideration of the above points, and an object of the present invention is to propose a video search apparatus and method capable of easily detecting a video of a desired subject from a huge video.

  In order to solve this problem, in the present invention, in the video search device, each subject existing in the video based on the first video information given from the outside is extracted, and the video information of the extracted subject is stored in the recording medium. Information relating to a subject extraction unit to be recorded, a feature part extraction unit that extracts a feature part of each subject extracted by the subject extraction part, and a feature part of each subject extracted by the feature part extraction unit The storage unit stores the feature information for each subject in association with the video information of each subject recorded in the recording medium, and stores the feature information in the storage unit based on the feature information of the subject specified as a search target. And a video search unit for searching for feature information corresponding to the subject specified as the search target from among the feature information. The features.

  In the present invention, in the video search method, each subject existing in the video based on the first video information given from outside is extracted, and the extracted video information of the subject is recorded on the recording medium and extracted. The extracted characteristic parts of each subject are extracted, and the characteristic information for each subject, which is information relating to the extracted characteristic parts of each subject, is stored in association with the video information of each subject recorded on the recording medium. And a second step of searching for feature information corresponding to the subject specified as the search target from among the stored feature information based on the feature information of the subject specified as the search target. It is characterized by providing.

  According to the present invention, it is possible to detect video information of a desired subject from video information of a plurality of subjects recorded on a recording medium, and thus easily detect a video of a desired subject from an enormous amount of video. A video search apparatus and method that can be implemented.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) First Embodiment (1-1) Configuration of Video Monitoring System According to this Embodiment In FIG. 1, reference numeral 1 denotes an overall video monitoring system according to this embodiment. The video monitoring system 1 extracts and collects information about characteristics of each subject (hereinafter referred to as feature information) from the captured video by image processing, and based on the collected feature information of each subject, the recording medium 8 The video of the subject specified by the user can be searched from the video stored in the screen.

  In practice, as shown in FIG. 2, the video monitoring system 1 includes a camera unit 3 including a plurality of cameras 3 </ b> A to 3 n that are fixedly arranged so that the same subject 2 can be photographed from multiple viewpoints. FIG. 2 is an example in which four cameras 3A to 3D are installed so that a subject (for example, a person) 2 located in the center from the four corners of the room 4 can be photographed, and an arrow a indicates the direction of the subject 2. .

  Then, for example, video signals (hereinafter referred to as camera video signals) of the captured video of the same subject 2 as shown in FIGS. 3A to 3D output from these cameras 3A to 3n are not shown. It is given to the video input unit 6 in the video search device 5 via a cable.

  The video input unit 6 performs predetermined signal processing such as synchronization processing for synchronizing the camera video signals with respect to the camera video signals respectively given from the cameras 3A to 3n. Each video signal is sent to the video controller 7.

  The video control unit 7 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and is based on each camera video signal supplied from the video input unit 6 in the normal mode. Shooting video information (hereinafter referred to as camera video information) is sequentially stored in a recording medium 8 composed of, for example, a hard disk device or a semiconductor memory.

  Thereafter, the camera video information stored in the recording medium 8 is sequentially read out frame by frame by the subject extraction unit 9. Then, the subject extraction unit 9 determines whether or not the subject 2 exists in the frame image based on the read camera image information for one frame. For example, when the subject 2 is a person, the presence or absence of the subject 2 can be determined by a conventional method such as a background difference method or a skin color detection method. When the subject 2 is a “moving object” such as a person or a car, the presence / absence of the subject 2 can be determined by a conventional method such as a motion detection process that extracts a difference between images between frame images.

  When the subject extraction unit 9 determines that the subject 2 is present in the frame image based on the determination, the subject extraction unit 9 extracts and extracts a video portion of the subject 2 (hereinafter referred to as a subject video) from the frame image. Video information of the subject video is sent to the feature part extraction unit 10 and the multi-view video generation unit 11. Further, the multi-view video generation unit 11 sequentially stores video information of the subject video provided from the subject extraction unit 9 in the recording medium 8.

  The subject extraction unit 9 repeatedly executes the above processing for each camera image information stored in the recording medium 8 for each frame. As a result, video information of a plurality of subject videos as shown in FIGS. 4A to 4D obtained by photographing the same subject from multiple viewpoints at the same timing is sequentially stored in the recording medium 8. However, the subject extraction unit 9 may perform the above processing every several to several tens of frames. By doing so, it is possible to reduce the load on the subject extraction unit 9 and to reduce the amount of information of the video information of the subject video accumulated in the recording medium 8.

  At this time, the feature part extraction unit 10 is based on the video information of the subject video provided from the subject extraction unit 9, and a predetermined characteristic part in the corresponding subject video (hereinafter referred to as a feature part). The video part of is extracted. For example, when the subject is a person, it is drawn on the eyes, eyebrows, nose, mouth, ears, the entire face, worn clothing (clothes, hats, etc.) and ornaments (necklaces, earrings, etc.), bags, clothes Characters and the like correspond to characteristic parts.

  Then, the feature part extraction unit 10 uses the extracted video information of the feature part and management information for managing the feature part as the feature information, and the video information of the subject video from which the feature part is extracted and the corresponding camera video information Are stored in the characteristic information storage unit 8A in the recording medium 8 in association with each other.

  On the other hand, when the operation unit 13 is operated by the user and the subject search mode is selected by the user, the video control unit 7 receives camera video signals (hereinafter referred to as “this”) from one camera 3A to 3n selected by the user at that time. (Referred to as “selected camera video signal”) is supplied to the monitor 15 via the video output unit 14. Accordingly, a camera video screen 16 as shown in FIG. 6A is displayed on the monitor 15 based on the selected camera video signal.

  The camera video screen 16 is a screen in which a frame 16A is displayed on a camera video based on the selected camera video signal so as to surround any one subject existing in the camera video. On the camera video screen 16, the frame 16 </ b> A can be moved so as to surround other subjects existing on the camera video screen 16 by operating the operation unit 13.

  After that, when the operation unit 13 is operated by the user, the video control unit 7 then receives a search command for a subject surrounded by the frame body 16A (hereinafter referred to as a search target subject as appropriate). Then, the video information of the characteristic part of the search target subject extracted by the characteristic part extraction unit 9 is read from the characteristic information storage unit 8A.

  Further, the video control unit 7 is based on the video information of the feature part, and the feature estimated to be the subject to be searched from the video information of the feature part of each subject accumulated so far in the feature information storage unit 8A. A request to detect video information of a part by, for example, pattern matching processing and generate a multi-view video based on a subject video associated with the video information of the characteristic part (hereinafter referred to as a multi-view video generation request) Is sent to the multi-view video generation unit 11.

  The multi-view video generation unit 11 includes management information (hereinafter referred to as camera information) for each of the cameras 3A to 3n such as the installation positions and shooting directions of the cameras 3A to 3n, for example, from a hard disk device or a semiconductor memory. Is stored in advance in the camera information storage unit 12. When the multi-view video generation request is given from the video control unit 7 as described above, the multi-view video generation unit 11 stores the camera information of each camera 3A to 3n held in the camera information storage unit 12 and the recording medium 8. For example, a multi-view video of the search target subject as shown in FIGS. 5A and 5B is generated based on the video information of the subject video of the search target subject stored in FIG.

  This multi-viewpoint video is a three-dimensional model of a subject when the subject is viewed from a viewpoint position different from the viewpoint position of the two subject videos based on the two subject videos. For example, FIG. And (C), a multi-view video as shown in FIG. 5 (A) is generated based on the video information of each subject video, and the video information of each subject video shown in FIGS. 4 (A) and 4 (B). A multi-viewpoint video as shown in FIG. Note that the viewpoint position of the multi-view video can be changed by operating the operation unit 13 as will be described later, but the viewpoint position at the initial stage of generation is a predetermined position.

  Then, the multi-view video generation unit 11 stores the video information of the multi-view video of the search target object generated in this way in the recording medium 8.

  Further, the video control unit 7 selects from among the recorded video based on the camera video information associated with the video information of the characteristic part that is assumed to be the subject of the search object among the camera video information stored in the recording medium 8. Then, a request to detect a video portion in which the search target subject is shown (hereinafter referred to as a subject video part detection request) is sent to the feature part extraction unit 10.

  When this subject video part detection request is given, the feature part extraction unit 10 is associated with the video information of the specific part by pattern matching processing using the video information of the feature part presumed to be the subject to be searched. The video portion in which the subject to be searched is recorded in the recorded video based on the camera video information is detected, and the video control unit 7 is notified of the detection result.

  Thus, the video control unit 7 includes the video information of the subject video of the search subject detected as described above, the video information of the multi-view video generated by the multi-view video generation unit 11 based on the subject video, The camera video information of the video portion in which the subject to be searched in the recorded video based on the corresponding camera video information is read from the recording medium 8, and based on these, the search result display screen 17 as shown in FIG. Is generated. Then, the video control unit 7 sends the screen signal of the search result display screen 17 to the monitor 15 via the video output unit 14 to display the search result display screen 17 on the monitor 15.

In FIG. 6B , 17A and 17B are output from predetermined cameras 3A to 3n among the subject videos of the search target subject generated based on the camera video information output from the cameras 3A to 3n. 17C is a subject video generated based on the obtained camera video information, and 17C is a multi-view video of the search target subject. The viewpoint position of this multi-viewpoint image can be freely changed by operating the operation unit 13. Further, 17D is a recorded video (moving image) showing a search target subject displayed based on the camera video information read from the recording medium 8.

(1-2) Specific processing contents feature site extracting section 10, where the characteristic portion extracting unit on the characteristics site extracting process, on the basis of the image information of the subject image supplied from the subject extraction unit 9 as described above, the A template image table 20 as shown in FIG. 7 is held in an internal memory (not shown) as means for extracting a characteristic part from the subject video.

  The template image table 20 is a table for managing template images of feature parts to be extracted (characters drawn on eyes, eyebrows, nose, mouth, clothes, etc.). A plurality of pieces of image information of images are stored.

  For example, for the characteristic parts of “eyes”, “eyebrows”, “nose” and “mouth”, template images of a plurality of shapes are stored for “eyes”, “nose” and “mouth”, respectively, and are called “characters”. With respect to the characteristic portions, template images of characters to be extracted such as alphabets, numbers, hiragana and katakana are stored.

  When the video information of the subject video is given from the subject extraction unit 9, the feature part extraction unit 10 executes pattern matching processing using each template image registered in the template image table 20 for the subject video. Thus, a characteristic part is extracted from the subject video.

The feature site extracting section 10, when detecting the feature portion by such pattern matching process, extracting a video portion of the characteristic portion from the object image, features video information of the video part and the management information about the characteristic site The information is stored in the feature information management table 21 shown in FIG. 8 held in the information storage unit 8A.

  The feature information management table 21 manages video information of a feature part extracted from a subject video and management information of the feature part in association with a video file of the subject video and a video file of corresponding camera video information. This table includes a feature part video information column 21A, a management information column 21B, a subject video file ID column 21C, and a captured video file ID column 21D.

  The feature part video information column 21A stores the video information of the feature part extracted from the subject video by the feature part extraction unit 10, and the subject video file ID column 21C stores the video information of the subject video. The file ID of the video file is stored. The camera video file ID column 21D stores the file ID of the video file of the recorded video from which the subject video is extracted.

On the other hand, the management information column 21B includes an attribute column 21E, a recognized character column 21F, a time code column 21G, and the like. The attribute field 21E stores the corresponding feature part attributes (eyes, nose, mouth, characters, etc.), and the recognized character field 21F recognizes the feature part when the feature part attribute is a character. Stores characters. Further, the time code column 21G stores the time code of the original frame image from which the subject video from which the corresponding characteristic part is detected is extracted.

  FIG. 9 shows the specific processing contents of the feature part extraction unit 10 regarding the above-described feature part extraction processing (hereinafter referred to as feature part extraction processing). The feature part extraction unit 10 executes the feature part extraction process shown in FIG. 9 every time video information of a subject video is given from the subject extraction unit 9 according to a control program stored in an internal memory (not shown).

  That is, the feature part extraction unit 10 starts the feature part extraction process when the video information of the subject video is given from the subject extraction unit 9, and first selects one template image from the template image table 20 (FIG. 7) ( SP1).

  Next, the feature part extraction unit 10 raster-scans the template image on the subject image received at that time, and the pixel value of each pixel in the region where the template image on the subject image overlaps, and each pixel in the template image Are compared with each other (SP2).

  Then, when the raster scan as described above is completed on the subject video, the feature part extraction unit 10 determines whether or not there is an area on the subject video that has a pixel value equal to or greater than a certain number of pixels of the template image. (SP3).

  When the characteristic part extraction unit 10 obtains a positive result in this determination, the video information of the region of the subject video is used as feature information together with the management information, and the video ID of the video file of the subject video and the video of the corresponding recorded video The file is registered in the feature information management table 21 in the feature information storage unit 8A in association with the file ID of the file (SP4), and then the process proceeds to step SP7.

  On the other hand, if the characteristic part extraction unit 10 obtains a negative result in this determination, it determines whether the template image has been enlarged or reduced (SP5). That is, in the case of the present embodiment, the pattern matching process as described above is performed a plurality of times while gradually expanding the template image, and similarly performed a plurality of times while gradually reducing the template image. For this reason, if the characteristic part extraction part 10 obtains a negative result in determination of step SP5, it will enlarge or reduce the template image currently used (SP6), and will return to step SP2 after this.

  When the feature part extraction unit 10 obtains a positive result in the determination of step SP5 by ending all the pattern matching processes as described above, a template image that has not yet been subjected to the pattern matching process exists on the template image table 20. It is determined whether or not to perform (SP7).

  If the characteristic part extraction unit 10 obtains a positive result in this determination, it exchanges the template image with another template image stored in the template image table 20 (SP8), and then returns to step SP2. Thus, the feature part extraction unit 10 thereafter executes the processing after step SP2 in the same manner as described above.

  And the characteristic part extraction part 10 will complete | finish this characteristic part extraction process, if a negative result is obtained in step SP7 by finishing the same process about all the template images registered into the template image table 20 before long.

(1-3) Specific processing contents of multi-view video generation unit regarding multi-view video generation processing Next, multi-view video generation processing in the multi-view video generation unit 11 (hereinafter referred to as multi-view video generation processing). ) Will be described in detail. At this time, first, camera information of each of the cameras 3A to 3n held in the camera information storage unit 12 by the multi-view video generation unit 11 will be described.

  FIG. 10 shows specific contents of the camera information of the cameras 3A to 3n held in the camera information storage unit 12 by the multi-view video generation unit 11. As is clear from FIG. 10, each camera information is composed of position information, orientation information, field angle information, and zoom magnification information of the corresponding cameras 3A to 3n.

The position information is information indicating the world coordinates of the corresponding cameras 3A to 3n when the place where the cameras 3A to 3n are installed is a single space, and the direction information is the shooting direction of the cameras 3A to 3n ( (Optical axis direction) information. In FIG. _{10, θ n (n = 1,2,3} , ......) indicates a tilt direction of the angle formed by the optical axis horizontal plane of the camera _{3A~3n, φ n (n = 1,2,3} , ... ...) indicates an angle formed by the optical axis of the cameras 3A to 3n with a reference direction on a horizontal plane.

The field angle information is information representing the field angles of the cameras 3A to 3n. In FIG. 10, α _n (n = 1, 2, 3,...) Indicates the horizontal angle of view of the cameras 3A to 3n, and β _n (n = 1, 2, 3,...) Indicates the camera. The angle of view in the vertical direction of 3A to 3n is shown. Further, the zoom magnification information is information representing the zoom magnification of the cameras 3A to 3n.

  Among these camera information, position information and angle of view information are stored in the camera information storage unit 12 in advance by the user. For the orientation information, for example, by the video control unit 7 or the like based on the orientation information of the corresponding cameras 3A to 3n given to the video search device 5 from each holding mechanism (not shown) that holds each camera 3A to 3n. Updated automatically. Further, the zoom magnification information is automatically updated based on the current zoom magnification that is sequentially notified to the video search device 5 from each of the cameras 3A to 3n.

Next, with reference to FIG. 11, the subject viewed from an arbitrary viewpoint position based on the video information of the subject videos 32A and 32B extracted from the shot images of the two cameras 31A and 31B that shoot the same subject 30 A method for generating 30 multi-view images 33 will be described.

  In this case, first, corresponding points on the subject 30 in each of the subject images 32A and 32B are detected. The corresponding point can be detected by detecting the same characteristic part in the subject video 32A obtained by the photographing of one camera 31A in the subject video 32B obtained by the photographing of the other camera 31B. it can. Specifically, for example, the position of “left eye” in one subject video 32A as a reference and the position of “left eye” in the other subject video 31B are used as corresponding points. Similarly, several corresponding points on the subject 30 in each of the subject images 32A and 32B are detected.

Subsequently, based on the coordinates of these detected corresponding points,
A planar projection matrix H satisfying the above is determined.

  Next, using this planar projection matrix H, matrix conversion is performed on all the pixels in the one subject video 32A as a reference. As a result, it is possible to detect the pixels in the other subject image 32B corresponding to the pixels in the one subject image 32A.

  Thereafter, a ratio between the distance from one camera 31A to the viewpoint position of the multi-view video 33 to be generated at that time and the distance from the other camera 31B to the viewpoint position is obtained. This ratio can be obtained based on the position information of each camera 3 </ b> A to 3 n stored in the camera information storage unit 12.

Here, it is assumed that a ratio between the distance from one camera 31A to the viewpoint position of the multi-view video 33 and the distance from the other camera 31B to the viewpoint position is γ: (1-γ) (where 0 ≦ γ ≦). If the position vector P ₁ of one pixel on one subject video 32A and the position vector of a pixel (hereinafter referred to as coordinates) on the other subject video 32B corresponding to the pixel are P ₂ , The position vector of the coordinates of the corresponding point on the multi-view video 33 at the position is given by

Can be calculated.

Therefore, when such a ratio is obtained, the coordinates (a ₁ , b ₁ ) of _one pixel on one subject image 32A and the coordinates (a _{2) of the} corresponding pixel on the other subject image 32B are thereafter obtained. , B ₂ ), the coordinates (a ₁₂ , b ₁₂ ) of the corresponding point on the multi-view video 33 at the viewpoint position are obtained from the equation (2).

Thereafter, a multi-view video is obtained by using the pixel value at the coordinates (a ₁ , b ₁ ) on the one subject video 32A and the pixel value at the coordinates (a ₂ , b ₂ ) on the other subject video 32B. The pixel value of the pixel at the corresponding point on 33 is obtained by linear interpolation.

  The above processing is executed for all pixels on one subject image 32A. As a result, a multi-view video 33 at an arbitrary viewpoint position can be obtained based on the video information of the subject videos 32A and 32B extracted from the shot videos of the two cameras 31A and 31B that shoot the same subject 30.

  Based on the above principle, the multi-view video generation unit 11 generates the multi-view video of the search target subject by executing the multi-view video generation processing shown in FIG. 12 according to the control program stored in the internal memory (not shown). To do.

  Actually, when the above-described multi-view video generation request is given from the video control unit 7, the multi-view video generation unit 11 starts the multi-view video generation process shown in FIG. 12, and first generates a multi-view video. Two cameras 3A to 3n positioned closest to the default viewpoint position among the camera information of each camera 3A to 3n stored in the camera information storage unit 12 as two subject images used for the purpose. To select a subject video based on the given camera video signal (SP10).

  Subsequently, the multi-view video generation unit 11 detects a plurality of corresponding points on the two subject videos by the method described with reference to FIG. 11 (SP11), and based on the detection result, the multi-view video generation unit 11 determines from the default viewpoint position. A multi-view video of the viewed subject is generated (SP12). Then, the multi-view video generation unit 11 ends this multi-view video generation process.

  When the user gives an operation input to change the viewpoint position of the multi-view video 17C displayed on the search result display screen 17 described above with reference to FIG. A multi-view video generation request (multi-view video generation request) is given to the multi-view video generation unit 11 with the specified position as the viewpoint position.

  The multi-view video generation unit 11 generates a multi-view video in the same manner as described above even when this multi-view video generation request is given. In this case, the content of the multi-view video generation process is as follows. In step SP10, the subject video based on the camera video signals respectively given from the two cameras 3A to 3n located closest to the viewpoint position designated by the user is obtained. Same as above except for selection.

(1-4) Specific Processing Contents of Video Control Unit for Subject Search Processing On the other hand, the video control unit 7 stores in an internal memory (not shown) when the subject search mode is selected by operating the operation unit 13 by the user. The subject search process shown in FIG. 13 is executed on the basis of the control program.

  That is, when the subject search mode is selected, the video control unit 7 sends video signals (selected camera video signals) given from the cameras 3A to 3n designated by the user to the monitor 15 via the video output unit 14 described above. By sending out, the camera video screen 16 described above with reference to FIG. 6A based on the selected camera video signal is displayed on the monitor 15 (SP20).

  In addition to this, the video control unit 7 periodically inquires the subject extraction unit 9 as to whether or not the subject has been detected in the camera video based on the selected camera video signal, and the subject extraction unit 9 It waits to detect the subject (SP21).

  At this time, the subject extraction unit 9 performs the same processing as in the normal mode, and if it detects at least one subject in the camera video based on the selected camera video signal, notifies the video control unit 7 to that effect. Thereafter, the position of each subject existing in each frame image of the camera video based on the selected camera video signal is sequentially reported to the video control unit 7.

  Thus, when the video control unit 7 receives the notification that the subject is detected from the subject extraction unit 9, the frame image signal generated based on the position information of each subject that is sequentially given from the subject extraction unit 9 is selected camera video. It is superimposed on the signal (SP22). Thus, the frame 16A (FIG. 6A) is displayed so as to surround one subject existing in the camera video based on the selected camera video signal.

  Thereafter, the video control unit 7 waits for selection of a search target subject (search target subject) while moving the position of the frame 16A to a position surrounding another subject in accordance with a user operation (SP23). . When the search target object is selected by the user, the video control unit 7 reads out the video information of the characteristic part of the search target object detected by the characteristic part detection unit 10 at this time from the feature information storage unit 8A (SP24). ).

  Subsequently, based on the read video information of the characteristic part of the search target subject, the video control unit 7 estimates that the video information of the other characteristic part stored in the feature information storage unit 8A belongs to the search target subject. The image information of the feature part to be searched is retrieved by pattern matching processing (SP25).

  Next, the video control unit 7 determines whether or not video information of a characteristic part estimated to be the subject of the search target has been detected by the search process (SP26). If the video control unit 7 obtains a negative result in this determination, it displays a message on the monitor 15 that the subject video or the like of the subject to be searched cannot be detected (SP27), and thereafter ends this subject search processing. To do.

  On the other hand, if the video control unit 7 obtains a positive result in the determination at step SP26, the video ID of the video file of the subject video associated with the video information of the characteristic part detected at that time, and the video of the characteristic part The file ID of the video file of the recorded video associated with the information is read from the feature information management table 21 (FIG. 8) stored in the feature information storage unit 8A (SP28).

  In addition, the video control unit 7 sends a multi-view video generation request for generating a multi-view video using the subject video specified by the file ID acquired in step SP28 to the multi-view video generation unit 11 (SP29). ) Sends a subject video part detection request to the feature part extraction unit 10 to detect a video part in which the subject to be searched is shown in the video of the recorded video specified by the file ID acquired in step SP28 ( SP30).

  Then, the video control unit 7 detects the video information of the subject video, the video information of the multi-view video generated by the multi-view video generation unit 11 based on the subject video, and the feature part extraction unit 10. Further, the camera video information of the video portion in which the subject to be searched in the corresponding recorded video is shown is read from the recording medium 8 (SP31).

  Further, the video control unit 7 outputs the screen signal of the search result display screen 17 described above with reference to FIG. 6B based on the video information of the subject video and the multi-view video read from the recording medium 8 and the camera video information. The search result display screen 17 is displayed on the monitor 15 by generating and supplying this screen signal to the monitor 15 via the video output unit 14 (SP32). Then, the video controller 7 thereafter ends this subject search process.

(1-5) Effects of this Embodiment As described above, in the video surveillance system 1 according to this embodiment, the camera video based on the camera video signals based on the camera video signals given from the cameras 3A to 3n. The video information and management information of the characteristic part of each subject existing therein are acquired, and these are stored as feature information of the subject in the feature information storage unit 8A in association with the subject video and multi-view video of the subject.

  Further, in the video surveillance system 1, when a subject (search target subject) existing in the camera video based on the camera video signal from any one of the cameras 3A to 3n is designated thereafter, the feature information of the characteristic part of the subject is designated. Based on the feature information, the feature information corresponding to the search subject is searched from the feature information stored in the feature information storage unit 8A. Based on the search result, the subject video and the multi-view video of the subject are searched. The information and the video of the video portion in which the subject of the camera video stored in the recording medium 8 is shown on the monitor 15.

  Therefore, according to the video monitoring system 1, since the subject specified by the user can be searched based on the feature information of each subject acquired from the captured images of the cameras 3A to 3n, a desired subject can be selected from a vast amount of videos. Can be easily detected, and can be applied practically as a video surveillance system in a facility where an unspecified number of people come and go, such as hotels and department stores.

(2) Second Embodiment In FIG. 1, reference numeral 40 denotes a video surveillance system according to a second embodiment as a whole. In this video monitoring system 40, the video search device 41 displays the camera video screen 16 described above with reference to FIG. 6A and the search result display screen 17 described above with reference to FIG. This is different from the video monitoring system 1 according to the first embodiment.

  That is, in the case of the video monitoring system 40 according to the present embodiment, when the subject search mode is selected by a user operation, a search screen 42 as shown in FIG.

  In this search screen 42, a camera video based on video signals (selected camera video signals) given from the cameras 3A to 3n designated by the user is displayed in the camera video display column 42A and displayed in the camera video display column 42A. Subject images of all subjects existing in the camera image are listed in the subject list display field 42B.

  On the search screen 42, the user operates the operation unit 13 to select a subject image of a subject to be searched (subject to be searched) from subject images displayed in the subject list display field 42B. The video portion of the recorded video in which the subject video, multi-view video, and the search target subject can be displayed in the basic video display column 42C, the multi-view video display column 42D, and the recorded video display column 42E, respectively.

  FIG. 15 shows the specific processing contents of the video control unit 43 (FIG. 1) regarding the subject search processing according to the second embodiment including the display of the search screen 42 as described above. When the operation unit 13 is operated by the user and the subject search mode is selected, the video control unit 43 executes the subject search process shown in FIG. 15 based on a control program stored in an internal memory (not shown).

  That is, when the subject search mode is selected, the video control unit 43 starts this subject search process, and first sends a predetermined screen signal to the monitor 15 via the video output unit 14 to thereby describe the above-described FIG. The search screen 42 is displayed on the monitor 15 (SP40).

  In the search screen 42 at this stage, the camera video based on the selected camera video signal is displayed in the camera video display column 42A of the search screen 42. However, the subject list display column 42B, the basic video display column 42C, and the multi-viewpoint video are displayed. The corresponding video is not displayed in the display column 42D and the recorded video display column 42E.

  Subsequently, the video control unit 43 periodically inquires the subject extraction unit 9 as to whether or not the subject has been detected in the camera video based on the selected camera video signal, and the subject extraction unit 9 detects the subject in the camera video. (SP41).

  Then, when the video control unit 43 eventually receives a notification from the subject extraction unit 9 that the subject has been detected, the video control unit 43 then generates each subject detected by the subject extraction unit 9 and stored in the recording medium 8 at that time. The video information of the subject video is read from the recording medium 8, and the subject video based on the video information is displayed in a list in the subject list display field 42B (SP42).

  Subsequently, the video control unit 43 operates the operation unit 13 to select the subject video of the subject to be searched (search target subject) from the subject videos displayed in the subject list display field 42B of the search screen 42. When a search target subject is eventually selected, step SP44 to step SP51 are processed in the same manner as step SP24 to step SP31 of the subject search process according to the first embodiment described above with reference to FIG.

  Next, the video controller 43 displays the subject video, multi-view video information, and recorded video based on the video information, multi-view video information, and camera video information read from the recording medium 8 in step SP51, respectively, on the search screen 42 as the basic video display. Displayed in the column 42C, the multi-view video display column 42D, and the recorded video display column 42E (SP52), and thereafter, the subject search process is terminated.

  As described above, in the video monitoring system 40 according to the present embodiment, the camera video based on the selected video signal, the subject video of each subject in the camera video, and the search result are displayed in one search screen 42. Therefore, the visibility and operability of the search target can be improved as compared with the video monitoring system 1 according to the first embodiment.

(3) Third Embodiment In FIG. 1, reference numeral 50 denotes a video surveillance system according to a third embodiment as a whole. The video monitoring system 50 is different from the video monitoring system 1 according to the first embodiment in that the video search device 51 can search for a desired subject using a keyword.

  That is, in the video monitoring system 50 according to the present embodiment, when the subject search mode is selected, a keyword input dialog 52 as shown in FIG. 16 is displayed. Then, after the user operates the operation unit 13 to input a desired keyword into the keyword input field 52A of the keyword input dialog 52, the user searches for a subject corresponding to the keyword by performing an operation of pressing the search button 52B. Can be executed. As keywords used at this time, shooting date and time, characters, and the like are applied. The search result at this time is displayed on the monitor as a search screen 54 shown in FIG.

  In this search screen 54, the keyword specified by the user using the keyword input dialog 52 at that time is displayed in the keyword display column 54A, and subject videos of each subject detected based on the keyword are displayed in a list in the candidate list display column 54B. Is done.

  On the search screen 54, the user then operates the operation unit 13 to select a subject image of a search target subject (search target subject) from the subject images displayed in the candidate list display field 54B. As shown in FIG. 17 (B), the subject video of the search subject, the multi-view video, and the video portion of the recorded video in which the search subject is shown, respectively, are areas that were previously empty spaces in the search screen 54. Can be displayed.

  FIG. 18 shows the specific processing contents of the video control unit 53 related to the subject search processing according to the third embodiment including the display of the search screen 54 as described above. When the operation unit 13 is operated by the user and the subject search mode is selected, the video control unit 53 executes the subject search process shown in FIG. 18 based on a control program stored in an internal memory (not shown).

  That is, when the subject search mode is selected, the video control unit 54 starts this subject search process, and first sends a predetermined screen signal to the monitor 15 via the video output unit 14 to thereby describe the above-described FIG. The keyword input dialog 51 is displayed on the monitor 15 (SP60).

  Subsequently, the video control unit 53 waits for an operation to be performed by pressing the search button 52B after the keyword is input to the keyword input field 52A of the keyword input dialog 52 (SP61), and the keyword is input to the keyword input field 52A. When an operation for pressing the search button 52B is input, the corresponding entry on the feature information management table 21 (FIG. 8) is searched (SP62).

  For example, when the keyword is the shooting date and time, the video control unit 53 sequentially compares the time code stored in the time code column 21G of each entry of the feature part management table 21 (FIG. 8) with the keyword, and the time code. Finds all entries that match the keyword. For example, when the keyword is a character, the video control unit 53 sequentially compares the character and the keyword respectively stored in the recognition result column 21F of each entry whose attribute is “character” in the feature information management table 21; Find all entries where the character matches the keyword.

  When the video control unit 53 detects a corresponding entry by this search, the video control unit 53 reads the file ID stored in the subject video file ID column 21C (FIG. 8) of the entry, and the video file to which the file ID is assigned is read. File information (video information of the subject video) is read from the recording medium 8. In addition, the video control unit 53 displays a search screen 54 (FIG. 17) on which the subject video based on the file information read from the recording medium 8 in this way and the keywords input via the keyword input dialog 52 are displayed. It is displayed on the monitor 15 (SP63).

  Thereafter, when one subject video is selected from the subject videos displayed in the candidate list display field 54B of the search screen 54, the video control unit 53 according to the first embodiment described above with reference to FIG. By processing step SP65 to step SP72 in the same manner as step SP24 to step SP31 of the subject search process, the subject video of the subject (search target subject) of the subject video selected in step SP64 and each video information of the multi-viewpoint video Then, the video information of the captured video in which the subject is shown is read from the recording medium 8.

  Then, the video control unit 54 adds the video information of the subject video read from the recording medium 8 in step SP72, the video information of the multi-view video, and the camera video information of the video portion in which the search target subject of the corresponding recorded video is shown. Based on this, the subject video and multi-view video of the subject to be searched and the camera video in which the subject to be searched among the camera videos recorded on the recording medium 8 are displayed at predetermined positions on the search screen 55, respectively. (SP73) Thereafter, the subject search process is terminated.

  As described above, in the video monitoring system according to the present embodiment, a search target subject can be searched using a keyword. Therefore, a desired subject can be searched based on the shooting date and the characteristics of the subject. Therefore, the usability can be further improved compared to the video monitoring system 1 according to the first embodiment.

(4) Fourth Embodiment In FIG. 19, in which parts corresponding to those in FIG. 1 are assigned the same reference numerals, 60 denotes a video surveillance system according to a fourth embodiment as a whole. This video monitoring system 60 is different from the video monitoring system 1 according to the first embodiment in that a video information communication unit 62 is provided in the video search device 61.

  The video information communication unit 62 is an interface for performing communication with the external database 63, and the video control unit 64 can exchange information with the external database 63 via the video information communication unit 62. It is made like that.

  The external database 63 can store various kinds of information related to video search as described above.

  For example, in the case where an image list of criminals who have been appointed is stored in the external database 63, the video control unit 64 reads out the image list from the external database 63 via the video information communication unit 62, and stores the image list in the recording medium. Record in 8. The video control unit 64 controls the subject extraction unit 9 to extract a criminal image (hereinafter referred to as a criminal image) from each image registered in the image list recorded on the recording medium 8. At the same time, the feature part extraction unit 10 is controlled to extract and collect feature information (image information of the feature part and its management information) from the criminal image.

  Then, the video control unit 64 patterns the video information of the characteristic part of the criminal image extracted at this time and the video information of each entry of the feature information management table 21 (FIG. 8) stored in the feature information storage unit 8A. Comparison is made by the matching process, and if there is a matching entry, that fact is displayed on the monitor 15.

  As described above, in the video monitoring system 60 according to the present embodiment, it is possible to search and collate using both the feature information of each subject accumulated in the feature information storage unit 8A and the data stored in the external database 63. Therefore, usability can be further improved compared to the video monitoring system 1 according to the first embodiment.

(5) Other Embodiments In the first to fourth embodiments described above, the case where the video surveillance systems 1, 40, 50, 60 are configured as shown in FIG. 1 or FIG. 19 will be described. However, the present invention is not limited to this, and various other configurations can be widely applied.

  In the first to fourth embodiments described above, the case where the feature information storage unit 8A is configured as a part of the recording medium 8 has been described. However, the present invention is not limited to this, and the feature information storage is performed. The unit 8A and the recording medium 8 may be configured separately.

  Further, in the first to fourth embodiments described above, the search result display screen 17 (FIG. 6B) and the search screens 42 and 54 (FIGS. 14 and 17) show the search target subject as the search results. Although the description has been given of the case where all of the video portions in which the subject to be searched is displayed among the subject video, the multi-view video, and the camera video recorded on the recording medium 8, the present invention is not limited to this. In the search result display screen 17 and the search screens 42 and 54, as a search result, a part of the subject video of the search target subject, the multi-view video, and the camera video recorded on the recording medium 8 may be displayed. good.

  Furthermore, in the first to fourth embodiments described above, a case where the video monitoring systems 1, 40, 50, 60 store the camera video information based on the cameras 3A-3n as they are in the recording medium 8 will be described. However, the present invention is not limited to this. For example, the feature information extraction unit 10 is recorded on the recording medium 8 as shown in FIG. 20 based on the feature information of each subject stored in the feature information storage unit 8A. The recorded video based on the camera video information may be divided into video portions for each subject, or processing such as deleting a video portion where the subject is not shown may be performed at an arbitrary timing. By doing so, the processing speed when displaying the search results on the search result display screen 17 and the search screens 42 and 54 can be improved, and the information amount of the video information to be recorded on the recording medium 8 can be reduced. Can be reduced.

  Further, in the first to fourth embodiments described above, the case has been described in which the multi-view video generation unit 11 generates a multi-view video after a user specifies a search target subject. However, the present invention is not limited to this. For example, the video information of the subject video given from the subject extraction unit 9 may be stored in the recording medium 8 and the multi-view video may be generated using the video information immediately after the video information is stored. Various other timings can be applied as the timing at which the generation unit 11 generates the multi-viewpoint video.

  Furthermore, in the above-described first to fourth embodiments, search is performed from the feature information stored in the feature information storage unit 10 based on the feature information of the subject (detection subject) designated as the search target. Although the case where the video control unit for searching for the feature information corresponding to the target subject is provided in the video control unit 7, 43, 53, 64 has been described, the present invention is not limited to this, and the function is provided. The functional blocks may be provided separately from the video control units 7, 43, 53, and 64.

It is a block diagram which shows the whole structure of the video surveillance system by the 1st-3rd embodiment. It is a basic diagram which shows an example of the installation position of a camera. (A)-(D) is a basic diagram which shows the picked-up image of each camera of FIG. (A)-(D) is a basic diagram which shows the to-be-photographed image based on the picked-up image of each camera of FIG. (A)-(B) is a basic diagram which shows the example of preparation of the multiview video based on the to-be-photographed image of FIG. (A) is a camera video screen, and (B) is a schematic diagram showing a configuration example of a search result display screen. It is a conceptual diagram with which it uses for description of a template image table. It is a conceptual diagram with which it uses for description of the characteristic information management table. It is a flowchart which shows the process sequence of a characteristic part extraction process. It is a graph which shows the structural example of camera information. It is a conceptual diagram with which it uses for description of the production | generation method of a multiview video. It is a flowchart which shows the process sequence of a multiview video production | generation process. It is a flowchart which shows the process sequence of the subject search process by 1st Embodiment. It is a basic diagram which shows the structure of the search screen by 2nd Embodiment. It is a flowchart which shows the process sequence of the subject search process by 2nd Embodiment. It is a basic diagram which shows the structural example of a keyword input dialog. It is a basic diagram which shows the structure of the search screen by 3rd Embodiment. 15 is a flowchart illustrating a processing procedure of subject search processing according to the third embodiment. It is a block diagram which shows the whole structure of the video surveillance system by 4th Embodiment. It is a conceptual diagram with which it uses for description of other embodiment.

Explanation of symbols

  1, 40, 50, 60 ... Video surveillance system, 2 ... Subject, 3, 3A-3n ... Camera, 5, 41, 51, 61 ... Video search device, 7, 43, 53, 64 ... Video Control unit, 8... Recording medium, 8 A. Feature information storage unit, 9... Subject extraction unit, 10... Feature part extraction unit, 11. ...... Operating unit, 15 ... monitor, 16 ... camera video screen, 17 ... search result display screen, 20 ... template image table, 21 ... feature information management table, 42, 54 ... search screen, 52 ... ... Keyword input dialog, 62 ... Video information communication unit, 63 ... External database.

Claims (12)

A subject extraction unit that extracts each subject existing in the video based on the first video information given from outside, and records the extracted video information of the subject on a recording medium;
A feature part extraction unit that extracts a feature part of each subject extracted by the subject extraction unit;
A storage unit for storing the feature information for each subject, which is information about the feature portion of each subject extracted by the feature portion extraction unit, in association with the video information of each subject recorded on the recording medium;
An output unit that outputs a subject extracted from the first video information to a display unit;
A designation unit for designating a subject to be searched from among the subjects displayed on the display unit based on a user operation;
Based on the subject specified as the search target, the feature information corresponding to the subject specified as the search target is searched from the feature information stored in the storage unit, and the feature information obtained as a search result A video search unit comprising: a video search unit that reads video information of a subject associated with the video from the recording medium and outputs the video information by the output unit. The video search unit
The video search device according to claim 1, wherein the video information of the subject associated with the feature information detected by the search is read from the recording medium, and a video based on the video information is displayed as a search result. . The first video information is
Consists of a plurality of second video information obtained by shooting the same subject from multiple viewpoints,
The subject extraction unit
For each of the second video information, extract each subject present in the video based on the second video information, record the extracted video information of the subject on the recording medium,
A multi-view video generation unit that generates a multi-view video composed of a three-dimensional model of the subject based on video information of the same subject existing in the video based on the second video information;
The video search unit
The video search apparatus according to claim 2, wherein the multi-view video generated by the multi-view video generation unit is displayed as a search result based on video information of a subject specified as the search target. The feature information of the subject includes video information of the feature part of the subject,
The video search unit
Based on the feature information of the subject specified as the search target, the feature information corresponding to the subject specified as the search target among the feature information stored in the storage unit is converted into a video pattern matching process. The video search device according to claim 1, wherein the video search device is searched for by the following. The video search unit
Recording the first video information on the recording medium;
The feature part extraction unit includes:
The video search according to claim 1, wherein the first video information recorded on the recording medium is divided into video parts for each subject based on the feature information stored in the storage unit. apparatus. The feature part extraction unit includes:
6. The video search apparatus according to claim 5 , wherein video information of a video portion in which any subject is not captured is deleted from the first video information recorded on the recording medium. Extracting each subject existing in the video based on the first video information given from outside, recording the extracted video information of the subject on a recording medium, and extracting the characteristic part of each extracted subject A first step of storing feature information for each subject, which is information relating to the extracted feature part of each subject, in association with video information of each subject recorded on the recording medium;
The subject extracted from the first video information is output to the display unit, and the subject to be searched is designated from among the subjects displayed on the display unit based on the user's operation . Based on the subject, the feature information corresponding to the subject specified as the search target is searched from the stored feature information, and the video information associated with the feature information detected by the search is stored in the recording medium. image retrieval method characterized by comprising a second step of outputting by the display unit to read out from. In the second step,
The video search method according to claim 7 , wherein the video information of the subject associated with the feature information detected by the search is read from the recording medium, and a video based on the video information is displayed as a search result. . The first video information is
Consists of a plurality of second video information obtained by shooting the same subject from multiple viewpoints,
In the first step,
For each of the second video information, extract each subject present in the video based on the second video information, record the extracted video information of the subject on the recording medium,
In the second step,
Displaying, as a search result of the search, a multi-view video that is generated based on video information of the same subject existing in each video based on the second video information and is a three-dimensional model of the subject. The video search method according to claim 8 . The feature information of the subject includes video information of the feature part of the subject,
In the second step,
Based on the feature information of the subject specified as the search target, the feature information corresponding to the subject specified as the search target is searched from the stored feature information by video pattern matching processing. The video search method according to claim 7 , wherein the video search method is characterized. In the first step,
Recording the first video information on the recording medium;
The video search method according to claim 7 , wherein the first video information recorded on the recording medium is divided into video parts for each subject based on the stored feature information. In the first step,
The video search method according to claim 11 , wherein video information of a video portion in which any subject is not captured is deleted from the first video information recorded on the recording medium.