JP6413134B2 - In-video activity visualization device, method and program - Google Patents

Description

  The present invention relates to an in-video activity level visualization apparatus, method, and program.

  Cameras are installed everywhere in the world, and the amount of video that humans cannot handle has been accumulated. The accuracy of automatic recognition by image processing is limited, and there are still many scenes where humans understand images. When analyzing images with traditional tools, you can assign fast-forward and rewind functions to shortcut keys, or use functions such as double-speed playback. There is a limit to the amount of video that one person can analyze, and the burden on humans is also great. There is a risk of missing an important event.

  As a conventional technique, there is a technique of Video Visualization that translates video into a form visible to human eyes. For example, in Non-Patent Document 1, the volume data shown in a donut shape can be used to color and display important positions in the video, and visualization of breaks in news video and the flow of people in the monitoring video is possible. It becomes possible. If this type of system is used, it is possible to understand the contents of the video without seeing the video, but a lot of experience and experience are required to interpret the visualized result.

  Non-Patent Document 2 discloses a technique for limiting the image to the image from the ceiling, dividing the entire room into blocks without gaps, and expressing the activity in the image of each block with a three-dimensional volume. The goal of this technology is to express activities only from the volume, but this limits the information that can be obtained by reviewing the video. For example, it is difficult to discriminate whether a gesture such as scratching the face or sleeping is based only on the generated three-dimensional volume.

G. Daniel and M. Chen. Video visualization. In IEEE Visualization, 409-416, 2003.

M. Romero, J. W. Summet, J. T. Stasko, and G. D. Abowd.Viz-a-Vis: Toward visualizing video through computer vision.IEEE Transactions onVisualization and Computer Graphics, 14 (6): 1261-1268, 2008.

  In the prior arts disclosed in Non-Patent Documents 1 and 2 described above, video is handled as volume data and the content is visualized and viewed from above, but learning is necessary to grasp the content. Or there is a problem that the display is insufficient to understand the content.

  An in-video activity visualization apparatus according to the present invention includes a display screen control unit that controls a display screen of a display, a video control unit that acquires video data and displays a video on a part of the display screen, A panel control unit that sets a region of interest as a panel; an activity calculation unit that calculates an activity level in the panel; and a seek bar control unit that displays the activity level in the panel with respect to an elapsed time of an image as a seek bar. It is characterized by providing.

  Also, the method for visualizing activity in a video according to the present invention includes a step of obtaining video data and displaying the video by the video data on a part of a display, and a designated signal from the outside in the displayed video. Setting a region as a panel, calculating an activity indicating a temporal change of an image in the panel, and displaying the activity in the panel with respect to an elapsed time of the image as a seek bar. It is characterized by providing.

  The program for visualizing the activity level in a video according to the present invention is a procedure for acquiring video data in a computer and displaying the video by the video data on a part of a display, and a signal from the outside in the displayed video. A procedure for setting the area designated by the above as a panel, a procedure for calculating an activity indicating a temporal change of the image in the panel, and the activity in the panel with respect to the elapsed time of the image are displayed as a seek bar. Is a program for executing the procedure.

  According to the present invention, it is possible to visualize the activity levels of a plurality of specific objects in a video and efficiently check the content of the video. Alternatively, it is possible to improve the efficiency of checking events that have occurred in the video.

These are the block diagrams of the activity visualization apparatus in the image | video by this invention. These are examples of the display screen of the in-video activity visualization device according to the present invention. These are figures which show the flow of the step of the whole process of the activity visualization method in the image | video of this invention. These are the figures which show the flow of the step of the process regarding the seek bar production | generation of the activity visualization method in a video of this invention. These are figures which show the flow of the step of the process regarding video browsing at the time of using the activity level visualization method in the present invention. These are figures for demonstrating Example 1 of this invention, and are figures which show the example of the display screen of a display in the case of examining the lesson content using an archive image | video. These are figures which show the position of the pixel in which a brightness | luminance difference exceeds a threshold value between adjacent frames in Example 1 of this invention. These are the figures for demonstrating Example 2 of this invention, and are the figures which show the example of the display screen of a display in the case of performing operation | movement observation of the laboratory animal in a gauge. These are the figures for demonstrating Example 3 of this invention, and are the figures which show the example of the display screen of a display in the case of observing a surveillance camera image | video.

  Hereinafter, embodiments of the present invention will be described in detail.

  FIG. 1 shows a configuration diagram of the in-video activity visualization device according to the present invention. An example of the display screen of the display displayed by the in-video activity visualization device according to the present invention is shown in FIG. Hereinafter, a detailed description will be given with reference to FIGS. 1 and 2.

  The display screen control unit receives an external instruction via the input device and controls the display screen of the display. The input device is a device that can specify a position of an image and give a predetermined instruction, and may be a normally used mouse, a tablet input device provided with a touch pen, a touch panel provided in a smartphone, or the like. The display may be a liquid crystal monitor or projector used in a normal computer.

  The video control unit acquires the stored video data and displays the video on a part of the display screen 1 of the display via the display screen control unit. The video data may be already recorded, or may be video data captured in real time from a fixed camera installed. The video data may include signals other than video and audio data synchronized with the video.

  The panel control unit sets an area designated by an external signal via the input device as an area to be noted in the video area 2 displayed on the display screen 1 of the display, and sets it as the panel 3. When the input device is a mouse, a rectangular area specified by dragging between two vertices positioned diagonally can be set as a panel. In the case of using a touch panel, a rectangular region having two vertices at positions where two points touching at the same time are diagonal can be used as a panel. Panel 3 is displayed on the screen as a frame indicating the outer periphery of the panel.

  The panel control unit can also display the metadata 5 corresponding to the panel 3 set as necessary when there is data prepared in advance. Metadata 5 is data corresponding to the panel, and indicates a name indicating an individual panel and other attributes.

  The panel control unit has a function of sending video data corresponding to the set panel to the activity calculation unit.

  The activity level calculation unit analyzes the transmitted in-panel image data, calculates an activity level indicating the degree of change of the image, and transfers the calculated activity level to the seek bar control unit. It is also transferred to the power gauge control unit as necessary.

  The degree of activity is a pixel in which the difference in the characteristic value of all pixels of a certain video frame in the video data in the panel exceeds a predetermined threshold value compared to the corresponding pixel of an adjacent video frame As a percentage of The characteristic value of the pixel can be various, such as brightness and brightness, or an angle difference representing a hue in a color space with a specific color. Adjacent video frames can be the immediately preceding or immediately following video frames, but a plurality of consecutive video frames are taken as one video frame group, and the average value of the corresponding pixels in the group is calculated, It is also possible to use the difference between the average values of the corresponding pixels of the group to be the difference in the characteristic value.

  The seek bar control unit displays the transferred activity as the seek bar 6 on a part of the display screen of the display as the activity of the corresponding panel. The seek bar 6 is a bar-like display whose length represents the time axis of the video, and indicates the activity in the color corresponding to the elapsed time of the video. For example, as in a thermograph, a red color can be displayed with high activity, and a blue color can be displayed with low activity.

  The seek bar 6 also has a function of specifying the time of the video to be played back. The seek bar control unit reproduces the video from the time corresponding to one point on the seek bar 6 selected by the input device. By selecting the vicinity of the point of time when the degree of activity is high, it is possible to efficiently reproduce the video of the part where the degree of activity is high. Further, the panel 3 corresponding to the selected seek bar 6 can be highlighted and displayed by changing the color or thickness of the frame or blinking via the panel control unit.

  The power gauge control unit displays the activity transferred from the activity calculation unit as a power gauge 4 in the vicinity of the panel 3. The power gauge 4 displays the activity corresponding to the time of the displayed video in real time, for example, by a rectangle having a length in the vertical direction. The elevation of the gauge may be displayed with the difference in the panel highlighted. In this case, there is an effect of amplifying a small movement in the panel.

  When the screen becomes complicated, it is possible not to display the power gauge, and the power gauge control unit need not be provided. Further, the presence or absence of the power gauge display, the length of the power gauge, and the degree of amplification may be selected as necessary.

  The seek bar control unit can also display a seek bar 7 for a group of a plurality of panels 3 selected in advance. The activity level in this case is calculated for the video area included in any of the selected plurality of panels. When one point on the seek bar 7 is selected, it is possible to reproduce a video from that point and highlight a panel belonging to the selected group.

  Further, the seek bar control unit can display one seek bar 8 corresponding to all the panels. When one point on the seek bar 8 is selected, it is possible to reproduce the video from that point and highlight all the panels.

  Further, the seek bar control unit can set a similar index of activity using a signal synchronized with video such as audio data, and display it as the seek bar 9. When the signal is a signal depending on any one of the panels 3, when one point on the seek bar 9 is selected, the video is reproduced from that point and the corresponding panel can be highlighted.

  Further, when the video is being reproduced, the time for the currently displayed video is displayed on the time display 10 on the screen.

  The various control units and activity calculation units described above may be realized using a normal computer or may be realized as a dedicated processing device.

  FIG. 3 shows a flow of steps of the entire processing of the method for visualizing activity in a video according to the present invention.

  First, in the first step, target video data is acquired, and video is displayed on a part of the display. At this time, the initial image screen or the representative screen may be displayed as a still image.

  In the next step, the region of interest in the video is set as a panel by external designation. If the video is known or can be predicted, a pre-registered panel can be set. In the subsequent step, if there is metadata belonging to the panel, that data is captured and displayed on the screen.

  In the next step, the video data in the panel is analyzed, and the activity indicating the degree of change of the video is calculated from the difference in the characteristic value of the corresponding pixel between the video frames.

  In the next step, the activity with respect to the elapsed time of the video is displayed as a seek bar. The shape of the seek bar is preferably a rod, but depending on the case, any shape may be used as long as it shows a time axis such as a circumferential shape, a spiral shape, or a polygonal shape. In addition, a seek bar representing a panel group and all panels, and a seek bar for audio and other signals are displayed as necessary.

  An example of steps for generating a detailed seek bar is shown in FIG. First, by analyzing the video frames in the panel to be noted, the difference between successive video frames is calculated to calculate the activity. When a panel group is designated, the activity level for the panel group and the activity level of the entire panel are also calculated. Based on the obtained activity, a seek bar for an individual panel, a seek bar for a panel group, and a seek bar for the entire panel are generated and displayed.

  When there is audio data synchronized with video data, a seek bar similar to the activity level can be generated and displayed using sound pressure. When there are a plurality of sound sources, a seek bar for the sound pressure group and a seek bar for the entire sound pressure can be generated and displayed together with a seek bar for each sound pressure.

  In the next step, a power gauge indicating the activity is displayed near the panel as necessary. The video analysis and necessary screen display settings are thus completed.

  When actually viewing the image, grasp the time when the activity is high by the color displayed on the seek bar corresponding to the panel you want to pay attention to, and select the time slightly before the time when the activity is high. Reproduce. If the event that is the cause of the increase in the activity level can be grasped by the image, the other time when the activity degree is high is grasped, and the image is similarly viewed.

  FIG. 6 shows an example of steps when viewing a video. After the first video frame is played back, the video frames are played back in sequence until a point on the seek bar is selected by clicking. When one point on any seek bar is selected, the corresponding frame is reproduced, and the panel corresponding to the selected seek bar is highlighted, and the subsequent frames are subsequently reproduced.

  Examples of the present invention will be described below.

  FIG. 6 shows an example of the display screen when the lesson content is examined using the archive video. In the conventional class content examination, other teachers actually visit the lecture and discuss the lecture later. By evaluating lessons for videos taken in the lecture, teachers who did not actually participate in the lecture can discuss.

  FIG. 6 shows an example of using a video of a class scene taken from the front of the classroom with a fixed camera. Panel 3 selects the face of the student taking the class as an area of note. In such a case, assuming that the student's seat is determined, the position of the image is known, so the panel can be set in advance. On the right side of the panel 3, a power gauge 4 is displayed. Reference numeral 5 is a display showing the metadata attached to the selected panel, and displays preset information such as the name and grade of the student corresponding to the selected panel.

  There is a seek bar at the bottom of the video. 6 is a seek bar corresponding to each panel, and the number of seek bars equal to the number of panels is displayed. 7 is a seek bar for the panel group. For example, a panel group for each gender or group is set and generated and displayed. No. 8 is a seek bar for all panels, indicating the movement of the whole class.

9 is a seek bar for sound, and in this example, it is based on the sound pressure of only one sound source.
Reference numeral 10 indicates the time point of the video being reproduced.

  FIG. 7 is a diagram illustrating pixel positions where the luminance difference between adjacent frames exceeds a threshold value as an example of calculating the activity in the panel. FIG. 7 (a) is an image including a plurality of panels, and here is a case where a point in time when the activity level of the seek bar corresponding to the panel 02 is large is selected. FIG. 7 (b) shows pixels in which the luminance of the corresponding pixel in the adjacent video frame is higher than a certain value in white. In this panel 02, since the face has moved, it can be seen that the luminance difference is large at the pixel at the position corresponding to the outline of the face, and is expressed in white.

  An example of calculating the activity level is shown below. The area indicated by the panel i is Ri, and the pixel value at the position (x, y) of the image of the t-th frame is I (x, y, t). Whether there is a time difference at each position can be expressed as a function of f (x, y, t) below. However, Ith is an arbitrary threshold value for determining the presence or absence of a difference. For example, the pixel value I may be luminance and Ith = 0.1. As the pixel value I, components other than luminance and multidimensional vectors can be used depending on the purpose.

  If it is known whether there is a time difference at each position, the activity vi (t) for the panel i can be defined by the following equation.

  For signals synchronized with video such as sound pressure and biological signals, almost the same formulation is possible. Regarding sound pressure, given a volume A (d, t) in a t frame for a certain direction d, a range of directions Qj for a certain target j can be defined. For example, d may be the direction of the student, and Qj may be a group of microphones directed to the student j. The following g (d, t) can be used to indicate whether the sound volume has changed over time in a certain direction. However, Ath is an arbitrary threshold value for determining the presence or absence of a volume difference. A can be a simple sound pressure, or can be replaced with a vector indicating the sound pressure for each frequency according to the purpose.

  If it is known whether there is a time difference at each position, the time difference amount aj (t) for the target j can be defined by the following equation.

  It is assumed that a time difference vi (t) and a sound pressure time difference aj (t) in the panel are obtained for the entire video composed of N frames. Consider generating a seek bar of width W and height H from now on.

The width of the seek bar should be similar to the video, and usually stays on the order of several hundred pixels. On the other hand, video consists of hundreds of thousands of frames per hour, and N may be more than 1000 times W. Therefore, simply placing vi (t) on the seek bar cannot express changes that occur within one second. Therefore, the frame t∈ [Nx / W, corresponding to the sample point x∈ [0, W-1] on the seek bar
The maximum value among the time differences generated by N (x + 1) / W) is arranged at the sample point. However, H / 2 indicates the upper and lower center on the seek bar.

  For the top and bottom on the seek bar, the value can be gradually reduced so that the continuity of the difference amount is visualized.

  Furthermore, the maximum value of activity in all time is obtained for each panel so that the activity is normalized in all time.

  The whole is normalized with this value to obtain the final pixel value.

  If the a priori maximum value is known, an arbitrary constant pcoeff can be used instead of pmax, and an online seek bar can be generated. In addition, a unified activity evaluation can be performed on all panels.

  A seek bar for the entire group can be generated in the same manner. For example, the region Rg1 for the group g1 is obtained as the sum of the regions Rg1-1, Rg1-2,... Occupied by individuals forming the group g1.

The activity level of the group / whole can be calculated in the same manner as in the case of an individual. Similarly for the whole region Rall, all groups Rg1, Rg2, ...
As the sum of

  A seek bar can be generated in the same manner for the volume.

  The use of the present invention for the purpose of studying lesson contents has the following advantages.

  First, you can observe the activity of not only individuals but also groups and the whole. Efficient video analysis can be performed by focusing on places with high activity and on places with low activity.

  Second, it does not require a strict shooting environment. There is also no need to share settings between different videos. By simply taking a lecture video with a home camera installed on a tripod, it can be used as a useful material for studying the content of the lesson.

  Next, individual activities can be observed for each student even for images with distortion, such as images from an omnidirectional camera. By setting a panel on the video, the target intended by the video provider can be noted.

  Finally, if metadata for a panel is prepared, the corresponding metadata can be displayed when a certain panel is selected. For example, the student name corresponding to the panel and the grade of the past report assignment can be displayed. In order to visualize the state of the class online, being able to list the students in front of you and their grades is useful for efficient lectures.

  Hereinafter, other embodiments of the present invention will be described.

  FIG. 8 is an example used for observing the motion of an experimental animal in the gauge. A panel is set for the water squeezer for the video of observing the mouse at a fixed point. The time when the mouse drank water is represented on the seek bar.

  Similarly, the movement of an animal can be grasped by setting a panel at a specific location in the gauge. In addition, when a biological signal is always embedded by separately embedding a biological sensor, it is possible to acquire a correlation between the behavior and the biological signal by displaying a seek bar related to the biological signal.

  In the following, another embodiment of the present invention will be described.

  FIG. 9 is an example applied to observation of surveillance camera video. By setting a panel even for fixed-point observation images that look down from a distance where the human detector does not work effectively, the time when a person flows into the panel can be visualized. In a present Example, the mode of the person who goes in and out of a park can be efficiently observed by setting a panel at a park entrance.

  According to the method, program and apparatus for visualizing activity in the image according to the present invention, it is possible to not only increase the examination efficiency by using it when examining lesson contents, but also to easily observe the behavior of experimental animals, Monitoring of surveillance cameras can be performed efficiently. Furthermore, it can be widely applied to applications such as education, research, security, and manufacturing industries, such as monitoring of objects that require long-term observation, analysis of fixed-point observation images, monitoring of manufacturing equipment in the manufacturing process, and appearance inspection of products. Is possible.

DESCRIPTION OF SYMBOLS 1 Display screen 2 Image | video area 3 Panel 4 Power gauge 5 Metadata 6 The seek bar corresponding to each panel 7 The seek bar corresponding to a panel group 8 The seek bar corresponding to all the panels 9 The seek bar corresponding to sound pressure 10 Time in playback

Claims (17)

A display screen control unit for controlling the display screen of the display;
A video control unit for acquiring video data and displaying the video on a part of the display screen;
A panel control unit for setting a region of interest in the video as a panel;
An activity level calculation unit for calculating an activity level in the panel;
A seek bar control unit for displaying the activity level in the panel with respect to an elapsed time of the video as a seek bar;
With
The seek bar control unit
The activity indicating the temporal change of the images in the plurality of panels is displayed as a seek bar,
The seek bar is displayed side by side along the time axis of the images in the plurality of panels ,
The intra- video activity visualization apparatus, further comprising: displaying a degree of activity with respect to a group indicating a temporal change of a group of pre-selected panels in the plurality of panels as a seek bar . Each of the activities indicating temporal changes of images in the plurality of panels is
The in-video activity visualization device according to claim 1, wherein each panel is normalized by a fixed value set in advance and converted into a pixel value and displayed.   3. The in-video activity level visualization apparatus according to claim 2, wherein the activity level for the group is displayed on the seek bar as a logical sum of the activity levels of a plurality of panels selected in advance. The difference in characteristic value between the activity level and the corresponding pixel of the immediately preceding or immediately following video frame among all the pixels of one video frame in the panel is a predetermined threshold value. The in-video activity visualization device according to any one of claims 1 to 3 , wherein a ratio of pixels exceeding the threshold is set. Of the panel, the activity for the video frame of the video being displayed, the image in the activities of any one of claims 1 to 4, further comprising a power gauge control unit for displaying in real time Degree visualization device. The scrubber, the video according to activity of the video frames in some point within the panel, the color pixels relative to the position indicating the time, any one of claims 1 to 5, characterized in that the display Internal activity visualization device. When the number of pixels in the time direction of the seek bar is smaller than the total number of video frames of the video, a plurality of video frames are assigned to one pixel in the time direction of the seek bar and assigned to one of the pixels. The in-video activity visualization device according to claim 6 , wherein a maximum value among the activity levels of the plurality of video frames is displayed in a pixel color. 8. The display device according to claim 6, wherein: by designating an arbitrary position of the seek bar, an image is reproduced from a corresponding time point, and a panel corresponding to the designated seek bar is highlighted. In-picture activity visualization device. The seek bar control unit
Other than the video, the video in the activity visualization apparatus according to any one of claims 1 to 8, characterized in that further displays the activity showing the temporal change of the video and sync signal as the scrubber. 10. The intra-video activity visualization device according to claim 9 , wherein the signal other than the video is a voice or a biological signal. The said image | video is an image | video which image | photographed lesson content, The said panel is set for every student and it is used in order to grasp | ascertain the operation | movement for every said student, Any one of Claim 1 thru | or 10 characterized by the above-mentioned. The in-video activity level visualization device described in Crab. The video is a video of the motion of an experimental animal in the gauge, and the panel is set for a specific location in the gauge and used to grasp the behavior of the animal at each specific location. The in-video activity visualization device according to any one of claims 1 to 10 , wherein: The video is an image of a person's behavior in a specific place, and the panel is set for a more limited location in the specific location, and the flow and behavior of the person in each limited location. The in-video activity visualization device according to any one of claims 1 to 10 , wherein the device is used for grasping. Obtaining video data and displaying the video with the video data on a part of a display;
A plurality of areas designated by external signals in the displayed video, each set as a panel;
Calculating each of the activities indicating temporal changes of the images in the panel;
Displaying each activity in the panel relative to the elapsed time of the video as a seek bar;
With
In the displaying step,
Activity levels indicating temporal changes in the images in the panel are displayed as seek bars,
The seek bar is displayed side by side along the time axis of the images in the plurality of panels ,
The intra- video activity visualization method, further comprising: displaying an activity level for the group as a seek bar that indicates temporal changes in the video group of the plurality of panels selected in advance in the plurality of panels . Each of the activities indicating temporal changes of images in the plurality of panels is
The in-video activity visualization method according to claim 14, wherein each panel is normalized by a fixed value set in advance and converted into a pixel value and displayed. On the computer,
A procedure for acquiring video data and displaying the video with the video data on a part of a display;
In the displayed video, a procedure for setting each of a plurality of areas designated by an external signal as a panel;
A procedure for calculating each activity indicating a temporal change of the image in the panel;
Displaying each of the activities in the panel relative to the elapsed time of the video as a seek bar;
In the displaying procedure,
Activity levels indicating temporal changes in the images in the plurality of panels are displayed as seek bars,
The seek bar is
A procedure for displaying the images in the panel side by side in alignment with the time axis;
Further , a program for visualizing the activity in the image for executing the procedure of displaying the activity for the group as a seek bar, showing temporal changes in the image group of the plurality of panels selected in advance in the plurality of panels. . Each of the activities indicating the temporal change of the image in the panel,
The in-video activity visualization program according to claim 16 , further comprising a procedure for displaying each of the panels normalized by a preset fixed value, converted into a pixel value, and displayed.