JP2002262233A - Behavior measurement system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a behavior measurement system that helps efficiently perform behavior evaluation and analysis work by automatically synchronizing a plurality of pieces of media data and reproducing the media data. SOLUTION: This system is provided with a media data inputting means 1 for storing one piece of media data or more together with an input time, a data integrating means 2 for setting a prescribed reference time and recording integrated data including an offset time against the input time of each of the media data, a master timer 4 for counting a reproduction elapse time from the reference time, a time converting means 5 for calculating the actual reproduction position of each media data being a reproduction object by using a count time in the master timer 4 and the offset time, a data reproducing means 6 for instructing synchronous reproduction of the media data by using the reproduction position, and a displaying means 3 for displaying the media data on the basis of an instruction from the data reproducing means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a behavior measurement system, and more particularly, to a system which evaluates and analyzes a behavior during a work in manufacturing or the like to improve work efficiency, and which makes the evaluation and analysis work more efficient. It is about.

[0002]

2. Description of the Related Art Conventionally, behaviors in work are evaluated,
In order to improve work efficiency, the following method was used. First, the working scene is photographed and recorded by a video camera. At the same time, the start time of each work procedure and the time of occurrence of a problem behavior (hereinafter collectively referred to as "action") are measured by a stopwatch or the like, and recorded together with comments on an observation sheet. At this time, environmental values such as changes in humidity and temperature, operation states of the devices, and the like (hereinafter, referred to as “measured values”) are also recorded as necessary. When performing the analysis, the unnecessary operation and the work location requiring the time are specified using the time and the action data recorded based on the observation sheet. Check the video again if necessary. When comparing the data of the two, a difference is found by the observation sheet, and it is determined which part is a problem, it takes time, and the like. When confirming with a video, the video is switched and observed with a switching device, or the video tape is replaced and observed.

This method has the following problems. First, there is no exact match between the video and the recorded time. Normal video equipment has a mechanism for recording a time code, but in order to accurately manage the time code, it is necessary to store the time code at the start of recording and to record the time code when an action occurs. However, it is impossible for a human to accurately read a fine unit such as a frame in a time code during recording because the cycle is so fast.

Second, there is a problem of synchronization between the measured value and the video image. Even if the first problem is solved, the measured value is recorded by another device, and thus the time is not synchronized with the video image. Even if synchronization has been achieved, it is not possible at the present time to view the video and the measured values simultaneously when recording is performed by different devices.

Third, there is a problem related to two-party comparison. On the comparative display screen, it is conceivable to simultaneously check the image by a screen division unit or the like, but there is still a problem in that a desired operation cannot be easily extracted on the device operation side.
Also, if you want to see how much difference there is in terms of work speed when comparing, it is possible to see it in a series of work, but it is very difficult to grasp the difference in detailed work units such as individual key operation. Have difficulty.

A fourth problem is the searchability. When checking the collected data at a later date, it is necessary to manage the storage location of the videotape and observation sheet. If you want to see a specific action, you need to start the corresponding time code, but it takes time to operate the device.

In order to evaluate the work scenery from various angles, it is necessary to install a large number of video cameras.
In addition, there is a fifth problem that it is difficult to synchronize the time.

Japanese Patent Application Laid-Open No. 2001-14016 solves these problems within a limited range. JP 20
In Japanese Patent Application Laid-Open No. 01-14016, it is possible to digitize an image of a working scene, take it into a computer, and define each operation section while reproducing the image on the computer. Accurate positioning is performed by storing the start position and the end position of the operation section in association with the time on the video data. This solves the first problem. In Japanese Patent Application Laid-Open No. 2001-14016, it is possible to perform labeling and attribute setting for an operation section, so that it is possible to relatively easily find the beginning of a scene to be viewed by looking at the label. Further, since a comment can be described in the operation section, it is also possible to enter a cautionary note on the work and give an instruction to the worker. Further, by adding an invalid attribute to the operation section, the corresponding section can be excluded from the reproduction target, and an ideal operation can be simulated by a function of manually rearranging the operation sections.

[0009]

SUMMARY OF THE INVENTION However, Japanese Patent Application Laid-Open
Japanese Patent Application Laid-Open No. 001-14016 does not solve the second to fifth problems, and has many problems in performing comparative analysis. As described above, in the conventional system, there are many problems as described above, and the work of evaluating and analyzing the behavior cannot be performed efficiently.

[0010] The present invention has been made to solve such a problem, and automatically performs a plurality of media data in synchronism with each other to support efficient evaluation and analysis of behavior. The purpose is to obtain a behavior measurement system.

[0011]

According to the present invention, media data input means for inputting one or more media data and storing the media data together with the input time of each media data, a predetermined reference time, and inputting each media data are set. A data integration means for calculating an offset time, which is a time difference between the time and the reference time, and recording integrated data obtained by integrating the offset times for the respective media data; and reproducing one or more of the media data to be reproduced from the media data The master timer that counts the elapsed time from the reference time, and the count time in the master timer is converted to the time in each media data using the offset time in the integrated data, so that the actual playback of each media data to be played is performed. Time conversion means for obtaining the position and time conversion means A data reproducing unit that instructs synchronous reproduction of one or more media data to be reproduced using an actual reproduction position of the obtained media data, and one or more of the one or more reproduction targets based on an instruction from the data reproducing unit. And a display unit for displaying a part or all of the media data.

In addition, the apparatus further includes device status input means for inputting the status of the device to be operated and storing the status together with the input time, and the data integration means integrates the device status data input to the device status input means into integrated data. Treat as a target.

Further, the apparatus further comprises measurement data input means for inputting the state of the worker and storing the data together with the input time, and the data integration means sets the worker state data input to the measurement data input means as an integration target of the integrated data. handle.

The measurement data input means inputs the physiological data of the worker as the worker state data.

[0015] The data integration means includes section definition means for defining a predetermined section into which each piece of media data is divided.

Further, the section definition means makes sections into a nested structure.

Further, there are a plurality of integrated data, and the display means simultaneously displays one or more integrated data.

Further, there are a plurality of integrated data, the display means simultaneously displays one or more of the integrated data, and the data integrating means selects a section to be associated in the displayed integrated data. The apparatus includes a section selecting unit and a section associating unit that associates a plurality of selected sections with each other.

The data reproducing means includes a section synchronous reproducing means for reproducing using the data associated by the section associating means.

The section synchronous playback means sets the data which has reached the end point of the section being played back to a standby state, and starts all the playback again when the playback of all the integrated data reaches the end point of the section. Standby reproducing means for performing the operation.

The section synchronous playback means includes a master selection means for selecting integrated data serving as a master and a playback time for each corresponding section of each media data, and adjusts the playback time of the corresponding section of the master data in accordance with the playback time of the corresponding section. Playback speed synchronization means for automatically increasing or decreasing the playback speed.

Further, the apparatus further comprises a section search means for searching a section set by itself or other integrated data according to given search conditions.

The section searching means includes an evaluation function defining means for defining an evaluation function for evaluating a relation between data in a defined section, and an evaluation function for obtaining a search result using an evaluation value obtained by a given evaluation function. Result presenting means.

Further, a plurality of media data input means are provided.

Further, a plurality of measurement data input means are provided.

The media data input means receives outputs from a plurality of video cameras arranged so as to surround a target worker, and the data display means includes a viewpoint designating means capable of designating an arbitrary viewpoint, and a viewpoint designating means. When there is no camera image that matches the viewpoint specified by the specifying means, a viewpoint image synthesizing means for synthesizing and displaying an image at the specified viewpoint from two or more nearby camera images is provided.

The data display means displays an image on a part of the spherical surface, and the viewpoint designating means selects a viewpoint by rolling the spherical surface.

The data display means converts the video data to the resolution of video × N (N is a positive number representing a floating point).
Is a rectangular parallelepiped that converted the time as the length and expressed as the depth, and displayed a graph of the temporal change of the measurement data or audio data specified in the time axis direction, and displayed on the graph Pop-up frame data in the video corresponding to the specified position is displayed in the vertical direction.

The device state input means receives an operation history from the device as input, and the section setting means includes automatic section setting means for automatically dividing a section according to the operation history information.

[0030]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 FIG. 1 is a block diagram showing a configuration of a behavior measurement system according to Embodiment 1 of the present invention. Hereinafter, this embodiment will be described. In the figure, reference numeral 1 denotes a media data input unit for inputting a plurality of videos or sounds during a work such as a manufacturing operation and storing each of them together with time. Reference numeral 2 denotes a predetermined reference time. Data integration means for calculating a start time in each input media data as an offset from the reference time, and integrating and recording the offset times. Reference numeral 3 denotes integrated data recorded in the data integration means 2. It is a display means for displaying a part or all of each piece of media data input to the media data input means 1. 4 is
A master timer 5 counts the playback elapsed time of the data playback unit, which will be described later, from a reference time. Reference numeral 5 denotes a time in the master timer 4 which is set to a time in each media data by using an offset time managed by integrated data. Time conversion means for calculating and outputting the actual playback position in each media data based on the count time of the master timer 4 and the offset time managed by the integrated data;
Reference numeral 6 denotes a data reproducing unit for instructing synchronous reproduction of each media data.

Next, the operation will be described. The media data input means 1 inputs and digitizes media data composed of a plurality of videos or sounds, acquires the time at which the video was shot, and stores it in a storage device. The data integration means 2 sets a predetermined reference time, and the difference between the time and the reference time in each piece of media data input by the media data input means 1 (hereinafter referred to as offset time).
Is calculated, and the information of the offset time is integrated and stored. The data reproducing unit 6 requests the master timer 4 for the elapsed time from the reference time, and outputs the received elapsed time to the time conversion unit 5. The time conversion unit 5 calculates an actual reproduction position in the media data to be reproduced based on the received elapsed time and the offset time managed by the integrated data, and returns the calculated position to the data reproduction unit 6. The data reproducing means 6 performs synchronous reproduction of the corresponding media data by the display means 3 using the information acquired by using the time converting means 6.

This will be described in more detail. In the present embodiment, it is assumed that two pieces of video data are input to the media data input unit 1 as media data. The input video data is recorded in association with the time at the start of the recording. For example, as shown in FIG. 2, video data is video file 1 as general-purpose data such as MPEG.
0, and the management information is recorded in the management file 11 as data in which a pointer to video data (that is, a path to a media data file), a video recording start time, and a video duration are recorded. Of course, a time management area may be secured in the video data and handled as one data.

FIG. 3 shows a management file 11 for media data.
An example of the structure is shown below. Since the management file 11 stores a reference to the media data entity as shown in the figure, the user can call up desired media data by looking at the management file 11. The video continuation time can be calculated and recorded when data input to the media data input means 1 is completed.

The data integration means 2 performs processing for integrating the two management files 11. At this time, the reference time is set. The reference time is set to a time suitable for evaluation, such as around the action recording start time. As shown in FIG. 4, there are two management files 11, and the reference time is set to, for example, 2001.
Assuming that 1/21/22: 01: 00.000 is set, the offset for the media data management file 1 is (22: 00: 30.187-22: 01: 00).
0.000 =) − 29 seconds 813, and the offset for the media data management file 2 is (22:01:
10.220-22: 01: 00.000 =) 10 seconds 2
It will be 20. This is integrated and recorded in a format as shown in integrated data 12 in FIG. That is, a label is attached to each piece of media data, and an offset time, a video duration, and a pointer to the media data are recorded. The reference time may be the one designated by the user, or may be the recording start time recorded in the media data management file 11 read first.

As the display means 3, for example, an interface as shown in FIG. 5 can be used. In the example of FIG. 5, the screen of the display means 3 is roughly composed of three regions. The respective images are displayed in image display areas A and B indicated by reference numerals 3a and 3b. In FIG. 5, the time control button group 3c serves as an interface of the data reproducing means 6. The time control button group 3c includes function buttons such as play / stop / pause / fast-forward / rewind (all not shown).

The master timer 4 counts and holds the playback elapsed time from the reference time in response to a start request, and sets and provides the elapsed time in response to a request from the data playback means 6. The master timer 4 counts the elapsed time during reproduction according to an instruction from the data reproduction means 6, and stops counting when stopped. In addition, it has a function of providing an elapsed time from the current reference time in response to a request from the data reproducing unit 6. Further, the master timer 4 can set a desired elapsed time in response to a request from the data reproducing means 6 (the details will be described later).

For example, consider the operation during reproduction. When the reproduction button is selected in the time control button group 3c, the data reproducing means 6 performs a preparation operation, and then performs synchronous reproduction. First, the preparation work will be described. The data reproducing means 6 requests the elapsed time from the reference time to the master timer 4. The master timer 4 returns the elapsed time held by itself to the request source. The data reproducing means 6 receives this, outputs it to the time converting means 5, and requests the corresponding position in each media data. Time conversion means 5
Calculates the time on the corresponding media data (that is, the actual playback position) based on the received elapsed time and the offset in the integrated data 12 and returns the time to the data playback unit 6. Preparation work is completed above. Synchronous playback is started based on the data obtained by the preparation work.
The data reproducing means 6 instructs the reproduction of each media data from the corresponding position and the start of counting to the master timer 4. Thereby, each media data is reproduced in synchronization with the master timer 4. When the stop or the pause is selected, the data reproducing means 6 instructs the stop of the master timer 4 and stops the reproduction of each media data. In the case of fast-forward, the data reproducing means 6 transmits the corresponding reproduction magnification to the master timer 4 and issues a count start command, and performs the fast-forward operation by setting the reproduction magnification for each medium and reproducing. In the case of rewinding, the data reproducing means 6 transmits the corresponding reproduction magnification to the master timer 4, issues a count subtraction start instruction, and issues a designated magnification rewind instruction for each medium to perform the operation.

For example, in the case of the data shown in FIG. 4, when the reproduction is started from the reference time, the data corresponding to the media data management file 1 is 29 seconds 81
The reproduction is started from the point 3 and the data corresponding to the media data management file 2 is not reproduced at the start of the reproduction. The data reproducing means 6 constantly monitors the elapsed time in the master timer 4. The data reproducing means 6 starts reproducing data corresponding to the media data management file 2 when the count of the master timer 4 reaches 220 for 10 seconds. In this way, a plurality of media data can be reproduced in synchronization. When the recording is temporarily stopped or stopped, the data reproducing unit 6 stops counting the master timer 4 and stops the media data as described above. For example, when a user requests to reproduce data 30 seconds after the reference time, the data reproducing means 6 issues a command to the master timer 4 to set the elapsed time to 30 seconds, It requests the time conversion means 5 to calculate and return the reproduction start position. The time conversion means 5 returns 59 seconds 813 and 19 seconds 780, respectively, and the data reproduction means 6 which has received them, starts reproduction from the position and instructs the master timer 4 to start counting.

As described above, in the action measuring system shown in this embodiment, a predetermined reference time is set,
An offset time from the reference time is obtained for each media data, the information is integrated and recorded as integrated data, and at the time of reproduction, synchronous reproduction is performed using the integrated data. Therefore, a plurality of videos can be automatically and synchronously reproduced, so that the action evaluation / analysis work can be made more efficient.

Embodiment 2 FIG. 6 is a diagram showing a configuration of a behavior measurement system according to Embodiment 2 of the present invention. In this embodiment, the system operates on a computer. In the figure, media data input means 21
An image is input and digitized, and a system clock in the computer is obtained and stored in a storage device. The device status input means 22 inputs the operating status (operations such as key operation status and device status, signal history) of the device such as a machine tool to be worked, and acquires the absolute time by the system clock in the computer. The operation state is stored in the storage device together with the absolute time. Measurement data input means 23
Acquires digital data of the state of the worker (ie, physiological data such as pulse, heart rate, breathing, blood pressure, etc.) and obtains the absolute time by the system clock in the computer and stores it in the storage device. Note that it is also possible to add an attribute to data in all of these input means 21 to 23.

The data integration means 24 sets a predetermined reference time, and arranges the information inputted by each of the input means 21 to 23 according to the reference time. If there is attribute information for each, this is also integrated and stored. The display unit 25 displays various data input according to the integrated data. The master timer 26 counts the elapsed playback time from the reference time. Time conversion means 27
Calculates the position in the video or audio data managed by the integrated data by the time of the master timer 26 and the time management unit in each media data. The data reproducing means 28 is displayed by the display means 25. Data such as video is reproduced synchronously using the time conversion means 27.

Next, the operation will be described. The video input by the media data input unit 21 is associated with an absolute time obtained from a system clock in the computer. Since the video data is composed of a temporally continuous fixed frame sequence, if the video recording start time and the video continuation time are stored, the position in the middle frame can be obtained by calculation. The same applies to audio data. The data in which the time information is recorded is stored in a management file different from the media file as in FIG. 2 in the first embodiment. Of course, the same file can be used. The management file 11A has a structure as shown in FIG. 7 and stores a reference to the media data entity. Therefore, the media file can be called by looking at the management file 11A. The video duration can be recorded when data input to the media data input means 21 is completed.

Since there is no rule as to when the operation / signal information is input in the equipment state input means 22, as shown in FIG. 8, each operation (equipment operation / signal output) is performed. And record the time information. Measurement data input means 2
The same applies to No. 3. Regarding the two persons, each data file (equipment status data file (see FIG. 9), measurement data file (FIG. 10)
See)).

If it is desired to set the attributes after each recording, the attributes for the data are set. As the attribute, a work name, a worker name, a comment, and the like can be used. Of course, other information can also be set as attributes. When the set attribute is video data, the device status is stored in the management file 11A.
And stored in the device status data file 13 and the measurement data file 14 in a format as shown in FIG.

Each created data is converted into integrated data by the data integrating means 24 according to the time. The integrated data 12A has a structure as shown in FIG. Since each integration target file is not limited to one file, it has a structure that can handle a plurality of target files. Since the input start time of each data is not always the same, the offset from each reference time is stored in the integrated data 12A. In the present embodiment, since the data is directly arranged in the data section for the device state data file and the measurement data file, no offset value is required.

FIG. 12 shows an example of creating integrated data. The integrated data 12A includes two media data management files 11A, one device status data file 13, and
The two measurement data files 14 are integrated.
For two media data, an offset from the reference time is recorded. Although the media file is outside of the integrated data, the media data can be specified from the information of the integrated data 12A, and the reproduction timing can be seen from the offset information.

The integrated data is displayed by the display means 25 as shown in FIG. The device status display area 25a displays device statuses arranged vertically in chronological order. For example, it is possible to make it easy for the user to understand which operation is being performed by highlighting the device state at the corresponding time. In the image display areas A and B indicated by reference numerals 25b and 25c, two integrated images are displayed. The display means 25 controls to read the path from the integrated data and display the path in this area. Symbols 25e, 25
The measurement data display areas 1 and 2 indicated by f display two sets of measurement data in the integrated data. In FIG. 13, the image and the measurement data display area are each two, but are not limited to two. When two or more video files and measurement data are handled, it is possible to switch the video and data to be displayed. In the measurement data display areas 1 and 2, it is possible to display a graph using the data of the data part or to display a numerical value corresponding to the time. The time control button 25d is an interface of the data reproducing unit 28. The data reproduction unit 28 performs synchronous reproduction of each data by the method described in the first embodiment. As for the device status data and the measurement data, the data reproducing unit 28 directly reads out the data at the time counted by the master timer 26 from the data portion in the integrated data and displays the data.

In this example, only video data is taken as media data, but audio data may be used in addition to video data. Audio data is available in the same way as video. Further, the present invention can be applied even when the measurement data input unit 23 does not exist or when the device state input unit 22 does not exist. Further, each of the media data input unit 21, the measurement data input unit 23, and the device state input unit 22 is not limited to a single unit, and may be provided in a plurality.

As described above, in the behavior measuring system according to the present embodiment, the device status input means 22 which inputs the status of the device to be worked and stores it together with the absolute time,
Measure the state of the worker, input the measured data,
It further comprises a measurement data input means 23 for storing the data together with the absolute time, and integrates and manages the video, the state of the device and the measurement data inputted to the media data input means 21 and enables synchronized reproduction thereof. The labor of performing analysis and evaluation can be reduced as compared with the related art, and the action evaluation and analysis work can be performed efficiently.

Embodiment 3 In the present embodiment,
In the behavior measurement system described in the second embodiment, FIG.
As shown in FIG. 4, in the data integration means 24, a section definition means 29 for defining a predetermined section in each data is provided. The section definition means 29 can store the section set in the integrated data. For example, a series of operations on a video may be defined as a section, a portion showing a characteristic change on the measurement data may be defined as a section, and in a device state, an operation history from the device may be defined. A series of operations may be defined as a section by automatically dividing the section according to an operation history as an input. It is also possible to automatically define each section in the device state as a section start position, and a section defined immediately before the start of the next operation as the section end point as a section on the video. . In this case, the device state input means 22 receives the operation history from the device as input, and
9 is provided with an automatic section setting means for automatically dividing the section according to the operation history information.

As shown in FIG. 15, the device status display area 2
It is also possible to display the automatically defined section in 5a. This makes it easier to understand semantic breaks in the data. Furthermore, by adding a mechanism for setting an attribute to a section in the section definition means 29, the meaning of the section can be made more understandable.

It is also possible for the section defined in this way to have a nested structure. This makes it possible to define the hierarchical structure of the work. For example, when observing and evaluating an operation including processes such as a large process, a middle process, and a small process as shown in FIG. 16, it is easy to understand because the process can be associated with each process.

As described above, in the present embodiment,
The data integration means 24 is provided with the section definition means 29 for storing the section set in the integrated data, and the section can be provided and stored in each media data, so that the synchronous reproduction for each section becomes possible. Evaluation and analysis of behavior can be performed efficiently.

Embodiment 4 In the first to third embodiments, a single integrated data is handled. However, in the present embodiment, a plurality of integrated data are handled at the same time, and the plurality of integrated data are simultaneously displayed on the display means for comparison. To be extended. FIG. 17 shows an example of the comparison display screen. Device status display area A (reference numeral 25a-1) / video display area A (25
b) The measurement data display area A (25e-1) is the subject A
, Device status display area B (25a-2), video display area B (25c), measurement data display area B (25e-
It is possible to simultaneously display data for two persons such as 2) for the subject B. Figure 1
As shown in FIG. 8, a method of simultaneously displaying a screen example as shown in FIG. 13 on the two display devices 1 and 2 using one control device 30 and two display devices 1 and 2 is also conceivable. In this case, the data of the subject A is displayed on the display device 1 and the data of the subject B is displayed on the display device 2.

Comparison display is performed by any of the display methods described above. Here, the data corresponding to the subject A will be referred to as teacher data, and the data corresponding to the subject B will be referred to as student data. The teacher data is assumed to be operated according to a model procedure. In order to ascertain where the problem points are in the student data, a video or the like is reproduced and compared and displayed. At this time, the analysis and evaluation work can be made more efficient if it is possible to easily associate and grasp what kind of procedure is different in the same process.

Therefore, as shown in FIG. 19, the system associates a plurality of selected sections with the section selecting means 32 for selecting a section to be associated in the displayed integrated data. Section associating means 31
And the section synchronous playback means 3 for automatically changing the playback order obtained by the section association means 31 and performing simultaneous playback.
3 is provided. For example, assume that the process is divided into sections as shown in FIG. In this example, the range to be associated with the same section is limited by the user's instruction by the section selecting means 32. Step 1-1 of the teacher data is selected on the display means 1 shown in FIG. 20, and step 1-1 is selected also for the student data. In such a nested structure, the section selecting means 32 can determine that when a parent node is selected, all of the child nodes are targeted. Of course, it is also possible to select individual sections. In this case, only the sections belonging to the steps 1-1 and below are targets for association. The section associating means, for example, according to a command from the user, processes 1-1-1 to 1-1-4 in process 1-1 in the teacher data.
Student data with the same name as before
Is found from the nodes belonging to by string matching. Then, Step 1-1-1, Step 1-1-2, Step 1-1
3. Step 1-1-4 is found, and is associated with a node having the same name in the teacher data. In this example, the association by the section name is used, but the association can also be performed by the attribute set for the section. For example, when the length of a section is defined as an attribute, it is possible to associate a section whose length is equal or close to each other, or to associate a section having the same or close change rate of measured data. Also, as shown in FIG. 20, the playback order on the student side is changed by associating the section name with the playback order on the teacher side according to the playback order on the teacher side. As a result, the association is automatically performed, and the efficiency of the analysis / evaluation work for instructing the student on the ideal work procedure is improved.

It is difficult to determine where the differences between the steps are, even if they are reproduced as they are by rearrangement during analysis and evaluation. For example, if the playback time of each process of the teacher is different from the playback time of each process of the student, there will be a difference between them. Therefore, it is necessary to perform playback control by stopping one of them to make a comparison judgment. Therefore, in the section synchronous reproduction means 33, standby reproduction means (not shown) which starts reproduction at the stage where the reproduction of each associated step has been completed first, waits for the reproduction completed last, and catches up. ) Is provided. As shown in FIG. 21, when the playback time of step 1-1-1 is longer for the teacher and the student is shorter than the teacher,
At the stage where the playback in the step 1-1-1 on the student side is completed, the playback on the student side is temporarily stopped. When the reproduction of the step 1-1-1 on the teacher side is completed, the reproduction of the step 1-1-2 is simultaneously started by the teacher and the student. This makes it easy to understand the difference in the time required for the work, so that the analysis and evaluation work is efficient.

Further, a master selecting means for selecting integrated data to be a master in the section synchronous reproducing means 33;
A reproduction speed synchronization means (not shown) for automatically changing the reproduction speed of data other than the master so as to synchronize with the reproduction time of the master is provided. As shown in FIG. 22, it is assumed that there is a difference in the reproduction time between the teacher side and the student side. Assuming that the teacher side is the master, the playback speed on the student side is changed for each process and playback is performed in association with each other. In this case, step 1-
1-1, 1-1-3, and 1-1-4 perform high-speed reproduction, and 1-1-1-2 perform low-speed reproduction. As a result, the playback speed is automatically adjusted to the ideal speed, so that the difference in the operation speed can be easily understood at the time of analysis and evaluation.

As described above, in the present embodiment,
A section selecting means for selecting two sections to be associated with each other so that the two integrated data can be simultaneously displayed on the display means, and a section association for associating the selected plurality of sections with each other Means and a section synchronous playback means for performing playback using the data associated by the section associating means, so that the two integrated data are simultaneously displayed on the display device.・ The analysis work can be performed efficiently.

Embodiment 5 FIG. In the fourth embodiment, the behavior measurement system capable of performing efficient analysis by comparison display and automatic association has been described. However, when comparison is performed using data similar to data to be evaluated, It is required to retrieve some data. Therefore, FIG.
As shown in FIG. 3, the behavior measurement system described in the fourth embodiment further includes a section search unit 34 capable of performing a search by section.

It is assumed that a section is defined in the integrated data as shown in FIG. An attribute can be added to this, and a search based on the attribute can be performed. As an attribute for a section, a measurement value can be used in addition to attributes such as “comment” and “section name”. For example, “minimum value of section” and “maximum value of section” can be stored as attribute information for the section.

The currently selected integrated data image in the section set in FIG.
It is considered that data having a portion whose heart rate (the reference numeral 40 in FIG. 24) and the transition of the heart rate are similar is searched. In such a case, it has conventionally been necessary to shift a portion having the same time width as the search condition for all data by a minute time to find a similar portion (see reference numeral 42 in FIG. 24). ). However, doing so takes a lot of time to get search results,
It is conceivable to detect a section that does not make much sense as it straddles the work.

In this embodiment, it is assumed that a similar section is searched for in the section divided into data (reference numeral 43 in FIG. 24). The similarity between the data corresponding to the search condition (reference numeral 40 in FIG. 24) and the other data is the same, but since the data to be searched is divided into sections in advance, this is indicated by reference numeral 43 in FIG. As described above, the evaluation for the search may be performed only for each divided section (in the example of FIG.
One). By doing so, a search can be performed only for a meaningful section, and the search execution time can be reduced. Also, by searching for sections,
Even for data evaluated by others, the meaning of the section of the search result can be known from the attribute information or the like. Further, data can be narrowed down in combination with a phrase search. Such a section search can be performed only in the integrated data after reading one integrated data, or the integrated data having the corresponding section can be searched from all the stored integrated data. . As a result, necessary data can be easily searched from a large amount of data, and the work of analysis and evaluation by comparison is made more efficient.

When searching for a similar section, the evaluation criterion may be changed depending on the meaning of the degree of similarity. For example, in the case of simply evaluating the similarity of data and the case of evaluating the stress value estimated from the heart rate and blood pressure, the former only determines the similarity between the corresponding indices, while the latter evaluates the similarity between the corresponding indices. It is necessary to calculate the stress value from the heart rate, blood pressure, and the like, and then evaluate the similarity. In the case of calculating a new index from a correlation between a plurality of data as in this example, the index is not known in advance, and the association is often found during analysis work or evaluation. Therefore, a mechanism is needed to add search criteria later. To cope with such a case, an evaluation function defining an evaluation function for providing a criterion of what is determined to be similar (that is, a function for evaluating the relevance between sections) in the section search means. A definition means is provided. The evaluation function can be freely defined in the form of, for example, an external function, or can be defined by a macro formula like spreadsheet software. Taking Microsoft's Windows operating system as an example,
It can be realized by implementing the main body of this system as a software program on Windows (Windows) and defining external functions as plug-ins. Plug-ins are provided in the form of dynamic link libraries that can be dynamically loaded when executing a function call. By defining a prototype for a function in the dynamic link library, various evaluation functions can be prepared if they are created according to the prototype, and can be switched and used depending on the case. As an example of the evaluation function, for example, “the step 1-1-1 is included in the section name”, “when the heart rate y decreases at y = 1 / t (t is unit time), the respiration rate x becomes x. = Increase in log (t) ". As a result, a search can be performed based on the relevance between indices having a complicated correlation.

Further, there is provided an evaluation result presenting means for presenting the result of the search using the attribute information of each section by the evaluation function together with its evaluation value. The evaluation result presenting means can display information such as an evaluation value, an integrated data file name, a section name, and a section offset position.

As described above, in the present embodiment,
In accordance with a given search condition, a section search means for searching for a section set by itself or other integrated data is provided, and a similar section is searched from search targets which have been divided into sections in advance. Therefore, since the evaluation for the search needs to be performed only for each divided section, the search execution time can be greatly reduced. In addition, by providing an evaluation function definition unit that defines an evaluation function that gives a criterion of what is judged to be similar in the section search unit, a function for evaluating the relevance between sections can be simplified. , So that a search for similar sections can be performed from multiple angles.

Embodiment 6 FIG. In the behavior measurement system described in the first to fifth embodiments, as shown in FIG. 25, a plurality of video cameras are arranged so as to surround a worker to be photographed, and input to a plurality of media data input means corresponding thereto. I do. In the case of FIG. 25, four cameras (reference numeral 5)
1, 52, 53, 54), it is assumed that there are four media data input means. Here, a viewpoint designating unit (not shown) capable of designating an arbitrary viewpoint is provided in the data display unit. Here, the arbitrary viewpoint is referred to as a virtual camera 55. The virtual camera 55 can be arranged at any point on the circumference connecting the cameras 1 to 4. Further, when there is no real camera image corresponding to the virtual camera position, viewpoint image synthesizing means (FIG. 1) extracts three-dimensional information from two adjacent real camera images, and synthesizes and displays an image of the virtual camera by calculation. Not shown)
Is provided. Thus, for example, when the user wants to view an image from an intermediate position between the camera 3 and the camera 4 in FIG. 25, the image from the camera 3 (reference numeral 53a in FIG. 25) and the image from the camera 4 (reference numeral 54a in FIG. 25). It is possible to extract three-dimensional information from the virtual camera and use it as a video (reference numeral 55a in FIG. 25) in the virtual camera. Several methods have been proposed for a method of extracting three-dimensional information from a camera and a method of synthesizing an image from a plurality of images.
Further, the camera to be arranged is not limited to a monocular camera, and it is also possible to use a stereoscopic image by using a stereo camera or the like. In this case, three-dimensional information can be obtained more accurately. This makes it possible to evaluate the motion at various angles without arranging a large number of cameras when measuring the behavior.

As another method, an image can be displayed on a part of the spherical surface by the data display means, and the viewpoint specifying means can specify the viewpoint by rotating the spherical surface. An image is attached to a sphere as shown in FIG. Although it looks like a circle on the display, it is a sphere as a treatment, and when dragging with a mouse or the like, the original image is displayed around the circle. When the image of the camera 3 is rotated 45 degrees to the left while being displayed on the front,
The viewpoint designating unit determines that the virtual camera is installed at the intermediate point between the cameras 2 and 3, and the viewpoint video synthesizing unit performs video display at the corresponding position. Here, as shown in FIG. 25, it is assumed that four cameras are arranged on the same circumference, but it is also possible to specify a vertical viewpoint by additionally installing cameras such as upward and downward. It becomes possible. As for these things, if the capability of the computer is sufficiently fast, the viewpoint synthesis can be performed in real time, and the synthesis result can be displayed as a moving image. As a result, an arbitrary viewpoint can be easily specified, so that the work efficiency at the time of evaluation analysis is improved.

As described above, in the present embodiment,
The media data input means receives images from a plurality of video cameras arranged so as to surround the shooting target, and the data display means includes a viewpoint designating means capable of designating an arbitrary viewpoint and a viewpoint designated by the viewpoint designating means. When there is no corresponding camera image, a viewpoint image synthesizing means for synthesizing and displaying an image at a specified viewpoint from two or more camera images in the vicinity is provided, so that an arbitrary viewpoint can be easily specified. In addition, the work efficiency at the time of evaluation analysis can be further improved.

Embodiment 7 FIG. In the second to fifth embodiments, the image display area and the measurement data display area are separately provided on the display screen of the display means, and each is displayed. In the present embodiment, a method of integrating and displaying these will be described.
As shown in FIG. 27, the image is cubic (reference numeral 60 in FIG. 27).
Expressed as The surface facing the near side in FIG. 27 is obtained by multiplying the resolution of the video by N in the vertical and horizontal directions (N is a positive number represented by a floating point). For example, it can be 0.5 times, 2 times, or the like. Thus, a cube can be expressed by treating the depth direction as a time axis. The measurement data expressed on the same time axis in the time axis direction (the side of the cube) is displayed as a graph. This makes it easy to understand the time on the video and the time of the measurement data. The reproduction of the video is performed by the front surface in FIG. At this time, in order to make the association between the frame being reproduced and the graph easy to understand, the corresponding position on the measurement data is highlighted (reference numeral 61), or a portion corresponding to the side of the frame to be reproduced is displayed in bold line. (Reference numeral 62)
And so on. Further, when a part of the graph of the measurement data is designated by a pointing device such as a mouse (reference numeral 63), the video frame at the corresponding time is displayed by popping up in the upward direction (reference numeral 64). As a result, the correspondence between the video and the measurement data can be more easily understood, and the analysis work can be made more efficient. In addition,
In the above description, the case where the measurement data is displayed has been described as an example, but the present invention is not limited to this case, and audio data may be displayed.

As described above, in the present embodiment,
The image data is defined as a rectangular parallelepiped that represents the resolution x N of the image as one side, and converts the time as the length to represent the depth.
In the time axis direction, a graph showing the temporal change of the specified measurement data or audio data is displayed, and the frame data in the video corresponding to the specified position on the graph is displayed in a vertical pop-up display As a result, the correspondence between the video and the measurement data or the audio data can be more easily understood, and the analysis work can be made more efficient.

[0072]

According to the present invention, media data input means for inputting one or more media data and storing the media data together with the input time of each media data, and setting a predetermined reference time,
A data integration unit that calculates an offset time that is a time difference between the input time and the reference time for each piece of media data, and records integrated data obtained by integrating the offset times for each piece of media data; A master timer that counts the elapsed playback time of the media data from the reference time, and converting the count time in the master timer to the time in each piece of media data by using the offset time in the integrated data to obtain each media to be played back. Time conversion means for determining an actual reproduction position of data; data reproduction means for instructing synchronous reproduction of one or more media data to be reproduced using the actual reproduction position of the media data obtained by the time conversion means; Based on an instruction from the data reproducing means, one or more Since the behavior measurement system includes a display unit for displaying a part or all of the media data to be recorded and reproduced, one or more pieces of media data can be automatically reproduced in synchronization with each other. Work can be made more efficient.

Further, the apparatus further comprises device status input means for inputting the status of the device to be worked and storing the status together with the input time, and the data integration means integrates the device status data input to the device status input device into integrated data. Since it is handled as an object, media data and device status data can be integrated and managed, and synchronized playback can be performed. Therefore, labor for analyzing and evaluating behavior can be reduced.

Further, there is further provided a measurement data input means for inputting the state of the worker and storing the data together with the input time, and the data integration means sets the worker state data input to the measurement data input means as an integration target of the integrated data. Because it handles, integrate and manage media data and measurement data,
Since the synchronous reproduction is possible, the labor for analyzing and evaluating the behavior can be reduced.

Further, the measurement data input means inputs the physiological data of the worker as the worker state data, so that the state of the worker can be accurately analyzed.

Further, since the data integration means is provided with the section definition means for defining a predetermined section into which each piece of media data is divided, it is easy to understand the meaningful delimitation in the data, and the analysis and evaluation can be performed more efficiently. Can be.

Further, since the section defining means has a nested section, it is possible to define a hierarchical structure of work.

Further, since there are a plurality of integrated data and the display means displays one or more integrated data at the same time, the analysis and evaluation work comparing the integrated data can be performed efficiently.

Further, when there are a plurality of integrated data, the display means simultaneously displays one or more of the integrated data, and the data integrating means selects a section to be associated in the displayed integrated data. Since the apparatus includes the section selecting means and the section associating means for associating a plurality of selected sections with each other, it is possible to automatically change the reproduction order by associating, and to specify an ideal work procedure. The efficiency of analysis and evaluation work, such as for performing

Further, since the data reproducing means has the section synchronous reproducing means for reproducing using the data associated by the section associating means, the reproducing order is automatically changed to perform the synchronous reproducing. Can improve the efficiency of analysis and evaluation work.

Further, the section synchronous playback means sets the data which has reached the end point of the section being played back to a standby state, and starts all the playback again when the playback of all the integrated data reaches the end point of the section. Since the standby reproduction means is provided, it is easy to understand the difference in the time required for the work to be compared, and the analysis and evaluation work becomes efficient.

Further, the section synchronous reproducing means includes a master selecting means for selecting integrated data serving as a master and a reproducing time for each corresponding section in each media data, and adjusts the reproducing time of the corresponding section of the master data. Since the playback speed is automatically adjusted to an ideal speed by the playback speed synchronizing means for automatically increasing / decreasing the playback speed, the playback speed is automatically adjusted to an ideal speed.

Further, since the apparatus further includes a section search means for searching for a section set by itself or other integrated data according to given search conditions, a search by section can be performed. Can be greatly reduced.

The section searching means includes an evaluation function defining means for defining an evaluation function for evaluating a relation between data in a defined section, and an evaluation function for obtaining a search result using an evaluation value obtained by a given evaluation function. Since the search function is provided with the result presenting means, the search can be performed by easily replacing the evaluation function.

Further, since a plurality of media data input means are provided, the workability of input can be improved.

Also, since a plurality of measurement data input means are provided, the workability of input can be improved.

The media data input means receives outputs from a plurality of video cameras arranged so as to surround a target worker, and the data display means includes a viewpoint designating means capable of designating an arbitrary viewpoint, and a viewpoint designating means. When there is no camera image that matches the viewpoint specified by the specifying means, the viewpoint measurement means includes a viewpoint image synthesizing means for synthesizing and displaying an image at the specified viewpoint from two or more nearby camera images. Is performed, the operation can be evaluated at various angles.

Further, the data display means displays an image on a part of the spherical surface, and the viewpoint designating means selects a viewpoint by rolling the spherical surface, so that an arbitrary viewpoint can be easily designated. Work efficiency during evaluation analysis is improved.

The data display means converts the video data to the resolution of video × N (N is a positive number representing a floating point).
Is a rectangular parallelepiped that converted the time as the length and expressed as the depth, and displayed a graph of the temporal change of the measurement data or audio data specified in the time axis direction, and displayed on the graph Since the frame data in the video corresponding to the designated position is displayed in a pop-up display in the vertical direction, the correspondence between the media data and the measurement data can be more easily understood, and the analysis work can be made more efficient.

The device state input means receives an operation history from the device and the section setting means includes automatic section setting means for automatically dividing a section in accordance with the operation history information. It is also possible to automatically define a section, and analysis and evaluation can be performed efficiently.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a behavior measurement system according to Embodiment 1 of the present invention.

FIG. 2 is an explanatory diagram showing files output by media data input means in the behavior measurement system according to Embodiment 1 of the present invention.

FIG. 3 is an explanatory diagram showing a structure of a media data management file in the behavior measurement system according to the first embodiment of the present invention.

FIG. 4 is an explanatory diagram illustrating creation of integrated data in the behavior measurement system according to the first embodiment of the present invention.

FIG. 5 is an explanatory diagram showing an example of an interface of a display unit in the behavior measurement system according to the first embodiment of the present invention.

FIG. 6 is a block diagram showing a configuration of a behavior measurement system according to Embodiment 2 of the present invention.

FIG. 7 is an explanatory diagram showing a structure of a media data management file in the behavior measurement system according to the second embodiment of the present invention.

FIG. 8 is an explanatory diagram showing correspondence between data and time in a device state input unit in a behavior measurement system according to Embodiment 2 of the present invention.

FIG. 9 is an explanatory diagram showing a structure of a device status data file in the behavior measurement system according to the second embodiment of the present invention.

FIG. 10 is an explanatory diagram showing a structure of a measurement data file in the behavior measurement system according to the second embodiment of the present invention.

FIG. 11 is an explanatory diagram showing a structure of integrated data in the behavior measurement system according to the second embodiment of the present invention.

FIG. 12 is an explanatory diagram showing an example of integrated data creation in the behavior measurement system according to the second embodiment of the present invention.

FIG. 13 is an explanatory diagram showing an example of an interface of a display unit in the behavior measurement system according to the second embodiment of the present invention.

FIG. 14 is a block diagram showing a configuration of a behavior measurement system according to Embodiment 3 of the present invention.

FIG. 15 is an explanatory diagram showing an example of a section definition display in a device state display area of a display unit in a behavior measurement system according to Embodiment 3 of the present invention.

FIG. 16 is an explanatory diagram showing a display example of a hierarchical structure in a device state display area of a display means in a behavior measurement system according to a third embodiment of the present invention.

FIG. 17 is an explanatory diagram showing an example of a comparison display screen of a display unit in the behavior measurement system according to Embodiment 4 of the present invention.

FIG. 18 is an explanatory diagram showing an example of a comparative display screen of a display unit using a multi-display in the behavior measurement system according to Embodiment 4 of the present invention.

FIG. 19 is a block diagram showing a configuration of a behavior measurement system according to Embodiment 4 of the present invention.

FIG. 20 is an explanatory diagram showing a section name associating method in the behavior measurement system according to Embodiment 4 of the present invention.

FIG. 21 is an explanatory diagram showing standby reproduction in the behavior measurement system according to Embodiment 4 of the present invention.

FIG. 22 is an explanatory diagram showing automatic variable speed playback in the behavior measurement system according to Embodiment 4 of the present invention.

FIG. 23 is a block diagram showing a configuration of a behavior measurement system according to Embodiment 5 of the present invention.

FIG. 24 is an explanatory diagram showing a section search in the behavior measurement system according to the fifth embodiment of the present invention.

FIG. 25 is an explanatory diagram showing combined display of virtual camera images in the behavior measurement system according to Embodiment 6 of the present invention.

FIG. 26 is an explanatory diagram showing viewpoint designation and display by a spherical interface in the behavior measurement system according to Embodiment 6 of the present invention.

FIG. 27 is an explanatory diagram showing superimposed display of moving image / measurement data by a cubic interface in the behavior measurement system according to the seventh embodiment of the present invention.

[Explanation of symbols]

1, 21 media data input means, 2 data integration means, 3 display means, 4 master timer, 5 time conversion means, 6 data reproduction means, 10 video files, 11
Management file, 12 integrated data, 22 device status input means, 23 measurement data input means, 24 data integration means, 25 display means, 26 master timer, 27
Time conversion means, 28 data reproduction means, 29 section definition means, 30 control device, 31 section association means, 32
Section selection means, 33 section synchronization playback means, 34 section search means, 51, 52, 53, 54 cameras, 55 virtual cameras.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Satoshi Tanaka 2-3-2 Marunouchi, Chiyoda-ku, Tokyo Mitsubishi Electric Corporation F-term (reference) 5C052 AC01 AC08 DD04 5C053 HA21 HA29 JA22 5C054 AA01 FE16 FE17 GB01 GB05 HA03 HA05

Claims (19)

[Claims] 1. Media data input means for inputting at least one piece of media data and storing the input time of each piece of media data, a predetermined reference time, and setting the input time and the reference time for each piece of media data. Data integration means for calculating an offset time, which is a time difference from the media data, and recording integrated data obtained by integrating the offset times for each of the media data; A master timer that counts the elapsed time from the reference time, and the count time in the master timer is converted into a time in each of the media data by using the offset time in the integrated data, whereby each of the reproduction targets is Time to find actual playback position of media data Conversion means; data reproduction means for instructing synchronous reproduction of one or more of the media data to be reproduced using an actual reproduction position of the media data obtained by the time conversion means; Display means for displaying a part or all of the one or more media data to be reproduced, based on an instruction. 2. The state of a device to be worked is input, and
2. The apparatus according to claim 1, further comprising a device status input unit that stores the device status data together with the input time, wherein the data integration unit handles the device status data input to the device status input unit as an integration target of the integrated data. 3. Behavior measurement system. 3. The system according to claim 1, further comprising: measurement data input means for inputting a state of the worker and storing the data together with the input time, wherein the data integration means converts the worker state data input to the measurement data input means into the integrated data. The behavior measurement system according to claim 1, wherein the behavior measurement system is handled as an integration target. 4. The behavior measurement system according to claim 3, wherein the measurement data input unit inputs physiological data of the worker as the worker state data. 5. The behavior measurement system according to claim 1, wherein said data integration means includes a section definition means for defining a predetermined section into which each of said media data is divided. . 6. The behavior measurement system according to claim 5, wherein said section definition means has a nested structure of said sections. 7. The behavior measurement system according to claim 1, wherein there are a plurality of the integrated data, and the display unit displays one or more of the integrated data at the same time. 8. There is a plurality of said integrated data, said display means simultaneously displays one or more of said integrated data, and said data integration means makes a target of association in said displayed integrated data. The behavior measurement system according to claim 6, further comprising: a section selecting unit that selects a section; and a section associating unit that associates a plurality of selected sections with each other. 9. The behavior measurement according to claim 8, wherein said data reproducing means includes a section synchronous reproducing means for reproducing using data associated by said section associating means. system. 10. The section-synchronous playback means sets the data that has reached the end point of the section being played back to a standby state, and performs all playback again when the playback of all integrated data reaches the end point of the section. 10. The behavior measuring system according to claim 9, further comprising a standby reproducing unit for starting. 11. The section-synchronous playback means includes: master selection means for selecting integrated data serving as a master; determining playback time of each corresponding section in each of the media data; 10. The behavior measuring system according to claim 9, further comprising: a reproducing speed synchronizing means for automatically increasing or decreasing the reproducing speed in accordance with the time. 12. The apparatus according to claim 5, further comprising a section search means for searching a section set by itself or other integrated data according to a given search condition. Behavior measurement system. 13. The section search means, comprising: an evaluation function defining means for defining an evaluation function for evaluating a relationship between data in a defined section; and an evaluation value obtained by a given evaluation function as a search result. The behavior measurement system according to claim 12, further comprising an evaluation result presentation unit. 14. The behavior measurement system according to claim 1, wherein a plurality of the media data input means are provided. 15. The behavior measurement system according to claim 1, wherein a plurality of the measurement data input units are provided. 16. The media data input unit receives outputs from a plurality of video cameras arranged so as to surround a target worker, and the data display unit includes: a viewpoint designating unit that can designate an arbitrary viewpoint; A viewpoint video synthesizing unit for synthesizing and displaying an image at a specified viewpoint from two or more nearby camera images when there is no camera image matching the viewpoint specified by the viewpoint specifying unit. The behavior measurement system according to claim 14, wherein: 17. The behavior measurement according to claim 16, wherein the data display unit displays an image on a part of the spherical surface, and the viewpoint designating unit selects a viewpoint by rolling the spherical surface. system. 18. The data display means according to claim 1, wherein the image data is a rectangular parallelepiped in which the resolution of the image × N (N is a positive number represented by a floating point) is defined as one surface, and the data converted as time is represented as depth. , Displays a graph of the temporal change of the measurement data or audio data specified in the time axis direction, and displays the frame data in the video corresponding to the specified position on the graph in a vertical pop-up display The behavior measurement system according to any one of claims 1 to 15, wherein: 19. The apparatus state input means receives an operation history from a device as input, and the section setting means includes an automatic section setting means for automatically dividing a section according to the operation history information. The behavior measuring system according to any one of claims 5 to 18, wherein