JP2004030623A - Work state management device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for easily retrieving a desired place in video information and voice information recorded during a work. <P>SOLUTION: A work state management device is provided with an input section 11 which inputs information regarding the work, a work state management section 13 which detects occurrence of a prescribed change to the inputted information and generates information indicating time when the prescribed change occurs and information specifying the prescribed change according to the detected prescribed change and a work state storage section 14 which stores the information indicating the time when the prescribed change occurs and the information specifying the prescribed change. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a work status management device that performs information processing between a single terminal or a plurality of terminals and manages information according to the work status of a user.
[0002]
[Prior art]
2. Description of the Related Art In recent years, a network conference system has been proposed and constructed which supports performing collaborative work including meetings and decision making while exchanging various kinds of information in real time.
[0003]
For example, Watanabe et al., "Multimedia Distributed Conference System MERMAID", IPSJ Transactions, Vol. 32, no. 9 (1991), Nakayama et al., "Multi-party Electronic Dialogue System ASSOCIA", Transactions of Information Processing Society of Japan, Vol. 32, no. 9 (1991).
[0004]
In the related art, a window is opened for personal use or information exchange between a plurality of terminals, and editing or presentation of meeting materials (documents composed of text, images, figures, and the like) in file units is performed.
[0005]
For this reason, after the meeting, memos and meeting materials during the meeting remain in the user's hand as minutes of the meeting, but are also kept as minutes of the meeting including dynamic information that is difficult to handle systematically such as the situation of the meeting. Cannot be performed (for example, dynamic information such as the lapse of time of finger position information when one of the participants indicates a material presented by the camera with a finger).
[0006]
Therefore, the conventional method is not sufficient from the viewpoint of helping the user's memory.
[0007]
In addition, a method of using a VTR or the like to record the status of the conference is conceivable. However, since all of the status of the conference is captured by the VTR or the like, a great deal of information is generated. Searching and editing audio information requires a great deal of effort from the user.
[0008]
Furthermore, in the conventional CAI (Computer Aided Education System) system, the purpose was to share teaching materials between teachers and students and to set up a place for conversation, so that students could review from a personal perspective after class. It was difficult for teachers to create teaching materials that reflect the class situation.
[0009]
[Problems to be solved by the invention]
In the conventional method, a window is opened for personal use or information exchange between a plurality of terminals, and a meeting material (a document composed of text, images, figures, and the like) is edited and presented in file units.
[0010]
For this reason, after the meeting, memos and meeting materials during the meeting remain in the user's hand as minutes of the meeting, but are also kept as minutes of the meeting, including dynamic information that is difficult to handle systematically, such as the situation of the meeting. I can't.
[0011]
In addition, since the state of the conference is all enormous even if the state of the conference is taken by a VTR or the like, searching and editing the information of the captured video and audio after the conference requires a great deal of labor for the user.
[0012]
Therefore, there is a problem that the conventional method is not sufficient from the viewpoint of helping the user's memory and a problem that it is necessary to record necessary information in a required amount.
[0013]
An object of the present invention is to manage various information created by a user with a work situation management device and to manage necessary information according to the work situation of the user.
[0014]
[Means for Solving the Problems]
The work situation management device according to the present invention requires a storage unit for storing information indicating the time course of the work, and the work required for the work based on the information indicating the time course of the work stored in the storage unit. A time slot identifying means for identifying a time slot to which a keyword is to be attached, a keyword candidate identifying means for identifying at least one keyword candidate for the time slot identified by the time slot identifying means; Keyword determining means for selecting one keyword candidate from at least one keyword candidate in accordance with a predetermined rule, and determining the selected keyword candidate as a keyword corresponding to the time zone. Objective is achieved.
[0015]
The information indicating the time lapse of the work is information for identifying a sound part and a silent part included in audio information generated during the work, and the time zone specifying unit corresponds to the sound part. May be specified as a time zone to which a keyword should be attached.
[0016]
The information indicating the time lapse of the work is information indicating a time zone in which a window displaying material information is estimated to have been noticed by a user among the time required for the work, and The specifying means may specify only the time zone in which the window is estimated to be noticed by the user as a time zone to which a keyword is to be attached.
[0017]
The information indicating the time lapse of the work is information indicating a time zone in which the instruction information is generated for the window displaying the material information, of the time required for the work, and the time zone identification unit includes: Only the time zone in which the instruction information is generated for the window may be specified as a time zone to which a keyword should be attached.
[0018]
The information indicating the time lapse of the work includes information for identifying a sound part and a silent part included in audio information generated during the work, and material information among time required for the work. At least one of information indicating a time period in which the window is estimated to be focused on by the user and information indicating a time period in which the instruction information is generated for the window, out of the time required for the work The time zone identification means generates the instruction information for the time zone and the window corresponding to the sound part and for the time zone and the window where it is estimated that the user is paying attention to the time zone and the window. Only a time zone determined based on at least one of the time zones described above may be specified as a time zone to which a keyword should be assigned.
[0019]
When material information including editable character information is used in the work, the keyword candidate specifying means may determine the first character information in the material information at a first time of the time required for the work and the first character information. Difference information storage means for storing difference information representing a difference between the second information and the second character information in the material information at the second time of the time required for the work, and the difference information stored in the difference information storage means Document keyword extracting means for extracting at least one keyword candidate from the keyword.
[0020]
The keyword candidate specifying means, when material information including character information is used in the work, position information storage means for storing position information indicating the position of the character information specified by the user during the work, An instruction keyword extracting means for extracting at least one keyword candidate from the material information based on the position information stored in the position information storage means may be provided.
[0021]
The keyword candidate specifying means, when the material information is displayed in a window having a portion for describing a title in the work, a title storage means for storing the title, and the title stored in the title storage means. And a title keyword extracting means for extracting at least one keyword candidate from the keyword.
[0022]
The keyword candidate specifying means, when the material information is displayed in a window having a portion for describing personal information in the work, a personal information storage means for storing the personal information; and a personal information storage means for storing the personal information in the personal information storage means. Personal information keyword extracting means for extracting at least one keyword candidate from the obtained personal information.
[0023]
The keyword candidate specifying unit recognizes voice information generated in the work and generates character information corresponding to the voice information, and voice recognition stores the character information corresponding to the voice information. An information storage unit and a voice keyword extraction unit for extracting at least one keyword candidate from the character information stored in the voice recognition information storage unit may be provided.
[0024]
The keyword candidate specifying unit may include a keyword candidate input unit that receives character information input by a user and uses the received character information as a keyword candidate.
[0025]
The predetermined rule may include a rule for determining a keyword based on an evaluation value related to a keyword appearance ratio.
[0026]
The predetermined rule may include a rule that specifies which keyword is to be selected from the plurality of keywords assigned to the competition section.
[0027]
Another work situation management device of the present invention is a storage means for storing information indicating a time course of work, a search keyword input means for inputting a search keyword from a user, and a search keyword input means for inputting a search keyword. A search means for searching the information indicating the time course of the work stored in the storage means, a search keyword storage means for storing the input search keyword and the search result, and A search keyword evaluation unit for evaluating whether the search keyword is appropriate based on the search keyword, thereby achieving the above object.
[0028]
The search keyword evaluation means may evaluate the search keyword based on at least the number of times the search keyword has been input by a user and the number of times the search result has been employed by the user.
[0029]
According to another aspect of the present invention, there is provided a work situation management device which divides first video information into a plurality of first video blocks and divides the second video information into a plurality of second video blocks. Determining whether one of the plurality of first video blocks and one of the plurality of second video blocks are present, and determining one of the plurality of first video blocks during the time period. If it is determined that one of the plurality of second video blocks exists, one of the video blocks existing in the time zone is determined to be preferentially selected according to a predetermined rule. Video block evaluation means for integrating the first video information and the second video information to generate one video information. Thereby, the above object can be achieved.
[0030]
The predetermined rule may include a rule for determining a video block to be selected based on a temporal pre / post relationship of video blocks existing in the time zone.
[0031]
The predetermined rule may include a rule for determining a video block to be selected based on a change in a work situation.
[0032]
In the present invention, various information created by conference participants is managed by the work status management device, and the user can efficiently extract necessary information (materials, comments, conference status) and work. It is possible to handle even dynamic information that is difficult to handle systematically, such as a conversation situation.
[0033]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0034]
FIG. 1A shows a configuration of a work situation management apparatus 10 according to an embodiment of the present invention. The work status management device 10 includes an input unit 11 for inputting information related to the work, a work status management unit 13 for managing the work status of the user, a work status storage unit 14 for storing the work status, and It has a material information storage unit 15 for storing, and a terminal control unit 12 for controlling the input unit 11 and the work situation management unit 13.
[0035]
Typically, “work” refers to one or more users presenting a material and explaining the material.
[0036]
In particular, in this specification, a typical task is an electronic conference in which a plurality of users review common materials in real time and exchange opinions. However, the operations described in this specification are not limited to such operations. In the present specification, the “work status” refers to a set of time-series information indicating the progress of the work. “Material information” refers to information related to the material presented by the user in the work.
[0037]
FIG. 1B shows a typical work scene when a user presents a material and explains the material. The user sits in front of the work status management device and explains materials. A camera 18 for photographing the material (hereinafter, this camera is referred to as a document camera), a camera 19 for photographing the user (hereinafter, this camera is referred to as an interpersonal camera), and a voice uttered by the user. A microphone 20 for recording is connected to the work status management device. The video information captured by the document camera 18 and the interpersonal camera 19 and the audio information recorded by the microphone 20 are supplied to the terminal control unit 12 via the input unit 11 of the work situation management device. In this way, information indicating the progress of the work, such as what kind of facial expression the user has described, what kind of material was presented in what order, and the like are input to the work situation management device. Become. Further, a keyboard, a mouse, a digitizer, a touch panel, and a light pen may be used as the input unit 11.
[0038]
As described above, various input devices can be connected to the terminal control unit 12 as the input unit 11. An identifier for specifying the input device connected to the terminal control unit 12 is set in the terminal control unit 12 in advance. When information is input from a plurality of input devices, the terminal control unit 12 identifies which information has been input from which input device based on a preset identifier. For example, when video information captured by the interpersonal camera 19 is supplied to the terminal control unit 12, the terminal control unit 12 stores a pair of an identifier for identifying the interpersonal camera 19 and the video information in the work status management unit 13 Output to
[0039]
The work situation management unit 13 detects that a predetermined change has occurred in the input information. When a plurality of pieces of information are input to the work status management unit 13, the work status management unit 13 detects that a predetermined change has occurred in each of the plurality of pieces of information. The predetermined change may be a change common to the plurality of pieces of information, or may be a change different from each other according to the plurality of pieces of information. When detecting that a predetermined change has occurred with respect to the input information, the work status management unit 13 sets information indicating the time at which the predetermined change occurred and information specifying the predetermined change as a work status. It is stored in the work status storage unit 14. By storing such information in the work status storage unit 14, it is possible to search for a desired part in the work by using a predetermined change to specific information as a search key. The input audio information and video information are also stored in the work situation storage unit 14 as work situations.
[0040]
The material information storage unit 15 stores material information. As the material information storage unit 15, a device such as a magnetic disk, a VTR, and an optical disk is used.
[0041]
The work status management device 10 may further include an output unit 16 that outputs the work status and the material information, and a transmission unit 17 that is connected to another device via a network. As the output unit 16, a device such as a display, a speaker, and a printer is used. As the transmission unit 12, a device such as a local area network (LAN), a cable television (CATV), a modem, and a digital PBX is used.
[0042]
FIG. 2 shows the work situation management device 10 connected to a plurality of terminal devices 20 via a network. Each of the plurality of terminal devices 20 includes an input unit 21 for inputting information related to work, a transmission unit 22 for connecting the work status management device via a network, and an output unit for outputting work status and material information. 24, and a terminal control unit 23 that controls the input unit 21, the transmission unit 22, and the output unit 24. Information input from the input unit 21 of the terminal device 20 is supplied to the terminal control unit 12 of the work situation management device 10 via the transmission unit 22 and the transmission unit 17. In the terminal control unit 12, an identifier for specifying an input device connected to the terminal control unit 12 via a network and an input device directly connected to the terminal control unit 12 are set in advance. When information is input from a plurality of input devices, the terminal control unit 12 identifies which information has been input from which input device based on a preset identifier. In this way, information indicating the time progress of the work is collected by the work status management device 10 from each of the plurality of terminal devices 20 used by the plurality of users. As the input unit 21 of the terminal device 20, devices such as a keyboard, a mouse, a digitizer, a touch panel, a light pen, a camera, and a microphone are used. As the output unit 24 of the terminal device 20, devices such as a display, a speaker, and a printer are used. As the transmission unit 22 of the terminal device 20, devices such as a local area network (LAN), a cable television (CATV), a modem, and a digital PBX are used.
[0043]
FIG. 3 shows a configuration example of the work status management unit 13. The work situation management unit 13 controls a video information management unit 31 that manages changes in video information, an audio information management unit 32 that manages changes in audio information, and controls the video information management unit 31 and the audio information management unit 32. And a work status control unit 33. In the present specification, the “video information” includes all pieces of information indicating a lapse of time of a work related to a video. For example, the video information includes not only a video composed of a plurality of frames captured by the camera but also a control signal generated by operating the camera. In the present specification, the “sound information” includes all pieces of information indicating the time lapse of work related to sound. For example, an audio signal generated by a microphone is included in audio information.
[0044]
The video information input from the input unit 11 is input to the video information management unit 31 via the work status control unit 33. The video information management unit 31 detects that a predetermined change has occurred in the input video information, and generates information indicating a time when the predetermined change has occurred and information specifying the predetermined change. .
[0045]
The voice information input from the input unit 11 is input to the voice information management unit 32 via the work status control unit 33. The video information management unit 31 detects that a predetermined change has occurred in the input audio information, and generates information indicating a time at which the predetermined change has occurred and information identifying the predetermined change. .
[0046]
The work status management unit 13 shown in FIG. 3 limits the targets to be managed as the work status to video information and audio information. As a result, the work status management unit 13 does not require a display device for displaying a window or an input device for instructing a window, and thus has an advantage that size reduction is easy. By expanding the functions of a normal VTR device, a work situation management device having a size almost equal to that of a normal VTR device will be realized. In addition, since video information can be used, it is possible to record facial expressions of conference participants and data of a three-dimensional shape that is difficult to be captured by a computer. Therefore, especially in the case of a bargain-strength conference where it is necessary to analyze the expression of the other party, or in the case of storing a process of assembling and operating a three-dimensional shape which is difficult to be taken into a computer, the work status management unit 13 is provided by the video information management unit. Preferably, it has 31.
[0047]
FIG. 4 shows another configuration example of the work status management unit 13. The work status management unit 13 controls the voice information management unit 32 that manages changes in voice information, the window information management unit 43 that manages changes in window information, and the voice information management unit 32 and the window information management unit 43. And a work status control unit 33 that performs the operation. In the present specification, “window information” refers to information indicating resources possessed by a window. For example, the number of windows, the size of the windows, and the positions of the windows are included in the window information. When the window information changes due to a user operation, a control signal indicating the change of the window information is input to the window information management unit 43 via the input unit 11. The terminal control unit 12 detects that the window information has changed due to a user operation. The part of the terminal control unit 12 that is in charge of detecting the window information is usually called a window management unit (not shown). The window information management unit 43 receives the input control signal, and generates information indicating the time at which the control signal was received and information specifying the control signal. The information generated by the window information management unit 43 is sent to the work situation control unit 33, and is stored in the work situation storage unit 14 by the work situation control unit 33. In this way, by storing the change of the window information while the user is working in the work status storage unit 14, the window operation of the user while the user is working is used as a key. It is possible to search for audio information and video information using the information. As a result, the user can easily look back on important points in the course of the work.
[0048]
The work status management unit 13 shown in FIG. 4 does not store video information requiring a large amount of storage capacity in the work status recording unit 14. Therefore, there is an advantage that the amount of information stored in the work status recording unit 14 can be significantly reduced. Further, the configuration of the work status management unit 13 shown in FIG. 4 extends the function of recording a conference status when users gather in the same place in a conference room or the like, or extending the function of a normal telephone mainly handling voice information. This is suitable for realizing a work situation management device.
[0049]
FIG. 5 shows another configuration example of the work status management unit 13. This configuration is a configuration in which a video information management unit 31 that manages changes in video information is added to the configuration shown in FIG. With such a configuration, it is possible to integrally manage the video / audio information in the real space and the window information which is a resource in the computer.
[0050]
FIG. 6 shows another configuration example of the work status management unit 13. The work status management unit 13 controls a voice information management unit 32 that manages changes in voice information, an instruction information management unit 53 that manages changes in instruction information, and controls the voice information management unit 32 and the instruction information management unit 53. And a work status control unit 33. In this specification, "instruction information" refers to information indicating an instruction for material information. For example, the position of the mouse pointer and the coordinate position detected by the touch panel are included in the instruction information.
[0051]
The instruction information input from the input unit 11 is input to the instruction information management unit 53 via the work status control unit 33. The instruction information management unit 53 detects that a predetermined change has occurred to the input instruction information, and generates information indicating a time at which the predetermined change has occurred and information specifying the predetermined change. .
[0052]
According to the work status management unit 13 shown in FIG. 6, since it is possible to detect a portion where a change in the instruction information and a change in the voice information occur at the same time, a search of the conference status is performed based on the position of the material explained by the user Can be easily performed. The reason is that, when a person tries to explain a certain matter (material), he or she often specifies the material almost at the same time as generating a sound. The work status management unit 13 shown in FIG. 6 does not store video information requiring a large amount of storage capacity in the work status recording unit 14 similarly to the work status management unit 13 shown in FIG. Therefore, there is an advantage that the amount of information stored in the work status recording unit 14 can be significantly reduced. Also, the configuration of the work status management unit 13 shown in FIG. 6 is similar to the configuration of the work status management unit 13 shown in FIG. 4, and records the status of a conference when users gather in the same place in a conference room or the like. It is suitable for a case or a case where a work situation management device is realized by extending the function of a normal telephone mainly handling voice information. Furthermore, the configuration of the work status management unit 13 shown in FIG. 6 is suitable for work in which the number of operations on windows is small as compared with the configuration of the work status management unit 13 shown in FIG. For example, a report-type meeting where writing to the material does not occur very often.
[0053]
FIG. 7 shows another configuration example of the work status management unit 13. This configuration is a configuration in which a video information management unit 31 that manages changes in video information is added to the configuration shown in FIG. With such a configuration, it is possible to integrally manage the video / audio information in the real space and the instruction information which is a resource in the computer.
[0054]
FIG. 8 shows another configuration example of the work status management unit 13. This configuration is obtained by integrating the configurations shown in FIGS. With such a configuration, there is an advantage that the advantages of each of the above-described configurations can be brought out.
[0055]
FIG. 9 shows the configuration of the video information management unit 31. The video information management unit 31 includes a camera operation detection unit 91 that detects a camera operation, a video scene change detection unit 92 that detects a change in a video scene, a video channel change detection unit 93 that detects a change in a video channel, A video information generation unit 94 that generates information indicating the time at which the change has occurred and information specifying the change in response to the change of the information, and a video information management control unit 95 are included.
[0056]
The camera operation detection section 91 detects a predetermined camera operation. The reason why the camera operation is detected is that, in many cases, before and after the camera operation has occurred, it can be considered that information to be noticed by the user has occurred. When a camera connected to the terminal control unit 12 is operated, a camera operation signal is input to the terminal control unit 12 according to the camera operation. Camera operation includes zoom operation to change the magnification of the image with respect to the subject, focus operation to focus on the subject, pan operation to change the camera direction horizontally with the camera position fixed, and camera position A tilt operation of changing the direction of the camera in the up and down direction in a fixed state. The camera operation signal includes a zoom operation signal indicating a zoom operation, a focus operation signal indicating a focus operation, a pan operation signal indicating a pan operation, and a tilt operation signal indicating a tilt operation. The terminal control unit 12 identifies the camera from which the camera operation signal has been input, and sends the camera identifier and the camera operation signal to the work status management unit 13. The identifier of the camera and the camera operation signal are input to the camera operation detection unit 91 via the work status control unit 33 and the video information management control unit 95. The camera operation detection unit 91 determines whether a predetermined change has occurred in the input camera operation signal. For example, when the camera operation signal is represented by an analog value proportional to the operation amount, when the camera operation signal exceeds a predetermined level, it is determined that a predetermined change has occurred. The predetermined level may be zero. When the camera operation signal is represented by a digital value of 0 or 1, when the camera operation signal changes from 0 to 1, it is determined that a predetermined change has occurred. Here, the digital value 0 indicates a state where the camera operation is not performed, and the digital value 1 indicates a state where the camera operation is performed. When it is determined that a predetermined change has occurred in the input camera operation signal, the camera operation detection unit 91 sends a detection signal indicating the predetermined change to the video information generation unit 94. The video information generation unit 94 generates information indicating a time when the camera operation has occurred and information specifying the camera operation in accordance with the detection signal from the camera operation detection unit 91. The information indicating the time at which the predetermined change occurred is a character string indicating at least one of year, month, day, hour, minute, and second. "12:15:10" and "5/3 18:03" are examples of the character string. Alternatively, the information indicating the time at which the predetermined change occurred may be binary data instead of a character string. Such information indicating the time is generated by inquiring the timer unit (not shown) for managing the current time about the current time.
[0057]
Next, the video scene change detection unit 92 will be described. It is assumed that an interpersonal camera for photographing a user's face and a document camera for photographing material information are connected to the terminal control unit 12. The purpose of the video scene change detection unit 92 is to detect the movement of a user sitting in front of the interpersonal camera, and to move the material information captured by the document camera or to indicate the material information. Is to detect the movement of the object. The video captured by the interpersonal camera and the document camera is input to the video scene change detection unit 92 via the work situation control unit 33 and the video information management control unit 95. The video scene change detection unit 92 calculates a difference between frames of the input video, and determines whether the difference is larger than a predetermined value. When it is determined that the difference is larger than the predetermined value, the video scene change detection unit 92 determines that a change in the video scene has occurred, and sends a detection signal indicating the change to the video information generation unit 94. The video information generation unit 94 generates information indicating the time at which the change of the video scene has occurred and information specifying the change of the video scene, according to the detection signal from the video scene change detection unit 92.
[0058]
When a sensor that detects the movement of the user's hand with respect to the document information is provided, the video scene change detection unit 92 detects the change in the video scene based on a difference between frames of the video instead of detecting the change. A change in a video scene may be detected according to an output signal from the sensor. For example, the sensor detects that the user's hand has blocked at least a part of the material information. Similarly, when a sensor for detecting the movement of a user sitting in front of the interpersonal camera is provided, the video scene change detection unit 92 changes the video scene based on the difference between video frames. , The change in the video scene may be detected according to the output signal from the sensor. For example, the sensor detects that the user has left. The sensor generates an output signal having a value of 1 only when a predetermined movement is detected. As such a sensor, an infrared sensor or an ultrasonic sensor can be used. The video scene change detection unit 92 receives an output signal from the sensor and determines whether or not the value of the output signal is 1. If it is determined that the value of the output signal is 1, the video scene change detection unit 92 determines that a change in the video scene has occurred, and sends a detection signal indicating the change to the video information generation unit 94. . The video information generation unit 94 generates information indicating the time at which the change of the video scene has occurred and information specifying the change of the video scene, according to the detection signal from the video scene change detection unit 92.
[0059]
Next, the video channel change detection unit 93 will be described. It is assumed that four cameras (first to fourth cameras) are connected to the terminal control unit 12. It does not matter whether those cameras are connected to the terminal control unit 12 via a network or are directly connected to the terminal control unit 12. The terminal control unit 12 has a function of allocating an input from the camera to a window and managing an allocation relationship between the input from the camera and the window. For example, the terminal control unit 12 assigns an input from the first camera to the first window, and assigns an input from the second camera to the second window. In the present specification, “change in video channel” refers to changing an assignment relationship between an input from a camera and a window. For example, when the above-mentioned assignment relationship is changed so that the input from the third camera is assigned to the first window and the input from the fourth camera is assigned to the second window, it is said that a change in the video channel has occurred. The terminal control unit 12 changes the assignment relationship between the input from the camera and the window according to a predetermined command input by the user or a predetermined control command from a program. For example, if the moderator of the conference wants to always display the face of the conference participant who wants to speak in the same window, the moderator of the conference inputs a command to switch the video channel each time the moderator changes. Maybe. Alternatively, the program may automatically switch the video channel at regular time intervals to display the participants' faces evenly in the same window. When detecting a predetermined command or a predetermined control command from a program, the video channel change detection unit 93 regards that a change in the video channel has occurred and sends a detection signal indicating the change to the video information generation unit 94. . The video information generation unit 94 generates information indicating the time at which the change of the video channel has occurred and information specifying the change of the video channel according to the detection signal from the video channel change detection unit 93. Detecting a change in a video scene is particularly effective when the purpose of using the video channel (for example, a video channel through which a video of a conference participant flows) is clear. Furthermore, according to the video channel change detection unit 93, even when information regarding camera operation is not stored at the time of shooting, it is possible to detect a change in a video scene based on only the shot video information.
[0060]
As described above, the functions of the camera operation detection unit 91, the video scene change detection unit 92, and the video channel change detection unit 93 are independent of each other. Therefore, the video information management unit 31 can be configured to include one or any two of the camera operation detection unit 91, the video scene change detection unit 92, and the video channel change detection unit 93. .
[0061]
FIG. 10 shows the configuration of the audio information management unit 32. The audio information management unit 32 includes an audio information division unit 101 that divides an input audio signal into a sound part and a silent part based on the power of the audio signal input from the microphone, In response to the change to the sound part, the voice information generating unit 102 generates information indicating the time at which the change occurred and information specifying the change, and controls the voice information dividing unit 101 and the voice information generating unit 102 And a voice information management control unit 103 that performs the operation.
[0062]
The audio information dividing unit 101 measures the power of an input audio signal, and divides the input audio signal into a sound part and a silent part based on the measurement result. A specific method of dividing the audio signal into a sound part and a silent part will be described later with reference to FIG. Based on the audio division, the audio information division unit 101 detects a change in the audio signal from a silent part to a sound part and the number of audio blocks in which the sound part continues. The audio information generation unit 102 receives information indicating the time at which the audio signal changed from a silent part to a sound part and information indicating the number of audio blocks in which the sound part continues according to the detection signal from the sound information division part 101. Generate Information indicating the time at which the audio signal changed from a silent part to a sound part and information indicating the number of audio blocks in which the sound part continues are stored in the work situation storage unit 14. In this way, by storing the time at which the audio signal changed from a silent part to a sound part and the number of audio blocks in which the sound part continues in the work status storage part 14, it is possible to deal with the sound part of the audio signal. It is possible to reproduce only the video information recorded or used by the user during the time period. As a result, the user can easily look back on important points in the course of the work.
[0063]
FIG. 11 is a diagram illustrating the configuration of the window information management unit 43. The window information management unit 43 includes a window generation / destruction detection unit 111 that detects generation / destruction of a window, a window size change detection unit 112 that detects a change in window size, and a window display that detects a change in the display position of the window. A position change detecting unit 113, a window focus change detecting unit 114 for detecting a change in focus on a window (a switching operation of a window to be edited (topic) between users), and a display area of information to be displayed in the window , A window display area change detecting unit 115 for detecting a change in the overlapping relationship between a plurality of windows, a display change detecting unit 116 for detecting a change in an overlapping relationship between a plurality of windows, and a time at which the change occurs in response to a change in window information. Window for generating information indicating the change and information specifying the change A broadcast generating unit 117, and a window information management control unit 118.
[0064]
The window generation / destruction detection unit 111 detects generation of a window or destruction of a window, and sends a detection signal to the window information generation unit 117. The other detection units 112 to 116 similarly detect a predetermined change and send a detection signal to the window information generation unit 117. The window information generation unit 117 receives the detection signal, and generates information indicating a time when the change has occurred and information specifying the change according to the detection signal.
[0065]
FIG. 12 shows the configuration of the instruction information management unit 53. The instruction information management unit 53 includes an instruction information detection unit 121 that detects a change in the instruction information, and an instruction that generates, in response to the change in the instruction information, information indicating a time when the change has occurred and information that specifies the change. It includes an information generation unit 122 and an instruction information management control unit 123.
[0066]
The operation of the instruction information management unit 53 will be described using an example of an instruction using a mouse pointer. When the mouse button is pressed by the user, a signal indicating that the mouse button is pressed and a signal indicating the coordinate position of the mouse pointer are input to the instruction information detection unit 121. The instruction information detection unit 121 detects a predetermined change in the coordinate position of the mouse pointer, and generates a detection signal indicating the predetermined change. For example, the predetermined change is that the mouse pointer moves from one position on the window to another position. Alternatively, the predetermined change may be that the mouse pointer moves from within a certain area on the window to outside the area. Alternatively, the predetermined change may be that the mouse button is double-clicked, or that the mouse is being dragged. The instruction information generation unit 122 generates information indicating the time at which the change has occurred and information specifying the change according to the detection signal from the instruction information detection unit 121.
[0067]
FIG. 13 shows an example of information generated by the voice information generation unit 102 and stored in the work status storage unit 14 by the work status control unit 33. In this example, the start time of the sound part is stored as information indicating the time when the change of the audio information occurs. Further, as information for specifying a change in audio information, an identifier of an audio block, a user who has emitted audio, and an audio block length of a sound part are stored. The user who has issued the voice is specified based on the correspondence between the identifier of the input device and the user. This correspondence is set in advance. For example, the first line in FIG. 13 has only “15:15:10” and “15 block length (seconds)” in the voice information input from the microphone connected to the terminal device of “Mr. Yamaguchi”. This indicates the work situation in which the sound continued.
[0068]
FIG. 14 illustrates an example of information generated by the video information generation unit 94 and stored in the work status storage unit 14 by the work status control unit 33. In this example, the event occurrence time is stored as information indicating the time at which the change in the video information has occurred. Further, as the information for specifying the change of the video information, an occurrence event, an event person, and an occurrence position are stored. In this specification, “event” is defined as having the same meaning as a predetermined change. The occurrence event includes a change in a video scene. The event person and the occurrence position are specified based on the correspondence between the identifier of the input device and the user and the use of the input device. This correspondence is set in advance. For example, in the first line of FIG. 14, in the video information input from the “document camera” connected to the terminal device of “Mr. Yamaguchi”, the event “change of video scene” at “5/318: 03” This indicates the work situation in which an error has occurred.
[0069]
In addition, as a method for detecting a change in video information, there is a method of adding an infrared sensor for detecting hand movement to a document camera for presenting materials, and a person-person for photographing a user's facial expression. There is a method of adding an ultrasonic sensor for checking a user's presence status to a camera. With these methods, a change in video information can be detected. As described above, by using various sensors according to the purpose, the motion information of the user can be obtained. Also, motion information can be obtained by using difference information between frames of video information obtained by a camera. Details will be described later with reference to FIG. 27 below.
[0070]
FIG. 15 shows another example of information generated by the video information generation unit 94 and stored in the work status storage unit 14 by the work status control unit 33. In this example, the occurrence event includes a change in the camera operation and a change in the video channel in addition to the change in the video scene described in FIG. For example, in the first line of FIG. 15, in the video information input from the “document camera” connected to the terminal device of “Mr. Yamaguchi”, the event “zoom enlargement” occurs at “5/3 18:03” Indicates the work situation that occurred.
[0071]
FIG. 16 illustrates an example of information generated by the window information generation unit 117 and the instruction information generation unit 122 and stored in the work status storage unit 14 by the work status control unit 33. In this example, the event occurrence time is stored as information indicating the time when the window information or the instruction information has changed. Further, as an information for specifying a change in the window information or the instruction information, an occurrence event, an event person, and an occurrence position are stored. The event person and the occurrence position are specified based on the correspondence between the identifier of the input device and the user and the use of the input device. This correspondence is set in advance. For example, the first line of FIG. 15 indicates that “mouse” is “5/3 18:03” in “Chapter 1” of the material of “Material No. 1” displayed in the window of the terminal device of “Mr. Yamaguchi”. This indicates a work situation in which an event "pointer by pointer" has occurred. Operations on a window may be performed on a logical page, chapter, or section basis. Further, when the window has a personal memo description section for describing a personal memo, attention may be paid to a change in the content of the personal memo description section. By storing the work status in the work status storage unit 14 in this manner, it becomes possible for the user to search for video information and audio information captured during the work based on the memory during the work. .
[0072]
Referring to FIGS. 17 to 20, when a plurality of users hold an electronic conference using a plurality of terminal devices interconnected by a network, it is preferable that the work status management unit 13 manage the electronic conference. Is exemplified.
[0073]
With reference to FIG. 17, a method of determining a window of interest to a user by detecting a change in window information will be described. Hereinafter, a window estimated by the work situation management unit 13 to be focused on by the user is referred to as a window of interest. A method for changing the window information will be described taking a change in window size as an example. It is assumed that the window has a window size changing unit for changing the window size. In a known window system, a window size changing unit is often provided in a peripheral portion of a window. Normally, the user changes the size of the window by dragging the mouse while pointing the window size changing unit with the mouse. The work situation management unit 13 detects a change in the window size, and determines that the window whose size has been changed is the window of interest. The work status management unit 13 stores information indicating which window is the window of interest in the work status storage unit 14 in chronological order. If the window size can be changed for a plurality of windows, the work status management unit 13 may determine the window whose size has been changed most recently as the window of interest. Alternatively, the work situation management unit 13 may determine that a window having a size larger than a predetermined size is the window of interest. Further, when the time interval in which the window is focused is shorter than the predetermined time interval, it may be determined that the user is searching for the material, and it may be determined that the window is not focused. This is because such a window is presumed not to be the subject of the main topic of the user. Similarly, the window of interest can be determined using a change in window information other than a change in window size (for example, a change in window focus or a change in display between windows).
[0074]
With reference to FIG. 18, a method of determining a window of interest by a user using the window owner information will be described. The editing area displayed on the display includes, as shown in FIG. 18, a co-editing area 181 that can be edited by a plurality of users and a personal editing area 182 that can be edited only by one user. It is assumed that the position of the area 181 and the position of the personal editing area 182 are set in advance. The work situation management unit 13 detects that the position of the window has been moved from the personal information editing area 182 to the joint information editing area 181 by the operation of the user, and determines that the moved window is the window of interest. The work status management unit 13 chronologically displays the information indicating which window is the window of interest and the information indicating which of the co-editing region 181 and the personal editing region 182 is located in the work status. The information is stored in the storage unit 14.
[0075]
With reference to FIG. 19, a method of determining information focused on by the user by detecting a change in the window display area will be described. It is assumed that the window has a window display area changing unit 191 for scrolling display contents. In a known window system, the window display area changing unit 191 often has a scroll bar type user interface. However, the window display area changing unit 191 may have another user interface such as a push button type. When the user operates the window display area change unit 191, the display contents of the window are scrolled. The work status management unit 13 detects that the window display area has changed. After the window display area changes, the work situation management unit 13 determines whether or not the audio signal of a predetermined level or more continues for a predetermined time (for example, one second or more). The reason why such a judgment is effective is that when explaining a material to another person, a person points to a specific position of the material and clarifies the object of the explanation, and then uses voice (words) to give the person his / her own voice. This is because they often try to convey their intentions. After the window display area is changed, if it is determined that the audio signal of a predetermined level or more has continued for a predetermined time or more, the work situation management unit 13 sets the time and data of the material information of interest to the user. The positional information (for example, a document name or an item name) is stored in the work status storage unit 14. Further, after the window display area changes, the work status management unit 13 detects that an instruction for the material information has occurred, and uses the temporal and positional information of the instruction as information indicating the point of interest of the user. The information may be stored in the storage unit 14. Further, by combining the above two detection methods, when the work situation management unit 13 detects the voice uttered by the user for a predetermined time or more and detects that the instruction to the material information has been issued, the user Temporal and positional information of the material information of interest may be stored in the work situation storage unit 14.
[0076]
With reference to FIGS. 20 and 21, a method of detecting the point of interest of the user for the video information will be described. As shown in FIG. 21, it is assumed that a document camera for capturing material information is connected to the terminal device. The work situation management unit 13 detects that voice information has been generated by the user after a predetermined camera operation has been performed by the user. The predetermined camera operation is, for example, switching of a video channel when there are a plurality of video sources, zoom operation of a camera, operation of a recording device such as a VTR device, and the like. The reason why such detection is effective is that, after performing a predetermined camera operation, the user often makes a sound to intentionally explain something. The work situation management unit 13 determines that the generation of the audio information at such a timing indicates the point of interest of the user, and the temporal and positional information indicating the point of interest of the user (for example, Information indicating when the position was instructed) is stored in the work status storage unit 14.
[0077]
FIG. 20 shows that during a teleconference, a certain user uses a document camera to display a document illustrating a “circuit board”, and another participant instructs himself / herself to an image of the “circuit board”. This shows the image being overlaid (overlaid). Here, the conversation state of the voice information (for example, who issued the information that can be regarded as a sound part) is stored for each user so that it is possible to determine who made the noticeable statement and when. You can easily search. The work situation management unit 13 detects that an instruction to material information has been issued after the user performed a camera operation. The work situation management unit 13 determines that the instruction to the material information at such a timing indicates the point of interest of the user, and stores the temporal and positional information of the instruction in the work situation storage unit 14. As a method of detecting an instruction for the material information, for example, a method of detecting an instruction with a mouse pointer, or as shown in FIG. There is a way to detect it. As a method of detecting an instruction for material information using video information captured by a document camera, a difference between frames in video information may be used. Alternatively, the work situation management unit 13 detects the audio information emitted by the user after the user performs the camera operation, and when it detects that the instruction to the material information has been generated, the time of the instruction is reduced. Alternatively, the positional information may be stored in the work status storage unit 14 as information indicating the point of interest of the user. The reason why such detection is effective is that when explaining a material to another person, a person points to a specific position of the material, clarifies the object of the explanation, and then uses voice (words) to give the other person his / her own voice. This is because they often try to convey their intentions. In particular, as shown in FIG. 20, when a plurality of users discuss the video while watching the video, an instruction for the audio generation time (a section that becomes a sound part of the voice) and the video is given. It is effective to memorize for each user. The reason is that it is easy to search and edit the material information because the time point at which the user is estimated to pay attention to the video is known for each user. Furthermore, by recording or outputting only the video information and the audio information at the time when it is estimated that the user is paying attention, it is possible to reduce the amount of information presented to the user and the storage capacity.
[0078]
Next, a work situation management device having a keyword management unit 220 for adding a keyword to video information or audio information using the work situation stored in the work situation storage unit 14 will be described. In this specification, "adding a keyword to video information or audio information" means determining a keyword corresponding to the time zone t for the time zone t. For example, the keyword management unit 220 ₁ For the keyword "A" and time zone t ₂ For the keyword "B" and time zone t ₃ Is assigned a keyword “C”. Since the video information or the audio information is represented by the function of the time t, it is possible to search for a desired portion of the video information or the audio information using the keyword as a search key.
[0079]
FIG. 22 shows the configuration of the keyword management unit 220. The keyword management unit 220 inputs information indicating the time progress of the work from the work status storage unit 14, and stores a set of a time zone t and a keyword K (t) corresponding to the time zone t in the keyword storage unit 224 (t, K (t)). The keyword management unit 220 reads out information indicating the time progress of the work from the work status storage unit 14 and, based on the information, identifies a time period to which a keyword is to be assigned, of the time required for the work. 221, a keyword candidate specifying unit 222 for specifying at least one keyword candidate for the time period specified by the time period specifying unit 221, and selecting one keyword candidate from the keyword candidates according to a predetermined rule; A keyword determining unit 223 that determines the selected keyword candidate as a keyword corresponding to the time zone. The time zone and the keyword corresponding to the time zone are stored in the keyword storage unit 224.
[0080]
As described above, in order for the keyword management unit 220 to add a keyword to video information or audio information, it is necessary that information indicating the time course of the work is stored in the work status storage unit 14 in advance. Information indicating the time progress of the work is generated by the work situation management unit 13 and stored in the work situation storage unit 14. Hereinafter, what kind of information should be stored in the work situation storage unit 14 will be described.
[0081]
FIG. 23A shows the flow of the work of editing a document. For example, editing work such as change, insertion, and deletion is performed on document A, and as a result, document A 'is created. The work status management unit 13 generates a difference between the document A before editing and the document A ′ after editing, and stores information indicating a time at which the difference occurred and information specifying the difference as a work status storage unit 14. Output to The information that specifies the difference is, for example, the name of a file that stores the difference character string. The work status management unit 13 may output information for specifying the edited document A ′ to the work status storage unit 14 instead of the information for specifying the difference. This is because there may be no difference. The timing of acquiring the difference between the document A before editing and the document A ′ after editing may be at regular intervals, or may be when the window is opened or the window is closed. Good.
[0082]
FIG. 23B shows an example of information stored in the work situation storage unit 14 by the work situation management unit 13 when the work shown in FIG. 23A is performed. In this example, the time zone in which the document was edited, the name of the document before editing, the name of the document after editing, and the difference are stored.
[0083]
FIG. 24A shows a scene where a part of the material information is instructed by the user in the work. The user specifies the range of the material information by pointing the material information using a mouse pointer, a touch panel, or the like. In FIG. 24A, the range specified by the user is highlighted. The work status management unit 13 detects the range specified by the user, and outputs information indicating the time at which the instruction has been issued by the user and information specifying the range specified by the user to the work status storage unit 14. I do.
[0084]
FIG. 24B shows an example of information stored in the work situation storage unit 14 by the work situation management unit 13 when the instruction shown in FIG. In this example, the name of the person who issued the instruction, the time zone in which the instruction occurred, and the range specified by the instruction are stored.
[0085]
FIG. 25A shows a scene in which material information is displayed in a window in the work. The window has a title description section 2501 for describing the title of the material information. As the title, for example, the names and numbers of chapters, sections, and sections are described. The work status management unit 13 detects a window of interest by the user, and stores information indicating the time at which the window of interest was detected and information described in the title description unit 2501 of the window in the work status storage unit 14. Output. Further, the window may have a personal information description section 2502 for describing a personal memo of the user. The work status management unit 13 detects a window of interest by the user, and stores information indicating the time at which the window of interest was detected and information described in the personal information description unit 2502 of the window in the work status storage unit 14. Output to
[0086]
FIG. 25B shows an example of information stored in the work status storage unit 14 by the work status management unit 13. In this example, a title, a subject, a time zone in which the window is focused, and a personal memo are stored.
[0087]
FIG. 26A shows the configuration of the voice keyword detection unit 2601. The voice keyword detection unit 2601 is included in the work status management unit 13. The voice keyword detection unit 2601 detects a predetermined voice keyword included in voice information input from the input unit 11, and converts information indicating a time at which the predetermined voice keyword was detected and information indicating the detected voice keyword. Output to the work status storage unit 14. The voice keyword detection unit 2601 includes a voice recognition unit 2602, a voice keyword extraction unit 2603, a voice keyword dictionary 2604, and a voice processing control unit 2605. The voice recognition unit 2602 receives voice information from the input unit 11 and converts the voice information into a character string corresponding to the voice information. The voice keyword extraction unit 2603 receives the character string corresponding to the voice information from the voice recognition unit 2602 and searches the voice keyword dictionary 2604 to extract a voice keyword from the character string corresponding to the voice information. The voice keyword dictionary 2604 stores voice keywords to be extracted in advance. For example, assume that a voice keyword “software” is stored in the voice keyword dictionary 2604 in advance. When the voice information “the feature of this software is to operate at high speed” is input to the voice recognition unit 2602, the voice recognition unit 2602 outputs the text “the feature of this software is to operate at high speed”. Generate a column. The voice keyword extraction unit 2603 receives the character string “the feature of this software is to operate at high speed”, and matches the “software” that is the voice keyword stored in the voice keyword dictionary 2604 from the received character string. Character string to be extracted. The audio processing control unit 2605 controls the above processing.
[0088]
FIG. 26B illustrates an example of information stored in the work status storage unit 14 by the work status management unit 13. In this example, the name of the uttered person, the time period during which the utterance was performed, and the speech keyword extracted from the uttered content are stored.
[0089]
FIG. 27 shows a flow of a keyword addition process to audio information or video information performed by the keyword management unit 220 shown in FIG. The time zone identification unit 221 identifies an evaluation target section (time zone) of the video information or the audio information (step S2701). The method of designating the evaluation target section (time zone) will be described later with reference to FIGS. The keyword candidate specifying unit 222 specifies at least one keyword candidate based on the processing result of each keyword extraction processing unit described later (step S2702). In order to adopt one of the keyword candidates, the keyword determination unit 223 selects a determination rule from keyword determination rules described later (step S2703). The keyword determination unit 223 determines a keyword corresponding to the evaluation target section (time zone) based on the selected determination rule (step S2704).
[0090]
With reference to FIGS. 28A to 28C, a method for specifying an evaluation target section (time zone) of video information or audio information will be described. There are mainly three methods. The first is a method of limiting a range to which a keyword is to be attached to a sound part of audio information. The second method is to limit the range to which a keyword is to be attached to a section in which the user focuses on the window. The method of detecting that the user is paying attention to a specific window has already been described with reference to FIGS. A third method is to limit the range to which a keyword is to be attached to a section where the instruction information has been generated. As described above, the instruction information includes an instruction by a mouse pointer, an instruction by a finger to material information, and the like. A method of combining these target range designation methods is shown in FIGS.
[0091]
FIG. 28A shows a method of limiting a range to which a keyword is to be attached based on window information and audio information. The time zone specifying unit 221 limits the range to which a keyword is to be attached to an overlapping portion between the sound portion of the audio information and the time zone in which the user focuses on the window. In the example shown in FIG. 28A, the time zone T is defined as an overlapping portion between the sound part of the audio information and the time zone in which the user focuses on the window. ₁ , T ₂ Is specified by the time zone specifying unit 221.
[0092]
FIG. 28B shows a method of limiting a range to which a keyword is to be attached based on window information and instruction information. The time zone specifying unit 221 limits the range to which a keyword is to be attached to an overlapping portion between the time zone in which the user focuses on the window and the time zone in which the instruction information is generated. In the example shown in (b) of FIG. 28, the time zone T is defined as an overlapping portion of the time zone in which the user focuses on the window and the time zone in which the instruction information is generated. ₁ , T ₂ , T ₃ Is specified by the time zone specifying unit 221.
[0093]
FIG. 28C shows a method of limiting the range to which a keyword is to be attached based on the instruction information and the voice information. The time zone specifying unit 221 limits the range to which the keyword is to be attached to the overlapping portion between the time zone in which the instruction information is generated and the sound part of the audio information. In the example shown in FIG. 28 (c), the time zone T is used as an overlapping portion between the time zone in which the instruction information is generated and the sound part of the audio information. ₁ , T ₂ , T ₃ Is specified by the time zone specifying unit 221.
[0094]
The above time zone T ₁ , T ₂ , T ₃ , Different keywords may be added, or the same keyword may be added. For example, in the example shown in (a) to (c) of FIG. ₁ , T ₂ , T ₃ To the same keyword "circuit board". As described above, by adding the same keyword to different time zones, it is possible to handle video information having different time zones as a video block which is a logical group having the same keyword. Similarly, by adding the same keyword to different time zones, it is possible to handle audio information with different time zones as audio blocks that are one logical group having the same keyword. As a result, it becomes easy to handle video information and audio information in logical information units.
[0095]
FIG. 29 shows a configuration of the keyword candidate specifying unit 222 shown in FIG. The keyword candidate specifying unit 222 extracts a keyword candidate based on a difference between a document before editing and a document after editing, and a designated keyword extracting unit that extracts a keyword candidate based on designated information. 2902, a personal keyword extraction unit 2903 that extracts keyword candidates based on the contents of the memo described in the personal information description unit 2502, and a title that extracts keyword candidates based on the contents of the title described in the title description unit 2501 It has a keyword extraction unit 2904, a voice keyword extraction unit 2905 for extracting keyword candidates based on voice information, a keyword input unit 2906 for inputting keyword candidates from a user, and a keyword control unit 2907.
[0096]
Next, the operation of the keyword candidate specifying unit 222 will be described. The time period T specified by the time period specifying unit 221 is input to the keyword control unit 2907. The keyword control unit 2907 sends the time period T to each of the extraction units 2901 to 2905 and the keyword input unit 2906. Each of the extraction units 2901 to 2905 extracts a keyword candidate to be added to the time zone T, and returns the extracted keyword candidate to the keyword control unit 2907. The keyword candidates input by the user are also sent to the keyword control unit 2907. In this way, the keyword control unit 2907 collects at least one keyword candidate for the time period T. At least one keyword candidate collected for the time period T is sent to the keyword determination unit 223.
[0097]
For example, it is assumed that a time zone of “10:00 to 10:01” has been input to the keyword candidate specifying unit 222. The document keyword extraction unit 2901 searches the table shown in FIG. 23B stored in the work status storage unit 14. As a result, a time zone of “10:00 to 10:03” (10: 00 → 10: 03) including the time zone of “10:00 to 10:01” is hit. The document keyword extraction unit 2901 extracts a keyword candidate from a difference between documents edited in a hit time slot. As a method of extracting a keyword candidate from a document difference, for example, there is a method of selecting only a character string corresponding to a noun among the character strings included in the document difference as a keyword candidate. To determine whether or not a character string corresponds to a noun, a “kana-kanji conversion dictionary” used by a word processor or the like may be used.
[0098]
The designated keyword extraction unit 2902 searches the table shown in FIG. 24B stored in the work status storage unit 14. As a result, a time zone of “10:00 to 10:01” (10: 00 → 10: 01) that matches the time zone of “10:00 to 10:01” is hit. The designated keyword extracting unit 2902 extracts a keyword candidate from a character string included in the specified range of the hit time zone.
[0099]
Similarly, the personal keyword extraction unit 2903 and the title keyword extraction unit 2904 search the table shown in FIG. 25B stored in the work status storage unit 14. The voice keyword extraction unit 2905 searches the table shown in FIG. 26B stored in the work status storage unit 14.
[0100]
Next, the operation of the keyword determination unit 223 will be described. The keyword determining unit 223 receives at least one keyword candidate from the keyword candidate specifying unit 222, and selects one of the received keyword candidates according to a predetermined keyword determining rule.
[0101]
FIG. 30 is an example of a keyword determination rule. The rules 1 to 4 define which of the extraction units should be preferentially selected from the extracted keyword candidates. Rule 5 defines which of the keyword candidates extracted from the plurality of extraction units should be selected based on the keyword evaluation value.
[0102]
Next, a method of selecting one keyword candidate from a plurality of keyword candidates based on the keyword evaluation value defined in FIG. 31 will be described. The methods are classified into the following four types depending on whether the evaluation by the keyword extraction unit or the difference in the evaluation section is taken into consideration. (1) A method of selecting a keyword candidate based on a keyword evaluation value: When a plurality of keyword candidates are extracted from one keyword extraction unit, one of the plurality of keyword candidates is selected. Used to The keyword evaluation value is a value of a keyword appearance ratio obtained by dividing the number of appearances in the keyword extraction unit by the number of keyword candidates obtained in the keyword extraction unit. (2) Method of selecting keyword candidates based on keyword comprehensive evaluation value: The keyword comprehensive evaluation value takes into account the evaluation results of a plurality of keyword extraction units. The keyword comprehensive evaluation value is obtained by obtaining the product of the keyword evaluation value and the evaluation value for the keyword extraction unit defined in advance by the user for each keyword extraction unit, and obtaining the sum of those products. (3) Method of selecting keyword candidates based on keyword importance: The keyword importance is to comprehensively evaluate keywords of the same name obtained from one keyword extraction unit. The keyword importance is obtained by dividing a keyword evaluation value by a keyword appearance time, which is the time length of a video block or an audio block, for each video block (audio block). Is obtained by calculating the sum total of the unit time keyword evaluation values for the video block (audio block). (4) Method of Selecting Keyword Candidates Based on Keyword Total Importance: The keyword overall importance takes into account the evaluation results of a plurality of keyword extraction units. The keyword overall importance is obtained by obtaining the product of the keyword importance and the evaluation value for the keyword extraction unit defined in advance by the user for each keyword extraction unit, and obtaining the sum of those products.
[0103]
Referring to FIG. 32, a procedure of a method for determining a keyword based on a keyword evaluation value and a keyword importance value will be described with reference to a specific example. First, (1) a keyword evaluation value is determined for each evaluation target section (time zone) to which a keyword is to be assigned. (2) A keyword is determined based on the keyword evaluation value. In the example of FIG. 32, the evaluation target section (time zone) T ₁ The keyword evaluation value of “Circuit Board” is 0.5, “Circuit Drawing” is 0.4, and “Safety” is 0.1 for each keyword. As a result, if the keyword with the highest keyword evaluation value is prioritized, the evaluation target section (time zone) T ₁ Is determined to be “circuit board”. Similarly, the evaluation target section (time zone) T ₂ Is determined to be “Circuit drawing”, and the evaluation target section (time zone) T ₃ Is determined to be “safety” and the evaluation target section (time zone) T ₄ Is determined to be “circuit board”. (3) The same keyword may be added to a plurality of evaluation target sections (time zones). In this case, in order to evaluate the keyword over the plurality of evaluation target sections (time zones), the time length in which the keyword appears is considered. In the example of FIG. 32, the “circuit board” having the keyword evaluation value 0.5 is an evaluation target section (time zone) T having a time length 5 ₁ , And a “circuit board” having a keyword evaluation value of 0.6 is an evaluation target section (time zone) T having a time length of 5 ₄ , The keyword importance of “circuit board” is (0.5 + 0.6) / (5 + 5) = 0.11. Similarly, the keyword importance of “circuit drawing” is 0.1, and the keyword importance of “safety” is 0.25. When the order in which keywords are presented to the user is controlled according to the keyword importance, the order is "security", "circuit board", and "circuit drawing". This makes it possible to control the number of keywords added to video information and audio information so as not to be unnecessarily large.
[0104]
Next, a method for automatically editing conversation information will be described with reference to FIG. This method is one example of using a keyword added to video information or audio information.
[0105]
FIG. 33 shows a procedure of a method for automatically editing conversation information when a keyword is added to video information or audio information based on audio information. The voice information generated by the conversation of the user is divided into a sound part and a soundless part (step S3301). In order to divide audio information into a sound part and a silent part, for example, a threshold value of audio power is determined in advance to distinguish between a sound state and a silent state of the audio information, and division is performed based on the threshold value. Just fine. This dividing method will be described later with reference to FIG. In particular, when a plurality of users work together in one work, the voice information generated during the conversation is recorded and managed for each user, so that the voice information during the conversation can be searched in more detail. It becomes possible to edit. Next, a noise portion is deleted from the audio information obtained in step S3301 (step S3302). For example, if the length of the sound portion of the audio information is shorter than a predetermined time (for example, one second), the audio information may be regarded as noise. When a noise portion is deleted from audio information, the corresponding audio information is replaced with silent information having the same time length. Based on the audio information from which noise has been removed, the video information is divided into a section corresponding to a silent part of the audio information and a section corresponding to a sound part of the audio information (step S3303). A keyword is added to video information (or audio information) based on the keyword adding method shown in FIG. 27 (step S3304). In order to add a keyword to video information (or audio information), for example, a keyword determination rule shown in FIG. 30 may be applied. When there are a plurality of video information channels (or a plurality of audio information channels), a plurality of video blocks (or audio blocks) may exist in one section indicating the same time zone. Hereinafter, in this specification, this section is called a competitive section. When different keywords are added to a plurality of video blocks (or audio blocks) existing in the conflicting section, one keyword is selected from the keywords according to a predetermined keyword integration rule described later. Selection is made (step S3305). Based on the keyword added to the video information (or audio information) and the time at which the video information (or audio information) was recorded, the conversation information is converted into text information (step S3306). Finally, the character information is converted into audio information and output (step S3307). The conversion from the character information to the voice information may use voice synthesis.
[0106]
FIG. 34 shows a procedure of a method of dividing audio information into a sound part and a silent part. A silence timer is set (MT = 0) to measure the time length of the silence section of the voice (step S3401). A status flag indicating whether the voice is a sound part or a silent part is set. That is, St = True (step S3402). If the sound level is lower than the threshold (ThV), the time (TB) at which the sound part starts is set (step S3403). Note that the threshold (ThV) is set in advance based on the level of the voice when no speech is made. Clear the audio status flag. That is, St = False is set (step S3404). If the audio level falls below the threshold (ThV) and the silent section exceeds the threshold time (TM), the audio status flag is set (step S3405). Note that the threshold time (TM) is set in advance to a length of about 400 milliseconds to one second. If the voice level is below the threshold (ThV), the silent section does not exceed the threshold time (TM), and the previous voice section is a sound section, the time (TE) at which the sound section ends is set. (Step S3406). The values of TB and TE are output to the work status storage unit 14 (step S3407). A silence timer is set (step S3408).
[0107]
Next, with reference to FIG. 35 and FIG. 36, the keyword integration rule in the conflict section will be described. Hereinafter, a keyword integration rule when video blocks compete will be described, but the same applies to a case where audio blocks compete. It is assumed that the video block A and the video block B are competing with each other, and there is a competition section C between the video block A and the video block B. Examples of the keyword integration rules include the following four rules (a) to (d). (A) A rule that gives priority to a video block whose start time is earlier. In the example shown in FIG. 35A, since the start time of the video information A is earlier than the start time of the video information B, the keyword “circuit board 1” added to the video information A is selected in the competitive section C. You. (B) A rule that gives priority to a video block whose start time is later. In the example shown in FIG. 35B, since the start time of the video block B is later than the start time of the video information A, the keyword “circuit board 2” added to the video block B is selected in the competitive section C. You. (C) A rule for determining a keyword based on the evaluation value of the user's operation history information (information indicating a situation change) in the competitive section C. In the example shown in FIG. 36 (c), information indicating a situation change is represented by an upward arrow. The number of the arrows indicates the number of occurrences of the situation change. The number of situation changes for video block A in the competition section C is greater than the number of situation changes for video block B in the competition section C. Therefore, in the competitive section C, the keyword “circuit board 1” added to the video block A is selected. (D) A rule for determining a keyword based on the evaluation value of the user's operation history information (information indicating a status change) included in each time slot of a video block. In the example shown in (d) of FIG. 36, the number of status changes for video block B is greater than the number of status changes for video block A. Therefore, in the competitive section C, the keyword “circuit board 2” added to the video block B is selected.
[0108]
FIG. 37 is an example in which a keyword integration rule in a competition section is described. Four rules including the keyword integration rule described above with reference to FIGS. 35 and 36 are described. The keywords in the competition section are determined based on these rules.
[0109]
Next, a description will be given of a documenting unit 380 that generates character information indicating a work situation by using a keyword stored in the keyword storage unit 224. The documentation unit 380 is included in the work status management device.
[0110]
FIG. 38 shows the configuration of the documenting unit 380. The documenting unit 380 includes a time information extracting unit 381 for extracting a relationship between a keyword and a time zone in which the keyword appears (information about When), and a subject extraction for extracting a relationship between the keyword and a subject (information about Who). A document extraction unit 383 for extracting the keyword itself, a documentation rule storage unit 385 for storing documentation rules, and a documentation control unit 384.
[0111]
With reference to FIG. 39, a method of generating character information indicating a work situation will be described. Hereinafter, a method of generating character information indicating a work situation based on video information will be described. The same applies to the case where character information indicating a work situation is generated based on voice information. (A) Attribute information for generating character information is assigned in advance to each video block. The attribute information includes information for specifying a person to be photographed (information about Who), information on the time when photographing was started and ended (information about When), and information for specifying a meeting place virtually set by a user. (Information about Where), information for specifying the target object (information about What), and information indicating whether or not an audio output is present (information about How). The keyword added to the video block may be used as the information for specifying the target. As described above, the attribute information is assigned to each video block in advance so that the work status can be expressed in a text by 5W1H (Who, Why, What, When, Where, How). (B) Select a specific video block from a plurality of video blocks included in the video information according to a predetermined documentation rule. The predetermined documentation rule is created in advance by the user. For example, as shown in rule 1 of FIG. 39 (b), if there is a documentation rule that “silent sections are not documented”, only the video block corresponding to the sound part of the audio information is selected. (C) Based on the attribute information assigned to the video block in advance, character information indicating a work situation corresponding to the selected video block is generated according to a predetermined documentation rule. For example, “Yamaguchi-san” is assigned to the specific video block as the information about Who, “about XX” is assigned as the information about When, and “about” is assigned as the information about What. Assume that "talked" is assigned as information about. In this case, for example, as shown in (c) of FIG. 39, character information “Mr. Yamaguchi talked about △△ around XX” is generated.
[0112]
With reference to FIG. 40, another method for generating character information indicating a work situation will be described. The method comprises the steps of: identifying a sound part in the audio information; identifying a video block corresponding to the sound part; detecting a change in the work situation; and And generating character information for the video block. For example, when a change in a video scene and an audio block are detected, character information “Mr. Yamaguchi, explained with a document camera” can be generated in accordance with rule 3 of FIG. Furthermore, if the keyword added to the video block is "circuit board", the keyword is used as information for identifying the object, and the character information "Mr. Yamaguchi, explanation of the circuit board with a document camera" is used. Can be generated. As a result, it is possible to generate character information indicating the work content according to the video information (or audio information), or to search for video information (or audio information) using the character information as a search key.
[0113]
Next, a description will be given of a keyword search unit 410 that searches for a work situation stored in the work situation storage unit 14 using a keyword stored in the keyword storage unit 224. The keyword search unit 410 is included in the work status management device.
[0114]
FIG. 41 shows the configuration of the keyword search unit 410. The keyword search unit 410 includes a search keyword input unit 411 for inputting a search keyword from a user, a search unit 412 for searching the work status storage unit 14 based on the input search keyword, and an input search. It has a search keyword storage unit 413 that stores keywords and search results, and a search keyword evaluation unit 414 that evaluates whether the search keywords are appropriate based on the search results.
[0115]
Next, the operation of the keyword search unit 410 will be described.
The search keyword input unit 411 inputs a search keyword from a user. In order to facilitate input of a search keyword by a user, the search keyword input unit 411 displays a plurality of keywords stored in the keyword storage unit 224 in a menu format, and uses one of the displayed keywords as a search keyword. The user may be allowed to selectively input. The search keyword input from the search keyword input unit 411 is stored in the search keyword storage unit 413.
[0116]
The search unit 412 searches the work status storage unit 14 based on the input search keyword. More specifically, the search unit 412 determines whether the search keyword matches one of the plurality of keywords stored in the keyword storage unit 224, and determines the video information to which the matched keyword is added. The result is output to the output unit 16 as a search result. Instead of the video information or in addition to the video information, arbitrary information stored in the work situation storage unit 14 may be output to the output unit 16 as a search result. The search unit 412 inquires of the user whether the detection result output to the output unit 16 is a desired result. The response of the user to the inquiry is stored in the search keyword storage unit 413. In this way, information indicating whether a desired search result has been obtained for the input search keyword is accumulated in the search keyword storage unit 413.
[0117]
FIG. 42 illustrates an example of information stored in the search keyword storage unit 413. In this example, in addition to the search keyword entered by the user, the group name to which the user belongs, the user name, the date and time when the search keyword was entered, the item name in which the search keyword was entered, The document name searched based on the search keyword and information indicating whether the searched document matches the document desired by the user are stored. In this example, if the retrieved document matches the document desired by the user, “adopted” is stored; otherwise, “not adopted” is stored. Alternatively, a number indicating the degree of matching between the retrieved document and the document desired by the user may be stored. For example, the degree of coincidence is “70%”. Here, an example in which a document is a search target has been described. Of course, any information stored in the work status storage unit 14 instead of or in addition to the document may be searched. In order to enable a search from a plurality of viewpoints, it is preferable that a plurality of items in which search keywords can be input are provided as shown in FIG. Further, a plurality of document names searched based on the search keyword may be stored in the search keyword storage unit 413.
[0118]
FIG. 43 shows an example of a search panel 430 for inputting a search keyword. The search panel 430 provides a user with a user interface for searching for information. The search panel 430 has a video keyword input unit 431, a document keyword input unit 432, and an event input unit 433. The video keyword input unit 431 displays a plurality of keywords added to the video information in a menu format, and allows the user to selectively input one of the displayed keywords as a search keyword. The document keyword input unit 432 allows a user to input a search keyword for searching for a document. The event input unit 433 displays a terminal state change (eg, a video scene change or a video channel change) generated by operating the document camera, or a terminal state change generated by a user operation on a window (eg, a terminal state change). , The movement of the mouse pointer, the open / closed state of the window, etc.) as search keywords.
[0119]
Next, the operation of the search keyword evaluation unit 414 shown in FIG. 41 will be described.
FIG. 44 shows the flow of processing executed by the search keyword evaluation unit 414. The processing includes a step of specifying an evaluation range (S4401) and a step of evaluating a search keyword in the specified evaluation range (S4402). At least one of the group name, the user name, and the date and time is input to the search keyword evaluation unit 414 to specify the evaluation range. The step of designating the evaluation range (S4401) includes, when a group name is input, extracting a search keyword used by a user belonging to the group from the search keyword storage unit 413 (S4403); When a name is input, a search keyword used by the user is extracted from the search keyword storage unit 413 (S4404), and when a date is input, the search keyword is used from the search keyword storage unit 413 for that date. Extracting the searched search keyword (S4405) and extracting the search keyword from the search keyword storage unit 413 according to a search condition defined by an operator (for example, a logical sum or a logical product) specified by the user. (S4406). The step of evaluating the search keyword in the specified evaluation range (S4402) includes, for the search keyword extracted in step S4401, calculating the hit rate of the search keyword from the number of adoptions and the number of uses of the search keyword (S4407). )including. Here, the hit rate (%) of the search keyword is calculated by the number of adoptions / the number of uses × 100. By presenting search keywords that have been input in the past to the user in descending order of the hit rate, the user can easily input search keywords that have a high probability of obtaining desired search results. As a result, the number of times a user inputs a search keyword before the user obtains a desired search result is reduced. Furthermore, if the evaluation value for the searched information (the degree of coincidence between the information desired by the user and the searched information, for example, a value between 0 and 1) is stored in the search keyword storage unit 413, the desired value can be obtained. It is possible to present to the user a search keyword with a higher probability of obtaining the search result. In this case, the hit rate (%) of the search keyword is calculated by the number of adoptions × the evaluation value / the number of uses × 100.
[0120]
FIG. 45 shows another configuration of the work status management unit 13. The work status management unit 13 controls a video information division unit 451 that divides video information into a plurality of video blocks, a video block evaluation unit 452 that evaluates video blocks, and a video information division unit 451 and a video block evaluation unit 452. And a video information integration control unit 453.
[0121]
Next, the operation of the work situation management unit 13 shown in FIG. 45 will be described.
The video information division unit 451 divides the video information into a plurality of logical video blocks based on the work status stored in the work status storage unit 14. Each video block includes at least one video scene. For example, the video information may be blocked according to the sound part of the audio information. The details of the method of blocking the video information have already been described, and the description is omitted here. In this way, the video information division unit 451 divides the first video information into a plurality of first video blocks, and divides the second video information into a plurality of second video blocks. For example, the first video information is video information captured by the user A, and the second video information is video information captured by the user B.
[0122]
The video block evaluation unit 452 determines whether or not there are a plurality of video blocks in the same time zone, and when it is determined that there are a plurality of video blocks in the same time zone, any one of the plurality of video blocks It is determined whether or not the video block is preferentially selected. Therefore, when one of the plurality of first video blocks and one of the plurality of second video blocks exist in the same time slot, the video block evaluation unit 452 sets the first and second video blocks existing in the same time slot. One of the video block and the second video block is selected. Thus, the first video information and the second video information are integrated, and one video information is generated. Accordingly, it is possible to generate video information indicating a dialogue state between the user A and the user B based on the video information captured by the user A and the video information captured by the user B. .
[0123]
FIG. 46 shows a procedure of the video information integration processing executed by the work situation management unit 13 shown in FIG. The video information division unit 451 generates a plurality of video blocks by dividing the video information into blocks (step S4601). The video block evaluation unit 452 determines whether a plurality of video blocks exist in the same time zone (Step S4602). When it is determined that there are a plurality of video blocks in the same time zone, the video block evaluation unit 452 determines which of the plurality of video blocks is preferentially selected according to a predetermined priority rule. (Step S4603). The predetermined priority rule is set in advance by the user.
[0124]
FIG. 47 shows an example of the priority rule. As shown in FIG. 47, there are various priority rules, such as a priority rule related to a change in the work situation and a priority rule based on the relationship of time.
[0125]
Next, the priority rules of rule numbers 1 to 10 shown in FIG. 47 will be specifically described with reference to FIGS.
[0126]
The priority rule of rule number 1 specifies that when there are a plurality of video blocks in the same time zone, the video block with the earliest start time is preferentially selected. In the example shown in FIG. 48A, the start time of the video block 1a is earlier than the start time of the video block 1b, so the video block 1a is selected.
[0127]
The priority rule of rule number 2 specifies that, when a plurality of video blocks exist in the same time zone, the video block whose start time is the latest is preferentially selected. In the example shown in FIG. 48B, the time period T ₂ In, since the start time of the video block 2b is the latest, the video block 2b is selected. However, time zone T ₁ In, since the start time of the video block 2a is the latest, the video block 2a is selected.
[0128]
The priority rule of rule number 3 specifies that when there are a plurality of video blocks in the same time zone, the video block with the longest time is preferentially selected. In the example shown in FIG. 48C, since the length of the video block 3a is longer than the length of the video block 3b, the video block 3a is selected.
[0129]
The priority rule of rule number 4 specifies that when a plurality of video blocks exist in the same time zone, the video block with the shortest time is preferentially selected. In the example shown in FIG. 49A, since the length of the video block 4b is shorter than the length of the video block 4a, the video block 4b is selected.
[0130]
The priority rule of rule number 5 specifies that, when a plurality of video blocks exist in the same time zone, the video block containing the most information indicating the change in the work situation per unit time is preferentially selected. In the example shown in (b) of FIG. 49, the time at which the information indicating the change in the work situation has occurred is indicated by a triangle. In this example, the video block 5b is selected because the video block 5b contains more information indicating a change in work status per unit time than the video block 5a.
[0131]
The priority rule of rule number 6 specifies that, when a plurality of video blocks exist in the same time zone, a video block that matches a predetermined combination rule of an occurrence event is preferentially selected. In the example shown in FIG. 49 (c), the video block 6b is selected because the video block 6b matches the predetermined combination rule of the event.
[0132]
FIG. 51 shows an example of an event combination rule. The event combination rule defines a combination of events that occur almost simultaneously in the work and an event name corresponding to the combination. For example, when a user explains a material using a document camera, the user often performs the operation while pointing at an object by hand. For this reason, the hand movement and the voice occur almost simultaneously. As shown in the first row of FIG. 51, for example, a combination of an event “change in video scene” and an event “audio block” is defined as an event “explanation with document camera”. When the user explains the material information displayed on the window, the instruction by the mouse pointer and the sound are generated almost simultaneously. As shown in the second line of FIG. 51, for example, a combination of the event “instruction by the mouse pointer” and the event “sound block” is defined as the event “explanation on the window”.
[0133]
Referring to FIG. 50, the priority rule of rule number 7 is that, when a plurality of video blocks exist in the same time zone, the video block corresponding to the time zone using the document information including the designated keyword is specified. Specifies that priority is selected. The priority rule of rule number 8 is that, when there are a plurality of video blocks in the same time zone, the video block corresponding to the time zone using the document information containing the most specified keyword is preferentially selected. Stipulates that In the example shown in FIG. 50A, since the specified keyword is included in the second page of the document information, the video block 7a is selected.
[0134]
The priority rule of rule number 9 specifies that, when a plurality of video blocks exist in the same time zone, the video block corresponding to the time zone in which the specified change in the work situation occurs is preferentially selected. The priority rule of rule number 10 specifies that, when a plurality of video blocks exist in the same time zone, the video block related to the designated subject is preferentially selected. In the example shown in FIG. 50B, the video block 9b is selected by applying the priority rule of the rule number 9, and the video block 9c is selected by applying the priority rule of the rule number 10.
[0135]
FIG. 52 shows an operation panel 5200 for operating information. The operation panel 5200 provides a user with a user interface for the work status management device. As shown in FIG. 52, the operation panel 5200 divides the video information into a video block composed of at least one video frame and displays a result of the division, and the audio is divided into a sound part and a silent part. A panel 5202 for displaying the result and information indicating a change in the work situation (switching of a video scene and switching of a video channel), a user's operation on the window (opening, closing, generating, deleting, etc. of the window), and a sticky note (Personal memo attached to the window) Panel 5203 for displaying information indicating the history of entry and instruction with the mouse pointer, Panel 5204 for displaying reference material, and video of search results And a panel 5205 to be used.
[0136]
FIG. 53 shows an operation panel 5300 for searching and editing information. The operation panel 5300 provides a user with a user interface for the work status management device. As shown in FIG. 53, an operation panel 5300 includes an operation panel 5301 for recording a work situation, an operation panel 5302 for searching information, an operation panel 5303 for operating information, and a plurality of information panels. And an operation panel 5305 for selecting a priority rule when a plurality of video blocks exist in the same time zone. By selecting a priority rule on the operation panel 5305, semi-automatic information editing by a computer becomes possible. The operation panel 5306 automatically converts a work situation (for example, the content of a meeting, etc.) into text information according to time information, an event name added to the video block, and information on an object for each video block. It is a panel.
[0137]
FIG. 54 shows an operation panel 5400 for integrating video information and audio information recorded for each participant. An operation panel 5400 includes a panel 5401 that displays video information captured by a certain user A and audio information based on speech, a panel 5402 that displays video information captured by another user B and audio information based on speech, And a panel 5403 for displaying integrated video information and audio information as a result of the automatic editing.
[0138]
The present invention can be applied not only to conferences but also to search and edit of multimedia mail when an editing device is used by an individual, and to create teaching materials by CAI (computer-assisted education) when a collaborative editing device is used. It is.
[0139]
【The invention's effect】
As described above, according to the work situation management device of the present invention, it is possible to manage various information indicating the time progress of the work. This makes it easy to search for a desired portion of the video information and audio information recorded during the work by focusing on the change in the work situation. It is possible to manage from a personal point of view by associating the information with the daily work of the individual so that the user can efficiently extract and work with necessary information (materials, comments, meeting status). In addition, it is possible to handle dynamic information such as a conversation situation that is difficult to handle systematically from a personal viewpoint. Furthermore, by recording or outputting only the video information and the audio information at the time when it is estimated that the user is paying attention, the amount of information presented to the user and the storage capacity can be reduced.
[0140]
Furthermore, according to the work situation management device of the present invention, it is possible to add a keyword to video information and audio information. By using the keyword, it becomes easy to search for a desired portion of the video information or the audio information. In addition, it is possible to generate character information indicating a work situation using a keyword.
[Brief description of the drawings]
FIG. 1A is a diagram showing a configuration of a work status management device of the present invention.
(B) shows a typical work scene
FIG. 2 is a diagram showing a configuration of a system including a plurality of terminal devices and a work status management device connected via a network.
FIG. 3 is a diagram showing a configuration of a work status management unit.
FIG. 4 is a diagram showing another configuration of the work status management unit.
FIG. 5 is a diagram showing another configuration of the work status management unit.
FIG. 6 is a diagram showing another configuration of the work status management unit.
FIG. 7 is a diagram showing another configuration of the work status management unit.
FIG. 8 is a diagram showing another configuration of the work status management unit.
FIG. 9 is a diagram showing a configuration of a video information management unit.
FIG. 10 is a diagram showing a configuration of a voice information management unit.
FIG. 11 is a diagram showing a configuration of a window information management unit.
FIG. 12 is a diagram showing a configuration of an instruction information management unit.
FIG. 13 is a diagram showing information indicating a work status stored in a work status storage unit.
FIG. 14 is a diagram showing information indicating a work status stored in a work status storage unit;
FIG. 15 is a diagram showing information indicating a work status stored in a work status storage unit.
FIG. 16 is a diagram showing information indicating a work status stored in a work status storage unit.
FIG. 17 is a view for explaining a method of determining a user's window of interest by using window size change information;
FIG. 18 is a view for explaining a method for determining a window of interest of a user using window owner information.
FIG. 19 is a view for explaining a method of determining information of interest of a user based on operation information of a display position changing unit.
FIG. 20 is a diagram illustrating a method for detecting a point of interest of a user with respect to video information.
FIG. 21 is a view for explaining a method of detecting a point of interest of a user for video information.
FIG. 22 is a diagram showing a configuration of a keyword information management unit.
FIG. 23A is a diagram showing a flow of work for editing a document.
(B) is a diagram showing an example of information stored in the work status storage unit by the work of (a).
FIG. 24A is a diagram showing a scene where a part of the material information is instructed by a user in an operation.
(B) is a diagram showing an example of information stored in the work status storage unit by the work of (a).
FIG. 25A is a diagram showing a scene in which material information is displayed in a window in a work;
(B) is a diagram showing an example of information stored in the work status storage unit by the work of (a).
FIG. 26A is a diagram illustrating a configuration of a voice keyword detection unit.
FIG. 4B is a diagram illustrating an example of information stored in a work status storage unit by a voice keyword detection unit.
FIG. 27 is a diagram showing a procedure of processing for adding a keyword to video information or audio information.
FIG. 28 is a view for explaining a method of designating an evaluation target section (time zone) of video information or audio information.
FIG. 29 is a diagram showing a configuration of a keyword candidate specifying unit.
FIG. 30 is a diagram showing a rule for determining a keyword to be added to video or audio information.
FIG. 31 is a view for explaining a method of calculating a keyword evaluation value.
FIG. 32 is a diagram illustrating a specific method of using the keyword evaluation value and the keyword important value.
FIG. 33 is a diagram showing a procedure of a method for automatically editing conversation information;
FIG. 34 is a diagram showing a procedure of a method of dividing audio information into a sound part and a silent part.
FIG. 35 is a diagram illustrating a keyword integration rule in a competitive section.
FIG. 36 is a view for explaining a keyword integration rule in a competitive section.
FIG. 37 is a diagram showing a keyword integration rule in a competitive section.
FIG. 38 is a diagram showing a configuration of a documenting unit.
FIG. 39 is a view for explaining a method of generating character information indicating a work situation;
FIG. 40 is a view for explaining another method for generating character information indicating a work situation;
FIG. 41 is a diagram showing a configuration of a keyword search unit.
FIG. 42 is a diagram showing an example of information stored in a search keyword storage unit.
FIG. 43 is a diagram showing an example of a search panel for inputting a search keyword.
FIG. 44 is a diagram showing a procedure of a search keyword evaluation process.
FIG. 45 is a diagram showing another configuration of the work status management unit.
FIG. 46 is a diagram showing a procedure for integrating video information;
FIG. 47 is a diagram showing a priority rule for preferentially selecting a video block.
FIG. 48 specifically illustrates a priority rule.
FIG. 49 is a diagram specifically illustrating a priority rule.
FIG. 50 is a diagram specifically illustrating a priority rule.
FIG. 51 is a diagram showing a combination rule of occurrence events;
FIG. 52 shows a screen image of an operation panel for operating information.
FIG. 53 is a diagram showing a screen image of an operation panel for searching and editing information;
FIG. 54 shows a screen image of an operation panel for integrating video information and audio information recorded for each participant.
[Explanation of symbols]
10 Work status management device
11 Input section
12 Terminal control unit
13 Work status management department
14 Work status storage
15 Document information storage
16 Output section
17 Transmission section

Claims (18)

Storage means for storing information indicating the time course of the work, and, based on the information indicating the time course of the work, stored in the storage means, a time to which a keyword is to be attached among the time required for the work Time zone specifying means for specifying a band, keyword candidate specifying means for specifying at least one keyword candidate for the time zone specified by the time zone specifying means, one of the at least one keyword candidate; A work situation management device comprising: a keyword determination unit that selects one keyword candidate according to a predetermined rule and determines the selected keyword candidate as a keyword corresponding to the time period. The information indicating the time lapse of the work is information for identifying a sound part and a silent part included in the audio information generated during the work, and the time zone identifying unit includes the information corresponding to the sound part. The work status management device according to claim 1, wherein only the time zone is specified as a time zone to which a keyword is to be attached. The information indicating the time lapse of the work is information indicating a time period in which a window displaying material information is estimated to be focused on by a user among the time required for the work, and 2. The work situation management device according to claim 1, wherein only the time period in which the window is estimated to be noticed by the user is specified as a time period to which a keyword is to be attached. The information indicating the time lapse of the work is information indicating a time zone in which the instruction information is generated for the window displaying the material information, of the time required for the work, and the time zone specifying means includes: The work situation management device according to claim 1, wherein only the time zone in which the instruction information is generated is specified as a time zone to which a keyword should be attached. The information indicating the time lapse of the work includes information for identifying a sound part and a silent part included in audio information generated during the work, and a window for displaying material information among time required for the work. At least one of information indicating a time zone estimated to be noticed by the user and information indicating a time zone in which the instruction information is generated for the window out of the time required for the work. And the time zone identification means includes: the time zone corresponding to the sound portion and the window in which the instruction information is generated for the time zone and the window in which it is estimated that the user is interested. 2. The work status management device according to claim 1, wherein only a time zone determined based on at least one of the time zones is specified as a time zone to which a keyword is to be attached. When material information including editable character information is used in the work, the keyword candidate specifying means determines whether the first character information in the material information at the first time in the time required for the work and the work Difference information storage means for storing difference information representing a difference between the second information and the material information at the second time of the required time; 2. The work situation management device according to claim 1, further comprising: a document keyword extracting unit that extracts one keyword candidate. Keyword candidate specifying means, when material information including character information is used in the work, position information storage means for storing position information indicating the position of the character information designated by the user during the work; 2. The work situation management device according to claim 1, further comprising: an instruction keyword extracting unit that extracts at least one keyword candidate from the material information based on the position information stored in the information storage unit. The keyword candidate specifying means includes a title storage means for storing the title when the material information is displayed in a window having a portion for describing the title in the work, and at least a title stored in the title storage means. 2. The work status management device according to claim 1, further comprising: a title keyword extracting unit that extracts one keyword candidate. The keyword candidate specifying means includes a personal information storage means for storing the personal information when the material information is displayed in a window having a portion for describing the personal information in the work; and a personal information storage means for storing the personal information. The work situation management device according to claim 1, further comprising personal information keyword extracting means for extracting at least one keyword candidate from the personal information. The keyword candidate specifying means recognizes voice information generated in the work and generates character information corresponding to the voice information, and a voice recognition information storage storing the character information corresponding to the voice information 2. The work status management device according to claim 1, further comprising: a voice keyword extracting unit that extracts at least one keyword candidate from the character information stored in the voice recognition information storage unit. 2. The work situation management apparatus according to claim 1, wherein the keyword candidate specifying unit includes a keyword candidate input unit that receives the character information input by the user and uses the received character information as a keyword candidate. The work situation management device according to claim 1, wherein the predetermined rule includes a rule for determining a keyword based on an evaluation value related to a keyword appearance ratio. The work situation management device according to claim 1, wherein the predetermined rule includes a rule that specifies which keyword is to be selected from a plurality of keywords assigned to the competition section. Storage means for storing information indicating the time course of the work, search keyword input means for inputting a search keyword from a user, and the storage means stored in the storage means based on the input search keyword. Search means for searching for the information indicating the time course of the work, search keyword storage means for storing the input search keywords and search results, and whether the search keywords are appropriate based on the search results. And a search keyword evaluation means for evaluating the work status. The operation according to claim 14, wherein the search keyword evaluation unit evaluates the search keyword based on at least the number of times the search keyword has been input by the user and the number of times the search result has been adopted by the user. Status management device. Video information dividing means for dividing the first video information into a plurality of first video blocks and dividing the second video information into a plurality of second video blocks; It is determined whether one and one of the plurality of second video blocks are present, and one of the plurality of first video blocks and one of the plurality of second video blocks are determined during the time period. A video block evaluation means for determining which of the video blocks existing in the time zone is preferentially selected in accordance with a predetermined rule when it is determined that one video block exists. An apparatus for managing work status, wherein the apparatus integrates the first video information and the second video information to generate one video information. 17. The work situation management device according to claim 16, wherein the predetermined rule includes a rule for determining a video block to be selected based on a temporal precedent / rear relationship of video blocks existing in the time zone. 17. The work situation management device according to claim 16, wherein the predetermined rule includes a rule for determining a video block to be selected based on a change in the work situation.

Images