WO2020209171A1

WO2020209171A1 - Iinformation processing device, information processing system, information processing method, and information processing program

Info

Publication number: WO2020209171A1
Application number: PCT/JP2020/015187
Authority: WO
Inventors: 慎江上; 竜米谷
Original assignee: オムロン株式会社
Priority date: 2019-04-09
Filing date: 2020-04-02
Publication date: 2020-10-15

Abstract

The present invention provides a technology capable of appropriately determining and presenting important information among a series of information. An information processing device (1) is provided with: a series information acquisition unit (112) that acquires annotation-added series information in which an annotation is added to each of elements included in series information relating to a subject: a degree-of- importance determination unit (114) that, with reference to the annotation-added series information relating to the subject, determines the degree of importance of each of the elements included in the annotation-added series information of the subject; a determination integration unit (116) that, with reference to determination results obtained by the degree-of-importance determination unit regarding a plurality of subjects, generates determination integration information; and a presentation information generation unit (118) that, with reference to the determination results obtained by the degree-of-importance determination unit and/or the determination integration information, generates presentation information.

Description

Information processing equipment, information processing systems, information processing methods, and information processing programs

The present invention relates to an information processing device, an information processing system, an information processing method, and an information processing program.

A system that supports learning and acquisition of work skills by the person to be learned is known as a conventional technique. For example, Patent Document 1 describes a technique of comparing a result of monitoring a learner's reading frequency and gaze retention with another learner and presenting the comparison result to a learner and an instructor.

Japanese Patent Publication "Japanese Unexamined Patent Publication No. 2005-338173 (published on December 8, 2005)"

Also, in recent years, it has been used to look back on videos recording learning and work in various situations and to improve the learning and work. In such a technique, it is preferable to be able to automatically identify important scenes useful for learning and work improvement from recorded moving images.

In addition to learning and work, there is a great need for technology that identifies and presents important information among information that has a series.

In view of the above problems, it is an object of the present invention to provide a technique capable of appropriately discriminating and presenting important information among information having series.

In order to solve the above problems, the information processing apparatus according to one aspect of the present invention includes a sequence information acquisition unit that acquires annotated sequence information in which annotations are added to each element included in the sequence information related to the target person. The importance determination unit for determining the importance of each element included in the annotated series information of the target person by referring to the annotated series information regarding the target person, and the determination result by the importance determination unit for a plurality of target persons. A discrimination integration unit that generates discrimination integration information by referring to, a presentation information generation unit that generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit and the discrimination integration information. It has.

In order to solve the above problems, in the terminal device according to one aspect of the present invention, an annotation series is added by annotating a series information acquisition unit that acquires series information and each element included in the acquired series information. It includes an annotation addition unit that generates information, a presentation information acquisition unit that acquires presentation information generated by referring to the annotation series information, and a presentation unit that presents the presentation information.

In order to solve the above problems, in the information processing system according to one aspect of the present invention, annotated series information is added by adding an annotation to an acquisition unit that acquires series information and each element included in the acquired series information. The importance determination unit that determines the importance of each element included in the annotated series information of the target person by referring to the annotated series information about a certain target person, and the plurality of target persons. The presentation information is generated by referring to at least one of the discrimination integration unit that refers to the discrimination result by the importance discrimination unit and generates the discrimination integration information, and the discrimination result by the importance discrimination unit and the discrimination integration information. It is equipped with a presentation information generation unit.

In order to solve the above problems, in the information processing method according to one aspect of the present invention, an annotation adding step of generating annotated series information by adding an annotation to each element included in the series information about a target person, With reference to the annotated series information, the importance determination step of determining the importance of each element included in the annotated series information of the target person and the determination result in the importance determination step for a plurality of target persons are referred to. The present invention includes a discrimination integration step of generating discrimination integration information, and a presentation information generation step of generating presentation information by referring to at least one of the discrimination result in the importance discrimination step and the discrimination integration information.

According to one aspect of the present invention, important information can be suitably discriminated and presented among the information having sequence.

It is a block diagram which schematically exemplifies the configuration example of the information processing system which concerns on one Embodiment of this invention. It is a sequence diagram which illustrates the operation example of the information processing system which concerns on one Embodiment of this invention. It is a figure which shows an example of the test question which is the object of the recording which concerns on one Embodiment of this invention. It is a figure which shows an example of the information for presentation which concerns on one Embodiment of this invention. It is a figure which shows an example of the information for presentation which concerns on one Embodiment of this invention. It is a figure which shows an example of the visual object of the subject person in learning of the customer service work which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the learner's visual object, utterance which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the veteran customer service person which concerns on one Embodiment of this invention. It is a figure which shows an example of the difference information between a learner and a veteran customer service person which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the learner's visual object and operation which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the veteran assembler which concerns on one Embodiment of this invention. It is a figure which shows an example of the difference information between a learner and a veteran assembler which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the learner's visual object and operation which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the veteran cook which concerns on one Embodiment of this invention. It is a figure which shows an example of the difference information between a learner and a veteran cook which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the visual object | movement of the player A which concerns on one Embodiment of this invention. It is a table which shows an example of the data of the player B which concerns on one Embodiment of this invention. It is a figure which shows an example of the difference information between the player A and the player B which concerns on one Embodiment of this invention. It is a block diagram which schematically exemplifies the configuration example of the information processing system which concerns on other embodiment of this invention. It is a sequence diagram which illustrates the operation example of the information processing system which concerns on other embodiment of this invention. It is a block diagram which schematically exemplifies the configuration example of the information processing system which concerns on still another Embodiment of this invention. It is a sequence diagram which illustrates the operation example of the information processing system which concerns on still another Embodiment of this invention.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings.

<Embodiment 1>
§1. Application Example First, an example of a situation in which the present invention is applied will be described with reference to FIG. Here, the case where the subject of the present invention is a learner who takes a lecture or a mock test at a cram school or the like is taken as an example, but this does not limit the present embodiment, and the subject is the series information. It is not particularly limited as long as it has some relation with.

FIG. 1 is a block diagram schematically illustrating a configuration example of the information processing system 1 according to the embodiment of the present invention. The information processing system 1 is, for example, an information processing system used in a cram school or the like. As shown in FIG. 1, the information processing system 1 according to the present embodiment includes a server 100 which is an information processing device, learner terminal devices 200A, 200B, 200C, and a lecturer terminal device 1000. The server 100, the learner terminal devices 200A, 200B, 200C, and the instructor terminal device 1000 are connected via a network and can communicate with each other. The type of network may be any, such as the Internet, a telephone network, and a dedicated network. In the following, the learner terminal devices 200A, 200B, and 200C may be collectively referred to as the learner terminal device 200.

The learner terminal devices 200A, 200B, and 200C according to the present embodiment are, for example, computers assigned to each learner (hereinafter, also referred to as a target person) A, B, C in a cram school. Each learner's terminal device 200A, 200B, and 200C is equipped with a camera 241, a microphone 242, and the like, and can record and record the state and utterance of each subject during a lecture or a mock test. .. The recording and the recorded moving image are examples of the series information in the present embodiment. Here, the series information means general information in which the order of each element included in the information is meaningful. In addition, the series information may include moving image data, audio data, text data, data indicating changes in numerical values with time, and the like. Further, each terminal device 200 can generate annotated sequence information by adding annotations to the sequence information and record it in each storage unit 230. Here, the annotation is meta information given to each element included in the series information. As an example, the annotation includes meta information regarding the state of the subject at each time point or each period during the recording of the series information. The type of annotation is not limited to this embodiment, but as an example, an index indicating the degree of concentration of the target person at that time can be used as an annotation.

Then, each learner terminal device 200A, 200B, and 200C transmits the series information about each target person with this annotation to the server 100.

The server 100 determines the importance of each element included in the series information based on the series information with annotations transmitted from each terminal device 200. Then, the determination result of the importance determined based on the annotated series information transmitted from each terminal device 200 is integrated to generate the presentation information. The server 100 stores the generated presentation information in the storage unit 130 and transmits it to each

terminal device

200 or 1000.

Each

terminal device

200 or 1000 presents the presentation information transmitted from the server 100 and transmits the feedback information to the server 100. At this time, the target person confirms the presentation information together with the instructor, and feeds back information regarding the update of the importance determination logic or the discrimination logic used in the discrimination integration process described later via each

terminal device

200 or 1000. It may be transmitted to 100. That is, the feedback information as a result of correcting and confirming the presentation information from each

terminal device

200 or 1000 may be transmitted to the server 100. The server 100 updates the importance determination logic based on the feedback information transmitted from each

terminal device

200 or 1000.

As described above, each time the transmission / reception of the sequence information is repeated between the server 100 and each

terminal device

200 or 1000, the information that can be used for determining the importance increases, so that the importance determination logic, And the discrimination logic used in the discrimination integration process is improved. As a result, the importance of the series information can be determined more appropriately. As a result, when looking back on the video, the instructor can look back on the video by focusing on the information that is likely to lead to improvement of the instruction content, and use time efficiently to improve the instruction. Can be done.

§2. Configuration Example Hereinafter, each configuration of the server 100, the learner terminal device 200, and the instructor terminal device 1000, which are the information processing devices constituting the information processing system 1, in the present embodiment will be described in detail with reference to FIG. To do.

<Learner terminal device 200>
The learner terminal devices 200A, 200B, and 200C according to the embodiment have the same configuration as an example. For example, the learner terminal device 200 includes a control unit 210, a communication unit 220, a storage unit 230, a camera 241 and a microphone 242, an operation reception unit 243, a display unit (presentation unit) 244, and a speaker 245.

The communication unit 220 performs communication processing with an external device such as the server 100. The storage unit 230 is a storage device that stores various types of data. The operation reception unit 243 is an interface for receiving an input operation of a target person or the like, and is a button such as a keyboard. The display unit 244 is a display panel for displaying a moving image. The operation reception unit 243 and the display unit 244 may be configured to be realized as a touch panel that accepts input operations of the target person or the like and displays a moving image. The control unit 210 is a control device that controls the entire learner terminal device 200, and for the sequence information acquisition unit 212 that acquires sequence information via the camera 241 and the like, and for each element included in the acquired sequence information. Annotation addition unit 214 that generates annotation series information by adding annotations, and presentation information acquisition unit 216 that acquires presentation information generated by referring to annotation series information via communication unit 220. It is provided with a discriminant integrated information acquisition unit (not shown) that acquires discriminant integrated information generated by referring to the discriminant result regarding the annotated series information of one or a plurality of target persons.

<Series information acquisition unit 212>
The series information acquisition unit 212 acquires series information about the target person. An example of the series information acquired by the series information acquisition unit 212 is as follows. As an example, these data are acquired as information during a mock test or a lecture, but this is not limited to this embodiment, and may be acquired in other situations.
-Subject imaged by camera 241 (may include the subject's face)
-Voice of the target person collected by the microphone 242-Operation input by the target person received by the operation reception unit 243 (text data may be included)
-Data of images, voices, texts, etc. acquired via the communication unit 220-Data of images, voices, texts, etc. read from the storage unit 230-Changes in temperature acquired from a temperature sensor (not shown) over time-Communication unit 220 Pulse data obtained from a sensor (not shown) that changes over time in the weather acquired through the system, and is used to read the degree of tension, stress, concentration, etc. of the subject.

Further, the series information may include information regarding the line of sight of the target person. For example, the series information acquisition unit 212 acquires the face information of the target person from the image acquired from the camera 241. The face information includes position information indicating the position of each part of the subject's face (for example, eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, size information indicating the size, and the like, as well as the target. The line of sight of the subject is detected as the state of the person. The line of sight is particularly important as an index showing the degree of concentration of the subject on the task. The detection of the line of sight of the subject will be described later.

<Annotation section 214>
The annotation unit 214 generates the annotated series information by automatically adding an annotation to each element included in the acquired series information or by adding an annotation based on a user's instruction. Annotations are attached to the elements included in the series information. As an example, annotation A is attached to the section from 00:01 to 00:02 of the moving image data, and annotation B is attached to the section from 00:05 to 00:06. As another example, the annotation X is attached to the clause AA in the sentence A included in the text data, and the annotation Y is attached to the clause BB.

Further, the annotation unit 214 may add information indicating whether or not the target person has visually recognized a specific area to the series information as annotations. Further, the operation reception unit 243 accepts an operation of adding annotations to the series information and an operation of inputting the importance described later for each part of the series information. For example, when the user gives an instruction to add an annotation to a certain part in the series information via the operation reception unit 243, the annotation unit 214 is instructed to the part of the series information. Annotate.

As a more specific example, the annotation unit 214 evaluates the degree of concentration of the target person at each time point from the recorded video in five stages, and concentrates as an annotation for each time point of the video which is series information. Give a degree. In addition, as another example, information indicating which question sentence or question is visually identified by identifying the question sentence or question that the subject is visually recognizing at each time point from the video about the subject during the mock test. As an annotation, is added to each time point of the moving image.

In this way, adding information indicating which question sentence or question is visually recognized as an annotation is identified as, for example, the coordinates indicating the tip of the subject's line of sight and the coordinates on the question sheet of the mock test. It is possible by that.

The annotation processing by the annotation unit 214 is not limited to the above example. For example, another image analysis algorithm or voice analysis algorithm is used to determine the points to be annotated, and the determined points. May be configured to annotate.

Further, as an example, the annotation may be defined by a start time, an end time, a tag, and a reliability. Here, the tag is information indicating the type of annotation, for example, information indicating the degree of concentration and the degree of understanding of the target person. The reliability is information indicating the certainty of the target annotation. The annotation unit 214 can calculate the reliability of each annotation added by the annotation processing, which indicates how reliable the annotation is. Further, in addition to the annotation, the annotation unit 214 may add information indicating the reliability of the annotation to each element in the series information.

Further, for example, the degree of concentration is set to a relatively high value when the variation of the visual target of the target person is small (when the variation of the line of sight is small), and when the variation of the visual target of the target person is large (for example). It may be set to a relatively low value (when the variation of the line of sight is large). The annotation unit 214 can specify the degree of concentration of the target person by acquiring and referring to the line-of-sight information of the target person.

Further, for example, the comprehension level may be set to a high value when the time for the subject to visually recognize the object to be understood is relatively long (or short).

Further, for example, when the series information is information that records the visual target of the target person during the mock test, the annotation unit 214 receives the score of the mock test input by the target person after the mock test via the operation reception unit 243. Alternatively, the comprehension level may be set by referring to the information indicating which question was answered correctly. Further, for example, the annotation unit 214 may set a high degree of understanding of the target portion in the series information when the target person satisfies a predetermined condition that a specific operation is performed at a specific timing. .. Further, the information indicating the above-mentioned predetermined conditions may be stored in advance in the storage unit 230 or the like.

Further, the annotation unit 214 may update the discrimination logic (annotation logic) used when annotating the series information by referring to the information included in the presentation information, for example.

Further, when the annotation addition unit 214 adds annotations by user input, the annotation addition logic may be updated by using the annotations attached by the user as feedback information. Further, the annotation unit 214 may update the annotation logic with reference to the discrimination integration information. That is, the annotation addition unit 214 may update the annotation addition logic by referring to at least one of the feedback information from the user and the discrimination integration information.

<Presentation information acquisition unit 216, display unit 244>
The presentation information acquisition unit 216 acquires the presentation information generated by referring to the annotated series information in the presentation information generation unit 118 of the server 100.

The display unit 244 displays the presentation information acquired by the presentation information acquisition unit 216. The presentation information is not limited to the moving image presented via the display unit 244, and may be the sound presented via the speaker 245. Further, the learner terminal device 200 may be configured to receive feedback information from the user via the operation reception unit 243, similarly to the instructor terminal device 1000 described later.

More specifically, as an example, the subject and the instructor confirm the presentation information displayed by the display unit 244, and confirm the importance determination result included in the presentation information while having a dialogue. .. And if there is something that needs to be corrected about importance,
-Enter information including which part should be corrected and how to correct it in the operation reception unit 243. Here, the point to be corrected regarding the importance is, for example, that the presentation information is not important even though the importance determination result corresponding to a certain presentation information shows a relatively high importance. It is the importance determination result etc. judged by the subject. The operation reception unit 243 may be configured to generate feedback information including the above-mentioned input information and transmit it to the server 100 via the communication unit 220.

<Server 100>
As shown in FIG. 1, the server 100 of the information processing system 1 includes a control unit 110, a communication unit 120, and a storage unit 130. The communication unit 120 can communicate with other information processing devices (learner terminal device 200, instructor terminal device 1000) included in the information processing system 1. In addition, the storage unit 130 can store information transmitted from other information processing devices, information integrated by the server, and the like.

The control unit 110 is a control device that controls the entire server 100, and includes a series information acquisition unit 112, an importance determination unit 114, a discrimination integration unit 116, and a presentation information generation unit 118. In addition, the communication unit 120 performs communication processing with an external device such as the learner terminal device 200. The storage unit 130 is a storage device that stores various types of data.

<Series information acquisition unit 112>
The series information acquisition unit 112 acquires annotated series information in which annotations are added to each element included in the series information regarding the target person. As an example, the series information acquisition unit 112 acquires information in which annotations are added to the series information about each target person transmitted from each learner terminal device 200 via the communication unit 120.

<Importance determination unit 114>
The importance determination unit 114 refers to the annotated series information about the target person, and determines the importance of each element included in the annotated series information of the target person.

The importance determination unit 114 may determine the importance based on the degree of concentration regarding the target person. As an example, in the importance determination unit 114, when the concentration level is equal to or higher than a predetermined value or the concentration level is equal to or lower than a predetermined value for a predetermined period or longer, the interval is not 0. Is determined to have.

For example, it is assumed that the server 100 acquires the annotated series information as shown in the table below from each terminal device 200.
Time 00:01 00:02 00:03 00:04 00:05
Concentration Target A 1 5 5 3 1
Target person B 1 5 5 5 1
Target person C 2 5 5 5 5
In this case, as an example, the importance determination unit 114 determines a section in which the concentration level 5 continues for 2 seconds or more as a section having a non-zero importance. For example, since the subject A maintains the state of concentration 5 for 2 seconds from 00:02 to 00:03, the importance determination unit 114 determines that the importance of the section is 1. .. Further, since the subject B maintains the state of concentration 5 for 3 seconds from 00:02 to 00:04, the importance determination unit 114 determines that the importance of the section is 2. .. Further, since the subject C maintains the state of concentration 5 for 4 seconds from 00:02 to 00:05, the importance determination unit 114 determines that the importance of the section is 3. ..

Similarly, the importance determination unit 114 can be configured to determine a section in which the

concentration levels

1 and 2 continue for 2 seconds or more as a section having a non-zero importance.

Further, for example, the importance determination unit 114 may determine that the lower (or higher) the degree of understanding of the target person is, the higher the importance of the target part is. Alternatively, the importance determination unit 114 may determine that the location where the subject's line of sight begins to disperse (or the location where the line of sight begins to concentrate) is of high importance in a lecture or the like.

Further, as another example, the importance determination unit 114 performs regression analysis (regression analysis) on the annotated series information, and refers to various parameters obtained by performing the regression analysis to refer to the importance. It may be configured to discriminate. For example, regression analysis is applied to annotated sequence information related to a certain target person, and a parameter indicating a regression curve indicating the annotated sequence information is derived. Then, the value of the parameter may be compared with the value of a predetermined parameter, and the importance may be determined according to the comparison result.

Further, the importance determination unit 114 updates the determination logic used for the importance determination process with reference to the feedback information acquired via the communication unit 120. As an example, when the importance determination unit 114 obtains feedback information indicating that the importance should be lowered for the element with the importance 5, the importance is lower for the element with the importance 5. Update the discriminant logic so that the degree is added.

Note that the importance determination unit 114 may determine that the higher the importance is, the more the line of sight or movement of the target person in a certain period deviates from that of another target person. Further, even if the importance determination unit 114 determines that the line of sight or movement of the subject during a certain period is more important as the line of sight or movement deviates from the line of sight or movement set as the norm. Good. In addition, the above-mentioned information indicating the line of sight or movement that serves as a norm may be stored in advance in the storage unit 130 or the like.

Further, the importance determination unit 114 performs the importance determination process on the assumption that the portion having a high degree of contribution to improving the identification logic when the annotation unit 214 annotates the series information has a high importance. You may.

<Discrimination integration unit 116>
The discrimination integration unit 116 refers to the discrimination results of the plurality of target persons A, B, C, ... By the importance discrimination unit 114, and generates discrimination integration information. For example, the discriminating and integrating unit 116 can more appropriately determine the importance of the series information by collecting the series information acquired from each terminal device 200 and integrating the series information.

As an example, the discrimination integration unit 116 may extract common information from the discrimination results by the importance discrimination unit 114 for each of the plurality of subjects A, B, and C and include it in the discrimination integration information.

For example, it is assumed that the importance determination unit 114 determines the importance of the subjects A, B, and C as follows.
Time 00:01 00:02 00:03 00:04 00:05
Importance Target person A 0 1 1 0 0
Target person B 0 2 2 2 0
Target person C 0 3 3 3 3
In this case, as an example, the discrimination integration unit 116 determines that 00:02, which is a time with a non-zero importance, is important by the subjects A, B, and C, and determines the determination result as discrimination integration information. Include in.

Further, for example, the discriminant integration unit 116 obtains information indicating that the line of sight or movement of the target person in a certain period is different from that of another target person, and information indicating the degree of dissociation. May be included in. In addition, the discrimination integration unit 116 provides information indicating that the line of sight or movement of the target person in a certain period deviates from the line of sight or movement set as a norm, and information indicating the degree of dissociation. , May be included in the discriminant integrated information.

Further, for example, the discrimination integration unit 116 provides meta information indicating matters related to the subject such as past performance records of the subject, or meta information indicating matters related to the subject's environment such as a mock test that the subject is taking. , May be included in the discrimination integrated information. Further, each of the above-mentioned meta information may be included in the determination result by the importance determination unit 114.

Further, as another example, when the importance determination unit 114 performs regression analysis on the annotated series information, the discrimination integration unit 116 determines the parameters obtained by the regression analysis on the subjects A, B, and C. Of these, parameters having common properties may be extracted and the parameters may be included in the discrimination integration information. In addition, the discrimination integration unit 116 may generate a regression model that is commonly used by referring to the regression model corresponding to a plurality of subjects. In other words, the discriminant integration unit 116 may use a plurality of discriminant algorithms or parameters referred to by them as input data, and output the discriminant algorithms after integration or the parameters referred to by them. Further, in the case of a configuration in which the importance can be input to the series information by the target person or the instructor, the discrimination integration unit 116 can generate discrimination integration information that reflects, for example, a portion considered to be important by a plurality of lecturers. ..

Further, the discrimination integration unit 116 updates the discrimination logic used for the discrimination integration process for generating the discrimination integration process with reference to the feedback information acquired via the communication unit 120. As an example, when the discrimination integration unit 116 obtains feedback information that the importance should be lowered for the element with the importance level 5, the importance level is lower than that of the element with the importance level 5. Update the discriminant logic so that is added.

Further, the discrimination integration unit 116 generates information for updating the importance discrimination logic or the updated discrimination logic, and supplies the information to the importance discrimination unit 114. The importance determination unit 114 updates the importance determination logic with reference to the acquired information. Thereby, for example, the result of the importance determination process can be updated to a more suitable one by the series information corresponding to a plurality of target persons.

Further, the discrimination integration unit 116 generates information for updating the discrimination logic used for adding annotations or the updated discrimination logic, and transmits the updated discrimination logic to the learner terminal device 200 via the communication unit 120. The annotation unit 214 updates the above-mentioned discrimination logic with reference to the acquired information. Thereby, for example, the process of annotating can be updated to a more suitable one by the series information corresponding to a plurality of target persons. The discriminant integration unit 116 may refer to the discriminant integration information generated by itself when updating each of the above-mentioned discrimination logics.

Note that the annotation logic by the annotation unit 214, the importance determination logic by the importance determination unit 114, and the integration logic by the identification integration unit 116 are not limited to the above examples. Further, the processing by the annotation unit 214, the importance determination unit 114, and the discrimination integration unit 116 may use rule-based logic, machine learning such as a neural network, or other methods. You may use it.

For example, any of the following machine learning methods or a combination thereof can be used.

・ Support Vector Machine (SVM)
・ Clustering
・ Inductive Logic Programming (ILP)
・ Genetic Programming (GP)
・ Bayesian Network (BN)
・ Neural network (NN)
When a neural network is used, the input data may be processed in advance for input to the neural network. For such processing, in addition to one-dimensional arrangement or multidimensional arrangement of data, for example, a method such as data augmentation can be used.

When using a neural network, a convolutional neural network (CNN: Convolutional Neural Network) including a convolution process may be used. More specifically, as one or more layers included in the neural network, a convolution layer for performing a convolution operation is provided, and a filter operation (product-sum operation) is performed on the input data input to the layer. It may be configured. Further, when performing the filter calculation, a process such as padding may be used together, or an appropriately set stride width may be adopted.

Further, as the neural network, a multi-layered or super-multilayered neural network having tens to thousands of layers may be used. The machine learning described above may be supervised learning or unsupervised learning.

<Presentation information generation unit 118>
The presentation information generation unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit 114 and the discrimination integrated information. For example, from the discrimination integration information integrated by the discrimination integration unit 116, the information processed for each target person is transmitted to each terminal device 200. The generated integrated information and presentation information are stored in the storage unit 130.

Further, for example, the presentation information generation unit 118 may preferentially include the corresponding information in the presentation information as the importance indicated by the determination result by the importance determination unit 114 becomes higher. In addition, the presentation information generation unit 118 indicates that the information included in the discriminant integrated information has a high degree of dissociation between the line of sight and movement of the target person during a certain period and the line of sight and movement of another target person. As such, the corresponding information may be preferentially included in the presentation information. In addition, the presentation information generation unit 118 has a high degree of dissociation between the line of sight and movement of the target person in a certain period and the line of sight and movement set as a norm in the information included in the discrimination integrated information. The more it indicates that, the more the corresponding information may be preferentially included in the presentation information.

<Terminal device 1000 for lecturers>
The instructor terminal device 1000 includes a control unit 1010, an operation reception unit 1043, a display unit 1044, a speaker 1045, a communication unit 1020, and a storage unit 1030.

The control unit 1010 is a control device that controls the entire instructor terminal device 1000, and includes a presentation information acquisition unit 1014 and a feedback information acquisition unit 1012.

The communication unit 1020 performs communication processing with an external device such as the server 100. The storage unit 1030 is a storage device that stores various data. The operation reception unit 1043 is an interface for receiving input operations of a lecturer or the like, and is, for example, a button such as a keyboard. The display unit 1044 is a display panel for displaying a moving image. The operation reception unit 1043 and the display unit 1044 may be realized as a touch panel that accepts input operations of a lecturer or the like and displays a moving image.

<Presentation information acquisition unit 1014, display unit 1044>
The presentation information acquisition unit 1014 acquires the presentation information generated by the presentation information generation unit 118 of the server 100 with reference to the annotated sequence information.

The display unit 1044 displays the presentation information acquired by the presentation information acquisition unit 1014.

<Feedback information acquisition unit 1012>
The feedback information acquisition unit 1012 acquires feedback information from the user. As an example, the instructor confirms the presentation information displayed by the display unit 1044, and confirms the importance determination result included in the presentation information. And if there is something that needs to be corrected about importance,
-Enter information including which part should be corrected and how to correct it in the operation reception unit 1043. Here, the part to be corrected with respect to the importance is, for example, that the presentation information is not important even though the importance determination result corresponding to a certain presentation information shows a relatively high importance. These are the results of importance determination judged by the instructor. The operation reception unit 1043 generates feedback information including the input information, and transmits the feedback information to the server 100 via the communication unit 1020.

The importance determination unit 114 of the server 100 updates the determination logic with reference to the feedback information from the user.

The feedback information acquisition unit 1012 may be provided in the instructor terminal device 1000 or in the learner terminal device 200.

<Detection of line of sight>
Hereinafter, the mechanism for detecting the line of sight of the target person in the series information acquisition unit 212 and the annotation unit 214 will be described.

First, the camera 241 takes a picture of the subject during the mock test. The series information acquisition unit 212 acquires the face information of the target person from the moving image taken by the camera 241. The face information of the subject includes, for example, position information indicating the position of each part of the face (for example, eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, size information indicating the size, and the like. Is done. In particular, from the eye information, it is possible to evaluate the degree of concentration of the subject on the subject to be watched by the subject. Examples of eye information include the end points of the inner and outer corners of the eye, the edges of the iris and the pupil, and the like. The sequence information acquisition unit 212 may appropriately perform correction processing such as noise reduction and edge enhancement on the moving image acquired from the camera 241. The sequence information acquisition unit 212 transmits the extracted face information to the annotation unit 214.

The annotation unit 214 detects the state of the target person based on the face information extracted by the series information acquisition unit 212. For example, the state of each part of the subject's face, which is at least one of the subject's line of sight, pupil state, number of blinks, eyebrow movement, cheek movement, eyelid movement, lip movement, and jaw movement. Detect one.

The method of detecting the line of sight of the subject is not particularly limited, but for example, by providing a point light source (not shown) in the terminal device 200 and photographing the corneal reflex image of the light from the point light source with the camera 241 for a predetermined time. , A method of detecting the movement destination of the line of sight of the subject can be mentioned. The type of the point light source is not particularly limited, and examples thereof include visible light and infrared light. For example, by using an infrared LED, the line of sight can be detected without causing discomfort to the subject. In the detection of the line of sight, if the line of sight does not move for a predetermined time or longer, it can be said that the same place is being watched.

The method of detecting the state of the pupil is not particularly limited, and examples thereof include a method of detecting a circular pupil from the image of the eye by using the Hough transform. In general, humans tend to open their pupils when they are concentrated, so it is possible to evaluate the degree of concentration of a subject by detecting the size of the pupil. For example, it can be said that there is a high possibility that the subject is gazing at a certain subject during the time when the pupil size is detected for a predetermined time and the pupil is enlarged within the predetermined time. A threshold value may be set for the pupil size and evaluated as "open" when the pupil size is equal to or larger than the threshold value and as "closed" when the pupil size is less than the threshold value.

The method for detecting the number of blinks is not particularly limited, but for example, a method of irradiating an eye to be infrared-lighted and detecting the difference in the amount of infrared light reflected between the eyes when the eyes are opened and when the eyes are closed. And so on. In general, when humans are concentrated, they tend to blink at low frequency and at stable intervals. Therefore, the degree of concentration of a subject can be evaluated by detecting the number of blinks. For example, if the number of blinks is detected for a predetermined time and the blinks are performed at stable intervals within a predetermined time, it can be said that there is a high possibility that the subject is gazing at a certain subject.

The annotation unit 214 detects at least one of the subject's line of sight, pupil condition and number of blinks, eyebrow movement, eyelid movement, cheek movement, nose movement, lip movement and jaw movement. However, it is preferable to combine these. By combining the detection methods in this way, the annotation unit 214 can suitably evaluate the degree of concentration of the target person when visually recognizing a certain object.

Other than the condition of the eyes, for example, eyebrow movements such as lifting the inside or outside of the eyebrows, eyebrow movements such as raising the upper eyelids and tensioning the eyebrows, and nose movements such as wrinkling the nose. , The state of each part of the face such as lip movements such as lifting the upper lip and purging the lips, cheek movements such as lifting the cheeks, and jaw movements such as lowering the nose. As the state of the subject, the state of a plurality of parts of the face may be combined.

Also, in addition to the line-of-sight information, as described above, refer to the detection results of pupil condition, number of blinks, eyebrow movement, eyelid movement, cheek movement, nose movement, lip movement and jaw movement. By doing so, the degree of concentration of the subject can be more preferably determined. Further, the pulse data of the subject may be further referred to to determine the degree of concentration of the subject.

The annotation unit 214 evaluates the degree of concentration of the determined target person in five stages of, for example, 1 to 5, and adds it to the series information of the target person as an annotation.

§3. Operation example <Operation of information processing system 1>
Next, the operation of the information processing system 1 will be described. FIG. 2 is a sequence diagram illustrating an operation example of the information processing system 1 according to the present embodiment. The process of generating presentation information in the information processing system 1 of the present embodiment will be described with reference to FIG.

(Step S102)
In step S102, the annotation-giving unit 214 generates the annotated series information by adding annotations to each element included in the series information regarding the target person acquired by the series information acquisition unit 212. The communication unit 220 transmits the generated annotated sequence information to the server 100.

(Step S104)
In step S104, the sequence information acquisition unit 112 of the server 100 acquires the sequence information with the annotation series information.

(Step S106)
In step S106, the importance determination unit 114 refers to the annotated series information about the target person and determines the importance of each element included in the annotated series information of the target person.

(Step S108)
In step S108, the discrimination integration unit 116 refers to the discrimination results of the importance of the plurality of target persons A, B, C ..., and generates the discrimination integration information.

(Step S110)
In step S110, the presentation information generation unit 118 generates the presentation information by referring to at least one of the discrimination result by the importance determination unit 114 and the discrimination integrated information. When the presentation information generation unit 118 generates the presentation information by referring only to the determination result by the importance determination unit 114, the process of step S108 does not necessarily have to be executed, and the process of step S106 The process of this step S110 may be executed subsequently. After that, the communication unit 120 transmits the generated presentation information to each learner terminal device 200.

(Step S112)
In step S112, the presentation information acquisition unit 216 of each learner terminal device 200 acquires the generated presentation information.

(Step S114)
In step S114, the presentation unit (display unit) 244 presents the acquired presentation information.

Further, in this step S114, the annotation unit 214 may update the discrimination logic used when annotating the series information by referring to the information included in the presentation information.

(Step S116)
In step S116, the terminal device 200 (or the instructor terminal device 1000) acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming / correcting the determined importance. The communication unit 220 of the terminal device 200 transmits feedback information to the server 100.

(Step S118)
In step S118, the importance determination unit 114 or the determination integration unit 116 of the server 100 updates each determination logic with reference to the feedback information from the learner terminal device 200. For example, the importance determination unit 114 updates the importance determination logic based on the feedback information including the confirmed / corrected importance from the terminal device 200.

(Step S120)
When the series information acquisition unit 212 of the terminal device 200 acquires the series information again, the annotation unit 214 adds annotations to the series information to generate the annotated series information. Hereinafter, the steps of steps S120 to S136 are the same as the steps of steps S102 to S118.

In the information processing system 1, every time the learner terminal device 200 acquires the series information, the steps from step S102 to step S118 are repeated.

Therefore, the server 100 updates each discrimination logic every time the series information is acquired from the target person. As the series information from the target person increases, each discrimination logic is improved, and each discrimination can be performed more preferably.

<Other examples 1 such as annotation processing, importance assignment processing, and presentation information generation processing>
The annotation processing, importance assignment processing, presentation information generation processing, and the like by the information processing system 1 according to the present embodiment are not limited to the above examples. Hereinafter, other examples such as annotation processing, importance assignment processing, and presentation information generation processing by the information processing system 1 according to the present embodiment will be described.

The series information in this example is information that records the visual target of the subject during the mock test. Further, FIG. 3 is a diagram showing an example of a test question to be recorded. In FIG. 3, regions R1 to R3 are pre-defined regions for distinguishing each range. Further, the marker 301 is a marker for identifying the coordinates indicating the tip of the line of sight of the subject as the coordinates on the question sheet of the mock test.

The annotation unit 214 according to this example gives information to the series information indicating which part of the test question the subject was visually recognizing during the recording of the series information. For example, the annotation unit 214 adds information or the like indicating that the target person has visually recognized the area R1 to the series information at time 00:05.

FIG. 4 is a diagram showing an example of presentation information. Further, FIG. 4A is a time-series diagram showing how to move the line of sight, which is a norm when solving the target mock test. FIG. 4B is a time-series diagram showing how the line of sight of the subject who solved the mock test moves. FIG. 4C is a diagram showing the importance at each time point given to the series information by the importance determination unit 114. In each figure of FIG. 4, the horizontal axis represents time. The vertical axis of FIG. 4C shows the high degree of importance.

For example, in FIG. 4C, when the subject read the entire examination question, the variation in the line of sight was larger than that of the standard line of sight, so that the importance determination unit 114 was highly important. Shows the set location. Further, in location 304, when the subject examined the answer, the ratio of visually recognizing the region R1 which is the point of the answer was low, so that the subject was judged to have a low degree of understanding and the importance was determined. Part 114 indicates a place where a high degree of importance is set.

The importance determination unit 114 may set a high importance in a place where the difference between the information shown in FIG. 4 (A) and the information shown in FIG. 4 (B) is large or small, or the target. When the ratio of the person visually recognizing the area that is the point of the answer is high, a high importance may be set for the corresponding part in the series information. Further, FIG. 4C includes, as the integrated discrimination information generated by the discrimination integration unit 116, information indicating the importance set in the series information corresponding to other users so that the users can compare. May be.

FIG. 5 is a diagram showing an example of presentation information displayed on the display unit 244 of the learner terminal device 200 (or the display unit 1044 of the instructor terminal device 1000). In FIG. 5, the

objects

311, 312 and 313 correspond to FIGS. 4 (A), 4 (B) and 4 (C), respectively.

The screen 315 is a playback screen of a video recording the visual target of the target person. Further, the marker 301 may not be displayed on the screen.

The object 317 and the object 314 in the seek bar 316 indicate which part of the video recording the visual target of the target person is being reproduced on the screen 315. Further, the position where the video is reproduced may be changed by sliding the object 316 or the object 317 left and right. The object 318 is an operation panel for controlling the reproduction of the video.

Further, the part where the video is reproduced may be changed by selecting a part of the text on the screen 315. That is, when a part of the sentence is selected, the reproduction of the image may start from the place where the target person visually recognizes the part of the sentence.

Further, the display unit 244 may display the screen illustrated in FIG. 5 by including a pie chart or the like showing the ratio of which part of the examination question the subject was looking at. The same applies to the above-mentioned normative pie chart and the like for the line of sight.

In addition, the operation reception unit 243 accepts input of comments and the like corresponding to each part of the video. The input comment or the like is transmitted to the server 100 as feedback information via the communication unit 220.

<Other examples 2 such as annotation processing, importance assignment processing, and presentation information generation processing>
Other examples of annotation processing, importance assignment processing, and presentation information generation processing by the information processing system 1 according to the present embodiment will be described. It should be noted that the duplicate explanation of the matters already explained in the above example will not be repeated.

The series information in this example is information that records the visual target of the target person who learns the customer service work in a restaurant or the like. Further, FIG. 6 is a diagram showing an example of a visual target of the target person during learning of customer service work. The image shown in FIG. 6 shows the inside of the restaurant as seen by a learner who is a clerk, and the image shows a customer, one's own hand, a clerk other than oneself, a table, and the like.

Further, FIG. 7 is a table showing an example of data on the learner's visual object, utterance (utterance content and voice tone), movement and concentration (blink, pupil state, line-of-sight dwell time and facial expression). It is a table which shows an example of the annotation which the annotation addition part 214 which concerns on this example gave to the series information at each time point.

As shown in FIG. 7, as an example, the annotation unit 214 assigns a visual object, an utterance, an action, and a degree of concentration as annotations at each time point. An event is a set of decisions (utterances, actions, etc.) made within a predetermined time from the timing when a learner or a veteran customer service person visually recognizes an object. In one example, an event includes timing, visual object, utterance, action, and concentration, and each event can be identified by an ID.

In addition to the condition of the eyes, for example, the movement of the eyebrows such as lifting the inside or the outside of the eyebrows, the movement of the eyebrows such as raising the upper eyelids and the tension of the eyebrows, and the nose such as wrinkling the nose. The state of each part of the face such as the movement of the lips, the movement of the lips such as lifting the upper lip and purging the lips, the movement of the cheeks such as lifting the cheeks, and the movement of the nose such as lowering the nose. As the state of the learner, the states of a plurality of parts of the face may be combined.

As a method of judging the degree of concentration of the learner from the facial expression, for example, when the eyebrows move upward, it can be judged that the learner is concentrated because he / she is paying more attention to the visual object. Further, for example, when the cheek moves upward while visually recognizing a person's face, it can be determined that the person is concentrating on the visual target as if he / she is making a facial expression for the other party.

Further, FIG. 8 is a table showing an example of data of veteran customer servicers stored in advance in the storage unit 130. FIG. 8 shows the timing at which the object is visually recognized, the object to be visually recognized, the content of the utterance, the tone of the voice, the movement, and the degree of concentration as an example of the data of the veteran customer service.

Further, in FIG. 9, information on “deviation” between the learner data shown in FIG. 7 and the veteran customer service data shown in FIG. 8 (“+/-” or “+/-” in the table was used ( Quantitative information indicated by the value of learner-experienced customer service), or information on whether or not the data matches the data of veteran customer service recipient (qualitative information indicated by "○ / ×" in the table). .. As shown in FIG. 9, when the learner's data and the veteran customer service data do not match, the veteran customer service data (“correct answer” in the table) may be included.

The importance determination unit 114 determines the importance of each element included in the annotated series information of the target person with reference to the information shown in FIGS. 7 to 9. For example, the information corresponding to the event ID "4" in FIG. 9 is the place where the veteran customer service person said "I will confirm your order", but the target person did not make such an utterance. It is shown that. The importance determination unit 114 may set a higher importance than the other locations in the series information where the difference in operation between the veteran customer service customer and the target person is large.

Further, for example, the information corresponding to the event ID "3" indicates that there is an error of as much as 7 seconds between the veteran customer service person and the target person in the timing of performing the operation corresponding to the information. The importance determination unit 114 may set a higher importance than the other locations in the series information in which the difference in the timing of the operation between the veteran customer service customer and the target person is large. Further, for example, the importance determination unit 114 is higher than other places where the concentration of the target person is lower than a predetermined threshold value, even if the difference from the concentration of the veteran customer service is relatively small. You may set the importance.

Further, the configuration of the playback screen of the presentation information illustrated in FIG. 5 is also applicable in this example. In addition, the presentation information generation unit 118 includes, in the screen 315 of FIG. 5, a text indicating that there is a difference in operation between the veteran customer service person and the target person at the target timing and the content of the difference. Information for presentation may be generated in.

<Other examples 3 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. It should be noted that the duplicate explanation of the matters already explained in the above example will not be repeated.

As an example, the series information in this example is video data that captures the movement of the assembler in FA (factory automation), image data taken by a wearable camera attached to the assembler, assembly sound, and information that records a visual object. is there. The assembler may be a plurality of assemblers who assemble the same product, or may be an assembler who assembles the same product on different lines.

FIG. 10 is a table showing an example of data on the target person's visual object, movement, concentration (blinking, line-of-sight retention time, and pupil state), and assembly sound, and the annotation unit 214 according to this example , Is a table showing an example of annotations given to series information at each time point. As shown in FIG. 10, as an example, the annotation unit 214 assigns a visual object, an utterance, an action, a degree of concentration, and an assembly sound as annotations at each time point.

Further, FIG. 11 is a table showing an example of data of a veteran assembler stored in advance in the storage unit 130. FIG. 11 shows, as an example of data of a veteran assembler, the timing of visually recognizing an object, the object to be visually recognized, the movement, the degree of concentration, and the sound of assembly. Further, in FIG. 12, information on the “deviation” between the data of the subject shown in FIG. 10 and the data of the veteran assembler shown in FIG. 11 (“+/-” or “+/-” in the table was used ( Quantitative information indicated by the value of the subject-experienced assembler), or information on whether or not the data matches the data of the veteran assembler (qualitative information indicated by "○ / ×" in the table). .. As shown in FIG. 12, when the data of the subject and the data of the veteran assembler do not match, the data of the veteran assembler (“correct answer” in the table) may be included.

The importance determination unit 114 determines the importance of each element included in the annotated series information of the target person with reference to the information shown in FIGS. 10 to 12. For example, in the information corresponding to the event ID "3" in FIG. 10, the veteran assembler picks up the end of the part B with his left hand, while the subject picks up the center of the part B with his right hand. It is shown that. The importance determination unit 114 may set a higher importance than the other locations in the series information in which the difference in operation between the veteran assembler and the target person is large.

Also, for example, the information corresponding to the event ID "7" indicates that there is an error of 18 seconds between the veteran assembler and the target person in the process of assembling the part B to the part A. Therefore, it is shown that the time required for the same assembly operation differs by as much as 18 seconds between the veteran assembler and the subject. The importance determination unit 114 may set a higher importance than the other locations for the locations in the series information in which the difference in time required for the same operation is large. Further, for example, the importance determination unit 114 is higher than other places where the concentration of the subject is lower than a predetermined threshold value, even if the difference from the concentration of the veteran assembler is relatively small. You may set the importance. Alternatively, the assembler may specify the importance of each event.

The presentation information generation unit may integrate the above importance determination results to generate, for example, the following presentation information.

・ Presentation information that presents points to be improved and points to be noted in each process may be generated for the target person. Based on the above presentation information, a video work manual for new assemblers may be created, in which highly important processes such as difficult processes and processes that are easily mistaken are reproduced in close-up or slow mode.

-For assembler management by the administrator, it is possible to generate presentation information that presents the time-series changes in the performance of each assembler and the points where multiple assemblers make mistakes.

-Furthermore, in order to improve each process, points that should be improved in the process itself, such as changing the process order, separating the process into multiple processes, or integrating multiple processes, where most assemblers make mistakes. You may generate the presentation information presented about.

Further, the configuration of the playback screen of the presentation information illustrated in FIG. 5 is also applicable in this example. In addition, the presentation information generation unit 118 includes, in the screen 315 of FIG. 5, a text indicating that there is a difference in operation between the veteran assembler and the target person at the target timing and the content of the difference. Information for presentation may be generated in.

<Other examples 4 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. Again, the duplicate description of the matters already described in the above example will not be repeated.

The series information in this example is, for example, video data that captures the movement of the cook in the cooking scene, data captured by the wearable camera attached to the cook, cooking sound, and information that records the visual object. The cook may be multiple cooks who cook the same menu, and may be cooks who want to learn cooking personally at each home, or employees who cook in the kitchen in a restaurant or supermarket. There may be.

FIG. 13 is a table showing an example of data on the subject's visual object, movement, concentration (blinking, line-of-sight retention time, and pupil state), and cooking sound, and the annotation unit 214 according to this example , Is a table showing an example of annotations given to series information at each time point. As shown in FIG. 13, the annotation unit 214 assigns a visual object, an utterance, an action, and a degree of concentration as annotations at each time point as an example.

Further, FIG. 14 is a table showing an example of data of a veteran cook stored in advance in the storage unit 130. FIG. 14 shows, as an example of data of a veteran cook, the timing of visually recognizing an object, the object to be visually recognized, the movement, the degree of concentration, and the sound of cooking. Further, in FIG. 15, information on “deviation” between the data of the subject shown in FIG. 13 and the data of the veteran cook shown in FIG. 14 (“+/-” or “+/-” in the table was used ( Quantitative information indicated by the value of the subject-experienced cook), or information on whether or not the data matches the data of the veteran cook (qualitative information indicated by "○ / ×" in the table). .. As shown in FIG. 15, when the data of the subject and the data of the veteran cook do not match, the data of the veteran cook (“correct answer” in the table) may be included.

The importance determination unit 114 determines the importance of each element included in the annotated series information of the target person with reference to the information shown in FIGS. 13 to 15. For example, in the information corresponding to the event ID “3” in FIG. 13, when a veteran cook holds a kitchen knife, he puts his index finger on the back of the kitchen knife and grabs the kitchen knife with his middle finger and thumb. It shows that the kitchen knife is being held with the thumb and index finger. The importance determination unit 114 may set a higher importance than the other locations in the series information in which the difference in operation between the veteran cook and the target person is large.

Also, for example, the information corresponding to the event ID "5" indicates that there is an error of 8 seconds between the veteran cook and the target person in the scene of cutting the ingredients. Therefore, it is shown that the time required for the same cooking operation differs by as much as 8 seconds between the veteran cook and the subject. The importance determination unit 114 may set a higher importance than the other locations for the locations in the series information in which the difference in time required for the same operation is large. Further, for example, the importance determination unit 114 is higher than other places where the concentration of the subject is lower than a predetermined threshold, even if the difference from the concentration of the veteran cook is relatively small. You may set the importance. Alternatively, the cook may specify a high degree of importance for difficult processes or processes that affect taste and quality.

The presentation information generation unit may integrate the above-mentioned importance determination results and present the following presentation information, for example.

・ For the target person, presentation information that presents points to be improved and points to be noted in each cooking scene may be generated. Based on the above-mentioned presentation information, a cooking manual may be created by a close-up or slow-playing video of highly important processes such as difficult processes and processes that are easily mistaken.

・ If the target person is an employee who cooks in a kitchen in a restaurant or supermarket, the time-series changes in the performance of each cook and the places where multiple cooks make mistakes in order to manage the cooks by the manager. You may generate the presentation information presented about.

-Furthermore, in order to improve each process, points that should be improved in the process itself, such as changing the process order, separating the process into multiple processes, or integrating multiple processes, for the parts that most cooks make mistakes. You may generate the presentation information presented about.

<Other examples 5 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. Again, the duplicate description of the matters already described in the above example will not be repeated.

The target person in this example includes a player in sports. The annotation may be added by the coach, may be added by the annotation unit 214, or may be a combination thereof.

Further, one coach may annotate a plurality of players, or a plurality of coaches may annotate one player. Further, a plurality of coaches may annotate a plurality of players.

Alternatively, the configuration may include a plurality of separately learned annotation units 214, and a plurality of annotation units may add annotations to one player. Further, a plurality of annotation units may add annotations to the plurality of players.

The series information in this example is, for example, player image, voice, and body sensing data. The player may be a player of any sport, but here, for example, a baseball player (batter) will be described as an example. In addition, the player may be a professional athlete, a student or a member of society.

FIG. 16 is a table showing an example of data on the visual object, movement, and degree of concentration (blinking, line-of-sight retention time, and pupil state) of player A (player with low batting average), and annotating according to this example. Part 214 is a table showing an example of annotations given to the sequence information at each time point. As an example, the annotation unit 214 assigns a visual object, an utterance, an action, and a degree of concentration as annotations at each time point.

Further, FIG. 17 shows, as an example of the data of player B (player with a high batting average), the timing at which the object is visually recognized, the visual target, the movement, the degree of concentration, and the sound of play.

Further, FIG. 18 uses information (“+/-” or “+/-” in the table of “deviation” between the data of player A shown in FIG. 16 and the data of player B shown in FIG. 17 (player). Quantitative information indicated by the value of A-player B), or information on whether or not the data matches the data of player B (qualitative information indicated by "○ / ×" in the table). As shown in FIG. 18, when the data of the player A and the data of the player B do not match, the data of the player B (“correct answer” in the table) may be included.

The importance determination unit 114 determines the importance of each element included in the annotated series information of each player with reference to the information shown in FIGS. 16 to 18. For example, the information corresponding to the event ID “1” in FIG. 16 indicates that the player B holds the bat short, while the player A holds the bat long. The importance determination unit 114 may set a higher importance than the other locations for the locations in the series information where the difference in operation between the players is large.

Further, for example, the information corresponding to the event ID "3" indicates that the swing speed of the player A is 135 Km / hour, while the swing speed of the player B has an error of 148 Km / hour and 13 km / hour. ing. The importance determination unit 114 may set a higher importance than the other locations for the locations in the series information in which the difference in speed at which the same operation is performed is large. Further, for example, the importance determination unit 114 may set a higher importance than other locations at a location where the player's concentration is lower than a predetermined threshold value. Alternatively, a high degree of importance may be specified for an operation or the like that makes a large difference depending on the player.

The presentation information generation unit integrates the above-mentioned importance determination results, for example, integrates information annotated by a plurality of coaches for each player to improve the play of each player, and should be improved. Presentation information that presents a part of play (scene of an image) may be generated.

Further, the information annotated by one or more coaches to a plurality of players may be integrated to generate presentation information for coaches and other coaches to present improvement points of guidance. Further, based on the above-mentioned presentation information, a manual for teaching may be created in which scenes important for successful batting are reproduced in close-up or slow-playing.

In the above example, the case where the play by each player such as batting is input as the series information is described as an example, but the series information is, for example, a play, an operation, etc. by a plurality of players such as a formation in soccer. May be.

Furthermore, the input data may include information for the spectators of the match. For example, moving image data that captures the state of the spectator, line-of-sight information of the spectator, and voice emitted by the spectator may be used as input data.

Based on the data related to these spectators, the degree of attention and excitement of the spectators may be added as annotations for each scene during the match.

The presentation information generation unit may generate presentation information for players and coaches that presents scenes that attracted a lot of attention from spectators.

In the following, other examples such as annotation processing, importance assignment processing, and presentation information generation processing will be described.

<Other examples 6 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. Again, the duplicate description of the matters already described in the above example will not be repeated.

In this example, an example is an electronic sport (e-sports) in which a character operated by a player existing in the real world competes in a virtual space. The above-mentioned player may be a plurality of players who play the same e-sports.

The series information in this example includes, as an example, an input from a player to a personal computer, a game operation unit, or the like. Further, the series information in this example may include information regarding the movement of the character in the virtual space generated based on these inputs. In addition, the series information may include player image, voice, and body sensing data. Further, the series information in this example may include information for spectators of e-sports as in the case of the above-mentioned sports. Information on the degree of attention and excitement of the spectator may be included.

The annotation unit 214 refers to the series information and, as an example, assigns the movement of the character, the degree of attention of the spectator, and the degree of excitement as annotations at each time point.

The importance determination unit 114 refers to the annotated series information of a plurality of players, and determines the importance of each element included in the annotated series information. For example, the importance determination unit 114 may set a higher importance than other points for a place in the series information in which the difference in operation between characters is large.

Further, for example, the importance determination unit 114 may set a higher importance than other locations at a location where the player's concentration is lower than a predetermined threshold value. Alternatively, a high degree of importance may be specified for an operation or the like that makes a large difference depending on the player.

Then, the presentation information generation unit may generate presentation information for the player to present a scene that attracted a lot of attention from the e-sports spectator.

In addition, the presentation information generation unit integrates the above-mentioned importance determination results to generate presentation information for each player, for example, presenting points to be improved and parts of play to be improved for each player. You may.

<Other examples 7 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. Again, the duplicate description of the matters already described in the above example will not be repeated.

The series information in this example is video data that captures the operation of an office worker in office work, data captured by a wearable camera attached to the office worker, work sound, and information that records a visual object. Further, the series information may include an operation history of a terminal device such as a keyboard or a mouse, a usage history of an application, and the like.

As an example, the annotation unit 214 adds information on a visual object, an utterance, an action, a degree of concentration, a work sound, an operation history of a terminal device, an application usage history, and the like as annotations at each time point.

The importance determination unit 114 refers to the annotated series information of the target office worker (target person) and the reference annotated series information as an example, and each element included in the annotated series information of the target person. Determine the importance of. Here, the reference annotated series information may include annotated series information of another office worker such as a veteran.

For example, the importance determination unit 114 may set a higher importance when the time required for text input by the target person is longer than that of other office workers. Further, the importance determination unit 114 may set a higher importance when the time required for the spreadsheet of the target person is longer than that of other office workers.

・ Presentation information that presents points to be improved and points to be noted in each work situation may be generated for the target person. Based on the above-mentioned presented information, a work manual may be created by a close-up or slow-playing video of highly important work such as difficult work or work that is easily mistaken.

-Furthermore, in order to improve each work content, the work itself should be improved by changing the work order, separating the work into multiple work, or integrating multiple work for the work that most office workers make a mistake. The presentation information presented about the points may be generated.

<Other examples 8 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. Again, the duplicate description of the matters already described in the above example will not be repeated.

As an example, the series information in this example includes video data that captures the movement of the telephone operator (target person), image data captured by a wearable camera attached to the telephone operator, voice data of communication with the telephone partner, work sound, and so on. This is information that records the visual target. Further, the series information may include the operation history of a terminal device such as a keyboard or a mouse.

As an example, the annotation unit 214 adds information on the target person's visual target, utterance, movement, concentration, work sound, terminal device operation history, application usage history, etc. as annotations at each time point. Further, the annotation unit 214 may estimate the tension level of the target person at each time point and add an index indicating the tension level as an annotation. A known algorithm can be used as the algorithm for estimating the degree of tension.

As an example, the importance determination unit 114 determines the importance of each element included in the annotation series information of the target person by referring to the annotation series information of the target person and the annotation series information as a reference. To do. Here, the reference annotated series information may include annotated series information of another telephone operator such as a veteran.

For example, the importance determination unit 114 may set a higher importance when the tension of the subject is higher than that of other telephone operators. Further, the importance determination unit 114 may set a higher importance when the tension with respect to a specific telephone partner is higher than the tension with respect to another telephone partner.

・ For the target person, presentation information that presents points to be improved and points to be noted in each telephone response situation may be generated. Based on the above-mentioned presented information, a work manual may be created by a close-up or slow-playing video of highly important work such as difficult telephone correspondence and telephone correspondence that is easy to make a mistake.

-Furthermore, for the administrator, it is possible to generate presentation information indicating the contents of the telephone correspondence and the telephone party with high tension of most telephone responders.

<Other examples 9 such as annotation processing, importance assignment processing, and presentation information generation processing>
Further examples of the annotation processing, the importance assignment processing, and the presentation information generation processing by the information processing system 1 according to the present embodiment will be described. Again, the duplicate description of the matters already described in the above example will not be repeated.

As an example, the series information in this example includes video data of the movement of a student (target person) in a driving school, image data of a wearable camera attached to the target person, voice data of communication with an instructor, and driving movement. This is information that records sound and visual objects. Further, the series information may include sensing data of an operation unit such as a steering wheel or a brake.

As an example, the annotation unit 214 adds information on the visual object, the operation, the degree of concentration, the utterance, the driving operation sound, the operation history of the operation unit, and the like as annotations at each time point. Further, the annotation unit 214 may estimate the tension level of the target person at each time point and add an index indicating the tension level as an annotation. A known algorithm can be used as the algorithm for estimating the degree of tension.

As an example, the importance determination unit 114 determines the importance of each element included in the annotation series information of the target person by referring to the annotation series information of the target person and the annotation series information as a reference. To do. Here, the reference annotated series information may include annotated series information of a model driver such as an instructor.

For example, the importance determination unit 114 sets a higher importance when the timing at which the target person starts to turn the handle and the timing at which the model driver starts to turn the handle are larger than a predetermined value when turning a certain curve. You may. In addition, the importance determination unit 114 may set a higher importance when the tension in a specific course is higher than the tension in other courses.

・ For the target person, presentation information that presents points to be improved and points to be noted in each driving scene may be generated. Based on the above-mentioned presented information, a driving manual may be created by a close-up or slow-playing video of highly important driving such as a difficult driving scene.

-Furthermore, you may generate presentation information for instructors that shows the courses and driving scenes where most students are highly nervous.

<Embodiment 2>
The configuration of the information processing system in the present invention is not limited to the above. FIG. 10 is a block diagram schematically illustrating a configuration example of the information processing system 1 according to the second embodiment. The information processing system 1 of the second embodiment is different from the information processing system 1 of the first embodiment in that the importance determination unit 215 is not provided in the server 100 but is provided in the learner terminal device 200. Other configurations are the same.

Next, the operation of the information processing system 1 of the second embodiment will be described. FIG. 11 is a sequence diagram illustrating an operation example of the information processing system 1 according to the second embodiment. The process of generating the presentation information in the information processing system 1 of the second embodiment will be described with reference to FIG.

(Step S202)
In step S202, the annotation-giving unit 214 generates the annotated series information by adding annotations to each element included in the series information regarding the target person acquired by the series information acquisition unit 212.

(Step S206)
In step S206, the importance determination unit 215 refers to the annotated series information about the target person generated by the annotation unit 214, and determines the importance of each element included in the annotation series information of the target person. The communication unit 220 of the learner terminal device 200 transmits the importance determination result to the server 100.

(Step S207)
In step S207, the series information acquisition unit 112 of the server 100 receives the importance determination result.

(Step S208)
In step S208, the discrimination integration unit 116 of the server 100 refers to the judgment results of the importance of the plurality of target persons A, B, C ..., And generates the discrimination integration information.

(Step S210)
In step S210, the presentation information generation unit 118 generates presentation information by referring to at least one of the discrimination result by the importance determination unit 215 and the discrimination integration information. When the presentation information generation unit 118 generates the presentation information by referring only to the determination result by the importance determination unit 215, the process of step S208 does not necessarily have to be executed, and the process follows step S207. The process of this step S210 may be executed. The communication unit 120 of the server 100 transmits the generated presentation information to each learner terminal device 200.

(Step S212)
In step S212, the presentation information acquisition unit 216 of each learner terminal device 200 acquires the generated presentation information.

(Step S214)
In step S214, the presentation unit (display unit) 244 presents the acquired presentation information.

Further, in this step S214, the annotation unit 214 may update the discrimination logic used when annotating the series information by referring to the information included in the presentation information.

(Step S216)
In step S216, the terminal device 200 acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming / correcting the determined importance. The feedback information is transmitted from the terminal device 200 to the server 100.

(Step S218)
In step S218, the importance determination unit 215 of the learner terminal device 200 updates the determination logic based on the feedback information. For example, the importance determination unit 215 updates the importance determination logic based on the feedback information including the confirmed / corrected importance. Further, the discrimination integration unit 116 of the server 100 also updates the discrimination integration logic based on the feedback information.

(Step S220)
When the series information acquisition unit 212 of the terminal device 200 acquires the series information again, in step S220, the annotation unit 214 adds annotations to the series information to generate the annotated series information. Hereinafter, the steps of steps S220 to S236 are the same as the steps of steps S202 to S218.

In the information processing system 1 of the second embodiment, every time the learner terminal device 200 acquires the series information, the steps from step S202 to step S218 are repeated.

Therefore, the importance determination unit 215 of the learner terminal device 200 updates the importance determination logic every time the sequence information is acquired from the target person. As the series information from the target person increases, the importance determination logic is improved, and the importance can be determined more preferably. The same applies to the discrimination logic used by the discrimination integration unit 116 for the discrimination integration process.

<Embodiment 3>
FIG. 12 is a block diagram schematically illustrating a configuration example of the information processing system 1 according to the third embodiment. The information processing system of the third embodiment is different from the information processing system of the first embodiment in that the presentation information generation unit 1013 is not provided in the server 100 but is provided in the instructor terminal device 1000. The configuration is the same.

Next, the operation of the information processing system 1 of the third embodiment will be described. FIG. 13 is a sequence diagram illustrating an operation example of the information processing system 1 according to the third embodiment. The process of generating the presentation information in the information processing system 1 of the third embodiment will be described with reference to FIG.

(Step S302)
In step S302, the annotation-giving unit 214 generates the annotated series information by adding annotations to each element included in the series information regarding the target person acquired by the series information acquisition unit 212. The communication unit 220 transmits the generated annotated sequence information to the server 100.

(Step S304)
In step S304, the sequence information acquisition unit 112 of the server 100 acquires the annotated sequence information.

(Step S306)
In step 306, the importance determination unit 114 refers to the annotated series information about the target person and determines the importance of each element included in the annotated series information of the target person.

(Step S308)
In step S308, the discrimination integration unit 116 refers to the discrimination results of the importance of the plurality of subjects A, B, C ..., And generates the discrimination integration information. The generated discriminant integrated information is transmitted to the instructor terminal device 1000.

(Step S310)
In step S310, the presentation information generation unit 1013 of the instructor terminal device 1000 generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit 114 and the discrimination integrated information. When the presentation information generation unit 1013 generates presentation information by referring only to the determination result by the importance determination unit 114, the process of step S308 does not necessarily have to be executed, and the determination in step S306 is required. The result may be transmitted to the instructor terminal device 1000, and subsequently the process of this step S310 may be executed. The generated presentation information is transmitted to each learner terminal device 200.

(Step S312)
In step S312, in each learner terminal device 200, the presentation information acquisition unit 216 acquires the generated presentation information.

(Step S314)
In step S314, the presentation unit (display unit) 244 presents the acquired presentation information.

Further, in this step S314, the annotation unit 214 may update the discrimination logic used when annotating the series information by referring to the information included in the presentation information.

(Step S316)
In step S316, the instructor terminal device 1000 acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming / correcting the determined importance. The feedback information is transmitted from the instructor terminal device 1000 to the server 100.

(Step S318)
In step S318, the importance determination unit 114 or the determination integration unit 116 of the server 100 updates each determination logic based on the feedback information from the instructor terminal device 1000. For example, the importance determination unit 114 updates the importance determination logic based on the feedback information including the confirmed / corrected importance from the instructor terminal device 1000.

(Step S320)
When the series information acquisition unit 212 of the terminal device 200 acquires the series information again, the annotation unit 214 adds annotations to the series information to generate the annotated series information. Hereinafter, the steps of steps S320 to S336 are the same as the steps of steps S302 to S318.

In the information processing system 1, every time the learner terminal device 200 acquires the series information, the steps from step S302 to step S318 are repeated.

[Example of realization by software]
The control block of the information processing device 100 (particularly, the sequence information acquisition unit 112, the importance determination unit 114, the discrimination integration unit 116, the presentation information generation unit 118), and each terminal device 200 (particularly the sequence information acquisition unit 212, annotation addition). The unit 214 and the presentation information acquisition unit 216) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the information processing device 100 and each terminal device 200 include a computer that executes a program instruction that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. Then, in the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, in addition to a “non-temporary tangible medium” such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (RandomAccessMemory) for expanding the above program may be further provided. Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[Summary]
In order to solve the above problems, the information processing apparatus according to one aspect of the present invention includes a sequence information acquisition unit that acquires annotated sequence information in which annotations are added to each element included in the sequence information related to the target person. The importance determination unit for determining the importance of each element included in the annotated series information of the target person by referring to the annotated series information regarding the target person, and the determination result by the importance determination unit for a plurality of target persons. A discrimination integration unit that generates discrimination integration information by referring to, a presentation information generation unit that generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit and the discrimination integration information. It has.

According to the above configuration, important information can be suitably discriminated and presented among the information having series.

In the information processing device according to the above aspect, the importance determination unit may update the determination logic with reference to the feedback information from the user.

According to the above configuration, since the discrimination logic in the importance determination unit is updated with reference to the feedback information from the user, it is possible to more accurately determine the importance of the series information each time the importance determination is repeated. it can.

In the information processing device according to the above aspect, the series information may include information regarding the line of sight of the target person.

According to the above configuration, the importance can be more preferably determined by referring to the information regarding the line of sight of the subject.

In the information processing apparatus according to the above aspect, the discrimination integration unit may extract common information from the discrimination results of the importance discrimination unit for each of the plurality of target persons and include it in the discrimination integration information. ..

According to the above configuration, even if the series information differs depending on the target person, common information is extracted, so that the importance can be determined more preferably.

In the information processing device according to the above aspect, the importance determination unit may determine the degree of concentration regarding the target person as the importance.

According to the above configuration, the importance of the series information can be determined by referring to the degree of concentration regarding the target person, so that the importance can be determined more preferably.

According to the above configuration, it is possible to acquire presentation information in which each series information is integrated and present it to the user.

The terminal device according to the above aspect further includes a discrimination integrated information acquisition unit that acquires discrimination integration information generated by referring to discrimination results related to annotation series information of a plurality of target persons, and the annotation addition unit is a user. The annotation logic may be updated with reference to at least one of the feedback information from the above and the discrimination integration information.

According to the above configuration, annotation can be more preferably performed by updating the annotation logic with reference to each information.

In the terminal device according to the above aspect, the annotation unit may add information indicating the reliability of the annotation to each element in addition to the annotation.

According to the above configuration, each process that refers to the annotation can be executed more preferably.

According to the above configuration, the same effect as that of the information processing device according to one aspect of the present invention can be obtained.

The information processing program according to one aspect of the present invention is an information processing program for operating a computer as the information processing device according to any one of the above, and includes the series information acquisition unit, the importance determination unit, and the discrimination integration unit. And make the computer function as a presentation information generator.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1 Information processing system 100 Information processing device 100

Server

110, 210, 1010

Control unit

112, 212 Series information acquisition unit 114, 215 Importance determination unit 116

Discrimination integration unit

118, 1013 Presentation

information generation unit

120, 220, 1020

Communication unit

130, 230, 1030 Storage unit 200 Learner terminal device 214

Annotating unit

216, 1014 Information acquisition unit for presentation 241 Camera 242

Microphone

243, 1043 Operation reception unit 244 Display unit (presentation unit)
1044

Display unit

245, 1045 Speaker 1000 Instructor terminal device 1012 Feedback information acquisition unit

Claims

A series information acquisition unit that acquires annotated series information with annotations attached to each element included in the series information related to the target person, and
An importance determination unit that refers to the annotated series information about the target person and determines the importance of each element included in the annotated series information of the target person.
A discrimination integration unit that generates discrimination integration information by referring to the discrimination results of the importance discrimination unit for a plurality of target persons, and
A presentation information generation unit that generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit and the discrimination integrated information.
An information processing device characterized by being equipped with.
The information processing apparatus according to claim 1, wherein the importance determination unit updates the determination logic with reference to at least one of the feedback information from the user and the determination integrated information.
The information processing apparatus according to claim 1 or 2, wherein the series information includes information about the line of sight of the target person.
Any of claims 1 to 3, wherein the discrimination integration unit extracts common information from the discrimination results by the importance discrimination unit for each of the plurality of target persons and includes the common information in the discrimination integration information. The information processing apparatus according to item 1.
The information processing apparatus according to any one of claims 1 to 4, wherein the importance determination unit determines the degree of concentration with respect to the target person as the importance.
The series information acquisition unit that acquires series information about the target person,
Annotation section that generates annotated series information by annotating each element included in the acquired series information,
The presentation information acquisition unit that acquires the presentation information generated by referring to the annotated series information,
A terminal device including a presentation unit that presents the presentation information.
It also has a discriminant integrated information acquisition unit that acquires discriminant integrated information generated by referring to the discriminant results related to annotated series information of multiple subjects.
The terminal device according to claim 6, wherein the annotation section refers to at least one of the feedback information from the user and the discriminant integration information to update the annotation logic.
The terminal device according to claim 6 or 7, wherein the annotation unit adds information indicating the reliability of the annotation to each element in addition to the annotation.
The acquisition unit that acquires series information,
Annotation section that generates annotated series information by annotating each element included in the acquired series information,
An importance determination unit that refers to the annotated series information about a target person and determines the importance of each element included in the annotated series information of the target person.
The discrimination integration unit that generates discrimination integration information by referring to the discrimination results by the importance discrimination unit for multiple subjects,
A presentation information generation unit that generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit and the discrimination integrated information.
Information processing system equipped with.
Annotation process that generates annotated series information by annotating each element included in the series information about the target person, and
The importance determination step of referring to the annotated series information and determining the importance of each element included in the annotated series information of the target person, and
A discrimination integration step that generates discrimination integration information by referring to the discrimination results in the importance discrimination step for a plurality of target persons, and
A presentation information generation step that generates at least one of the discrimination result in the importance discrimination step and the discrimination integrated information, and
An information processing method characterized by including.
A program for operating a computer as the information processing device according to any one of claims 1 to 5, wherein the series information acquisition unit, the importance determination unit, the discrimination integration unit, and the presentation information. An information processing program that allows a computer to function as a generator.