WO2024111386A1

WO2024111386A1 - Information processing device, information processing method, program, and information processing system

Info

Publication number: WO2024111386A1
Application number: PCT/JP2023/039825
Authority: WO
Inventors: 弘泰馬場; 宇紀深澤; 奈々河村
Original assignee: ソニーグループ株式会社
Priority date: 2022-11-21
Filing date: 2023-11-06
Publication date: 2024-05-30

Abstract

The present technology relates to an information processing device, an information processing method, a program, and an information processing system which make it possible to efficiently look back on the situation of a patient. This information processing device calculates a total feature amount on the basis of a data set comprising patient data that is at least an image feature amount being the feature amount of image data obtained by capturing an image of a patient or the clinical data of the patient, and generates a first display screen for displaying, in chronological order, the calculated total feature amount together with a display component indicating the playback position of the image data. The present technology can be applied to a look-back display system.

Description

Information processing device, information processing method, program, and information processing system

This technology relates to an information processing device, an information processing method, a program, and an information processing system, and in particular to an information processing device, an information processing method, a program, and an information processing system that enable efficient review of a patient's condition.

In intensive care units (ICUs) and other locations where careful observation of patients is required, doctors, nurses, and other medical professionals often assess a patient's condition not only by looking at changes in vital signs and test data, but also by comprehensively judging information from the patient's appearance, such as facial expressions and breathing patterns, to estimate the severity of the patient's condition.

For example, Patent Document 1 discloses a technology that analyzes a patient's movements based on hospital bed image data and estimates the severity of the patient's condition based on the analyzed movements.

In addition, cases frequently occur in the field where a significant change occurs in the measured vital signs, but in reality it is just the nurse intervening to temporarily remove the vital sensor.

Although the information obtained from external appearance is important, it is difficult to constantly observe patients and their bedside in the medical field due to factors such as manpower shortages and overwork.

Some hospitals have cameras installed on the ceilings above the beds, but the primary purpose of these is to prevent blind spots from the nurses' stations, and currently there is no review of recorded footage or analysis of the video data. Furthermore, even if medical staff were to review the recorded footage, it would be difficult to thoroughly review footage covering a huge number of patients and over a huge period of time.

International Publication No. 2020/203015

Therefore, there is a need for technology that can clearly display important scenes from continuously recorded patient footage.

This technology was developed in light of these circumstances, and makes it possible to efficiently review a patient's condition.

An information processing device according to one aspect of the present technology includes a data processing unit that calculates an overall feature amount based on a data set consisting of video feature amounts, which are feature amounts of video data obtained by imaging a patient, and patient data, which is at least one of the patient's clinical data, and a display screen generating unit that generates a first display screen that displays the calculated overall feature amount in chronological order together with a display component that indicates the playback position of the video data.

In one aspect of the present technology, an overall feature is calculated based on a data set consisting of image features, which are features of video data obtained by imaging a patient, and patient data, which is at least one of the patient's clinical data. Then, a first display screen is generated that displays the calculated overall feature in chronological order together with a display component that indicates the playback position of the video data.

1 is a diagram illustrating a configuration example of a review display system according to an embodiment of the present technology. 2 is a block diagram showing a detailed configuration of the review display system of FIG. 1 . 3 is a block diagram showing an example of a functional configuration of a data processing unit in FIG. 2 . FIG. 13 is a diagram showing an example of the configuration of a review display screen used for reviewing video data. FIG. 5 is an enlarged view of the video operation unit in FIG. 4 . 6 is a diagram showing an example of the configuration of patient features displayed on the playback seek bar in FIG. 5 . FIG. 7 is a diagram showing an example of display of patient features in the case of the configuration data set of FIG. 6 . 7 is a diagram showing an example of display of patient features in the case of the comprehensive feature in FIG. 6; FIG. FIG. 13 is a diagram illustrating an example of a comprehensive feature amount and a constituent data set. FIG. 13 is a diagram illustrating an example of customization of a comprehensive feature amount. FIG. 13 is a diagram showing an example of a superimposed display of comprehensive feature amounts. 13 is a flowchart illustrating a process of setting patient features. 13 is a flowchart illustrating a process of generating an event label. FIG. 13 is a diagram showing an example of threshold setting in the time-series data display operation unit. FIG. 13 is a diagram showing an example of a display of the playback position of video data currently being played back. FIG. 13 is a diagram showing an example of displaying correlated data. FIG. 2 is a block diagram illustrating an example of the configuration of a computer.

Hereinafter, an embodiment of the present technology will be described in the following order.
1. System configuration and device configuration 2. Screen configuration example 3. Other

1. System configuration and device configuration
<Configuration of the retrospective display system>
FIG. 1 is a block diagram showing an example of a configuration of a review display system according to an embodiment of the present technology.

The review display system 1 in Figure 1 is a system used in places such as intensive care units (ICUs) where the patient's condition needs to be carefully observed, allowing medical staff such as doctors and nurses to efficiently review images of the patient's condition around the bed. "Efficiently" here means that since images of patients (hereafter referred to as patient images) that are constantly being recorded can last for an enormous amount of time, only important scenes can be selected and played back.

The retrospective display system 1 displays information such as video, video features extracted from the video, and clinical data such as vital signs recorded in an existing patient information system 2, etc., all together on a display screen. The retrospective display system 1 also uses the patient's video features and event labels (tags) indicating events to clearly display important scenes of the patient, and can play back video data from those important scenes.

Here, video features include information obtained through image analysis (facial feature point detection, facial expression estimation, whole body skeletal point estimation, etc.) (e.g. facial expression of the subject, joint point positions, number of people, etc.) and information converted into values per unit time such as the amount of movement of the entire image.

More specifically, video features include the average value of the agony facial expression score per unit time, the frequency peak and amplitude of the positional change of the relevant area associated with shoulder breathing or mandibular breathing, the total amount of movement of each joint point per unit time associated with the patient's movement, the average number of people on the screen per unit time, and the total amount of image shift per unit time.

The retrospective display system 1 is broadly configured to include a sensing block 11, a data processing block 12, and a display block 13.

The sensing block 11 uses sensing equipment such as an RGB camera, an IR camera, a depth camera, and a microphone to capture images and sounds of the patient and the area around the bed.

The data processing block 12 processes the patient data recorded in the patient information system 2 and the video data and audio data acquired by the sensing block 11 in response to a specific program or an instruction signal sent from the display block 13, and generates display screen data.

The display block 13 displays a display screen corresponding to the display screen data generated by the data processing block 12. The display block 13 also transmits an instruction signal to the data processing block 12 in response to the operation of the medical staff.

Patient information system 2 is an information system within the hospital that records patient attributes, vital sign data, and test result data, such as electronic medical records and ICU information systems.

<Detailed configuration of the retrospective display system>
FIG. 2 is a block diagram showing a detailed configuration of the review display system 1 of FIG.

The retrospective display system 1 in FIG. 2 is configured to include a bedside terminal 21, an information server 22, and a display terminal 23. In FIG. 2, the dashed rectangular lines indicate that the corresponding devices may be configured as separate hardware or separate systems.

Each device is configured to include a communication unit (not shown). Although not specifically mentioned, information and data are sent and received between the devices via each communication unit.

The bedside terminal 21 is a terminal that is installed at the bedside where the patient is sleeping. The bedside terminal 21 is configured to include a sensing block 11.

In FIG. 2, the sensing block 11 is configured to include a sensing data acquisition unit 31 and the sensing device 32 described above. The sensing data acquisition unit 31 acquires video data, audio data, and depth information obtained by sensing performed by the sensing device 32. The acquired data and information are transmitted to the information server 22.

In the case of a sensing device 32 that can directly transmit video data, such as an IP camera, the data and information acquired by the sensing device 32 is transmitted to the information server 22 without passing through the sensing data acquisition unit 31.

The information server 22 is a device installed in a hospital or on the cloud. The information server 22 is configured to include a patient information system 2 and a data processing block 12. Note that the patient information system 2 may be installed in a server other than the information server 22.

In FIG. 2, the data processing block 12 is configured to include a data collection unit 41, a data processing unit 42, and a display screen generation unit 43.

The data collection unit 41 collects video data, audio data, and depth information, as well as data recorded in the patient information system 2. The data collected from the patient information system 2 is, for example, clinical data such as the patient's vital sign data, respiratory data, blood test data, blood gas test data, medication record data, nursing record data, and patient attribute data (gender, height, age, medical history, etc.). The data collection unit 41 outputs the collected clinical data to the data processing unit 42.

The data processing unit 42 performs data processing on the data supplied from the data collection unit 41, including image analysis such as facial expression estimation and whole body skeletal point estimation, as well as voice analysis, text analysis, and correlation evaluation.

The display screen generation unit 43 generates display screen data using the results of data processing by the data processing unit 42 in response to instruction signals and other signals transmitted from the display terminal 23. The generated display screen data is transmitted to the display terminal 23.

The display terminal 23 is a device on which a medical professional actually looks at the display screen and operates it. The display terminal 23 is, for example, a PC located at the bedside, nurse's station, or medical office, or a mobile device such as a tablet or smartphone. The display terminal 23 is configured to include the display block 13.

The display block 13 is configured to include a display control unit 51, a display unit 52, and an operation input unit 53.

The display control unit 51 causes the display unit 52 to display a display screen corresponding to the display screen data transmitted from the display screen generation unit 43. The display unit 52 is composed of an LCD (Liquid Crystal Display) panel, an organic EL (Electro-Luminescence) panel, or the like. The operation input unit 53 inputs instruction signals in response to operations performed by a medical professional using a touch panel, mouse, or the like. The input instruction signals are transmitted to the information server 22 (the display screen generation unit 43).

<Example of functional configuration of data processing unit>
FIG. 3 is a block diagram showing an example of the functional configuration of the data processing unit 42 in FIG.

The functional configuration in FIG. 3 is realized by the CPU of the information server 22 expanding a specific program into memory, etc.

The data processing unit 42 is configured to include a memory unit 71, an image analysis unit 72, a voice analysis unit 73, a text analysis unit 74, a correlation analysis unit 75, and a comprehensive feature calculation unit 76.

The storage unit 71 stores data collected by the data collection unit 41 and data processed by the data processing unit 42 as necessary. For example, the storage unit 71 stores an event list generated by the text analysis unit 74. The event list will be described later.

The image analysis unit 72 performs image analysis such as facial feature point detection, facial expression estimation, and whole body skeletal point estimation on the video data collected by the data collection unit 41 to obtain video feature amounts. The obtained video feature amounts are supplied to the memory unit 71 and the overall feature amount calculation unit 76.

The voice analysis unit 73 performs voice analysis on the voice data collected by the data collection unit 41 to obtain voice features. The obtained voice features are supplied to the storage unit 71 and the overall feature calculation unit 76.

The text analysis unit 74 performs character recognition on the patient's nursing record data collected by the data collection unit 41, and generates an event list consisting of event labels indicating events of the patient. The generated event list is registered in the memory unit 71, etc.

The correlation analysis unit 75 analyzes other image features or clinical data that are correlated with the image features or clinical data for the time period specified by the instruction signal from the display terminal 23 obtained via the display screen generation unit 43. Information on the image features or clinical data that are analyzed to be correlated is supplied to the display screen generation unit 43.

The overall feature calculation unit 76 calculates an overall feature, which is a comprehensive feature, using default video features and clinical data specified by an instruction signal from the display terminal 23 obtained via the display screen generation unit 43 or set in advance. Note that audio features may also be used to calculate the overall feature. Details of the overall feature will be described later. The calculated overall feature is supplied to the display screen generation unit 43.

<2. Screen configuration example>
<Configuration of the review display screen>
FIG. 4 is a diagram showing an example of the configuration of a review display screen used for reviewing video data.

In FIG. 4, the review display screen 100 is configured to include a video display section 101, a video operation section 102, and a time-series data display operation section 103.

The video display unit 101 is located in the upper left of the review display screen 100. Video captured of the patient around the bed is displayed on the video display unit 101. In the case of FIG. 4, as shown by the LIVE icon displayed on the video display unit 101, real-time live video of the patient is displayed on the video display unit 101.

The video operation unit 102 is located at the bottom left of the review display screen 100, directly below the video display unit 101. The location of the video operation unit 102 is not limited to below the video display unit 101, but it may be above or next to it. However, it is preferable that the video operation unit 102 is located near the video display unit 101 so that operations can be performed while viewing the image displayed on the video display unit 101.

The video operation unit 102 is composed of an operation button display unit 111 and a playback seek bar display unit 112.

The operation button display section 111 displays various operation buttons that enable medical personnel to perform operations related to the playback of the video displayed on the video display section 101.

The playback seek bar display unit 112 displays a playback seek bar that medical personnel can operate to search for a scene to view from the video data of the video displayed on the video display unit 101 and to specify the playback position (playback start position).

The time-series data display operation unit 103 is located to the right of the video display unit 101. The time-series data display operation unit 103 displays time-series data of clinical data and video features from the past to the present.

The playback position of the video displayed on the video display unit 101 is displayed on the time series data of the time series data display operation unit 103 in conjunction with the playback seek bar of the video operation unit 102.

In addition, the review display screen 100 may also be configured with other components such as a data set reading section and a display section that works in conjunction with the patient information system 2 to read and display patient attributes (age, sex, patient ID number, etc.), underlying diseases, medical history, etc.

<Example of video operation unit configuration>
FIG. 5 is an enlarged view of the video operation unit 102 in FIG.

Operation button display section 111 has operation buttons arranged thereon, such as a (back to) previous event button, a 10 second rewind button, a rewind button, a play button, a fast forward button, a 15 second forward button, a (forward to) next event button, a pause button, and a (back to) live button.

Directly below the operation button display section 111, a playback seek bar display section 112 is displayed, which displays a playback seek bar 120. A graph showing the time series changes in the patient characteristics is superimposed on the playback seek bar 120. As described above, the playback seek bar 120 is a tool (display component) that medical staff operates to search for a scene to be viewed from the video data of the video displayed on the video display section 101 and to specify the playback start position.

The patient features consist of constituent datasets consisting of clinical data and image features, or comprehensive features calculated comprehensively from the constituent datasets. Figure 5 shows a case where the patient features are comprehensive features. Note that both the constituent datasets and the comprehensive features may be displayed as patient features.

Also displayed on the playback seek bar 120 are event labels 121a to 121c that indicate events related to the patient that were generated using nursing record data recorded in the patient information system 2.

Event label 121a is an event label that indicates a route management event such as an IV drip. Event label 121a is composed of the event title "Route Management", an event point 122a that indicates the start time of the event on the playback seek bar 120, and an event preview image 123a obtained from the video data around the event time.

Event label 121b is an event label that indicates a position change event. Event label 121b is composed of the event title "Position change", an event point 122b that indicates the start time of the event on the playback seek bar 120, and an event preview image 123b obtained from the video data around the event time.

Event label 121c is an event label that indicates a disturbing event. Disturbing refers to a restless movement. Event label 121b is composed of the event title "disturbing," an event point 122c that indicates the start time of the event on the playback seek bar 120, and an event preview image 123c obtained from the video data around the event time.

Note that, hereinafter, when there is no need to distinguish between the event labels 121a to 121c, the event points 122a to 122c, and the event preview images 123a to 123c, they will be referred to as the event label 121, the event point 122, and the event preview image 123, respectively.

A play icon, which is a right-facing triangle within a rectangular frame, is superimposed on the event preview image 123. When the play icon is clicked, the video data is played from the event point 122 in the video display unit 101. In other words, the event point 122 can also be considered the playback start point of the video data.

<Example of patient feature configuration>
FIG. 6 is a diagram showing an example of the configuration of patient features displayed on the playback seek bar in FIG.

Patient features consist of constituent datasets or overall features, as described above.

In the case of Figure 6, the constituent dataset is composed of clinical data A, clinical data B, patient feature A, and patient feature B.

In addition, the overall feature is calculated comprehensively using the constituent data sets, clinical data A, clinical data B, patient feature A, and patient feature B.

<Display example of patient features>
FIG. 7 is a diagram showing an example of display of patient features in the case of the configuration data set of FIG.

In FIG. 7, a graph showing the time series changes in the constituent data sets, clinical data B, clinical data F, patient characteristic C, and patient characteristic G, is displayed on the playback seek bar 120 of the playback seek bar display section 112.

Furthermore, on the playback seek bar 120, a playback point 151 indicating the point currently being played back and a preview image 152 at the playback point 151 are displayed. The preview image 152 is displayed by selecting the playback point 151.

<Display example of patient features>
FIG. 8 is a diagram showing an example of display of patient features in the case of the comprehensive features in FIG.

In FIG. 8, a graph showing the time series change in the overall feature calculated comprehensively from the constituent data sets, clinical data B, clinical data F, patient feature C, and patient feature G, is displayed on the playback seek bar 120 of the playback seek bar display section 112.

As in FIG. 7, a playback point 151 indicating the point currently being played back and a preview image 152 at the playback point 151 are displayed on the playback seek bar 120. The preview image 152 is displayed by selecting the playback point 151.

<Examples of comprehensive features and constituent datasets>
FIG. 9 is a diagram showing an example of the comprehensive feature amount and the constituent data sets.

The overall features and constituent data sets can be selected according to the type of observation to be made, the disease of the target patient, and the doctor operating the retrospective display system 1.

In Figure 9, an example of a combination of comprehensive features and constituent data sets is shown. In Figure 9, comprehensive features are shown as solid rectangles. Clinical data are shown as hatched rectangles. Video features are shown as dashed rectangles.

The periodic fluctuation of the shoulders, which is an overall feature, is a movement that can occur when shoulder breathing, which is a type of labored breathing.

The periodic fluctuation of the shoulders can be calculated from the respiratory rate (clinical data), the amount of vertical shoulder movement, and the periodicity of the vertical shoulder movement (image feature). The amount of vertical shoulder movement is composed of the amount of vertical shoulder movement of the right shoulder and the amount of vertical shoulder movement of the left shoulder.

The overall feature of suspected agitation can be calculated from blood pH (clinical data), the amount of whole-body movement, the amount of change in gaze, and the volume of the voice (visual features). The amount of whole-body movement is composed of the amount of head movement, shoulder movement, arm movement, and leg movement.

The total feature, heart rate excluding intervention, is the heart rate excluding abnormal heart rate values during intervention.

Heart rate excluding intervention times can be calculated from the heart rate (clinical data) and the number of interventions (video feature).

The comprehensive feature, Doctor A's original, is the parameter set that the veteran Doctor A focuses on.

The Doctor A original can be calculated from clinical data X1, clinical data X2, and image feature X. Image feature X is composed of image feature X1 and image feature X2.

The overall feature for sepsis patients is the parameter set that should be focused on in the case (in this case, sepsis).

For sepsis patients, it can be calculated from clinical data Y1, clinical data Y2, and image feature Y. Image feature Y is composed of image feature Y1 and image feature Y2.

In this way, a default configuration dataset prepared in advance may be used for the comprehensive feature (e.g., suspicion of a disturbance), or a configuration dataset customized (created) by medical personnel may be used (e.g., original by Doctor A).

The constituent datasets are composed of clinical data and/or image features. The overall feature is a feature (value) calculated based on the calculation basis set using the constituent datasets. However, the overall feature does not necessarily have to be a "value" calculated from the constituent datasets, and may simply be the "group name" of the constituent datasets in order to display the constituent datasets as patient features. In that case, the value of the overall feature is assumed to be empty. The overall feature may also be a severity score used in the ICU, etc.

An example of a severity score is the SOFA (Sepsis-related Organ Failure Assessment) score, which evaluates multiple organ failure based on vital signs and test data.

<Customization of comprehensive features>
FIG. 10 is a diagram showing an example of customization of the comprehensive feature amount.

The overall feature amount can be customized using simple programming methods or existing programming languages. In addition, the calculation basis for the default overall feature amount can also be displayed, allowing medical professionals to view and confirm the calculation basis for the overall feature amount.

In Figure 10, the basis for calculating the overall feature when the overall feature is the heart rate excluding intervention and the constituent data sets are the heart rate (clinical data) and the number of interventions (video feature) is shown on the right.

The inputs include heart rate and number of participants.

If the number of interventions is zero, the heart rate excluding interventions is set to the actual heart rate of the patient. If the number of interventions is not zero, the heart rate excluding interventions is set to None.

The output is set to the heart rate excluding intervention.

<Example of superimposing comprehensive features>
FIG. 11 is a diagram showing an example of a superimposed display of comprehensive feature amounts.

Time series data can be displayed on top of each other.

In Figure 11, the heart rate excluding the time of intervention, which is the overall feature, and the heart rate (hatched), which is the clinical data, are superimposed. During intervention, the heart rate excluding the time of intervention is not displayed, so only the heart rate (hatched) is displayed.

In this way, by superimposing the heart rate (hatch) and the heart rate excluding intervention, the heart rate with and without intervention can be displayed in an easy-to-understand manner.

<Setting process of patient features>
FIG. 12 is a flowchart illustrating the process of setting patient features.

In step S11, the display screen generation unit 43 selects the patient features to be displayed in response to the operation of an input unit (not shown) by a medical professional.

In step S12, the display screen generating unit 43 determines whether or not the selected patient feature has already been registered in the existing list in the storage unit 71. If it is determined in step S12 that the selected patient feature has not already been registered in the existing list, the process proceeds to step S13.

In step S13, the display screen generation unit 43 creates new patient features. For example, as the patient features, a constituent data set and an overall feature that make up the patient features, and the calculation basis for the overall feature are created.

In step S14, the display screen generator 43 registers the created patient features in an existing list. Then, the process proceeds to step S15.

If it is determined in step S12 that the selected patient feature is already registered in the existing list, steps S13 and S14 are skipped and processing proceeds to step S15.

In step S15, the display screen generation unit 43 reads the existing list from the storage unit 71.

In step S16, the display screen generation unit 43 selects the patient features selected in step S11 from the existing list that was loaded.

In step S17, it is determined whether display of the constituent data set has been selected as the display method for the selected patient feature. If it is determined in step S17 that display of the constituent data set has been selected, the process proceeds to step S18.

In step S18, the display screen generation unit 43 displays the configuration data set as the patient features.

If it is determined in step S17 that display of the configuration data set has not been selected, processing proceeds to step S19.

In step S19, the display screen generation unit 43 displays the overall feature as the patient feature.

After step S18 or S19, the patient feature setting process ends.

<Event label generation process>
FIG. 13 is a flowchart illustrating the process of generating an event label.

As described above with reference to Figure 5, the event label is linked to the nursing record data in the patient information system 2, and the text information entered by the nurse or medical staff is recognized, generated, and displayed on the playback seek bar 120.

Figure 13 shows the process of generating an event label.

In step S31, the data collection unit 41 imports nursing record data from the patient information system 2.

In step S32, the text analysis unit 74 performs character recognition using the nursing record data, and generates an event label 121 based on the recognized characters. For example, if there is nursing record data stating "At 1:32, XX behavior occurred. There is a possibility of agitation," characters such as "1:32" and "agitation" are recognized.

In step S33, the text analysis unit 74 registers the generated event label 121 in the event list of the storage unit 71.

In step S34, the display screen generating unit 43 displays the event label 121 registered in the event list in the storage unit 71 on the playback seek bar 120.

Through the above process, event labels indicating important scenes are generated without changing the workflow of doctors and nurses on-site.

As an alternative to automatic generation, although this would impose some burden on the field, event labels could be generated by medical staff entering nursing records into a terminal.

Alternatively, the time on the event label may be recorded by pressing a button such as a nurse call button placed at the bedside. In this case, the event label does not necessarily need to have a title, and only the time the event occurred may be recorded.

<Example of threshold setting>
FIG. 14 is a diagram showing an example of threshold setting in the time-series data display operation unit 103. As shown in FIG.

The time series data display operation unit 103 shown on the right side of A in Figure 14 displays the time series data X.

On the time series data X, an adjustment bar 181 is displayed at a position indicating the threshold value 100. On the time series data X, the time period where the value exceeds the adjustment bar 181, i.e., the time period 191a where the value is higher than the threshold value 100, is shown in a different color.

This time period 191a is reflected on the playback seek bar 120 of the video operation unit 102 as time period 192a during which the time series data X is greater than the threshold value 100, as shown on the left side of A in FIG. 14.

At that time, an event preview image 193a is displayed in association with the time period 192a.

A play icon, which is a right-facing triangle within a rectangular frame, is superimposed on the event preview image 193a. When the play icon is clicked, the video data in the video display section 101 is played from the beginning of the time period 192a.

This makes it easy to see from the video what is happening during the time period when the time series data X is higher than the threshold, i.e., during the time period when the time series data X is not normal.

In addition, by dragging and dropping the adjustment bar 181 up and down, medical personnel can set any threshold value for the time series data X.

The right side of FIG. 14B shows a time-series data display operation unit 103 that displays the time-series data X when a medical professional moves the adjustment bar 181, which is at the position indicating the threshold value 100 shown in FIG. 14A, to a position indicating the threshold value 80 by dragging and dropping it down.

In the case of FIG. 14B, on the time series data X, time periods 191a to 191d whose values exceed the adjustment bar 181, i.e., whose values are higher than the threshold value 80, are displayed in different colors. In this case, by lowering the threshold value compared to the case of FIG. 14A, more time periods are picked up (displayed in different colors).

These time periods 191a to 191d are reflected on the playback seek bar 120 of the video operation unit 102 as time periods 192a to 192d whose values are higher than the threshold value 80, as shown on the left side of FIG. 14B.

At that time, preview images 193a to 193d are displayed in association with time periods 192a to 192d, respectively.

Similar to event preview image 193a, the play icon is also superimposed on event preview images 193b to 193d.

By clicking on each of the play icons superimposed on the event preview images 193b to 193d, the video data in the video display section 101 can be played from the beginning of each of the time periods 192b to 193d, in the same way as in the case of the event preview image 193a.

In FIG. 14, the time periods when the time series data X exceeds the threshold are displayed in different colors, but depending on the threshold that is set, the time periods when the time series data X falls below the threshold may be displayed in different colors. The method of displaying them differently is not limited to different colors, and may be in a different format.

The threshold value was set by dragging and dropping the adjustment bar 181 up and down, but a text box can also be displayed and a numerical value can be entered directly.

<Example of playback position display>
FIG. 15 is a diagram showing an example of a display of the playback position of a video currently being played back.

On the left side of FIG. 15, the playback position 200 of the video currently being played is displayed on the playback seek bar 120 of the playback seek bar display section 112 of the video operation section 102.

The playback position corresponding to playback position 200 displayed on the video operation unit 102 is also displayed on each piece of time series data on the time series data display operation unit 103, as shown on the right side of Figure 15.

For example, playback positions corresponding to playback position 200 are displayed as playback positions 201a to 201c on the time-series data of clinical data A to C.

Similarly, playback positions corresponding to playback position 200 are displayed as playback positions 211a to 211c on the time-series data of video features A to C.

By displaying the playback position displayed on the video operation unit 102 on each piece of time series data on the time series data display operation unit 103, it is easy to check the relationship between changes in the time series data and the patient's condition in the video.

Displaying correlated data
FIG. 16 is a diagram showing an example of displaying correlated data.

FIG. 16 shows the time series data display operation unit 103 on which the time series data of clinical data A to C and the time series data of image features A to C are displayed.

When an arbitrary time period is selected in the time series data of clinical data A as shown in box 251 and a correlation evaluation is performed, a small screen 252 is displayed showing other data candidates with positive or negative correlation from among the clinical data and image features as shown by arrow P1.

The small screen 252 shows other data candidates, namely clinical data K and image feature N, which are positively correlated, and image feature J, which is negatively correlated.

In addition, the small screen 252 also shows the time difference with clinical data A and the degree of correlation coefficient for each of the other data candidates.

In other words, as other data candidates, not only the correlation of waveforms in the same time period is evaluated, but also the correlation (cross-correlation) of other time periods when shifted by Δt.

For example, when clinical data K is selected on the small screen 252, a cross-correlation display screen 253 with clinical data A is displayed, as shown by arrow P2.

The cross-correlation display screen 253 shows a graph of the correlation coefficient between clinical data A and clinical data K shifted in time by Δt.

As described above, it is possible to search for other time series data that are correlated with the time series data for the time period selected in the time series data display operation unit 103.

The usefulness of this function is that by understanding the mutual relationships between clinical data and video features, it can lead to the explicitization of tacit knowledge in the field, which has traditionally relied on the experience of doctors and nurses. Furthermore, it is thought that this relationship could be used to develop technology for predicting sudden changes using patient videos and clinical data.

<3. Other>
<Effects of this technology>
As described above, in the present technology, an overall feature is calculated based on a dataset consisting of video features, which are features of video data obtained by imaging a patient, and patient data, which is at least one of the patient's clinical data, and a first display screen is generated that displays the calculated overall feature in chronological order together with a display component that indicates the playback position of the video data.

In other words, a display screen is generated that shows changes over time along with a display component that indicates the playback position of the video data, allowing users to efficiently review the patient's condition.

This technology makes it possible to grasp the condition of patients efficiently, making it possible to check the condition of multiple patients efficiently with a small number of staff at night when there are fewer staff on duty, in hospitals with a shortage of medical staff, and in support centers for remote ICUs.

This will contribute to resolving the shortage of doctors, reforming work styles, and advancing remote medical care.

When medical staff hand over information between day and night shifts, they can easily communicate the patient's condition to others using video.

Medical professionals can load feature datasets that are prepared in advance or that they have created themselves, allowing them to focus on the data that interests them. It also allows young doctors to learn the perspectives of veteran doctors.

The discovery of correlations between clinical data and video features will lead to the conversion of tacit knowledge into explicit knowledge. It will also lead to the development of technology to predict sudden changes using video.

<Example of computer configuration>
The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

FIG. 17 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program.

CPU (Central Processing Unit) 301, ROM (Read Only Memory) 302, and RAM (Random Access Memory) 303 are interconnected by bus 304.

Further connected to the bus 304 is an input/output interface 305. Connected to the input/output interface 305 are an input unit 306 consisting of a keyboard, mouse, etc., and an output unit 307 consisting of a display, speakers, etc. Also connected to the input/output interface 305 are a storage unit 308 consisting of a hard disk or non-volatile memory, a communication unit 309 consisting of a network interface, etc., and a drive 310 that drives removable media 311.

In a computer configured as described above, the CPU 301 performs the above-mentioned series of processes, for example by loading a program stored in the storage unit 308 into the RAM 303 via the input/output interface 305 and the bus 304 and executing the program.

The programs executed by the CPU 301 are provided, for example, by being recorded on removable media 311, or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and are installed in the storage unit 308.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or a program in which processing is performed in parallel or at the required timing, such as when called.

In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiment of this technology is not limited to the above-mentioned embodiment, and various modifications are possible without departing from the gist of this technology.

For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

In addition, each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices.

Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices.

<Examples of configuration combinations>
The present technology can also be configured as follows.
(1)
a data processing unit that calculates a comprehensive feature based on a data set including image feature amounts that are feature amounts of image data obtained by imaging a patient and patient data that is at least one of clinical data of the patient;
a display screen generating unit that generates a first display screen that displays the calculated comprehensive feature amount in chronological order together with a display component that indicates a playback position of the video data.
(2)
The data processing unit generates an event label indicating an event related to the patient based on the nursing record data of the patient;
The information processing device according to (1), wherein the display screen generating unit superimposes the event label together with the display component on the first display screen.
(3)
The information processing device according to (2), wherein the event label includes a point indicating a start time of the event and a preview image of the event.
(4)
The information processing device according to (3), wherein the event label includes a title of the event.
(5)
The information processing device according to any one of (1) to (4), wherein the display screen generation unit displays the patient data together with the display part on the first display screen.
(6)
The information processing device according to (5), wherein the display screen generating unit displays the patient data on the first display screen by superimposing the patient data on the comprehensive feature amount.
(7)
The information processing device according to any one of (1) to (6), wherein the display screen generation unit displays, on the first display screen, a point corresponding to a playback position of the video data together with the display part.
(8)
The information processing device according to any one of (1) to (7), wherein the display screen generation unit generates the first display screen in a periphery of a second display screen that displays an image corresponding to the video data being played back.
(9)
The information processing device according to any one of (1) to (8), wherein the display screen generation unit generates a third display screen that displays the patient data in chronological order.
(10)
The data processing unit extracts correlation data, which is other patient data that is correlated with the patient data for the time period selected on the third display screen,
The information processing device according to (9), wherein the display screen generation unit generates a fourth display screen that displays the extracted correlation data.
(11)
The information processing device according to (10), wherein the correlation data is other patient data in a time zone different from the time zone selected on the third display screen.
(12)
A threshold value for the patient data can be set on the third display screen,
The information processing device according to (9), wherein the display screen generating unit displays, on the third display screen, a time period in which the patient data is equal to or greater than the threshold and a time period in which the patient data is less than the threshold in different formats.
(13)
The information processing device according to (12), wherein the display screen generating unit displays, on the first display screen, a time period in which the patient data is equal to or greater than the threshold and a time period in which the patient data is less than the threshold in different formats.
(14)
the display screen generation unit displays, on the first display screen, a point corresponding to a playback position of the video data together with the display part;
The information processing device according to (9), wherein a point corresponding to a playback position of the video data is displayed on the third display screen on the patient data.
(15)
The information processing device according to any one of (1) to (14), wherein the data set used for calculating the comprehensive feature amount is set by a user.
(16)
The information processing device according to (14), wherein a basis for calculating the comprehensive feature amount is set by a user.
(17)
The information processing device according to any one of (1) to (16), wherein the display part is a playback designation tool that is operated by a user to search for a scene to be viewed from the video data of the displayed video and to designate a playback position.
(18)
An information processing device,
Calculating a comprehensive feature based on a data set including image feature amounts, which are feature amounts of image data obtained by imaging a patient, and patient data, which is at least one of clinical data of the patient;
generating a first display screen that displays the calculated comprehensive feature amount in chronological order together with a display component that indicates a playback position of the video data.
(19)
a data processing unit that calculates a comprehensive feature based on a data set including image feature amounts that are feature amounts of image data obtained by imaging a patient and patient data that is at least one of clinical data of the patient;
a display screen generating unit that generates a first display screen that displays the calculated comprehensive feature amount in time series together with a display component that indicates a playback position of the video data,
The programs that make a computer function.
(20)
A sensor for capturing an image of a patient to obtain video data;
a data processing unit that calculates a comprehensive feature based on a data set including image feature amounts that are feature amounts of the image data and patient data that is at least one of clinical data of the patient;
an information processing device including a display screen generation unit that generates a first display screen that displays the calculated comprehensive feature amount in time series together with a display component that indicates a playback position of the video data;
and a terminal that controls the display of the first display screen.

1. Retrospective display system, 2. Patient information system, 11. Sensing block, 12. Data processing block, 13. Display block, 21. Bedside terminal, 22. Information server, 23. Display terminal, 41. Data collection unit, 42. Data processing unit, 43. Display screen generation unit, 51. Display control unit, 52. Display unit, 53. Operation input unit, 71. Memory unit, 72. Image analysis unit, 73. Audio analysis unit, 74. Text analysis unit, 75. Correlation analysis unit, 76. Overall feature calculation unit, 100. Retrospective display screen, 101. Video display unit, 102. Video operation unit, 103. Time series data display operation unit, 111. Operation button display unit, 112. Playback seek bar display unit, 120. Playback seek bar

Claims

a data processing unit that calculates a comprehensive feature based on a data set including image feature amounts that are feature amounts of image data obtained by imaging a patient and patient data that is at least one of clinical data of the patient;
a display screen generating unit that generates a first display screen that displays the calculated comprehensive feature amount in chronological order together with a display component that indicates a playback position of the video data.
The data processing unit generates an event label indicating an event related to the patient based on the nursing record data of the patient;
The information processing apparatus according to claim 1 , wherein the display screen generating unit superimposes the event label together with the display component on the first display screen.
The information processing device according to claim 2 , wherein the event label includes a point indicating a start time of the event and a preview image of the event.
The information processing device according to claim 3 , wherein the event label includes a title of the event.
The information processing apparatus according to claim 1 , wherein the display screen generating unit displays the patient data together with the display parts on the first display screen.
The information processing apparatus according to claim 5 , wherein the display screen generating unit displays the patient data on the first display screen in a manner superimposed on the comprehensive feature amount.
The information processing device according to claim 1 , wherein the display screen generating section displays, on the first display screen, a point corresponding to a playback position of the video data together with the display part.
The information processing device according to claim 1 , wherein the display screen generating section generates the first display screen in a periphery of a second display screen that displays an image corresponding to the video data being reproduced.
The information processing apparatus according to claim 1 , wherein the display screen generating unit generates a third display screen that displays the patient data in chronological order.
The data processing unit extracts correlation data, which is other patient data that is correlated with the patient data for the time period selected on the third display screen,
The information processing apparatus according to claim 9 , wherein the display screen generating unit generates a fourth display screen that displays the extracted correlation data.
The information processing apparatus according to claim 10 , wherein the correlation data is other patient data in a time period different from the time period selected on the third display screen.
A threshold value for the patient data can be set on the third display screen,
The information processing device according to claim 9 , wherein the display screen generating unit displays, on the third display screen, a time period in which the patient data is equal to or greater than the threshold value and a time period in which the patient data is less than the threshold value in different formats.
The information processing device according to claim 12 , wherein the display screen generating unit displays, on the first display screen, a time period in which the patient data is equal to or greater than the threshold value and a time period in which the patient data is less than the threshold value in different formats.
the display screen generation unit displays, on the first display screen, a point corresponding to a playback position of the video data together with the display part;
The information processing apparatus according to claim 9 , wherein a point corresponding to a playback position of the video data is displayed on the third display screen on the patient data.
The information processing apparatus according to claim 1 , wherein the data set used for calculating the comprehensive feature amount can be set by a user.
The information processing apparatus according to claim 15 , wherein a basis for calculating the comprehensive feature amount can be set by a user.
The information processing apparatus according to claim 1 , wherein the display part is a playback designation bar that is operated by a user to search for a scene to be viewed from the video data of the displayed video and to designate a playback position.
An information processing device,
Calculating a comprehensive feature based on a data set including image feature amounts, which are feature amounts of image data obtained by imaging a patient, and patient data, which is at least one of clinical data of the patient;
generating a first display screen that displays the calculated comprehensive feature amount in chronological order together with a display component that indicates a playback position of the video data.
a data processing unit that calculates a comprehensive feature based on a data set including image feature amounts that are feature amounts of image data obtained by imaging a patient and patient data that is at least one of clinical data of the patient;
a display screen generating unit that generates a first display screen that displays the calculated comprehensive feature amount in time series together with a display component that indicates a playback position of the video data,
The programs that make a computer function.
A sensor for capturing an image of a patient to obtain video data;
a data processing unit that calculates a comprehensive feature based on a data set including image feature amounts that are feature amounts of the image data and patient data that is at least one of clinical data of the patient;
an information processing device including a display screen generation unit that generates a first display screen that displays the calculated comprehensive feature amount in time series together with a display component that indicates a playback position of the video data;
and a terminal that controls the display of the first display screen.