JP7570228B2 - Machine learning execution program, dry eye test program, machine learning execution device, and dry eye test device - Google Patents

Description

The present invention relates to a machine learning execution program, a dry eye test program, a machine learning execution device, and a dry eye test device.

Currently, especially in developed countries, electronic devices equipped with displays, such as personal computers, smartphones, and tablets, are becoming widespread, and as a result, the number of people who are aware of eye problems is rapidly increasing. Furthermore, due to the increase in online services via the Internet and the increase in companies promoting teleworking, the number of people who are aware of eye problems is expected to increase further.

One example of an eye disorder that is relatively frequently caused by such lifestyle habits is dry eye, which is accompanied by an instability of the tear film. For example, to examine the tear film breakup time, which is an index of tear film stability, an ophthalmologist must instill a fluorescein dye reagent into the eye and, while observing the eye using a slit lamp, measure the time from immediately after blinking to the breakdown of the tear film. Alternatively, the patient must be examined using the ophthalmic diagnostic support device disclosed in Patent Document 1.

This ophthalmological diagnosis support device includes a case data storage unit, a machine learning unit, a patient data acquisition unit, a comparison judgment unit, and a result display unit. The case data storage unit stores a plurality of case basic image data consisting of a combination of ophthalmological diagnostic image data and diagnosis results. The machine learning unit extracts characteristic image elements and classifies the case basic image data. The patient data acquisition unit acquires patient ophthalmological diagnostic image data of a patient. The comparison judgment unit compares the patient ophthalmological diagnostic image data with the case basic image data and performs a similarity judgment between the data. The result display unit displays the result of the similarity judgment.

JP 2020-036835 A

However, the above-mentioned ophthalmic diagnostic support device requires obtaining ophthalmic diagnostic data using testing equipment used in medical institutions, and therefore cannot easily test for the presence or severity of dry eye in people who are aware of eye problems.

The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a machine learning execution program, a dry eye test program, a machine learning execution device, and a dry eye test device that can predict the results of a test related to dry eye symptoms without actually conducting the test.

One aspect of the present invention is a machine learning execution program that causes a computer to execute a teacher data acquisition function that acquires teacher data in which the answer is at least one of the following: training image data showing training images depicting the eyes of a training subject and training eyelid open data showing the maximum eyelid open time, which is the time the training subject was able to continuously open the eye in which the training image was captured; training test image data showing an image depicting the eyes of the training subject when a test for dry eye symptoms is performed on the eyes of the training subject, and training test result data showing the results of the test for dry eye symptoms; and a machine learning execution function that inputs the teacher data into a machine learning program and trains the machine learning program.

One aspect of the present invention is the machine learning execution program described above, in which the training data acquisition function acquires the training data based on the training image data showing a training tear meniscus image obtained by cutting out an area depicting the tear meniscus of the training subject from the training image.

One aspect of the present invention is the machine learning execution program described above, in which the training data acquisition function acquires the training data based on training image data showing a training corneal image extracted from the training image to depict an area depicting illumination reflected on the cornea of the training subject.

One aspect of the present invention is the machine learning execution program described above, in which the teacher data acquisition function acquires the teacher data with at least one of the learning test image data showing an image depicting the eye of the learning subject when a test is performed to examine the tear film breakup time of the eye of the learning subject, and the learning test result data showing the results of the test to examine the tear film breakup time of the eye of the learning subject as an answer.

One aspect of the present invention concerns a data acquisition function that acquires inference image data showing an inference image depicting the eye of an inference subject and inference eyelid opening data showing the maximum eyelid opening time, which is the time the inference subject can continuously open the eye in which the inference image was captured, and learning image data showing a learning image depicting the eye of a learning subject and learning eyelid opening data showing the maximum eyelid opening time, which is the time the learning subject can continuously open the eye in which the learning image was captured, and the learning image data showing the maximum eyelid opening time, which is the time the learning subject can continuously open the eye in which the learning image was captured, when an examination for dry eye symptoms is performed on the eye of the learning subject. A dry eye testing program that implements in a computer a symptom estimation function that inputs the inference image data and the inference eyelid open data into a machine learning program that has been trained using teacher data that uses as an answer at least one of learning test image data showing an image depicting the eye of a learning subject and learning test result data showing the results of a test regarding dry eye symptoms, causes the machine learning program to estimate the dry eye symptoms present in the eye of the inference subject, and outputs symptom data showing the dry eye symptoms present in the eye of the inference subject.

One aspect of the present invention is the dry eye test program described above, in which the data acquisition function acquires, as the image data for inference, tear meniscus image data for inference showing an inference tear meniscus image obtained by cutting out an area depicting the tear meniscus of the inference subject's eye from the inference image depicting the tear meniscus of the inference subject's eye, and the symptom estimation function inputs the tear meniscus image data for inference to the machine learning program trained using the teacher data, in which the symptom estimation function uses as the training image data learning tear meniscus image data showing a learning tear meniscus image obtained by cutting out an area depicting the tear meniscus of the learning subject's eye from the training image depicting the tear meniscus of the learning subject's eye, and causes the machine learning program to estimate the symptoms of dry eye appearing in the eye of the inference subject.

One aspect of the present invention is the dry eye test program described above, in which the data acquisition function acquires, as the inference image data, the inference illumination image data showing an inference illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the inference subject from the inference image depicting the illumination reflected on the cornea of the inference subject, and the symptom estimation function inputs the inference illumination image data to the machine learning program trained using the teacher data, and has the machine learning program estimate the symptoms of dry eye appearing in the eyes of the inference subject, in which the data acquisition function acquires, as the inference image data, the inference illumination image data showing an inference illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the learning subject from the learning image depicting the illumination reflected on the cornea of the learning subject, and the symptom estimation function inputs the inference illumination image data to the machine learning program trained using the teacher data, and has the machine learning program estimate the symptoms of dry eye appearing in the eyes of the inference subject.

One aspect of the present invention is a machine learning execution device that has a training data acquisition unit that acquires training data in which training image data showing training images depicting the eyes of a training subject and training eyelid open data showing the maximum eyelid open time, which is the time the training subject was able to continuously open the eye in which the training image was captured, are questions, and the training test image data showing an image depicting the eyes of the training subject when a test for dry eye symptoms is performed on the eyes of the training subject and training test result data showing the results of the test for dry eye symptoms are answers, and a machine learning execution unit that inputs the training data into a machine learning program and trains the machine learning program.

One aspect of the present invention concerns a data acquisition unit that acquires inference image data showing an inference image depicting the eye of an inference subject and inference eyelid opening data showing the maximum eyelid opening time, which is the time the inference subject can continuously open the eye in which the inference image is captured, and learning image data showing a learning image depicting the eye of a learning subject and learning eyelid opening data showing the maximum eyelid opening time, which is the time the learning subject can continuously open the eye in which the learning image is captured, when an examination for dry eye symptoms is performed on the eye of the learning subject and a symptom estimation unit that inputs the inference image data and the inference eyelid open data into a machine learning program that has been trained using teacher data that uses as an answer at least one of learning test image data showing an image depicting the eye of the learning subject and learning test result data showing the results of a test regarding dry eye symptoms, causes the machine learning program to estimate the dry eye symptoms present in the eye of the inference subject, and causes the machine learning program to output symptom data showing the dry eye symptoms present in the eye of the inference subject.

One aspect of the present invention is a machine learning execution program that causes a computer to execute a teacher data acquisition function that acquires teacher data having as an answer at least one of learning test image data showing an image depicting the eye of the learning subject when a test for dry eye symptoms is performed on the eye of the learning subject and learning test result data showing the results of the test for dry eye symptoms, and a machine learning execution function that inputs the teacher data into a machine learning program and trains the machine learning program, with at least one of learning tear meniscus image data showing a learning tear meniscus image obtained by cutting out an area depicting the tear meniscus of the learning subject from a learning image depicting the eye of the learning subject and learning illumination image data showing a learning illumination image obtained by cutting out an area depicting illumination reflected on the cornea of the learning subject from the learning image.

One aspect of the present invention is a data acquisition function for acquiring at least one of inference tear meniscus image data showing an inference tear meniscus image obtained by cutting out an area depicting the tear meniscus of the inference subject from an inference image depicting the eye of the inference subject, and inference illumination image data showing an inference illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the inference subject from the inference image, and learning tear meniscus image data showing a learning tear meniscus image obtained by cutting out an area depicting the tear meniscus of the learning subject from a learning image depicting the eye of the learning subject, and learning illumination image data showing a learning illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the learning subject from the learning image. and a symptom estimation function that inputs at least one of the tear meniscus image data for inference and the illumination image data for inference into a machine learning program trained using teacher data that uses as an answer at least one of training test image data showing an image of the training subject's eye when a test for dry eye symptoms is performed on the training subject's eye and training test result data showing the results of the test for dry eye symptoms, causes the machine learning program to estimate the dry eye symptoms appearing in the training subject's eye, and causes the machine learning program to output symptom data showing the dry eye symptoms appearing in the training subject's eye.

One aspect of the present invention is a machine learning execution device that has a teacher data acquisition unit that acquires teacher data in which at least one of learning tear meniscus image data showing a learning tear meniscus image obtained by cutting out an area depicting the tear meniscus of the learning subject from a learning image depicting the eye of the learning subject and learning illumination image data showing a learning illumination image obtained by cutting out an area depicting illumination reflected on the cornea of the learning subject from the learning image is an answer, and at least one of learning test image data showing an image depicting the eye of the learning subject when a test for dry eye symptoms is performed on the eye of the learning subject and learning test result data showing the results of the test for dry eye symptoms, and a machine learning execution unit that inputs the teacher data into a machine learning program and trains the machine learning program.

One aspect of the present invention includes a data acquisition unit that acquires at least one of inference tear meniscus image data showing an inference tear meniscus image obtained by cutting out an area depicting the tear meniscus of the inference subject from an inference image depicting the eye of the inference subject, and inference illumination image data showing an inference illumination image obtained by cutting out an area depicting illumination reflected on the cornea of the inference subject from the inference image, and a data acquisition unit that acquires at least one of inference tear meniscus image data showing an inference tear meniscus image obtained by cutting out an area depicting illumination reflected on the cornea of the inference subject from the inference image, and a learning illumination image data showing learning tear meniscus image data showing an learning tear meniscus image obtained by cutting out an area depicting the tear meniscus of the learning subject from a learning image depicting the eye of the learning subject, and a learning illumination image showing an learning illumination image obtained by cutting out an area depicting illumination reflected on the cornea of the learning subject from the learning image. and a symptom estimation unit that inputs at least one of the tear meniscus image data for inference and the illumination image data for inference into a machine learning program that has been trained using teacher data that takes as its answer at least one of learning test image data showing an image of the eye of the learning subject when a test for dry eye symptoms is performed on the eye of the learning subject and learning test result data showing the results of the test for dry eye symptoms, causes the machine learning program to estimate the dry eye symptoms appearing in the eye of the learning subject, and causes the machine learning program to output symptom data showing the dry eye symptoms appearing in the eye of the learning subject.

The present invention makes it possible to predict the results of a test for dry eye symptoms without actually conducting the test.

FIG. 2 is a diagram illustrating an example of a hardware configuration of a machine learning execution device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a software configuration of a machine learning execution program according to the first embodiment. 4 is a flowchart illustrating an example of processing executed by a machine learning execution program according to the first embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of a dry eye test apparatus according to a first embodiment. FIG. 4 is a diagram showing an example of a software configuration of a dry eye test program according to the first embodiment. 5 is a flowchart showing an example of processing executed by a dry eye test program according to the first embodiment. FIG. 11 is a diagram illustrating an example of a hardware configuration of a machine learning execution device according to a second embodiment. A figure showing an example of the software configuration of a machine learning execution program related to the second embodiment. FIG. 11 is a diagram showing an example of a learning tear meniscus image obtained by cutting out a region depicting the tear meniscus from a learning image according to the second embodiment. 13 is a diagram showing an example of a learning illumination image obtained by cutting out a region depicting illumination reflected on the cornea of a learning subject from a learning image according to the second embodiment. FIG. 11 is a flowchart illustrating an example of processing executed by a machine learning execution program according to the second embodiment. FIG. 11 is a diagram illustrating an example of a hardware configuration of a dry eye test apparatus according to a second embodiment. FIG. 11 is a diagram showing an example of a software configuration of a dry eye test program according to the second embodiment. 13 is a flowchart showing an example of processing executed by a dry eye test program according to the second embodiment. A figure showing an example of a portion of a training image depicting the eye of a training subject that is given priority when a machine learning program according to the third embodiment predicts the results of a test to examine tear film breakup time.

[First embodiment]
Specific examples of the machine learning execution program, the machine learning execution device, and the machine learning execution method according to the first embodiment will be described with reference to FIGS. 1 to 3.

Figure 1 is a diagram showing an example of the hardware configuration of a machine learning execution device according to the first embodiment. The machine learning execution device 10a shown in Figure 1 is a device that causes the machine learning device 700a to execute machine learning in the learning phase of the machine learning device 700a, which will be described later. As shown in Figure 1, the machine learning execution device 10a also includes a processor 11a, a main memory device 12a, a communication interface 13a, an auxiliary memory device 14a, an input/output device 15a, and a bus 16a.

The processor 11a is, for example, a CPU (Central Processing Unit), and reads and executes the machine learning execution program 100a described below to realize each function of the machine learning execution program 100a. The processor 11a may also read and execute programs other than the machine learning execution program 100a to realize functions necessary to realize each function of the machine learning execution program 100a.

The main memory device 12a is, for example, a RAM (Random Access Memory), and pre-stores the machine learning execution program 100a and other programs that are read and executed by the processor 11a.

The communication interface 13a is an interface circuit for communicating with the machine learning device 700a and other devices via the network NW shown in FIG. 1. The network NW is, for example, a LAN (Local Area Network) or an intranet.

The auxiliary storage device 14a is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or a read only memory (ROM).

The input/output device 15a is, for example, an input/output port (Input/Output Port). For example, the keyboard 811a, mouse 812a, and display 910a shown in FIG. 1 are connected to the input/output device 15a. The keyboard 811a and mouse 812a are used, for example, to input data required to operate the machine learning execution device 10a. The display 910a displays, for example, a graphical user interface (GUI) of the machine learning execution device 10a.

The bus 16a connects the processor 11a, the main memory device 12a, the communication interface 13a, the auxiliary memory device 14a, and the input/output device 15a so that they can send and receive data to each other.

Figure 2 is a diagram showing an example of the software configuration of a machine learning execution program according to the first embodiment. The machine learning execution device 10a reads and executes the machine learning execution program 100a using the processor 11a, and realizes the teacher data acquisition function 101a and the machine learning execution function 102a shown in Figure 2.

The teacher data acquisition function 101a acquires teacher data in which a piece of learning image data and learning open eyelid data are used as a question, and at least one piece of learning test image data and learning test result data is used as an answer.

The training image data constitutes part of the training data problem, and is data showing training images depicting the eyes of a training subject. The training images are captured, for example, using a camera mounted on a smartphone.

The training eyelid-open data constitutes part of the training data problem and indicates the maximum eyelid-open time, which is the time that the training subject was able to continuously open the eye from which the training image was taken.

For example, the learning eyelid opening data may indicate a maximum eyelid opening time calculated based on eyelid movement captured in a video that captures at least the period from immediately after the learning subject blinks and opens his eyes until the next blink.

Alternatively, the learning eyelid-open data may be based on the learning subject's self-reporting and may indicate a maximum eyelid-open time input using a user interface. The user interface may be displayed, for example, on the display 910a shown in FIG. 1. Alternatively, the user interface may be displayed on a touch panel display mounted on a smartphone subscribed to by the learning subject.

The learning test image data constitutes part of the answer of the teacher data, and is data showing a learning test image depicting the learning subject's eye when a test for dry eye symptoms is conducted on the learning subject's eye. An example of such a test is a test in which a fluorescein dye reagent is dropped into the eye and the eye is observed using a slit lamp to check the tear film breakup time. Therefore, for example, the learning test image data is data showing an image or video depicting an eye stained with fluorescein and observed using a slit lamp.

The learning test result data constitutes part of the answers to the teacher data, and is data showing the results of tests related to dry eye symptoms. Such test results include, for example, a numerical value between 0 and 1 that indicates the length of tear film breakup time based on a test in which a fluorescein dye is dropped into the eye and the eye is observed using a slit lamp to examine the tear film breakup time.

If this value is greater than or equal to 0 and less than 0.5, the tear film breakup time is 10 seconds or more, indicating that the study subject's eyes are normal and no examination by an ophthalmologist is necessary. If this value is greater than or equal to 0.5 and less than or equal to 1, the tear film breakup time is less than 10 seconds, indicating that the study subject's eyes are abnormal and an examination by an ophthalmologist is necessary.

For example, the teacher data acquisition function 101a acquires 1280 pieces of teacher data. In this case, the 1280 pieces of training image data represent 1280 training images acquired by performing four sets of a process in which each of the two eyes of eight training subjects is photographed 20 times using a camera mounted on a smartphone. In addition, in this case, the 1280 pieces of training eyelid opening data represent 1280 values each indicating a value representing the maximum eyelid opening time of the training subject.

In this case, the 1,280 pieces of learning test image data each represent a learning test image captured when an examination for dry eye symptoms was performed on the eye of the learning subject depicted in each of the 1,280 learning images described above. The 1,280 pieces of learning test result data each represent 1,280 numerical values between 0 and 1 indicating the results of the examination for dry eye symptoms performed when the 1,280 learning test images were captured.

In the following explanation, an example will be given in which the 1,280 pieces of learning test result data include 620 pieces of learning test result data showing a value between 0 and less than 0.5, indicating that the learning subject's eyes are normal, and 660 pieces of learning test result data showing a value between 0.5 and 1, indicating that the learning subject's eyes are abnormal.

The machine learning execution function 102a inputs training data into the machine learning program 750a implemented in the machine learning device 700a, and trains the machine learning program 750a. For example, the machine learning execution function 102a trains the machine learning program 750a, which includes a convolutional neural network (CNN), by backpropagation.

For example, the machine learning execution function 102a inputs 540 pieces of training data, including training test result data showing values between 0 and less than 0.5, indicating that the eyes of the training subject are normal, into the machine learning program 750a. These 540 pieces of training data are training data obtained from seven people selected from the eight people mentioned above.

Also, for example, the machine learning execution function 102a inputs 580 pieces of training data, including training test result data showing a numerical value between 0.5 and 1, indicating that the eyes of the training subject are abnormal, into the machine learning program 750a. These 580 pieces of training data are training data obtained from seven people selected from the eight people described above.

Then, the machine learning execution function 102a trains the machine learning program 750a using these 1,120 pieces of training data.

Also, for example, the machine learning execution function 102a may represent that the eyes of the training subject are normal, and 80 pieces of training data that were not used in the training of the machine learning program 750a may be input to the machine learning program 750a as test data. These 80 pieces of training data are training data obtained from one person who was not selected as one of the seven people mentioned above.

Also, for example, the machine learning execution function 102a may indicate that the eyes of the training subject are abnormal, and 80 pieces of training data that were not used in the training of the machine learning program 750a may be input to the machine learning program 750a as test data. These 80 pieces of training data are training data obtained from one person who was not selected as one of the seven people mentioned above.

As a result, the machine learning execution function 102a may evaluate the characteristics of the machine learning program 750a obtained by training the machine learning program 750a using the above-mentioned 1,120 pieces of teacher data.

The machine learning execution function 102a may use, for example, class activation mapping (CAM) when using test data to evaluate the characteristics of the machine learning program 750a. Class activation mapping is a technique that clarifies the part of the data input to a neural network that is the basis for the results output by the neural network. An example of class activation mapping is gradient-weighted class activation mapping. Gradient-weighted class activation mapping is a technique that uses the gradient of the classification score related to the features of the convolution executed by the convolutional neural network to identify areas of the image input to the convolutional neural network that have had a certain degree of influence on the classification.

Next, an example of processing executed by the machine learning execution program 100a according to the first embodiment will be described with reference to FIG. 3. FIG. 3 is a flowchart showing an example of processing executed by the machine learning execution program according to the first embodiment. The machine learning execution program 100a executes the processing shown in FIG. 3 at least once.

In step S11, the teacher data acquisition function 101a acquires teacher data in which the questions are learning image data and learning eyelid open data, and the answers are at least one of learning test image data and learning test result data.

In step S12, the machine learning execution function 102a inputs the training data into the machine learning program 750a and causes the machine learning program 750a to learn.

Next, specific examples of the dry eye test program, dry eye test device, and dry eye test method according to the first embodiment will be described with reference to Figures 4 to 6.

Figure 4 is a diagram showing an example of the hardware configuration of a dry eye test device according to the first embodiment. The dry eye test device 20a shown in Figure 4 is a device that estimates dry eye symptoms appearing in the eye of an inference subject using a machine learning device 700a in the inference phase of the machine learning device 700a that has been trained by a machine learning execution program 100a. As shown in Figure 5, the dry eye test device 20a includes a processor 21a, a main memory device 22a, a communication interface 23a, an auxiliary memory device 24a, an input/output device 25a, and a bus 26a.

The processor 21a is, for example, a CPU, and reads and executes the dry eye test program 200a described below to realize each function of the dry eye test program 200a. The processor 21a may also read and execute a program other than the dry eye test program 200a to realize functions necessary to realize each function of the dry eye test program 200a.

The main memory device 22a is, for example, a RAM, and prestores the dry eye test program 200a and other programs that are read and executed by the processor 21a.

The communication interface 23a is an interface circuit for communicating with the machine learning device 700a and other devices via the network NW shown in FIG. 5. The network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24a is, for example, a hard disk drive, a solid state drive, a flash memory, or a ROM.

The input/output device 25a is, for example, an input/output port. For example, the keyboard 821a, mouse 822a, and display 920a shown in FIG. 5 are connected to the input/output device 25a. The keyboard 821a and mouse 822a are used, for example, to input data required to operate the dry eye test device 20a. The display 920a displays, for example, a graphical user interface of the dry eye test device 20a.

The bus 26a connects the processor 21a, the main memory device 22a, the communication interface 23a, the auxiliary memory device 24a, and the input/output device 25a so that data can be sent and received among them.

Figure 5 is a diagram showing an example of the software configuration of a dry eye test program according to the first embodiment. The dry eye test device 20a reads and executes the dry eye test program 200a using the processor 21a, and realizes the data acquisition function 201a and the symptom estimation function 202a shown in Figure 5.

The data acquisition function 201a acquires image data for inference and open eyelid data for inference.

The image data for inference is data showing an image depicting the eye of the subject for inference. The image for inference is captured, for example, using a camera mounted on a smartphone.

The eyelid opening data for inference is data that indicates the maximum eyelid opening time, which is the time that the subject for inference was able to continuously open the eye from which the inference image was taken.

For example, the eyelid opening data for inference may indicate a maximum eyelid opening time calculated based on eyelid movement shown in a video that captures at least the period from immediately after the subject for inference blinks and opens his/her eyes until the next blink.

Alternatively, the inferred eyelid opening data may be based on the inferred subject's self-reporting and may indicate a maximum eyelid opening time entered using a user interface. The user interface may be displayed, for example, on the display 920a shown in FIG. 4. Alternatively, the user interface may be displayed on a touch panel display mounted on a smartphone subscribed to by the inferred subject.

The symptom estimation function 202a inputs the inference image data and inference eyelid open data to the machine learning program 750a that has been trained by the machine learning execution function 102a, and causes the machine learning program 750a to estimate the symptoms of dry eye appearing in the eye of the inference subject. For example, the symptom estimation function 202a inputs these two pieces of data to the machine learning program 750a, and causes it to estimate a numerical value representing the length of tear film breakup time in the eye of the inference subject.

The symptom estimation function 202a then causes the machine learning program 750a to output symptom data indicating the symptoms of dry eye appearing in the eye of the inference subject. For example, the symptom estimation function 202a causes the machine learning program 750a to output symptom data indicating a numerical value indicating the length of the tear film breakup time of the eye of the inference subject. In this case, the symptom data is, for example, indicated by the function itself and is used to display a numerical value indicating the length of the tear film breakup time of the eye of the inference subject on the display 920a.

Next, an example of processing executed by the dry eye test program 200a according to the first embodiment will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of processing executed by the dry eye test program according to the first embodiment.

In step S21, the data acquisition function 201a acquires image data for inference and open eyelid data for inference.

In step S22, the symptom estimation function 202a inputs the inference image data and inference open eyelid data to the trained machine learning program 750a, estimates the symptoms of dry eye appearing in the eyes of the inference subject, and causes the machine learning program 750a to output symptom data indicating the symptoms of dry eye appearing in the eyes of the inference subject.

The above describes the machine learning execution program, dry eye test program, machine learning execution device, dry eye test device, machine learning execution method, and dry eye test method according to the first embodiment.

The machine learning execution program 100a includes a teacher data acquisition function 101a and a machine learning execution function 102a.

The teacher data acquisition function 101a acquires teacher data in which the questions are learning image data and learning eyelid-open data, and the answer is at least one of learning test image data and learning test result data. The learning image data is data showing an image depicting the eye of the learning subject. The learning eyelid-open data is data showing the maximum eyelid-open time, which is the time the learning subject was able to continuously open the eye in which the learning image was captured. The learning test image data is data showing an image depicting the eye of the learning subject when a test for dry eye symptoms is performed on the eye of the learning subject. The learning test result data is data showing the results of the test for dry eye symptoms.

The machine learning execution function 102a inputs the training data into the machine learning program 750a and causes the machine learning program 750a to learn.

This allows the machine learning execution program 100a to generate a machine learning program 750a that predicts the results of a test regarding dry eye symptoms based on the learning image data and the learning open eyelid data.

In addition, the machine learning execution program 100a trains the machine learning program 750a using training data that includes not only training image data but also training open-eyelid data as a problem. Therefore, the machine learning execution program 100a can generate a machine learning program 750a that can predict the results of tests related to dry eye symptoms with even greater accuracy.

The dry eye test program 200a includes a data acquisition function 201a and a symptom estimation function 202a.

The data acquisition function 201a acquires image data for inference and eyelid-open data for inference. Image data for inference is data showing an image depicting the eyes of the subject for inference. Eyelid-open data for inference is data showing the maximum eyelid-open time, which is the time that the subject for inference was able to continuously open the eye in which the image for inference was captured.

The symptom estimation function 202a inputs the inference image data and inference eyelid open data to the machine learning program 750a that has been trained by the machine learning execution program 100a, and causes the machine learning program 750a to estimate the dry eye symptoms present in the eyes of the inference subject. The symptom estimation function 202a then causes the machine learning program 750a to output symptom data indicating the dry eye symptoms present in the eyes of the inference subject.

This allows the dry eye test program 200a to predict the results of a test regarding dry eye symptoms without actually conducting the test.

Next, we will provide an example in which the machine learning program 750a was actually trained to estimate symptoms of dry eye, and explain specific examples of the effects achieved by the machine learning execution program 100a by comparing a comparative example with an example according to the first embodiment.

The comparative example in this case is an example in which a machine learning program is trained using as training data 540 pieces of training data indicating that the eyes of the training subject described above are normal and 580 pieces of training data indicating that the eyes of the training subject are abnormal, with the training eyelid open data removed.

The machine learning program trained in this way showed a prediction accuracy of 72% and a false negative rate of 28% when its characteristics were evaluated using 80 test data pieces representing normal eyes of the training subjects described above and 80 test data pieces representing abnormal eyes of the training subjects.

On the other hand, when the characteristics of the machine learning program 750a trained by the machine learning execution program 100a were evaluated using the same test data, it showed a prediction accuracy of 80% and a false negative rate of 20%. These characteristics exceed those of the machine learning program according to the comparative example.

The prediction accuracy is the ratio of the total number X+Y of the training images in the abnormal group that were predicted to be abnormal by the machine learning program and the number Y of the training images in the normal group that were predicted to be normal by the machine learning program to the total number of test data. Therefore, in this comparative example, the prediction accuracy is calculated as ((X+Y)/160)×100.

The false negative rate is the ratio of the number of training images in the abnormal group that are predicted to be normal by the machine learning program to the total number of training images in the abnormal group. Therefore, in this comparative example, the false negative rate is calculated as ((80-X)/80) x 100.

At least a part of the functions of the machine learning execution program 100a may be realized by hardware including a circuitry. Similarly, at least a part of the functions of the dry eye test program 200a may be realized by hardware including a circuitry. Examples of such hardware include a large scale integration (LSI), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and a graphics processing unit (GPU).

In addition, at least some of the functions of the machine learning execution program 100a may be realized by a combination of software and hardware. Similarly, at least some of the functions of the dry eye test program 200a may be realized by a combination of software and hardware. In addition, these pieces of hardware may be integrated into one, or may be divided into multiple pieces.

In the first embodiment, the machine learning execution device 10a, the machine learning device 700a, and the dry eye test device 20a are independent devices, but this is not limiting. These devices may be realized as a single device.

[Second embodiment]
Specific examples of the machine learning execution program, machine learning execution device, and machine learning execution method according to the second embodiment will be described with reference to Figures 7 to 11. Unlike the machine learning execution program, machine learning execution device, and machine learning execution method according to the first embodiment, the machine learning execution program, machine learning execution device, and machine learning execution method according to the second embodiment use teacher data that includes, as a problem, at least one of learning tear meniscus image data and learning illumination image data cut out from learning image data, which will be described later.

Figure 7 is a diagram showing an example of the hardware configuration of a machine learning execution device according to the second embodiment. The machine learning execution device 10b shown in Figure 7 is a device that causes the machine learning device 700b to execute machine learning in the learning phase of the machine learning device 700b, which will be described later. As shown in Figure 8, the machine learning execution device 10b includes a processor 11b, a main memory device 12b, a communication interface 13b, an auxiliary memory device 14b, an input/output device 15b, and a bus 16b.

The processor 11b is, for example, a CPU, and reads and executes the machine learning execution program 100b described below to realize each function of the machine learning execution program 100b. The processor 11b may also read and execute programs other than the machine learning execution program 100b to realize functions necessary to realize each function of the machine learning execution program 100b.

The main memory device 12b is, for example, a RAM, and pre-stores the machine learning execution program 100b and other programs that are read and executed by the processor 11b.

The communication interface 13b is an interface circuit for communicating with the machine learning device 700b and other devices via the network NW shown in FIG. 8. The network NW is, for example, a LAN or an intranet.

The auxiliary storage device 14b is, for example, a hard disk drive, a solid state drive, a flash memory, or a ROM.

The input/output device 15b is, for example, an input/output port. For example, the keyboard 811b, mouse 812b, and display 910b shown in FIG. 1 are connected to the input/output device 15b. The keyboard 811b and mouse 812b are used, for example, to input data necessary to operate the machine learning execution device 10b. The display 910b displays, for example, a graphical user interface of the machine learning execution device 10b.

The bus 16b connects the processor 11b, the main memory device 12b, the communication interface 13b, the auxiliary memory device 14b, and the input/output device 15b so that data can be sent and received among them.

Figure 8 is a diagram showing an example of the software configuration of a machine learning execution program according to the second embodiment. The machine learning execution device 10b reads and executes the machine learning execution program 100b using the processor 11b, and realizes the teacher data acquisition function 101b and the machine learning execution function 102b shown in Figure 8.

The teacher data acquisition function 101b acquires teacher data in which at least one of a piece of learning tear meniscus image data and a piece of learning illumination image data is used as a question, and at least one of a piece of learning test image data and a piece of learning test result data is used as an answer.

The training image data is data showing a training image depicting the eye of a training subject. The training image is captured, for example, using a camera mounted on a smartphone.

The learning tear meniscus image data is data showing a learning tear meniscus image obtained by cutting out an area showing the tear meniscus of the learning subject from a learning image showing the eye of the learning subject, and is a type of learning image. The tear meniscus is a layer of tears formed between the cornea and the lower eyelid, and reflects the amount of tears. When the tear meniscus is low, there is little tear, and when the tear meniscus is high, there is much tear. The learning tear meniscus image is generated, for example, by cutting out an area showing the tear meniscus from the learning image. FIG. 9 is a diagram showing an example of a learning tear meniscus image obtained by cutting out an area showing the tear meniscus from a learning image according to the second embodiment. The teacher data acquisition function 101b acquires, for example, learning tear meniscus image data showing the learning tear meniscus image shown in FIG. 9.

The learning illumination image data is data showing a learning illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the learning subject from a learning image depicting the eye of the learning subject, and is a type of learning image. Such illumination is, for example, a fluorescent lamp installed on the ceiling of the room in which the learning image is taken. The learning illumination image is generated, for example, by cutting out an area of the learning image depicting the illumination reflected on the cornea of the learning subject. FIG. 10 is a diagram showing an example of a learning illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the learning subject from a learning image according to the second embodiment. The teacher data acquisition function 101b acquires, for example, learning illumination image data showing the learning illumination image shown in FIG. 10.

The learning test image data constitutes part of the answer of the teacher data, and is data showing a learning test image depicting the learning subject's eye when a test for dry eye symptoms is conducted on the learning subject's eye. An example of such a test is a test in which a fluorescein dye reagent is dropped into the eye and the eye is observed using a slit lamp to check the tear film breakup time. Therefore, for example, the learning test image data is data showing an image or video depicting an eye stained with fluorescein and observed using a slit lamp.

The learning test result data constitutes part of the answers to the teacher data, and is data showing the results of tests related to dry eye symptoms. Such test results include, for example, a numerical value between 0 and 1 that indicates the length of tear film breakup time based on a test in which a fluorescein dye is dropped into the eye and the eye is observed using a slit lamp to examine the tear film breakup time.

If this value is greater than or equal to 0 and less than 0.5, the tear film breakup time is 10 seconds or more, indicating that the study subject's eyes are normal and no examination by an ophthalmologist is necessary. If this value is greater than or equal to 0.5 and less than or equal to 1, the tear film breakup time is less than 10 seconds, indicating that the study subject's eyes are abnormal and an examination by an ophthalmologist is necessary.

For example, the teacher data acquisition function 101b acquires 1280 pieces of teacher data. In this case, the 1280 pieces of learning tear meniscus image data and learning illumination image data represent images generated by cutting out the area in which the tear meniscus is depicted and the area in which the illumination reflected on the cornea is depicted from 1280 learning images acquired by performing four sets of processing in which each of the two eyes of eight learning subjects is photographed 20 times using a camera mounted on a smartphone.

In this case, the 1,280 pieces of learning test image data each represent a learning test image captured when an examination for dry eye symptoms was performed on the eye of the learning subject depicted in each of the 1,280 learning images described above. The 1,280 pieces of learning test result data each represent 1,280 numerical values between 0 and 1 indicating the results of the examination for dry eye symptoms performed when the 1,280 learning test images were captured.

In the following explanation, an example will be given in which the 1,280 pieces of learning test result data include 620 pieces of learning test result data showing a value between 0 and less than 0.5, indicating that the learning subject's eyes are normal, and 660 pieces of learning test result data showing a value between 0.5 and 1, indicating that the learning subject's eyes are abnormal.

The machine learning execution function 102b inputs training data to the machine learning program 750b implemented in the machine learning device 700b, and trains the machine learning program 750b. For example, the machine learning execution function 102b trains the machine learning program 750b, which includes a convolutional neural network, by backpropagation.

For example, the machine learning execution function 102b inputs 540 pieces of training data, including training test result data showing values between 0 and less than 0.5, indicating that the eyes of the training subject are normal, into the machine learning program 750b. These 540 pieces of training data are training data obtained from seven people selected from the eight people mentioned above.

Also, for example, the machine learning execution function 102b inputs 580 pieces of training data, including training test result data showing a numerical value between 0.5 and 1, indicating that the eyes of the training subject are abnormal, into the machine learning program 750b. These 580 pieces of training data are training data obtained from seven people selected from the eight people described above.

Then, the machine learning execution function 102b trains the machine learning program 750b using these 1,120 pieces of training data.

Also, for example, the machine learning execution function 102b may represent that the eyes of the training subject are normal, and 80 pieces of training data that were not used in the training of the machine learning program 750b may be input to the machine learning program 750b as test data. These 80 pieces of training data are training data obtained from one person who was not selected as one of the seven people mentioned above.

Also, for example, the machine learning execution function 102b may indicate that the eyes of the training subject are abnormal, and 80 pieces of training data that were not used in the training of the machine learning program 750b may be input to the machine learning program 750b as test data. These 80 pieces of training data are training data obtained from one person who was not selected as one of the seven people mentioned above.

As a result, the machine learning execution function 102b may evaluate the characteristics of the machine learning program 750b obtained by training the machine learning program 750b using the above-mentioned 1,120 pieces of teacher data.

The machine learning execution function 102b may use, for example, class activation mapping when using test data to evaluate the characteristics of the machine learning program 750b. Class activation mapping is a technique that clarifies the part of the data input to a neural network that is the basis for the results output by the neural network. An example of class activation mapping is gradient-weighted class activation mapping. Gradient-weighted class activation mapping is a technique that uses the gradient of the classification score related to the features of the convolution performed by the convolutional neural network to identify areas of the image input to the convolutional neural network that have had a certain level of influence on the classification.

Next, an example of processing executed by the machine learning execution program 100b according to the second embodiment will be described with reference to FIG. 11. FIG. 11 is a flowchart showing an example of processing executed by the machine learning execution program according to the second embodiment. The machine learning execution program 100b executes the processing shown in FIG. 11 at least once.

In step S31, the teacher data acquisition function 101b acquires teacher data in which at least one of the learning tear meniscus image data and the learning illumination image data is used as a question, and at least one of the learning test image data and the learning test result data is used as an answer.

In step S32, the machine learning execution function 102b inputs the training data into the machine learning program 750b and causes the machine learning program 750b to learn.

Next, a specific example of a dry eye test program, a dry eye test device, and a dry eye test method according to the second embodiment will be described with reference to Figures 12 to 14. The dry eye test program, the dry eye test device, and the dry eye test method according to the second embodiment differ from the dry eye test program, the dry eye test device, and the dry eye test method according to the first embodiment in that they acquire at least one of tear meniscus image data for inference and illumination image data for inference, which are extracted from learning image data for inference, which will be described later.

Fig. 12 is a diagram showing an example of the hardware configuration of a dry eye test device according to the second embodiment. The dry eye test device 20b shown in Fig. 12 is a device that estimates dry eye symptoms appearing in the eye of an inference subject using a machine learning device 700b in the inference phase of the machine learning device 700b that has been trained by a machine learning execution program 100b. As shown in Fig. 12, the dry eye test device 20b also includes a processor 21b, a main memory device 22b, a communication interface 23b, an auxiliary memory device 24b, an input/output device 25b, and a bus 26b.

The processor 21b is, for example, a CPU, and reads and executes the dry eye test program 200b described below to realize each function of the dry eye test program 200b. The processor 21b may also read and execute a program other than the dry eye test program 200b to realize functions necessary to realize each function of the dry eye test program 200b.

The main memory device 22b is, for example, a RAM, and prestores the dry eye test program 200b and other programs that are read and executed by the processor 21b.

The communication interface 23b is an interface circuit for communicating with the machine learning device 700b and other devices via the network NW shown in FIG. 12. The network NW is, for example, a LAN or an intranet.

The auxiliary storage device 24b is, for example, a hard disk drive, a solid state drive, a flash memory, or a ROM.

The input/output device 25b is, for example, an input/output port. For example, the keyboard 821b, mouse 822b, and display 920b shown in FIG. 12 are connected to the input/output device 25b. The keyboard 821b and mouse 822b are used, for example, to input data required to operate the dry eye test device 20b. The display 920b displays, for example, a graphical user interface of the dry eye test device 20b.

The bus 26b connects the processor 21b, the main memory device 22b, the communication interface 23b, the auxiliary memory device 24b, and the input/output device 25b so that data can be sent and received among them.

Figure 13 is a diagram showing an example of the software configuration of a dry eye test program according to the second embodiment. The dry eye test device 20b reads and executes the dry eye test program 200b using the processor 21b, and realizes the data acquisition function 201b and the symptom estimation function 202b shown in Figure 13.

The data acquisition function 201b acquires at least one of tear meniscus image data for inference and illumination image data for inference.

The image data for inference is data showing an image depicting the eye of the subject for inference. The image for inference is captured, for example, using a camera mounted on a smartphone.

The inference tear meniscus image data is data showing an inference tear meniscus image obtained by cutting out an area depicting the tear meniscus of the inference subject from an inference image depicting the eye of the inference subject. The inference tear meniscus image is generated, for example, by cutting out an area depicting the tear meniscus from the inference image.

The illumination image data for inference is data showing an illumination image for inference obtained by cutting out an area depicting the illumination reflected on the cornea of the subject for inference from an image for inference depicting the eye of the subject for inference. The illumination image for inference is generated, for example, by cutting out an area from the image for inference depicting the illumination reflected on the cornea of the subject for inference.

The symptom estimation function 202b inputs at least one of the tear meniscus image data for inference and the illumination image data for inference to the machine learning program 750b that has been trained by the machine learning execution function 102b, and causes the machine learning program 750b to estimate the symptoms of dry eye appearing in the eye of the subject for inference. For example, the symptom estimation function 202b inputs these data to the machine learning program 750b, and causes it to estimate a numerical value representing the length of tear film breakup time in the eye of the subject for inference.

Then, the symptom estimation function 202b causes the machine learning program 750b to output symptom data indicating the symptoms of dry eye appearing in the eye of the inference subject. For example, the symptom estimation function 202b causes the machine learning program 750b to output symptom data indicating a numerical value indicating the length of the tear film breakup time of the eye of the inference subject. In this case, the symptom data is, for example, indicated by itself and is used to display a numerical value indicating the length of the tear film breakup time of the eye of the inference subject on the display 920b.

Next, an example of processing executed by the dry eye test program 200b according to the second embodiment will be described with reference to FIG. 14. FIG. 14 is a flowchart showing an example of processing executed by the dry eye test program according to the second embodiment.

In step S41, the data acquisition function 201b acquires at least one of tear meniscus image data for inference and illumination image data for inference.

In step S42, the symptom estimation function 202b inputs at least one of the inference tear meniscus image data and the inference illumination image data to the trained machine learning program, estimates the symptoms of dry eye appearing in the eyes of the inference subject, and causes the machine learning program to output symptom data indicating the symptoms of dry eye appearing in the eyes of the inference subject.

The above describes the machine learning execution program, dry eye test program, machine learning execution device, dry eye test device, machine learning execution method, and dry eye test method according to the second embodiment.

The machine learning execution program 100b includes a teacher data acquisition function 101b and a machine learning execution function 102b.

The teacher data acquisition function 101b acquires teacher data in which at least one of learning tear meniscus image data and learning illumination image data is used as a question, and at least one of learning test image data and learning test result data is used as an answer. The learning tear meniscus image data is data showing a learning tear meniscus image depicting the tear meniscus of the learning subject. The learning illumination image data is data showing a learning illumination image depicting illumination reflected on the cornea of the learning subject. The learning test image data is data showing an image depicting the eyes of the learning subject when an examination regarding dry eye symptoms is performed on the eyes of the learning subject. The learning test result data is data showing the results of an examination regarding dry eye symptoms.

The machine learning execution function 102b inputs the training data into the machine learning program 750b and causes the machine learning program 750b to learn.

As a result, the machine learning execution program 100b can generate a machine learning program 750b that predicts the results of a test regarding dry eye symptoms based on at least one of the learning tear meniscus image data and the learning illumination image data.

The machine learning execution program 100b also trains the machine learning program 750b using training data that includes, as a problem, at least one of the learning tear meniscus image data and the learning illumination image data extracted from the learning image data. Therefore, the machine learning execution program 100b can generate the machine learning program 750b that can predict the results of tests related to dry eye symptoms with even greater accuracy.

The dry eye test program 200b includes a data acquisition function 201b and a symptom estimation function 202b.

The data acquisition function 201b acquires at least one of inference tear meniscus image data and inference illumination image data. The inference tear meniscus image data is data showing an inference tear meniscus image depicting the tear meniscus of the inference subject. The inference illumination image data is data showing an inference illumination image depicting illumination reflected on the cornea of the inference subject.

The symptom estimation function 202b inputs at least one of the tear meniscus image data for inference and the illumination image data for inference to the machine learning program 750b that has been trained by the machine learning execution program 100b, and causes the machine learning program 750b to estimate the symptoms of dry eye appearing in the eye of the subject for inference. The symptom estimation function 202b then causes the machine learning program 750b to output symptom data indicating the symptoms of dry eye appearing in the eye of the subject for inference.

This allows the dry eye test program 200b to predict the results of a test for dry eye symptoms without actually conducting the test.

Next, we will provide an example in which the machine learning program 750b was actually trained to estimate symptoms of dry eye, and explain specific examples of the effects achieved by the machine learning execution program 100b by comparing a comparative example with an example of the second embodiment.

The comparative example in this case is an example in which a machine learning program is trained using as training data data to which training image data has been added, excluding the training tear meniscus image data and the training illumination image data from the 540 training data pieces representing that the training subject's eyes are normal and the 580 training data pieces representing that the training subject's eyes are abnormal.

The machine learning program trained in this way showed a prediction accuracy of 72% and a false negative rate of 28% when its characteristics were evaluated using data in which the training tear meniscus image data and training illumination image data were removed from the 80 test data representing the above-mentioned training subject's eyes being normal and training image data was added, and data in which the training tear meniscus image data and training illumination image data were removed from the 80 test data representing the training subject's eyes being abnormal and training image data was added.

On the other hand, when the machine learning program 750b, which was trained using the learning tear meniscus image data by the machine learning execution program 100b, was evaluated for its characteristics using test data containing only the learning tear meniscus image data, it showed a prediction accuracy of 77% and a false negative rate of 23%. These characteristics exceed those of the machine learning program according to the comparative example.

In addition, when the machine learning program 750b, which was trained using the learning illumination image data by the machine learning execution program 100b, was evaluated for its characteristics using test data containing only the learning illumination image data, it showed a prediction accuracy of 77% and a false negative rate of 23%. These characteristics exceed those of the machine learning program according to the comparative example.

Furthermore, when the machine learning program 750b trained using the learning tear meniscus image data and the learning illumination image data by the machine learning execution program 100b was evaluated for its characteristics using test data including the learning tear meniscus image data and the learning illumination image data, it showed a prediction accuracy of 82% and a false negative rate of 18%. These characteristics exceed those of the machine learning program according to the comparative example, the characteristics of the machine learning program 750b trained using the learning tear meniscus image data, and the characteristics of the machine learning program 750b trained using the learning illumination image data.

The prediction accuracy is the ratio of the total number X+Y of the training images in the abnormal group that were predicted to be abnormal by the machine learning program and the number Y of the training images in the normal group that were predicted to be normal by the machine learning program to the total number of test data. Therefore, in this comparative example, the prediction accuracy is calculated as ((X+Y)/160)×100.

The false negative rate is the ratio of the number of training images in the abnormal group that are predicted to be normal by the machine learning program to the total number of training images in the abnormal group. Therefore, in this comparative example, the false negative rate is calculated as ((80-X)/80) x 100.

At least a part of the functions of the machine learning execution program 100b may be realized by hardware including a circuit unit. Similarly, at least a part of the functions of the dry eye test program 200b may be realized by hardware including a circuit unit. Such hardware may be, for example, an LSI, an ASIC, an FPGA, or a GPU.

Furthermore, at least some of the functions of the machine learning execution program 100b may be realized by a combination of software and hardware. Similarly, at least some of the functions of the dry eye test program 200b may be realized by a combination of software and hardware. Furthermore, these pieces of hardware may be integrated into one, or may be divided into multiple pieces.

In the second embodiment, the machine learning execution device 10b, the machine learning device 700b, and the dry eye test device 20b are independent devices, but this is not limiting. These devices may be realized as a single device.

[Third embodiment]
In the first embodiment described above, the case where the teacher data acquisition function 101a acquires teacher data that does not include the learning tear meniscus image data and the learning illumination image data described in the second embodiment as part of the problem has been exemplified, but is not limited thereto. The teacher data acquisition function 101a may acquire teacher data in which at least one of the learning tear meniscus image data and the learning illumination image data is used as the above-mentioned learning image data in addition to the learning eyelid open data. In this case, the data acquisition function 201a acquires at least one of the inference tear meniscus image data and the inference illumination image data as the above-mentioned inference image data in addition to the inference eyelid open data. In this case, the symptom estimation function 202a causes the machine learning program 750a to estimate the symptoms of dry eye appearing in the eye of the inference subject based on at least one of the inference tear meniscus image data and the inference illumination image data in addition to the inference eyelid open data.

In the second embodiment described above, the teacher data acquisition function 101b acquires teacher data that does not include the learning eyelid open data described in the first embodiment as part of the problem, but this is not limited to this. The teacher data acquisition function 101b may acquire teacher data that has learning eyelid open data as a problem, in addition to learning tear meniscus image data and learning illumination image data. In this case, the data acquisition function 201b acquires inference eyelid open data in addition to at least one of inference tear meniscus image data and inference illumination image data. In this case, the symptom estimation function 202b causes the machine learning program 750b to estimate the symptoms of dry eye appearing in the eye of the inference subject based on the inference eyelid open data in addition to at least one of inference tear meniscus image data and inference illumination image data.

Furthermore, the machine learning execution function 102a according to the third embodiment may use, for example, class activation mapping when using test data to evaluate the characteristics of the machine learning program 750a. Similarly, the machine learning execution function 102b according to the third embodiment may use, for example, class activation mapping when using test data to evaluate the characteristics of the machine learning program 750b.

Figure 15 is a diagram showing an example of a portion of a training image depicting the eye of a training subject that is considered intensively when a machine learning program according to the third embodiment predicts the results of a test to examine tear film breakup time. For example, the machine learning execution function 102a according to the third embodiment uses gradient-weighted class activation mapping to evaluate that the area surrounded by the ellipse C11, the area surrounded by the ellipse C12, and the area surrounded by the ellipse C2 in the training image shown in Figure 15 have a certain degree of influence on the prediction of the test to examine the tear film breakup time by the machine learning program 750a. The matters described with reference to Figure 15 also apply to the machine learning execution function 102b according to the third embodiment.

The three grayscales contained in the areas enclosed by these ellipses indicate the degree to which they influenced the prediction of the test results related to corneal epithelial damage. The areas enclosed by ellipses C11 and C12 are both displayed superimposed on the area depicting the tear meniscus of the learning subject. The area enclosed by ellipse C2 is displayed superimposed on the area depicting the illumination reflected on the cornea of the learning subject.

Next, a specific example of the effect achieved when training is performed using teacher data that includes as problems the learning open eyelid data and at least one of the learning tear meniscus image data and the learning illumination image data, and the symptoms of dry eye appearing in the eye of the inference subject are estimated based on the inference open eyelid data and at least one of the inference tear meniscus image data and the inference illumination image data will be described. In the following explanation, an example will be given in which the teacher data acquisition function 101a acquires the learning open eyelid data, and the data acquisition function 201a acquires the inference open eyelid data.

The comparative example in this case is an example in which a machine learning program is trained using as training data data to which training image data has been added, excluding training eyelid open data, training tear meniscus image data, and training illumination image data from the 540 training data pieces representing that the training subject's eyes are normal and the 580 training data pieces representing that the training subject's eyes are abnormal.

The machine learning program trained in this way showed a prediction accuracy of 72% and a false negative rate of 28% when its characteristics were evaluated using data to which the training image data was added, excluding the training open eye data, training tear meniscus image data, and training illumination image data from 80 test data representing the above-mentioned training subject's eyes being normal, and data to which the training image data was added, excluding the training open eye data, training tear meniscus image data, and training illumination image data from 80 test data representing the training subject's eyes being abnormal.

On the other hand, the machine learning program 750a, which was trained by the machine learning execution program 100a using training open eyelid data, training tear meniscus image data, and training illumination image data, showed a prediction accuracy of 90% and a false negative rate of 10% when its characteristics were evaluated using test data including training tear meniscus image data and training illumination image data. These characteristics exceed those of the machine learning program according to the comparative example, the machine learning program 750a according to the first embodiment, and the machine learning program 750b according to the second embodiment.

The prediction accuracy is the ratio of the total number X+Y of the training images in the abnormal group that were predicted to be abnormal by the machine learning program and the number Y of the training images in the normal group that were predicted to be normal by the machine learning program to the total number of test data. Therefore, in this comparative example, the prediction accuracy is calculated as ((X+Y)/160)×100.

The false negative rate is the ratio of the number of training images in the abnormal group that are predicted to be normal by the machine learning program to the total number of training images in the abnormal group. Therefore, in this comparative example, the false negative rate is calculated as ((80-X)/80) x 100.

The above-mentioned dry eye test program, dry eye test device, and dry eye test method may be used before or after the dry eye eye drops are applied to the eyes of the subject to be inferred. By using the above-mentioned dry eye test program, dry eye test device, and dry eye test method at such timing, they can also serve as a means of verifying the effectiveness of the dry eye eye drops for the dry eye symptoms suffered by the subject to be inferred.

Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and various combinations, modifications, substitutions, and/or design changes may be made without departing from the spirit of the present invention.

The effects of the above-described embodiments of the present invention are shown as examples. Therefore, in addition to the effects described above, the embodiments of the present invention may also have other effects that a person skilled in the art may recognize from the description of the above-described embodiments.

10a, 10b...machine learning execution device, 100a, 100b...machine learning execution program, 101a, 101b...teaching data acquisition function, 102a, 102b...machine learning execution function, 20a, 20b...dry eye test device, 200a, 200b...dry eye test program, 201a, 201b...data acquisition function, 202a, 202b...symptom estimation function, 700a, 700b...machine learning device, 750a, 750b...machine learning program

Claims (11)

a training data acquisition function for acquiring training data having as an answer at least one of training test image data showing an image depicting the training subject's eye when a test for dry eye symptoms is performed on the training subject's eye and training test result data showing the results of the test for dry eye symptoms, the training image data being training illumination image data showing a training illumination image obtained by cutting out a region depicting the illumination reflected on the cornea of the training subject from a training image depicting the eye of the training subject, and training eyelid open data showing a maximum eyelid open time which is the time the training subject was able to continuously open the eye in which the training image was captured;
A machine learning execution function that inputs the teacher data into a machine learning program and trains the machine learning program;
A machine learning execution program that causes a computer to execute the above. The teacher data acquisition function acquires the teacher data, the teacher data being training image data that indicates a training tear meniscus image obtained by cutting out an area depicting the tear meniscus of the training subject from the training image.
The machine learning execution program according to claim 1 . The teacher data acquisition function acquires teacher data having as an answer at least one of the learning test image data showing an image depicting the eye of the learning subject when a test is performed to examine the tear film breakup time of the eye of the learning subject, and the learning test result data showing a result of the test to examine the tear film breakup time of the eye of the learning subject.
The machine learning execution program according to claim 1 or 2 . a data acquisition function for acquiring inference image data, which is inference illumination image data showing an inference illumination image obtained by cutting out an area depicting the illumination reflected on the cornea of the subject for inference from an inference image depicting the eye of the subject for inference, the area depicting the illumination reflected on the cornea of the subject for inference, and inference eyelid opening data showing the maximum eyelid opening time, which is the time the subject for inference could continuously open the eye in which the inference image was captured;
a symptom estimation function for inputting the inference image data and the inference eye-open data into a machine learning program trained using teacher data having as answer at least one of: learning image data showing a learning image depicting the eye of a learning subject, and learning eyelid-open data showing a maximum eyelid-open time, which is the time the learning subject could continuously open the eye in which the learning image was captured; learning test image data being learning illumination image data showing a region depicting illumination reflected on the cornea of the learning subject from the learning image depicting the eye of the learning subject when a test for dry eye symptoms is performed on the eye of the learning subject, the learning image showing an image depicting the eye of the learning subject and depicting illumination reflected on the cornea of the learning subject; and learning test result data showing the result of the test for dry eye symptoms; and
A dry eye testing program that enables a computer to achieve this. The data acquisition function acquires, as the image data for inference, tear meniscus image data for inference showing a tear meniscus image for inference obtained by cutting out a region depicting the tear meniscus of the subject for inference from the image for inference depicting the tear meniscus of the eye of the subject for inference,
the symptom estimation function inputs the inference tear meniscus image data to the machine learning program trained using the teacher data, the training image data being training tear meniscus image data showing a training tear meniscus image obtained by cutting out a region depicting the tear meniscus of the training subject from the training image depicting the tear meniscus of the training subject's eye, and causes the machine learning program to estimate a symptom of dry eye appearing in the eye of the inference subject;
The dry eye test program according to claim 4 . a training data acquisition unit that acquires training data having as an answer at least one of training test image data showing an image depicting the training subject's eye when a test for dry eye symptoms is performed on the training subject's eye and training test result data showing the results of the test for dry eye symptoms, and training image data being training illumination image data showing a training illumination image obtained by cutting out a region depicting the illumination reflected on the cornea of the training subject from a training image depicting the eye of the training subject and training eyelid open data showing a maximum eyelid open time which is the time the training subject was able to continuously open the eye in which the training image was captured;
a machine learning execution unit that inputs the teacher data into a machine learning program and trains the machine learning program;
A machine learning execution device comprising: a data acquisition unit that acquires inference image data, which is inference illumination image data showing an inference illumination image obtained by cutting out an area depicting illumination reflected on the cornea of the subject for inference from an inference image depicting the eye of the subject for inference, the area depicting illumination reflected on the cornea of the subject for inference, and inference eyelid opening data showing a maximum eyelid opening time that is the time the subject for inference can continuously open the eye in which the inference image is captured;
a symptom estimation unit that inputs the inference image data and the inference eye-open data into a machine learning program trained using teacher data that has as its answer at least one of: training image data showing a training image depicting the eye of a training subject, and training eyelid-open data showing a maximum eyelid-open time that is the time that the training subject could continuously open the eye in which the training image was captured; training test image data that is training illumination image data showing a region depicting illumination reflected on the cornea of the training subject from the training image depicting the eye of the training subject when a test for dry eye symptoms is performed on the eye of the training subject, the training image showing an image depicting the eye of the training subject and depicting illumination reflected on the cornea of the training subject; and training test result data showing a result of the test for dry eye symptoms; and causes the machine learning program to infer symptoms of dry eye that are present in the eye of the inference subject, and causes the machine learning program to output symptom data showing symptoms of dry eye that are present in the eye of the inference subject.
A dry eye examination device comprising: a training data acquisition function for acquiring training data having training test image data showing an image depicting the training subject's eye when a test for dry eye symptoms is performed on the training subject's eye and training test result data showing the results of the test for dry eye symptoms, as answers, and training test image data showing an image depicting the training subject's eye when a test for dry eye symptoms is performed on the training subject's eye; and
A machine learning execution function that inputs the teacher data into a machine learning program and trains the machine learning program;
A machine learning execution program that causes a computer to execute the above. a data acquisition function for acquiring tear meniscus image data for inference showing a tear meniscus image for inference obtained by cutting out an area depicting the tear meniscus of the subject for inference from an image for inference depicting the eye of the subject for inference, and illumination image data for inference showing an illumination image for inference obtained by cutting out an area depicting illumination reflected on the cornea of the subject for inference from the image for inference;
a symptom estimation function that inputs the inference tear meniscus image data and the inference illumination image data into a machine learning program trained using teacher data having as questions training tear meniscus image showing a training tear meniscus image obtained by cutting out a region depicting the tear meniscus of the training subject from a training image depicting the eye of the training subject and training illumination image data showing a region depicting illumination reflected on the cornea of the training subject from the training image, and answers training test image data showing an image depicting the eye of the training subject when a test for dry eye symptoms is performed on the eye of the training subject and training test result data showing the results of the test for dry eye symptoms, causes the machine learning program to estimate a symptom of dry eye appearing in the eye of the inference subject, and causes the machine learning program to output symptom data showing the symptom of dry eye appearing in the eye of the inference subject;
A dry eye testing program that enables a computer to achieve this. a training data acquisition unit that acquires training data having training tear meniscus image data showing a training tear meniscus image obtained by cutting out a region depicting the tear meniscus of the training subject from a training image depicting the eye of the training subject and training illumination image data showing a training illumination image obtained by cutting out a region depicting illumination reflected on the cornea of the training subject from the training image, as questions, and has training test image data showing an image depicting the eye of the training subject when a test for dry eye symptoms is performed on the eye of the training subject and training test result data showing the results of the test for dry eye symptoms as answers;
a machine learning execution unit that inputs the teacher data into a machine learning program and trains the machine learning program;
A machine learning execution device comprising: a data acquisition unit that acquires tear meniscus image data for inference showing a tear meniscus image for inference obtained by cutting out an area depicting the tear meniscus of the subject for inference from an image for inference depicting the eye of the subject for inference, and illumination image data for inference showing an illumination image for inference obtained by cutting out an area depicting illumination reflected on the cornea of the subject for inference from the image for inference;
a symptom estimation unit that inputs the inference tear meniscus image data and the inference illumination image data into a machine learning program trained using teacher data having as problems training tear meniscus image showing a training tear meniscus image obtained by cutting out a region depicting the tear meniscus of the training subject from a training image depicting the eye of the training subject and training illumination image data showing a region depicting illumination reflected on the cornea of the training subject from the training image, and answers training test image data showing an image depicting the eye of the training subject when a test for dry eye symptoms is performed on the eye of the training subject and training test result data showing a result of the test for dry eye symptoms, causes the machine learning program to estimate a symptom of dry eye appearing in the eye of the inference subject, and causes the machine learning program to output symptom data showing the symptom of dry eye appearing in the eye of the inference subject;
A dry eye examination device comprising:

Images