KR102477694B1 - A method for guiding a visit to a hospital for treatment of active thyroid-associated ophthalmopathy and a system for performing the same - Google Patents

Abstract

According to the present application, disclosed is a method for predicting thyroid-associated ophthalmopathy, executable by a computer. The method comprises: preparing a conjunctival injection prediction model, a conjunctival edema prediction model, a lacrimal caruncle edema prediction model, an eyelid redness prediction model, and an eyelid edema prediction model; acquiring an image of the face of a subject; acquiring a first processed image and a second processed image from the image, wherein the first processed image is different from the second processed image; inputting the first processed image to the conjunctival injection prediction model, the conjunctival edema prediction model, and the lacrimal caruncle edema prediction model to acquire predicted values for conjunctival injection, conjunctival edema, and lacrimal caruncle edema; inputting the second processed image to the eyelid redness prediction model and the eyelid edema prediction model to acquire predicted values for eyelid redness and eyelid edema; and determining the possibility that the subject has a thyroid eye disease, based on the predicted values for the conjunctival injection, the conjunctival edema, the lacrimal caruncle edema, the eyelid redness, and the eyelid edema. Therefore, the method can predict clinical activity scores for the thyroid eye disease by using images acquired with a digital camera that can be used by the general public.

Description

Method for guiding visits to the hospital for treatment of active thyroid eye disease and system for performing the same

The present invention relates to a method for guiding visits to the hospital for treatment of active thyroid ophthalmopathy and a system for performing the same.

BACKGROUND OF THE INVENTION Ophthalmic diseases are diseases that occur in the eyeball and the periphery surrounding it. Since eye diseases are occurring to many people around the world and cause great inconvenience in daily life, such as causing visual impairment in serious cases, it is necessary to monitor the occurrence or degree of eye diseases.

Meanwhile, an eye disease may be one of several complications caused by other diseases. For example, there is thyroid ophthalmopathy as a complication caused by abnormal thyroid function.

When thyroid eye disease becomes severe, it is very important to diagnose thyroid eye disease early because the eyeball protrudes and cannot be treated without surgery. However, thyroid ophthalmopathy is difficult to diagnose early because there are no clear prognostic symptoms. In the medical world, the clinical activity score (CAS), which has been proposed since 1989, is used to evaluate thyroid ophthalmopathy at an early stage. Efforts are being made to diagnose.

A total of 7 items are considered in determining the clinical activity score for thyroid ophthalmopathy, and a total of 7 items are considered: 1) Spontaneous retrobulbar pain, 2) Pain on attempted eye movement upward or downward gaze), 3) Redness of eyelid, 4) Redness of conjunctiva, 5) Swelling of eyelid, 6) Swelling of conjunctiva, and 7) Swelling of lacrimal caruncle.

In order to determine such a clinical activity score, an individual directly visits a hospital or clinic, and a doctor's medical examination and visual observation are essential. For example, spontaneous pain in the posterior part of the mouth and pain during eye movement can be confirmed through a doctor's medical examination, and eyelid redness, conjunctival congestion, eyelid edema, conjunctival edema, and lacrimal edema can be visually observed by a doctor. can be confirmed by This method of visual examination and interview with a doctor for confirming the clinical activity score requires a direct visit of the patient to the hospital for the diagnosis of thyroid eye disease as a prerequisite, so there was difficulty in diagnosing thyroid eye disease at an early stage.

Therefore, without a direct visit to the hospital, it is necessary to develop a method for inducing patients to visit the hospital by guiding the risk of eye diseases, if necessary, as well as enabling continuous monitoring by checking the risk of eye diseases more conveniently and quickly for each individual. is being demanded

(Patent Document 0001) Korean Patent Registration No. 10-2223478 (registered on February 26, 2021)
(Patent Document 0002) Korean Patent Registration No. 10-2047237 (registered on November 15, 2019)
(Patent Document 0003) Korean Patent Registration No. 10-2058883 (registered on December 18, 2019)
(Patent Document 0004) Republic of Korea Patent Registration No. 10-2347551 (registered on January 2, 2022)

The problem to be solved by the contents disclosed by this application is the learning used to predict the clinical activity score related to thyroid eye disease using images obtained with a digital camera that can be used by the general public rather than a professional medical diagnostic device. to provide a model.

Another problem to be solved by the contents disclosed by this application is to provide a method and system that can continuously monitor clinical activity scores related to thyroid ophthalmia without the help of doctors and without direct visits to hospitals.

Another problem to be solved by the contents disclosed by the present application is to provide a method for recommending a hospital visit for treatment of active thyroid eye disease according to a monitoring result of a clinical activity score and a system for performing the same.

The problem to be solved by this application is not limited to the above-mentioned problem, and the problem not mentioned is clearly made clear to those skilled in the art from this specification and the accompanying drawings to which the technology disclosed by this application belongs. You will be able to understand.

According to one aspect of the present application, a computer-executable method for predicting thyroid eye disease is disclosed. The method includes preparing a conjunctival hyperemia prediction model, a conjunctival edema prediction model, a tear hill edema prediction model, an eyelid erythema prediction model, and an eyelid edema prediction model, obtaining a face image of a subject, and a first processed image from the face image. Obtaining a (first processed image) and a second processed image, where the first processed image is different from the second processed image, and the first processed image as the conjunctival congestion Prediction values for conjunctival hyperemia, conjunctival edema, and lacrimal edema are input to the prediction model, the conjunctival edema prediction model, and the tear hill edema prediction model, respectively, and the second processed image is used as the eyelid erythema prediction model and the eyelid edema prediction model. Input to an edema prediction model to obtain predicted values for eyelid redness and eyelid edema, respectively, the predicted value for conjunctival redness, the predicted value for conjunctival edema, the predicted value for lacrimal edema, and the predicted value for eyelid redness. and determining a possibility that the subject has thyroid eye disease based on the predicted value and the predicted value for the eyelid edema. At this time, the first processed image is based on positional information of pixels corresponding to the outline of the eye and positional information of pixels corresponding to the outline of the iris included in the eye, outside and outside the outline of the eye. An image cropped along a first region including the outline of the eye by masking regions corresponding to the inside of the outline of the iris, and the second processed image includes pixels corresponding to the outline of the eye. Based on location information and location information of pixels corresponding to the outline of the iris included in the eye, the image is cropped along a second area that extends wider than the first area.

In some embodiments, location information of pixels corresponding to the outline of the eye and location information of pixels corresponding to the outline of the iris included in the eye may be obtained by a segmentation model.

In some embodiments, the first processed image includes a first processed left eye image and a first processed right eye image, and the second processed image includes a first processed left eye image and a first processed right eye image. The image may include a second processed left eye image and a second processed right eye image.

In some embodiments, the conjunctival edema prediction model includes a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model, the conjunctival edema prediction model includes a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model, wherein the The lacrimal edema prediction model includes a left eye lacrimal edema prediction model and a right eye lacrimal edema prediction model, wherein the eyelid redness prediction model includes a left eye eyelid redness prediction model and a right eye eyelid redness prediction model, wherein the eyelid edema prediction model comprises: A left eye eyelid edema prediction model and a right eye eyelid edema prediction model may be included.

In some embodiments, the predicted value for the conjunctival hyperemia is a result obtained by inputting the first preprocessed left eye image to the left eye conjunctival hyperemia model and inputting the first preprocessed right eye image to the right eye conjunctival hyperemia model. It is determined based on the result obtained by inputting the first preprocessed left eye image to the left eye conjunctival edema model and the first preprocessed right eye image as the right eye conjunctival edema It is determined based on a result obtained by inputting the lacrimal hillock edema, and the predicted value for the tear hill edema is determined based on a result obtained by inputting the first preprocessed left eye image to the left eye lacrimal hill edema model and the first preprocessed right eye image. is determined based on a result obtained by inputting the second preprocessed left eye image into the left eye eyelid erythema model and the predicted value for the eyelid erythema is determined based on a result obtained by inputting the second preprocessed left eye image into the left eye eyelid erythema model and the second It is determined based on a result obtained by inputting the preprocessed right eye image into the right eye eyelid edema model, and the predicted value for the eyelid edema is determined based on a result obtained by inputting the second preprocessed left eye image into the left eye eyelid edema model. It may be determined based on a result obtained by inputting the second preprocessed right eye image to the right eye eyelid edema model.

In some embodiments, the method performs left-right reversal processing of one of the first processed left eye image and the first processed right eye image, and the second processed right eye image. The method may further include horizontally inverting one of the second processed left eye image and the second processed right eye image.

In some embodiments, the predicted value for the conjunctival congestion is determined based on result values obtained by inputting the horizontally inverted image and the left-right inverted image to the conjunctival congestion model, respectively, and the conjunctiva The predicted value for edema is determined based on result values obtained by inputting the left-right inverted image and the non-left-right reversed image to the conjunctival edema model, respectively, and the predicted value for the tear hill edema is It is determined based on result values obtained by inputting the inverted image and the non-left-right inverted image to the tear hill edema model, respectively, and the predicted value for the eyelid redness is the left-right inverted image and the left-right inversion process. The prediction value for the eyelid edema is determined based on the result values obtained by inputting the non-reversed image into the eyelid edema model, respectively, and the predicted value for the eyelid edema is obtained by combining the horizontally inverted image and the left-right inverted image into the eyelid edema model. It may be determined based on the result values obtained by inputting each.

In some embodiments, the method resizes the first processed left eye image and the first processed right eye image, and the second processed left eye image. Resizing the second processed left eye image and the second processed right eye image may be further included.

According to the contents disclosed by this application, it is possible to predict the clinical activity score related to thyroid eye disease using an image acquired by a digital camera usable by the general public, not a professional medical diagnostic device.

In addition, according to the contents disclosed by this application, the general public can continuously monitor the clinical activity score for thyroid eye disease without the help of a doctor and without a direct visit to the hospital, and, if necessary, receive a recommendation to visit a hospital. be able to

1 is a diagram illustrating a system for predicting a clinical activity score related to thyroid eye disease according to an embodiment disclosed by the present application.
2 is a block diagram of a user terminal provided by the present application.
3 is a block diagram of a server disclosed by the present application.
4 is a view for explaining an eye exposed to the outside and its surrounding tissue so that it can be captured by a camera when a face is photographed using a camera.
5 is a view for explaining an eyeball exposed to the outside.
6 is a diagram for explaining the outline of an eye.
7 is a view for explaining the cornea exposed to the outside.
8 is a view for explaining the conjunctiva exposed to the outside.
9 is a diagram for explaining a face image and a binocular image.
10 is a diagram for explaining a left eye image and a right eye image.
11 is a diagram illustrating X _max , X _min , Y _max , and Y _min of outline pixels.
12 is a diagram illustrating a determined second cropped area.
13 is an exemplary diagram of a second cropped image.
14 is an exemplary diagram of a third cropped image.
15 and 16 are diagrams for explaining iris segmentation.
17 is a diagram for explaining eye outline segmentation.
18 is an exemplary diagram of a first masking image.
19 is an exemplary diagram of a second masking image.
20 to 22 are diagrams illustrating various examples of an original image and a horizontally reversed image.
23 is a flowchart for explaining a method for predicting conjunctival congestion.
24 is a flowchart for explaining a method for predicting conjunctival edema.
25 is a flowchart illustrating a method for predicting tear hill edema.
26 is a flowchart for explaining a method for predicting eyelid redness.
27 is a flowchart for explaining a method for predicting eyelid edema.
28 is a diagram for explaining a method for predicting clinical activity scores related to thyroid ophthalmopathy.
29 is a view for explaining a method for continuously monitoring clinical activity scores related to thyroid eye disease and a method for recommending a hospital visit based thereon.

The foregoing objects, features and advantages of the present application will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, the present application can apply various changes and can have various embodiments. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

Like reference numerals designate essentially like elements throughout the specification. In addition, components having the same function within the scope of the same idea appearing in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present application may unnecessarily obscure the subject matter of the present application, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of this specification are only identifiers for distinguishing one component from another component.

In addition, the suffixes "module" and "unit" for components used in the following embodiments are given or used interchangeably in consideration of ease of writing the specification, and do not have meanings or roles that are distinguished from each other by themselves.

In the following examples, expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean that features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added.

In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present invention is not necessarily limited to those shown.

If an embodiment is otherwise implementable, the order of specific processes may be performed differently from the order described. For example, two processes that are described in succession may be performed substantially concurrently, or may proceed in an order reverse to that described.

In the following embodiments, when components are connected, a case in which the components are directly connected as well as a case in which components are interposed between the components and connected indirectly is included. For example, when it is said that components are electrically connected in this specification, not only the case where the components are directly electrically connected, but also the case where the components are interposed and electrically connected indirectly is included.

According to one aspect of the present application, a method for predicting thyroid ophthalmopathy that can be performed with putero is disclosed. The method includes preparing a conjunctival hyperemia prediction model, a conjunctival edema prediction model, a tear hill edema prediction model, an eyelid erythema prediction model, and an eyelid edema prediction model, obtaining a face image of a subject, and a first processed image from the face image. Obtaining a (first processed image) and a second processed image, where the first processed image is different from the second processed image, and the first processed image as the conjunctival congestion Prediction values for conjunctival hyperemia, conjunctival edema, and lacrimal edema are input to the prediction model, the conjunctival edema prediction model, and the tear hill edema prediction model, respectively, and the second processed image is used as the eyelid erythema prediction model and the eyelid edema prediction model. Input to an edema prediction model to obtain predicted values for eyelid redness and eyelid edema, respectively, the predicted value for conjunctival redness, the predicted value for conjunctival edema, the predicted value for lacrimal edema, and the predicted value for eyelid redness. and determining a possibility that the subject has thyroid eye disease based on the predicted value and the predicted value for the eyelid edema. At this time, the first processed image is based on positional information of pixels corresponding to the outline of the eye and positional information of pixels corresponding to the outline of the iris included in the eye, outside and outside the outline of the eye. An image cropped along a first region including the outline of the eye by masking regions corresponding to the inside of the outline of the iris, and the second processed image includes pixels corresponding to the outline of the eye. Based on location information and location information of pixels corresponding to the outline of the iris included in the eye, the image is cropped along a second area that extends wider than the first area.

In some embodiments, location information of pixels corresponding to the outline of the eye and location information of pixels corresponding to the outline of the iris included in the eye may be obtained by a segmentation model.

In some embodiments, the first processed image includes a first processed left eye image and a first processed right eye image, and the second processed image includes a first processed left eye image and a first processed right eye image. The image may include a second processed left eye image and a second processed right eye image.

In some embodiments, the conjunctival edema prediction model includes a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model, the conjunctival edema prediction model includes a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model, wherein the The lacrimal edema prediction model includes a left eye lacrimal edema prediction model and a right eye lacrimal edema prediction model, wherein the eyelid redness prediction model includes a left eye eyelid redness prediction model and a right eye eyelid redness prediction model, wherein the eyelid edema prediction model comprises: A left eye eyelid edema prediction model and a right eye eyelid edema prediction model may be included.

In some embodiments, the predicted value for the conjunctival hyperemia is a result obtained by inputting the first preprocessed left eye image to the left eye conjunctival hyperemia model and inputting the first preprocessed right eye image to the right eye conjunctival hyperemia model. It is determined based on the result obtained by inputting the first preprocessed left eye image to the left eye conjunctival edema model and the first preprocessed right eye image as the right eye conjunctival edema It is determined based on a result obtained by inputting the lacrimal hillock edema, and the predicted value for the tear hill edema is determined based on a result obtained by inputting the first preprocessed left eye image to the left eye lacrimal hill edema model and the first preprocessed right eye image. is determined based on a result obtained by inputting the second preprocessed left eye image into the left eye eyelid erythema model and the predicted value for the eyelid erythema is determined based on a result obtained by inputting the second preprocessed left eye image into the left eye eyelid erythema model and the second It is determined based on a result obtained by inputting the preprocessed right eye image into the right eye eyelid edema model, and the predicted value for the eyelid edema is determined based on a result obtained by inputting the second preprocessed left eye image into the left eye eyelid edema model. It may be determined based on a result obtained by inputting the second preprocessed right eye image to the right eye eyelid edema model.

In some embodiments, the method performs left-right reversal processing of one of the first processed left eye image and the first processed right eye image, and the second processed right eye image. The method may further include horizontally inverting one of the second processed left eye image and the second processed right eye image.

In some embodiments, the predicted value for the conjunctival congestion is determined based on result values obtained by inputting the horizontally inverted image and the left-right inverted image to the conjunctival congestion model, respectively, and the conjunctiva The predicted value for edema is determined based on result values obtained by inputting the left-right inverted image and the non-left-right reversed image to the conjunctival edema model, respectively, and the predicted value for the tear hill edema is It is determined based on result values obtained by inputting the inverted image and the non-left-right inverted image to the tear hill edema model, respectively, and the predicted value for the eyelid redness is the left-right inverted image and the left-right inversion process. The prediction value for the eyelid edema is determined based on the result values obtained by inputting the non-reversed image into the eyelid edema model, respectively, and the predicted value for the eyelid edema is obtained by combining the horizontally inverted image and the left-right inverted image into the eyelid edema model. It may be determined based on the result values obtained by inputting each.

In some embodiments, the method resizes the first processed left eye image and the first processed right eye image, and the second processed left eye image. Resizing the second processed left eye image and the second processed right eye image may be further included.

According to the present application, a system for predicting a user's clinical activity score (Clinical Activity Score, CAS) for thyroid eye disease and providing guidance on the need for a user's visit to a hospital based thereon is disclosed.

1. Full system

(1) Hardware configuration of the system

1 is a diagram illustrating a system for predicting a clinical activity score related to thyroid eye disease according to an embodiment disclosed by the present application.

Referring to FIG. 1 , the system 1 includes a plurality of user terminals 10 and a server 20 .

Hereinafter, the plurality of user terminals 10 and the server 20 will be described in more detail.

(2) User terminal function

The plurality of user terminals 10 transmit information to the server 20 through various networks and also receive information from the server 20 .

The plurality of user terminals 10 are images of the user's upper and lower eyelids and eyeballs exposed to the outside by the upper and lower eyelids (hereinafter referred to as eye images). ) may be obtained, and necessary processing of the obtained eye image may be performed, or the obtained eye image or the processed eye image may be transmitted to the server 20 .

The plurality of user terminals 10 may receive, from the server 20 , prediction results related to clinical activity scores processed by the server 20 .

(3) Server function

The server 20 transmits information to the plurality of user terminals 10 through various networks and also receives information from the plurality of user terminals 10 .

The server 20 may receive the eye image from the plurality of user terminals 10 . At this time, the server 20 may process the eye image. Alternatively, the server 20 may receive the processed eye image.

The server 20 may obtain a prediction result for a clinical activity score related to the user's thyroid eye disease based on the processed eye image.

The server 20 may transmit the prediction result for the clinical activity score to the plurality of user terminals 10 .

(4) Software configuration of the system

In order for the system 1 to operate, several software configurations are required.

To perform communication between the user terminals 10 and the server 20, terminal software needs to be installed in the plurality of user terminals 10, and server software needs to be installed in the server 20. need to be

Various pre-processing algorithms may be used to perform the necessary pre-processing of the eye image.

A plurality of learning models for predicting a clinical activity score based on the preprocessed eye image may be used.

The plurality of preprocessing algorithms may be driven by terminal software installed in the user terminals 10 or may be driven by software installed in the server 20 . Alternatively, some of the plurality of pre-processing algorithms may be executed by the user terminals 10 and other parts may be executed by the server 20 .

The plurality of learning models may be driven by software installed in the server 20 . Alternatively, the plurality of learning models may be driven by terminal software installed in the user terminals 10 . Alternatively, some of the plurality of learning models may be executed by the user terminals 10 , and other portions may be executed by the server 20 .

(5) Components of user terminal

2 is a block diagram of a user terminal disclosed by the present application.

Referring to FIG. 2 , a user terminal 10 disclosed by the present application includes an output unit 110 , a communication unit 120 , a memory 130 , a camera 140 and a controller 150 .

The output unit 110 outputs various types of information according to the control command of the controller 150 . According to one embodiment, the output unit 110 may include a display 112 that visually outputs information to a user. Alternatively, although not shown in the drawings, a speaker that aurally outputs information to the user and a vibration motor that tactilely outputs information to the user may be included.

The communication unit 120 may include a wireless communication module and/or a wired communication module. Here, the wireless communication module may include a Wi-Fi communication module, a cellular communication module, and the like.

The memory 130 stores execution codes that can be read by the controller 150, processed result values, and necessary data. The memory 130 may include a hard disk drive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM, and the like. The memory 130 may store the aforementioned terminal software and may store execution codes for implementing various preprocessing algorithms and/or learning models described above. Furthermore, the memory 130 may store the eye image obtained through the camera 140 and the preprocessed eye image.

The camera 140 is a digital camera and may include an image sensor and an image processing unit. An image sensor is a device that converts an optical image into an electrical signal, and may be composed of a chip in which a plurality of photo diodes are integrated. Illustratively, the image sensor may include a Charge Coupled Device (CCD), Complementary Metal Oxide Semiconductor (CMOS), and the like. Meanwhile, the image processing unit may generate image information by processing a photographed result.

The controller 150 may include at least one processor. At this time, each processor may execute a predetermined operation by executing at least one command stored in the memory 130 . Specifically, the controller 150 may process information according to terminal software, a preprocessing algorithm, and/or a learning model driven in the user terminal 10 . Meanwhile, the controller 150 controls overall operations of the user terminal 10 .

Although not shown in the drawing, the user terminal 10 may include a user input unit. The user terminal 10 can receive input from a user through the user input unit for various information necessary for the operation of the user terminal 10 .

(6) Components of the server

3 is a block diagram of a server disclosed by the present application.

Referring to FIG. 3 , a server 20 disclosed by the present application includes a communication unit 210 , a memory 220 and a controller 230 .

The communication unit 210 may include a wireless communication module and/or a wired communication module. Here, the wireless communication module may include a Wi-Fi communication module, a cellular communication module, and the like.

The memory 220 stores execution codes that can be read by the controller 230, processed result values, and necessary data. The memory 220 may include a hard disk drive (HDD), solid state disk (SSD), silicon disk drive (SDD), ROM, RAM, and the like. The memory 220 may store the above-described server software and may store execution codes for implementing the above-described various preprocessing algorithms and/or learning models. Furthermore, the memory 220 may store the eye image received from the user terminal 10 and the preprocessed eye image.

The controller 230 may include at least one processor. At this time, each processor may execute a predetermined operation by executing at least one command stored in the memory 220 . Specifically, the controller 230 may process information according to server software running in the server 20, a preprocessing algorithm, and/or a learning model. Meanwhile, the controller 230 controls overall operations of the server 20 .

Hereinafter, in order to more clearly and easily understand the technology disclosed by this application, the eye, eyeball, and upper eyelid, lower eyelid, and lacrimal caruncle are included. The tissues around the eyeball are briefly described, and terms related to the eye and its surroundings used herein are defined.

2. Structure of the eye and definition of terms

(1) Eyeball and surrounding tissue

FIG. 4 is a diagram for explaining an eye exposed to the outside and its surrounding tissue so that it can be captured by a camera when a face is photographed using a camera.

4, the eyelids (upper eyelid), lower eyelid, lacrimal caruncle, and the conjunctiva and cornea (conjunctiva) partially covered and partially exposed by the upper eyelid, lower eyelid, and lacrimal caruncle ( cornea) is shown.

Typically the eye or eyeball is larger than shown in FIG. 4 . However, the eyeball is protected from the outside by tissues such as the upper and lower eyelids, and accordingly, only a part of the eyeball is exposed to the outside even when the person's eyes are open.

(2) Definition of terms

conjunctiva, white

Hereinafter, since the conjunctiva generally corresponds to the position of the white of the eye, the terms conjunctiva and white of the eye may be used interchangeably.

cornea, iris

Hereinafter, since the cornea generally corresponds to the position of the iris of the eye, the terms cornea and iris may be used interchangeably. Meanwhile, in the present specification, the term 'iris' is used to include the pupil area.

eyelid

It is the upper and lower two folds of skin covering the anterior part of the eyeball. It is also called an eyelid. The upper eyelid is called the upper eyelid, and the lower eyelid is called the lower eyelid. The outer surface is made of skin, and the inner surface is made of conjunctiva. In between, there are muscles that move the eyelids and tissue that contains meibomian glands, which maintain the shape of the eyelids. have. The eyelids protect the eyeball and at the same time clean the eyeball with tears by blinking or make the cornea shiny and transparent.

eyebrows

Eyebrows are hairs that grow in an arch along the bony prominence above the eye.

eyelashes

It refers to hairs of about 10 mm in length that grow on the edge of the lower and upper eyelids.

eyeball exposed

Hereinafter, "eyeballs exposed to the outside" means a part that is not covered by the upper eyelid, the lower eyelid and the tear hill, that is, the part exposed to the outside by the upper eyelid, the lower eyelid and the tear hill when the person's eyes are open. . For example, the inside of the dotted line shown in FIG. 5 is referred to as “the eyeball exposed to the outside”.

outline of eye

Hereinafter, "eye outline" refers to the outline of a portion including both the eyeball and the tear hill region exposed to the outside when a person's eyes are open. That is, the outline of the area where the eyeball exposed to the outside and the tear hill are combined is referred to as the “eye outline”. For example, the dotted line shown in Fig. 6 is referred to as "eye outline".

Externally exposed cornea (exposed iris)

Hereinafter, "exposed cornea" refers to a portion of the cornea that is not covered by the upper and lower eyelids when the person's eyes are open, that is, a portion of the cornea exposed to the outside by the upper and lower eyelids. For example, the inside of the dotted line shown in FIG. 7 is referred to as “exposed cornea”.

Exposed conjunctiva (exposed white of the eye)

Hereinafter, "conjunctiva exposed to the outside" refers to the part of the conjunctiva that is not covered by the upper eyelid, the lower eyelid, and the tear hill, that is, the part of the conjunctiva exposed to the outside by the upper eyelid, the lower eyelid, and the tear hill, when the person's eyes are open. it means. For example, the inside of the dotted line shown in FIG. 8 is referred to as "exposed conjunctiva".

Hereinafter, various image preprocessing algorithms for performing image preprocessing disclosed by the present application will be described.

3. Image pre-processing algorithm

(1) Necessity of image pre-processing

One object of the present application is to provide a learning model capable of predicting a clinical activity score related to thyroid eye disease by using images acquired by a digital camera usable by the general public rather than a professional medical diagnostic device.

To this end, in predicting the clinical activity score for thyroid ophthalmopathy, an image that the general public can easily acquire about the eyeball and tissues around the eyeball should be used. For example, digital images obtained by digital cameras that can be easily used by ordinary people or built-in smartphones should be used for image analysis, rather than digital images obtained by specialized medical devices that can be used in medical institutions. do.

Under these circumstances, it is difficult to standardize or standardize digital images acquired by users, etc., and in order to more accurately and quickly recognize digital images acquired by users, etc., it is necessary to perform various preprocessing on the acquired images. there is

(2) 1st crop (crop both eyes image)

Images used to predict the clinical activity score for thyroid ophthalmopathy should include the left and right eyes and their surrounding areas.

However, for faster and more accurate analysis, several areas unnecessary for image analysis (eg, areas corresponding to the nose, mouth, forehead, etc.) , It is more effective to use images in which only both eyes and their surrounding areas are captured (hereinafter, both eyes images).

Therefore, it is necessary to cut out an image (hereinafter, both eyes image) including both eyes (left eye/right eye) from an image in which the entire face acquired by the user is captured (hereinafter, face image).

For example, as shown in FIG. 9(b) from the face image obtained by the user shown in FIG. Hereinafter, acquiring a binocular image from a face image obtained by a user in this way is referred to as a binocular image cropping or first cropping.

(3) Necessity of applying an additional cropping method

The inventors of the present application have tried to build a system for predicting scores for 5 items related to thyroid eye disease using the prediction models to be described later using the first cropped image (both eyes image) described above, but the accuracy of prediction is poor. It was confirmed that it is not high.

Accordingly, the inventors of the present application have determined that the reason for the low prediction accuracy is that there are many areas unnecessary for analysis even in the binocular image, and thus it is necessary to secure a more detailed cropped image. That is, as shown in (a) of FIG. 10, rather than using a binocular image in which the left eye and the right eye are included in one image, as shown in (b) of FIG. 10, the left eye image and the right eye image are respectively secured. It was decided that using it would be more effective.

(4) Necessity of applying different cropping methods

It has already been described that five of the seven items for evaluating the clinical activity score for thyroid eye disease are items evaluated according to the doctor's visual observation of the user's eyeball and its surrounding area. The above five items are as follows.

1) Redness of conjunctiva,

2) Swelling of conjunctiva,

3) Swelling of lacrimal caruncle,

4) Redness of eyelid, and

5) Swelling of eyelid,

As will be described below, in order to evaluate the clinical activity score for thyroid ophthalmia provided by the present application, predictive models independent of each other were applied to the above five symptoms.

Therefore, there may be a method of utilizing images with different cropping methods applied to five independent prediction models, but the inventors of the present application use an image cropping method for analyzing the conjunctiva and the tear hill and an image cropping method for analyzing the eyelid. It was judged that sufficient prediction accuracy could be obtained by applying .

(5) 2nd Crop (Eye Outline Base Crop)

Hereinafter, an eye-outline-based crop (second crop) will be described. The second cropping can be applied to both the right eye image and the left eye image, but for convenience of explanation, it will be described based on securing the right eye cropped image.

Purpose of the 2nd Crop

The purpose of the second crop is a model for predicting redness of conjunctiva, a model for predicting swelling of conjunctiva, and a swelling of conjunctiva among prediction models to be described later. lacrimal caruncle) has the purpose of generating an image to be used as an input image to a model for predicting whether or not, and to generate an image in which information on the cornea and tear hill is maximized and information on other areas is minimized. It has a purpose.

input image

The second cropping may be applied to the face image or may be applied to the binocular image (first cropped image).

Eye outline detection

According to an embodiment, in order to detect the outline of the right eye, pixels corresponding to the boundary between the upper eyelid and the eyeball and the boundary between the lower eyelid and the eyeball may be detected. In addition, in order to detect the outline of the right eye, pixels corresponding to points where the upper and lower eyelids meet each other may be detected. Furthermore, in order to detect the outline of the right eye, pixels corresponding to the tear hill may be detected.

According to another embodiment, outline pixels corresponding to the outermost periphery of the eye outline may be detected using an eye outline segmentation model to be described later.

Check the maximum and minimum values of the X,Y coordinates of the outline pixels

By checking the detected pixels, the maximum value of X coordinate values (X _max ), the minimum value of X coordinate values (X _min ), the maximum value of Y coordinate values (Y _max ), and the minimum value of Y coordinate values (Y _min ) are checked. do.

11 is a diagram illustrating X _max , X _min , Y _max , and Y _min of outline pixels.

Crop area determination

Based on X _max , X _min , Y _max , and Y _min of the identified outline pixels, a rectangle having the following four points as vertices is created, and an area included inside the rectangle is determined as a cropped area. .

(X _min , Y _max ),

(X _max , Y _max ),

(X _max , Y _min ), and

(X _min , Y _min )

12 is a diagram illustrating a determined second cropped area.

As described above, the second cropped area may be determined in the same way for the left eye.

Generation of the second cropped image

When the second crop area is determined, based on the determined second crop area, as shown in FIG. Two cropped images (a second cropped right eye image and a second cropped left eye image) may be generated.

Hereinafter, the term “second cropped right eye image” may be used interchangeably with the term “right eye outline cropped image”, and the term “second left eye cropped image” may be interchanged with the term “left eye cropped outline image”.

In addition, without special mention below, the term "second cropped image (or outline cropped image)" may mean either one of a second right eye cropped image and a second left eye cropped image, or both depending on the context. have.

The second cropped image means an image cropped based on the 'eye outline', and even if it is a cropped image generated according to a method different from the above method, the uppermost and lowermost sides of the 'eye outline' , rightmost, and leftmost pixels are generated to be included in the cropped area, it should be referred to as a second cropped image (outline cropped image).

On the other hand, since the size and direction of the X coordinate value and Y coordinate value in this application vary according to the relative position to the reference point, the terms maximum value and minimum value should be understood in a relative sense, not in an absolute sense. It won't. That is, as the position of the origin of the coordinate system is changed, the maximum value of the above-described X coordinate value may be the minimum value of the X coordinate value in the coordinate system whose origin is changed, and the minimum value of the X coordinate value is the X coordinate value in the coordinate system whose origin is changed. It may be the maximum value of coordinate values. This can also be applied to the Y coordinate value in the same way.

(6) 3rd crop (crop including eyelids)

Hereinafter, an eyelid-included crop (third crop) will be described. The third crop can be applied to both the right eye image and the left eye image, but for convenience of explanation, it will be described based on securing the right eye cropped image.

Purpose of the third crop

The purpose of the third crop is an image to be used as an input image for a model for predicting redness of eyelids and a model for predicting swelling of eyelids among prediction models to be described later. It has the purpose of generating, and the purpose is to allow information about the eyelid to be included in the image. At this time, it may be better to generate a cropped image so that all pixels included in the eye outline may be included rather than cropping only pixels corresponding to the eyelid. This is because color values must be inferred and determined in order to predict redness of the eyelids, and at this time, color values of pixels corresponding to the iris and/or the white of the eye can be used.

input image

The third cropping may be applied to the face image or may be applied to the binocular image (first cropped image).

Eye outline detection and checking of maximum and minimum values of X,Y coordinates of outline pixels

According to an embodiment, the eye outline detection method described in the second crop may be applied as it is, outline pixels corresponding to the outermost periphery of the eye outline may be detected, and the detected pixels may be checked to determine the X coordinate. The maximum value of values (X _max ), the minimum value of X coordinate values (X _min ), the maximum value of Y coordinate values (Y _max ), and the minimum value of Y coordinate values (Y _min ) can be checked.

Crop Area Determination #1

Based on the identified Y _max value and Y _min value, a first expansion value (Y _e ) determined according to a predetermined criterion is added to the Y _max value, and a second expansion value (determined according to a predetermined criterion) After Y _e ') is subtracted from the Y _min value, the third cropped area may be determined similarly to the above-described method for determining the second cropped area.

That is, a rectangle having the following four points as vertexes may be created, and an area included in the rectangle may be determined as the third cropping area.

(X _min , Y _max +Y _e ),

(X _max , Y _max +Y _e ),

(X _max , Y _min -Y _e ') and

(X _min , Y _min -Y _e ')

When the third cropped area is determined in this way, more pixels corresponding to the upper and lower eyelids can be included in the image compared to the second cropped area.

In this case, the first extension value and the second extension value may be equal to each other, but do not necessarily have to be equal to each other.

Meanwhile, the criteria for determining the first extension value and the second extension value may be the same, but do not necessarily have to be the same.

The first extension value and the second extension value may be determined based on the size of the second crop area. For example, the first extension value and the second extension value may be determined by calculating lengths obtained by multiplying the horizontal length of the second cropped region by an extension ratio, and determining the number of pixels corresponding to the calculated lengths. For another example, the first extension value and the second extension value may be determined by calculating lengths obtained by multiplying the extension ratio to the vertical length of the second cropped area, and determining the number of pixels corresponding to the calculated lengths.

At this time, the specific ratio is 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60% can be one

An expansion ratio used when determining the first expansion value and an expansion ratio used when determining the second expansion value may be the same, but may not necessarily be the same or may be different from each other.

If the horizontal length of the second crop area is used when determining the first extension value, the horizontal length may also be used when determining the second extension value, but is not limited thereto, and determines the second extension value. When the vertical length can be used.

Crop Area Determination #2

Based on the identified X _max value and X _min value, a first width expansion value (X _we ) is determined according to a predetermined criterion, and a second horizontal expansion value (X _we ') is determined according to a predetermined criterion. decide

Based on the checked Y _max value and Y _min value, a first vertical expansion value (Y _he ) determined according to a predetermined criterion is determined, and a second vertical expansion value (Y _he ) is determined according to a predetermined criterion. ') to determine

A value obtained by adding the first horizontal expansion value (X _we ) to an X _max value, a value obtained by subtracting the second horizontal expansion value (X _we ') from an X _min value, and a value obtained by subtracting the Y _max value from the first vertical expansion value (Y _he ) and a value obtained by subtracting the second vertical expansion value (Y _he ') from the Y _min value, the third cropped area may be determined similarly to the above-described method for determining the second cropped area.

That is, a rectangle having the following four points as vertexes may be created, and an area included in the rectangle may be determined as the third cropping area.

(X _min - X _we ', Y _max + Y _he ),

(X _max +X _we , Y _max +Y _he ),

(X _max +X _we , Y _min -Y _he ') and

(X _min- X _we ', Y _min -Y _he ')

When the third cropped area is determined in this way, more pixels corresponding to the upper and lower eyelids can be included in the image compared to the second cropped area.

Furthermore, more pixels can be included in the cropped image in the left and right than in the cropped image by the "determination of crop area #1" method. As a result, more information about the upper and lower eyelids can be included in the cropped image. Because the widths of the upper and lower eyelids are generally wider than the widths of the eyeballs exposed to the outside, pixels corresponding to the upper and lower eyelids are increased through not only vertical expansion but also left and right expansion.

In this case, the first horizontal extension value and the second horizontal extension value may be equal to each other, but do not necessarily have to be equal to each other.

Meanwhile, criteria for determining the first vertical expansion value and the second vertical expansion value may be the same, but do not necessarily have to be the same.

The first horizontal expansion value and the second horizontal expansion value may be determined based on the size of the second cropped area. For example, the first horizontal expansion value and the second horizontal expansion value may be determined by calculating a length obtained by multiplying the horizontal length of the second cropped area by an expansion ratio, and determining the number of pixels corresponding to the calculated length. have. For another example, the first horizontal expansion value and the second horizontal expansion value may be determined by calculating a length obtained by multiplying the expansion ratio to the vertical length of the second cropped region, and determining the number of pixels corresponding to the calculated length. have.

At this time, the specific ratio is 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60% can be

An expansion ratio used when determining the first horizontal expansion value and an expansion ratio used when determining the second horizontal expansion value may be the same, but may be different from each other.

If the horizontal length of the second crop area is used when determining the first horizontal expansion value, the horizontal length may also be used when determining the second horizontal expansion value, but is not limited thereto, and the second expansion value When determining , the vertical length may be used.

Meanwhile, a method of determining the first and second vertical extension values is the same as the method of determining the first extension value and the second extension value, and thus a detailed description thereof will be omitted.

Generation of third cropped image

When the third crop area is determined, as shown in FIG. 14 , based on the determined third crop area, pixels included in the second crop area determined as described above from the facial image or the both eyes image are used to generate the third crop area. Three cropped images (a third cropped right eye image and a third cropped left eye image) may be generated.

For reference, (a) of FIG. 14 is a third cropped image cropped by the method of 'determining crop area #1' described above, and (b) of FIG. 14 is a method of 'determining crop area #2' described above. This is the third cropped image cropped by

Hereinafter, the term "third right eye cropped image" may be used interchangeably with the term "right eyelid-included-cropped image", and the term "third left eye cropped image" may be interchanged with the term "left eyelid-included-cropped image". Can be used interchangeably with "left eyelid-included-cropped image".

In addition, without special mention below, the term “third cropped image (or eyelid-including cropped image)” may refer to either a third right eye cropped image or a third left eye cropped image, or both depending on the context. can

The third cropped image means an image generated so that information about the eyelid is included in the image, and even if the cropped image is generated according to a method different from the above method, the cropped area so that pixels corresponding to the eyelid can be additionally included If the boundary of is determined, it should be referred to as a third cropped image (an eyelid-included cropped image).

(7) iris segmentation

Hereinafter, iris segmentation will be described.

Iris segmentation may be performed by a model that distinguishes a region corresponding to the iris or cornea from an image of the eyeball and its surroundings.

Using iris segmentation, as shown in FIG. 15 , it is possible to infer pixels corresponding to an iris in an image.

Using iris segmentation, as shown in FIG. 16 , it is possible to infer pixels corresponding to an externally exposed iris in an image.

The model for iris segmentation receives a face image as input data, outputs '1' for pixels inferred to be pixels corresponding to the iris in the face image, and outputs '0' for other pixels. can be printed out.

The iris segmentation model may be learned using training data including a face image and an image in which the pixel value of a pixel corresponding to the iris in the face image is '1' and the pixel value of the remaining pixels is '0'.

However, although it has been described that iris segmentation is performed by receiving a face image as input data, it is also possible to perform iris segmentation by using the above-described binocular image as input data.

(8) Eye outline segmentation

In the following, outline segmentation of the eye will be described.

Eye outline segmentation may be performed by a model that distinguishes a region corresponding to the inside of the eye outline from an image of the eye and its surroundings.

Using the eye outline segmentation, as shown in FIG. 17 , it is possible to infer pixels corresponding to the inside of the eye outline in the image.

The model for eye outline segmentation receives a face image as input data, outputs '1' for pixels inferred to be pixels corresponding to the inside of the eye outline in the face image, and outputs '1' for other pixels. For , '0' can be output.

The eye outline segmentation model uses training data including a face image and an image in which the pixel value of a pixel corresponding to the inside of the eye outline in the face image is '1' and the pixel value of the remaining pixels is '0'. so it can be learned.

(9) Masking

1st masking

In the present application, the first masking means leaving pixels corresponding to the conjunctiva and the tear hill in the image and removing information reflected in pixel values corresponding to other regions.

The meaning of removing information reflected in pixel values may mean changing pixel values of pixels whose information is to be removed to predetermined specific values. For example, pixel values of pixels whose information is to be removed may all be changed to 0.

The first masking may be performed before inputting an image to a model for predicting symptoms related to the conjunctiva and lacrimal hill among prediction models for predicting a clinical activity score related to thyroid ophthalmopathy.

2nd masking

In this application, the second masking means leaving pixels corresponding to the externally exposed cornea (externally exposed iris) in the image and removing information reflected in pixel values corresponding to other regions. do.

The second masking may be performed before the image is input to a model for predicting symptoms related to the eyelids (upper and lower eyelids) among prediction models for predicting the clinical activity score related to thyroid ophthalmopathy.

Method of the first masking

The first masking may be performed on a first masking target image selected from any one of a face image, a first cropped image (both eyes image), and a second cropped image (outline cropped image).

A first masking image may be generated based on the first masking target image, the eye outline segmentation result, and the iris segmentation result. For example, values of pixels other than pixels corresponding to the inside of the eye outline and pixels corresponding to iris (or iris exposed to the outside) from the first masking target image are removed. can do.

18 is an exemplary diagram of a first masking image.

Second masking method

Second masking may be performed on a second masking target image selected from any one of a face image, a first cropped image (both eyes image), and a third cropped image (eyelid-included cropped image).

A second masking image may be generated based on the second masking target image, the eye outline segmentation result, and the iris segmentation result. For example, pixel values of pixels corresponding to the inside of the eye outline and corresponding to the iris (or the iris exposed to the outside) may be removed from the second masking target image.

19 is an exemplary diagram of a second masking image.

Another embodiment of the first masking

According to the above description, the first masking has been described as leaving pixels corresponding to the conjunctiva and the tear hill and removing all pixel values of pixels corresponding to other regions, but if necessary, pixels corresponding to the cornea (iris) Pixel values of may not be removed.

However, since the color of the iris may have different colors depending on race, it may be advantageous for faster learning and higher accuracy to remove pixel values of pixels corresponding to the iris.

Second masking selectivity

According to the above description, the second masking has been described as removing all pixel values of pixels corresponding to the iris, but the second masking may not be performed at all.

However, since the color of the iris may have different colors according to race, it may be advantageous for faster learning and higher accuracy to remove pixel values of pixels corresponding to the iris by performing the second masking.

(10) Left and right reversal

Necessity of left-right reversal

According to the method for predicting the clinical activity score for thyroid ophthalmopathy provided by the present application, cropped images for each of the left and right eyes are used instead of using images of both eyes.

On the other hand, the outline of the eye is asymmetrical. For example, based on the right eye, there is a tear hill at the left end of the right eye, but there is a point where the upper and lower eyelids naturally meet at the right end of the right eye.

Accordingly, for faster learning and more accurate prediction, it is more effective to distinguish between a learning model learned based on the right eye and a learning model learned based on the left eye.

However, due to the line symmetry between the left and right eyes, when the left eye is reversed to the right eye, the shape characteristics of the right eye and the left eye become similar to each other.

Accordingly, according to an embodiment of the present application, if one of the right eye and the left eye is used without horizontal reversal and the other eye is used with left and right reversal, only one learning model can be used.

How to reverse left and right

Converting left and right of an image means that, when the left and right reference lines dividing the image into left and right halves crossing the image to be reversed vertically and X = a, When the first pixel value corresponds to the a+Δ, Y) pixel and the second pixel value corresponds to the (a-Δ, Y) pixel, the pixel value of (a+Δ, Y) is subtracted from the first pixel value. It means changing to 2 pixel values and changing the pixel value of (a-Δ, Y) from the second pixel value to the first pixel value.

Left/Right Flip Target Image

It is sufficient to invert only one of the images for the left eye and the image for the right eye. Which one of the left-eye image and the right-eye image is to be reversed is determined according to which one of the left-eye image and the right-eye image is learned as a standard when learning predictive models to be described later.

Meanwhile, an image for which both masking and cropping (second cropping or third cropping) has been completed may be horizontally reversed, but an image in which only cropping is completed and masking is not completed may be horizontally reversed.

20 to 22 are diagrams illustrating various examples of an original image and a horizontally reversed image.

Selectivity of left-right reversal

However, since left-right reversal is introduced to unify the prediction model for the left eye and the prediction model for the right eye as described above, if the prediction model for the left eye and the prediction model for the right eye are implemented as different models, , the left-right reversal preprocessing can be omitted.

(11) Resizing

Need for resizing

As described above, when an image is cropped and used based on an eye outline, each person has a different eye size, so the cropped image has a different size for each person.

Meanwhile, when the left eye image and the right eye image are obtained by independently cropping, the left eye cropped image and the right eye cropped image of the same person are different from each other due to a size difference between the left eye and the right eye.

For this reason, it is necessary to resize the size of the eye image to a standard size corresponding to each predictive model before inputting an eye image to a predictive model to be described later.

Standard size for each prediction model

Standard sizes corresponding to the first to fifth prediction models may be different from each other.

Standard sizes corresponding to prediction models using the second cropped image as an input image may be the same.

Standard sizes corresponding to prediction models using the third cropped image as an input image may be the same.

A standard size corresponding to prediction models using the second cropped image as an input image may be different from a standard size corresponding to prediction models using the third cropped image as an input image.

Alternatively, all standard sizes corresponding to the first to fifth prediction models may be the same.

How to resize

Adjusts the size of the image to be resized to the standard size.

When the width or height of the image to be resized is larger than the width or height of the standard size, the width or height of the image to be resized may be increased.

When the width or height of the image to be resized is smaller than the width or height of the standard size, the width or height of the image to be resized may be reduced.

During resizing, an aspect ratio of an image before resizing may be different from an aspect ratio of an image after resizing.

4. Prediction Model

(1) 1st prediction model

Purpose and operation of the first predictive model

The first predictive model is a model for predicting whether or not conjunctiva is congested.

The first predictive model may receive an eye image as input data and output a probability value that the conjunctiva captured in the input eye image is congested.

When the first prediction model includes a first left eye prediction model and a first right eye prediction model, the first left eye prediction model receives a left eye image and outputs a probability value that the conjunctiva captured in the left eye image is congested, and The right eye prediction model may receive a right eye image and output a probability value that the conjunctiva captured in the right eye image is congested.

When the first predictive model is not binary but implemented as a single model, the first predictive model receives one of the right eye image and the left eye image and outputs a probability value that the conjunctiva captured in the input image is congested, and the other one A probability value that the conjunctiva captured in the input image is congested may be output.

The eye image may be an image pre-processed by the above-described pre-processing algorithms.

For example, the eye image may be an image preprocessed according to the second cropping.

For another example, the eye image may be a preprocessed image including second cropping and resizing.

As another example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

As another example, the eye image may be a preprocessed image including second cropping, first masking, horizontal reversal, and resizing.

In the present specification, the first prediction model may be referred to as a conjunctival congestion prediction model.

Learning of the first predictive model

In order to train the first prediction model, a plurality of training data sets may be prepared. The training data set may include an eye image and an evaluation value for conjunctival congestion captured in the eye image. The eye image may be an image pre-processed by the above-described pre-processing algorithm. For example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

In order to train the first prediction model, an artificial intelligence model may be prepared.

The artificial intelligence model may be a Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

Subsequently, eye images included in a plurality of training data sets prepared for the artificial intelligence model may be input, and learning may be performed using an evaluation value corresponding to each of the input eye images and an output value output from the artificial intelligence model.

If the first prediction model includes the first left eye prediction model and the first right eye prediction model, the plurality of learning data sets for learning the first left eye prediction model are the left eye image and congestion of the conjunctiva captured in the left eye image , and the plurality of training data sets for training the second right eye prediction model may include a right eye image and an evaluation value regarding congestion of the conjunctiva captured in the right eye image. Meanwhile, in order to increase the number of learning data sets, the plurality of learning data sets for learning the first left eye prediction model may include a left-right inverted right eye image and an evaluation value for conjunctival congestion captured in the right eye image. In addition, the plurality of training data sets for learning the first right eye prediction model may include a left eye image processed by inverting left and right and evaluation values of conjunctival congestion captured in the left eye image.

If the first predictive model is to be implemented as a single model without being binary, the plurality of training data sets include a right eye image and an evaluation value for conjunctival congestion captured in the right eye image, or a left and right inverted left eye image and an evaluation value of congestion of the conjunctiva captured in the left eye image. Alternatively, the plurality of training data sets may include a left eye image and an evaluation value for conjunctival congestion captured in the left eye image, or a left-right inverted right eye image and an evaluation value for conjunctival congestion captured in the right eye image. have.

On the other hand, in learning the first predictive model, in order to be able to predict whether or not conjunctiva is congested without distinguishing between the right eye image and the left eye image, the right eye image, the left and right inverted right eye image, the left eye image and the left and right inverted left eye All images can be used as training data for training one model.

For example, when the first prediction model includes a first left eye prediction model and a first right eye prediction model, a plurality of training data sets for learning the first left eye prediction model include a left eye image and the conjunctiva captured in the left eye image may include an evaluation value for congestion of the right eye image, a left and right inverted right eye image, and an evaluation value for congestion of the conjunctiva captured in the right eye image, and a plurality of training data sets for training the first right eye predictive model include the right eye image and An evaluation value for conjunctival congestion captured in the right eye image, a left-right inverted left eye image, and an evaluation value for conjunctival congestion captured in the left eye image may be included.

If the first predictive model is to be implemented as a single model without being binary, the plurality of training data sets are the right eye image and the evaluation value of the conjunctival congestion captured in the right eye image, the left and right inverted right eye image, and the right eye An evaluation value for conjunctival congestion captured in an image, a left eye image and an evaluation value for conjunctival congestion captured in the left eye image, and a left-right inverted left eye image and an evaluation value for conjunctival congestion captured in the left eye image can include

(2) 2nd prediction model

Purpose and operation of the second predictive model

The second predictive model is a model for predicting whether or not the conjunctiva is edematous.

The second predictive model may receive an eye image as input data and output a probability value of existence of edema in the conjunctiva captured in the input eye image.

When the second predictive model includes the second left eye predictive model and the second right eye predictive model, the second left eye predictive model receives the left eye image and outputs a probability value that edema exists in the conjunctiva captured in the left eye image, and 2 The right eye prediction model may receive a right eye image and output a probability value of the presence of edema in the conjunctiva captured in the right eye image.

When the second predictive model is not binary but implemented as a single model, the second predictive model receives one of the right eye image and the left eye image and outputs a probability value of the existence of edema in the conjunctiva captured in the input image, and other It is possible to receive one input again and output a probability value of the presence of edema in the conjunctiva captured in the input image.

The eye image may be an image pre-processed by the above-described pre-processing algorithms.

For example, the eye image may be an image preprocessed according to the second cropping.

For another example, the eye image may be a preprocessed image including second cropping and resizing.

As another example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

As another example, the eye image may be a preprocessed image including second cropping, first masking, horizontal reversal, and resizing.

In the present specification, the second prediction model may be referred to as a conjunctival edema prediction model.

Learning of the second predictive model

In order to train the second predictive model, a plurality of training data sets may be prepared. The training data set may include an eye image and an evaluation value regarding whether edema exists in the conjunctiva captured in the eye image. The eye image may be an image pre-processed by the above-described pre-processing algorithm. For example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

In order to train the second prediction model, an artificial intelligence model may be prepared.

The artificial intelligence model may be a Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

Subsequently, eye images included in a plurality of training data sets prepared for the artificial intelligence model may be input, and learning may be performed using an evaluation value corresponding to each of the input eye images and an output value output from the artificial intelligence model.

If the second prediction model includes the second left eye prediction model and the second right eye prediction model, the plurality of learning data sets for learning the second left eye prediction model are the left eye image and swelling in the conjunctiva captured in the left eye image may include an evaluation value for whether there is edema in the right eye image and the conjunctiva captured in the right eye image, and the plurality of learning data sets for learning the second right eye prediction model may include an evaluation value for whether edema exists can On the other hand, in order to increase the number of learning data sets, a plurality of training data sets for learning the second left eye predictive model evaluates the presence of edema in the right eye image and the conjunctiva captured in the right eye image The plurality of training data sets for learning the second right eye prediction model may include a left eye image processed in a left-right inversion process and an evaluation value regarding whether edema is present in the conjunctiva captured in the left eye image.

If the second prediction model is to be implemented as one model without being binary, the plurality of training data sets include the right eye image and an evaluation value for whether or not edema exists in the conjunctiva captured in the right eye image, or invert left and right and an evaluation value regarding whether edema exists in the conjunctiva captured in the left eye image and the left eye image. Alternatively, the plurality of learning data sets include a left eye image and an evaluation value regarding whether or not edema is present in the conjunctiva captured in the left eye image, or a left and right inverted right eye image and whether edema exists in the conjunctiva captured in the right eye image. An evaluation value may be included.

On the other hand, in learning the second predictive model, in order to be able to predict whether or not the conjunctiva is edema without distinguishing between the right eye image and the left eye image, the right eye image, the left and right inverted right eye image, the left eye image and the left and right inverted left eye All images can be used as training data for training one model.

For example, when the second prediction model includes the second left eye prediction model and the second right eye prediction model, the plurality of learning data sets for learning the second left eye prediction model include the left eye image and the conjunctiva captured in the left eye image may include an evaluation value for edema of the left and right inverted right eye image and an evaluation value for edema of the conjunctiva captured in the right eye image, and a plurality of training data sets for training the second right eye prediction model may include the right eye image and An evaluation value for conjunctival edema captured in the right eye image, a left-right inverted left eye image, and an evaluation value for conjunctival edema captured in the left eye image may be included.

If the second predictive model is to be implemented as one model without being binary, the plurality of learning data sets are the right eye image and the evaluation value of conjunctival edema captured in the right eye image, the left and right inverted right eye image, and the right eye An evaluation value for the edema of the conjunctiva captured in the image, an evaluation value for the edema of the conjunctiva captured in the left eye image and the left eye image, and an evaluation value for the edema of the conjunctiva captured in the left and right inverted left eye image and the left eye image can include

(3) The 3rd Prediction Model

Purpose and operation of the 3rd predictive model

The third predictive model is a model for predicting whether or not the tear hill is edematous.

The third predictive model may receive an eye image as input data and output a probability value of existence of edema in the tear hill captured in the input eye image.

When the third predictive model includes a third left eye predictive model and a third right eye predictive model, the third left eye predictive model receives a left eye image and outputs a probability value that edema exists in the tear hill captured in the left eye image, The third right eye predictive model may receive a right eye image and output a probability value of existence of edema in the tear hill captured in the right eye image.

When the third predictive model is not binary but implemented as a single model, the third predictive model receives one of the right eye image and the left eye image and outputs a probability value of the presence of edema in the tear hill captured in the input image, Another input is received again, and a probability value of the presence of edema in the tear hill captured in the input image may be output.

The eye image may be an image pre-processed by the above-described pre-processing algorithms.

For example, the eye image may be an image preprocessed according to the second cropping.

For another example, the eye image may be a preprocessed image including second cropping and resizing.

As another example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

As another example, the eye image may be a preprocessed image including second cropping, first masking, horizontal reversal, and resizing.

In this specification, the third prediction model may be referred to as a tear hill edema prediction model.

Learning of the 3rd predictive model

In order to train the third predictive model, a plurality of training data sets may be prepared. The training data set may include an eye image and an evaluation value regarding whether edema exists in the tear hill captured in the eye image. The eye image may be an image pre-processed by the above-described pre-processing algorithm. For example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

In order to train the third predictive model, an artificial intelligence model may be prepared.

The artificial intelligence model may be a Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

Subsequently, eye images included in a plurality of training data sets prepared for the artificial intelligence model may be input, and learning may be performed using an evaluation value corresponding to each of the input eye images and an output value output from the artificial intelligence model.

If the third prediction model includes the third left eye prediction model and the third right eye prediction model, the plurality of training data sets for learning the third left eye prediction model are the left eye image and the tear hill captured in the left eye image. It may include an evaluation value regarding whether edema exists, and the plurality of learning data sets for training the third right eye prediction model evaluates whether edema exists in the right eye image and the tear hill captured in the right eye image. can include On the other hand, in order to increase the number of learning data sets, a plurality of training data sets for learning the third left eye prediction model are the left and right inverted right eye image and the evaluation value of whether edema exists in the tear hill captured in the right eye image , and the plurality of training data sets for learning the third right eye prediction model may include a left eye image processed in a left-right inversion process and an evaluation value regarding whether edema exists in the tear hill captured in the left eye image. .

If the third predictive model is to be implemented as one model without being binary, the plurality of learning data sets may include the right eye image and an evaluation value regarding whether or not edema exists in the tear hill captured in the right eye image, or An inverted left eye image and an evaluation value regarding whether edema exists in the tear hill captured in the left eye image may be included. Alternatively, the plurality of training data sets include an evaluation value for whether edema exists in the left eye image and the tear hill captured in the left eye image, or whether there is edema in the right eye image and the tear hill captured in the right eye image. may include an evaluation value for

On the other hand, in learning the third predictive model, the right eye image, the left-right inverted right-eye image, the left-eye image, and the left-right inverted image are used to predict whether the tear hill is edema without distinguishing between the right eye image and the left eye image. All of the left eye images may be used as training data for training one model.

For example, when the third prediction model includes a third left eye prediction model and a third right eye prediction model, the plurality of training data sets for learning the third left eye prediction model include a left eye image and tears captured in the left eye image. The plurality of learning data sets for training the third right eye prediction model may include an evaluation value for swelling of the hill, a left-right inverted right eye image, and an evaluation value for swelling of the tear hill captured in the right eye image. It may include an image and an evaluation value for swelling of the tear hill captured in the right eye image, a left-right inverted left eye image, and an evaluation value for swelling of the tear hill captured in the left eye image.

If the third predictive model is to be implemented as one model without being binary, the plurality of training data sets are the right eye image and the evaluation value of the edema of the tear hill captured in the right eye image, the left and right inverted right eye image, and the right eye image. Evaluation values for edema of the tear hills captured in the right eye image, evaluation values for the edema of the tear hills captured in the left eye image and the left eye image, and left and right inverted left eye images and edema of the tear hills captured in the left eye image evaluation values may be included.

(4) The 4th predictive model

Purpose and operation of the 4th predictive model

The fourth predictive model is a model for predicting whether the eyelids are red.

The fourth predictive model may receive an eye image as input data and output a probability value of eyelid redness captured in the input eye image.

When the fourth prediction model includes a fourth left eye prediction model and a fourth right eye prediction model, the fourth left eye prediction model receives a left eye image and outputs a probability value of eyelid redness captured in the left eye image. The right eye prediction model may receive a right eye image and output a probability value of eyelid redness captured in the right eye image.

If the fourth predictive model is not binary but implemented as a single model, the fourth predictive model receives one of the right eye image and the left eye image and outputs a probability value of eyelid redness captured in the input image, It is possible to output a probability value of eyelid redness captured in the input image by receiving input again.

The eye image may be an image pre-processed by the above-described pre-processing algorithms.

For example, the eye image may be an image preprocessed according to the third crop.

For another example, the eye image may be a preprocessed image including third cropping and resizing.

As another example, the eye image may be a preprocessed image including third cropping, horizontal reversal, and resizing.

As another example, the eye image may be a preprocessed image including third cropping, second masking, and resizing.

As another example, the eye image may be a preprocessed image including third cropping, second masking, horizontal reversal, and resizing.

In the present specification, the fourth prediction model may be referred to as an eyelid redness prediction model.

Learning of the 4th predictive model

In order to train the fourth predictive model, a plurality of training data sets may be prepared. The training data set may include eye images and evaluation values of eyelid redness captured in the eye images. The eye image may be an image pre-processed by the above-described pre-processing algorithm. For example, the eye image may be a preprocessed image including second cropping, first masking, and resizing.

In order to train the fourth predictive model, an artificial intelligence model may be prepared.

The artificial intelligence model may be a Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

Subsequently, eye images included in a plurality of training data sets prepared for the artificial intelligence model may be input, and learning may be performed using an evaluation value corresponding to each of the input eye images and an output value output from the artificial intelligence model.

If the fourth prediction model includes the fourth left eye prediction model and the fourth right eye prediction model, the plurality of learning data sets for learning the fourth left eye prediction model are the left eye image and the redness of the eyelid captured in the left eye image , and the plurality of training data sets for training the second right eye prediction model may include a right eye image and an evaluation value for redness of the eyelid captured in the right eye image. Meanwhile, in order to increase the number of learning data sets, the plurality of learning data sets for learning the fourth left eye prediction model may include a right eye image processed in a left-right inversion process and an evaluation value for redness of the eyelid captured in the right eye image. In addition, the plurality of training data sets for learning the fourth right eye prediction model may include a left eye image processed by inverting the left eye and an evaluation value of redness of the eyelid captured in the left eye image.

If the fourth prediction model is to be implemented as a single model without being binary, the plurality of training data sets include the right eye image and the evaluation value of the redness of the eyelid captured in the right eye image, or the left and right inverted left eye image and an evaluation value of redness of the eyelid captured in the left eye image. Alternatively, the plurality of training data sets may include a left eye image and an evaluation value for redness of the eyelid captured in the left eye image, or may include an inverted right eye image and an evaluation value for redness of the eyelid captured in the right eye image. have.

On the other hand, in learning the first predictive model, the right eye image, the left-right inverted right-eye image, the left-eye image, and the left-right inverted left eye in order to be able to predict the redness of the eyelid without distinguishing between the right eye image and the left eye image. All images can be used as training data for training one model.

For example, when the fourth prediction model includes a fourth left eye prediction model and a fourth right eye prediction model, a plurality of learning data sets for learning the fourth left eye prediction model include a left eye image and an eyelid captured in the left eye image may include an evaluation value for redness of the right eye image, a left and right inverted right eye image, and an evaluation value for redness of the eyelid captured in the right eye image, and the plurality of training data sets for training the fourth right eye prediction model include the right eye image and An evaluation value for redness of the eyelid captured in the right eye image, a left eye image inverted left and right, and an evaluation value for redness of the eyelid captured in the left eye image may be included.

If the fourth predictive model is to be implemented as a single model without being binary, the plurality of training data sets are the right eye image and the evaluation value of redness of the eyelid captured in the right eye image, the left and right inverted right eye image, and the right eye An evaluation value for redness of the eyelids captured in an image, an evaluation value for redness of the eyelids captured in the left eye image and the left eye image, and an evaluation value for redness of the eyelids captured in the left-right inverted left eye image and the left eye image can include

(5) The 5th predictive model

Purpose and operation of the 5th predictive model

The fifth prediction model is a model for predicting whether or not the eyelid is edematous.

The fifth predictive model may receive an eye image as input data and output a probability value of existence of edema in the eyelid captured in the input eye image.

When the fifth predictive model includes a fifth left eye predictive model and a fifth right eye predictive model, the fifth left eye predictive model receives a left eye image and outputs a probability value of existence of edema in the eyelid captured in the left eye image, 5 The right eye prediction model may receive a right eye image and output a probability value of the existence of edema in the eyelid captured in the right eye image.

When the fifth predictive model is not binary but implemented as a single model, the fifth predictive model receives one of the right eye image and the left eye image and outputs a probability value of the presence of edema in the eyelid captured in the input image, and the other It is possible to receive one input again and output a probability value of the existence of edema in the eyelid captured in the input image.

The eye image may be an image pre-processed by the above-described pre-processing algorithms.

For example, the eye image may be an image preprocessed according to the third crop.

For another example, the eye image may be a preprocessed image including third cropping and resizing.

As another example, the eye image may be a preprocessed image including third cropping, horizontal reversal, and resizing.

As another example, the eye image may be a preprocessed image including third cropping, second masking, and resizing.

As another example, the eye image may be a preprocessed image including third cropping, second masking, horizontal reversal, and resizing.

In this specification, the fifth prediction model may be referred to as an eyelid edema prediction model.

Learning of the 5th predictive model

In order to train the fifth predictive model, a plurality of training data sets may be prepared. The training data set may include an eye image and an evaluation value regarding whether edema exists in the eyelid captured in the eye image. The eye image may be an image pre-processed by the above-described pre-processing algorithm. For example, the eye image may be a preprocessed image including third cropping, second masking, and resizing.

In order to train the fifth prediction model, an artificial intelligence model may be prepared.

The artificial intelligence model may be a Support Vector Machine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG, GoogLeNet, MobileNet, and Vision Transformer.

Subsequently, eye images included in a plurality of training data sets prepared for the artificial intelligence model may be input, and learning may be performed using an evaluation value corresponding to each of the input eye images and an output value output from the artificial intelligence model.

If the fifth prediction model includes the fifth left eye prediction model and the fifth right eye prediction model, the plurality of learning data sets for learning the fifth left eye prediction model are the left eye image and swelling on the eyelid captured in the left eye image The plurality of training data sets for learning the fifth right eye prediction model may include an evaluation value regarding whether edema exists in the right eye image and the eyelid captured in the right eye image. can On the other hand, in order to increase the number of learning data sets, a plurality of training data sets for learning the fifth left eye prediction model are the left and right inverted right eye image and the evaluation value of whether edema exists in the eyelid captured in the right eye image In addition, the plurality of training data sets for learning the fifth right eye prediction model may include a left eye image processed by inverting the left eye and an evaluation value regarding whether edema is present in the eyelid captured in the left eye image.

If the fifth prediction model is to be implemented as one model without being binary, the plurality of training data sets include the right eye image and the evaluation value of whether or not edema exists in the eyelid captured in the right eye image, or invert left and right and an evaluation value regarding whether edema exists in the eyelid captured in the left eye image and the left eye image. Alternatively, the plurality of training data sets include a left eye image and an evaluation value regarding whether edema exists in the eyelid captured in the left eye image, or a right eye image reversed left and right and whether edema exists in the eyelid captured in the right eye image. An evaluation value may be included.

On the other hand, in learning the fifth predictive model, the right eye image, the left-right inverted right-eye image, the left-eye image, and the left-right inverted left eye in order to be able to predict the edema of the eyelid without distinguishing between the right eye image and the left eye image. All images can be used as training data for training one model.

For example, when the fifth prediction model includes a fifth left eye prediction model and a fifth right eye prediction model, the plurality of learning data sets for learning the fifth left eye prediction model include a left eye image and an eyelid captured in the left eye image may include an evaluation value for edema, a right-left inverted right eye image, and an evaluation value for edema of the eyelid captured in the right eye image, and the plurality of training data sets for training the fifth right eye prediction model include the right eye image and An evaluation value for eyelid edema captured in the right eye image, a left-right inverted left eye image, and an eyelid edema evaluation value captured in the left eye image may be included.

If the fifth prediction model is to be implemented as a single model without being binary, the plurality of training data sets are the right eye image and the evaluation value of the eyelid edema captured in the right eye image, the left and right inverted right eye image, and the right eye An evaluation value for the edema of the eyelid captured in the image, an evaluation value for the edema of the eyelid captured in the left eye image and the left eye image, and an evaluation value for the edema of the eyelid captured in the left and right inverted left eye image and the left eye image can include

Learning of the predictive models may be performed by an electronic device, and in particular, may be performed by the server 20 described above. In addition, the meaning of learning the predictive models by the electronic device or the server 20 means a series of processes that make the output value of the predictive model for the input data output a value similar to the output value labeled for the input data, , To this end, the electronic device or the server 20 may change the weight value of each node included in the prediction model using the difference between the output value of the prediction model and the labeling values. At this time, the electronic device or the server 20 may determine the amount of change in the weight value of each node using various feedback functions.

Hereinafter, an eye image is preprocessed through the above-described system (1), and the preprocessed eye image is input to the above-described prediction model to predict each symptom related to the thyroid ophthalmopathy clinical activity score, and each symptom A method of predicting the clinical activity score based on the prediction result of the clinical activity score, and furthermore, a method of monitoring the prediction result of the clinical activity score and guiding or recommending the user to visit a hospital to receive a checkup according to the monitoring result will be described.

5. Conjunctival congestion prediction method

The conjunctival congestion prediction method disclosed by the present application may be performed by the server 20 .

23 is a flowchart for explaining a method for predicting conjunctival congestion.

Referring to FIG. 23, the server 20 acquires a face image (S100), pre-processes the acquired face image (S110), and converts the pre-processed image to the above-described first prediction model (conjunctival congestion prediction model). input (S120), and an output value of the first predictive model is obtained (S130).

Acquisition of face image

The server 20 obtains a face image (S100). The server 20 may acquire the face image from the user terminal 10 .

Preprocessing of face images

The server 20 may pre-process the acquired face image (S110). The server 20 may perform the aforementioned iris segmentation, eye outline segmentation, masking, cropping, and resizing on the acquired face image.

segmentation processing

The server 20 performs the iris segmentation and the eye outline segmentation, and according to the result, it is possible to identify pixels corresponding to the iris and pixels corresponding to the inside of the eye outline in the acquired face image. The server 20 may check coordinate values of pixels corresponding to the iris and coordinate values of pixels corresponding to the inside of the eye outline.

masking treatment

The server 20 may process the first masking on the face image based on the information about the identified pixels. The server 20 may remove pixel values of pixels included in the face image except for pixels corresponding to the externally exposed conjunctiva and tear hill through the first masking process. Accordingly, pixel values of pixels corresponding to the conjunctiva and lacrimal hill of the left eye and the conjunctiva and lacrimal hill of the right eye may be maintained as original pixel values, but the iris (or cornea) of the left eye and the iris of the right eye may be maintained. (or the cornea), pixel values of pixels corresponding to outside the outline of the left eye and outside the outline of the right eye may be removed or changed to other values.

cropped

The server 20 may crop the masked face image. The server 20 may generate a left eye cropped image and a right eye cropped image by cropping the masked face image. When performing the conjunctival congestion prediction method, the server 20 may use the second cropping (eye outline cropping) method among the two cropping methods described above. Since the second cropping method has already been described in detail, a detailed description thereof will be omitted.

Resizing processing and left-right reversal processing

The server 20 may resize the left eye cropped image and the right eye cropped image to a predetermined size.

Meanwhile, when the first prediction model is implemented as a single model without dualization, the server 20 may horizontally invert one of the left-eye cropped image and the right-eye cropped image as described above. The server 20 does not horizontally reverse the other of the left eye cropped image and the right eye cropped image. At this time, the criterion for determining which of the left-eye image and the left-eye image is to be reversed is determined by the same criterion as the criterion applied when training the first prediction model. That is, when learning the first prediction model, if the left eye image is inverted and the right eye image is not inverted, the server 20 inverts the left eye image and does not invert the right eye image.

If, as described above, in implementing the first prediction model, if the first left eye prediction model and the first right eye prediction model are binary, the server 20 may not perform left-right reversal processing.

On the other hand, in performing the preprocessing, it has been described that segmentation, masking process, cropping process, resizing process, and horizontal reversal process are performed, but the order of each preprocessing can achieve the purpose of the method for predicting conjunctival congestion disclosed by the present application. It can be changed within scope.

Input of preprocessed image

The server 20 may input the preprocessed image to the first prediction model (S120).

When the first prediction model is implemented as a single model without dualization, the server 20 sequentially inputs the right eye preprocessing image and the left eye preprocessing image inverted to the left into the first prediction model.

If, in implementing the first prediction model, the first left eye prediction model and the first right eye prediction model are binary, the server 20 inputs the left eye preprocessing image to the first left eye prediction model, and the right eye preprocessing An image is input to the first right eye prediction model. Alternatively, the server 20 inputs the left eye preprocessing image to the first left eye prediction model, inputs the left-right inverted left eye preprocessing image to the first right eye prediction model, and inputs the right eye preprocessing image to the first right eye prediction model. A model, and the left-right inverted right-eye preprocessing image may be input to the first left-eye prediction model.

If, in implementing the first predictive model, the first predictive model is implemented as one model without being binary and at the same time learned to determine whether the conjunctiva is congested without distinguishing between the left eye image and the right eye image, the server (20) may input the left eye preprocessing image and the right eye preprocessing image to the first predictive model without horizontal reversal. Alternatively, the server 20 may input the left eye pre-processing image, the left-right inverted left-eye pre-processing image, the right-eye pre-processing image, and the left-right inverted right eye pre-processing image to the first prediction model.

Prediction of conjunctival hyperemia

The server 20 may obtain a result value output from the first predictive model (S130). The resulting value may be a predicted probability value that the conjunctiva captured in the image is congested. The server 20 determines that the conjunctiva is congested if the predicted probability value is greater than or equal to the threshold value based on a predetermined threshold value, and if the predicted probability value is less than the threshold value, the conjunctiva is congested. It can be judged that this is not congested.

The server 20 may obtain both prediction results for the left eye and prediction results for the right eye.

If the server 20 inputs the left eye preprocessing image to the first left eye prediction model, inputs the horizontally inverted left eye preprocessing image to the first right eye prediction model, and inputs the right eye preprocessing image to the first right eye prediction model, When input to the prediction model and the left-right inverted right eye preprocessing image is input to the first left eye prediction model, the server 20 inputs the left eye preprocessing image to the first left eye prediction model and obtains the result and the A prediction result for the left eye may be obtained by considering all results obtained by inputting the left-right inverted left eye preprocessing image to the first right eye prediction model. At this time, the server 20 considers both the result obtained by inputting the right eye preprocessing image to the first right eye prediction model and the result obtained by inputting the left and right inverted right eye preprocessing image to the first left eye prediction model. A prediction result for the right eye may be obtained.

For example, the server 20 is an average value of a result obtained by inputting the left eye preprocessing image to the first left eye prediction model and a result obtained by inputting the left-right inverted left eye preprocessing image to the first right eye prediction model. A prediction result for the left eye may be obtained based on whether is greater than or equal to the threshold value.

For another example, the server 20 may select a result obtained by inputting the left eye preprocessing image into the first left eye prediction model and a result obtained by inputting the left/right inverted left eye preprocessing image into the first right eye prediction model. If any one value is equal to or greater than the aforementioned threshold value, it can be predicted that the conjunctiva of the left eye is congested.

For another example, the server 20 may input a result obtained by inputting the left eye preprocessing image to the first left eye prediction model and a result obtained by inputting the left and right inverted left eye preprocessing image to the first right eye prediction model. If all of them are above the above-mentioned threshold value, it can be predicted that the conjunctiva of the left eye is congested.

If the server 20 inputs the left eye preprocessed image, the left eye inverted left eye preprocessed image, the right eye preprocessed image, and the left eye inverted right eye preprocessed image to the first prediction model that is not binary, the server ( 20) calculates a prediction result for the left eye by considering both a result obtained by inputting the left eye preprocessing image to the first prediction model and a result obtained by inputting the left-right inverted left eye preprocessing image to the first prediction model. can be obtained At this time, the server 20 considers both a result obtained by inputting the right eye pre-processing image to the first prediction model and a result obtained by inputting the right-left inverted right eye pre-processing image to the first prediction model, and then the right eye A prediction result for can be obtained.

For example, the server 20 calculates an average value of a result obtained by inputting the left eye preprocessing image into the first prediction model and a result obtained by inputting the left/right inverted left eye preprocessing image into the first prediction model. A prediction result for the left eye may be obtained based on whether the value is greater than or equal to a threshold value.

For another example, the server 20 may select one of a result obtained by inputting the left eye preprocessing image to the first prediction model and a result obtained by inputting the left and right inverted left eye preprocessing image to the first prediction model. If the value of is greater than or equal to the above threshold value, it can be predicted that the conjunctiva of the left eye is congested.

As another example, the server 20 may determine both a result obtained by inputting the left eye preprocessing image to the first prediction model and a result obtained by inputting the left/right inverted left eye preprocessing image to the first prediction model. If it is greater than the above-mentioned threshold value, it can be predicted that the conjunctiva of the left eye is congested.

The above method can be similarly applied to determining whether the conjunctiva of the right eye is congested.

6. Conjunctival edema prediction method

The conjunctival edema prediction method disclosed by the present application may be performed by the server 20 .

24 is a flowchart for explaining a method for predicting conjunctival edema.

Referring to FIG. 24, the server 20 acquires a face image (S200), pre-processes the acquired face image (S210), and converts the pre-processed image to the aforementioned second prediction model (conjunctival edema prediction model). input (S220), and an output value of the second predictive model is obtained (S230).

The conjunctival edema prediction method is the same as the conjunctival congestion prediction method, except that the second predictive model is used instead of the first predictive model, and the finally obtained result is a predictive value for whether or not there is edema in the conjunctiva. Since they are very similar, a detailed description will be omitted.

7. Tears Hill Edema Prediction Method

The tear hill edema prediction method disclosed by the present application may be performed by the server 20 .

25 is a flowchart illustrating a method for predicting tear hill edema.

Referring to FIG. 25, the server 20 acquires a face image (S300), pre-processes the obtained face image (S310), and uses the pre-processed image as the above-mentioned third prediction model (tear hill edema prediction model). (S320), and an output value of the third predictive model is obtained (S330).

Since the method for predicting conjunctival edema is the same as or very similar to the method for predicting conjunctival congestion, except that the third predictive model is used instead of the first predictive model, a detailed description thereof will be omitted.

As described above, the image preprocessing methods used in the method for predicting conjunctival congestion, the method for predicting conjunctival edema, and the method for predicting tear hill edema are the same, only the prediction models into which the preprocessed images are input are different. Therefore, after image pre-processing as described above, the image can be input to different predictive models.

However, it is described that the same method as the image pre-processing method used in the method for predicting conjunctival hyperemia and conjunctival edema is applied to this tear hill edema, but in some cases, in the method of predicting tear hill edema, a different method is used. A preprocessed image may be used. For example, a pre-processed image may be used that is cropped so that a part of the iris and a tear hill are included in the image. Alternatively, a preprocessed image cropped so that the iris is not included but the tear hill and the tear hill are included in the image may be used.

8. Eyelid redness prediction method

The eyelid redness prediction method disclosed by this application may be performed by the server 20 .

26 is a flowchart for explaining a method for predicting eyelid redness.

Referring to FIG. 26, the server 20 acquires a face image (S400), pre-processes the acquired face image (S410), and converts the pre-processed image to the aforementioned fourth prediction model (eyelid redness prediction model). input (S420), and an output value of the fourth predictive model is obtained (S430).

Acquisition of face image

The server 20 obtains a face image (S400). The server 20 may acquire the face image from the user terminal 10 .

Preprocessing of face images

The server 20 may pre-process the acquired face image (S410). The server 20 may perform the aforementioned iris segmentation, eye outline segmentation, masking, cropping, and resizing on the acquired face image.

segmentation processing

The server 20 performs the iris segmentation and the eye outline segmentation, and according to the result, it is possible to identify pixels corresponding to the iris and pixels corresponding to the inside of the eye outline in the acquired face image. The server 20 may check coordinate values of pixels corresponding to the iris and coordinate values of pixels corresponding to the inside of the eye outline.

However, in performing the eyelid redness prediction method, as will be described later, if a separate masking process is performed, iris segmentation must be performed. However, if a separate masking process is not performed, it is not necessary to perform iris segmentation.

masking treatment

The server 20 may process the second masking on the face image based on the information about the identified pixels. The server 20 may remove pixel values of pixels corresponding to an externally exposed iris (cornea) among pixels included in the face image through the second masking process. Accordingly, pixel values of pixels corresponding to regions other than the iris (cornea) of the left eye and the iris (cornea) of the right eye may be maintained as original pixel values, but the iris (or cornea) of the left eye, Pixel values of pixels corresponding to the iris (or cornea) of the right eye may be removed or changed to other values.

However, in performing the eyelid redness prediction method, performing preprocessing of masking the iris (cornea) has advantages in various aspects, but it is okay not to mask the iris.

cropped

The server 20 may crop the masked face image. The server 20 may generate a left eye cropped image and a right eye cropped image by cropping the masked face image. When performing the method of predicting conjunctival congestion, the server 20 may use a third cropping method (cropping including eyelids) among the two cropping methods described above. Since the third cropping method has already been described in detail, a detailed description thereof will be omitted.

Resizing processing and left-right reversal processing

The server 20 may resize the left eye cropped image and the right eye cropped image to a predetermined size.

Meanwhile, when the fourth prediction model is implemented as a single model without dualization, the server 20 may horizontally invert one of the left-eye cropped image and the right-eye cropped image as described above. The server 20 does not horizontally invert the other of the left eye cropped image and the right eye cropped image. At this time, the criterion for determining which of the left eye image and the left eye image is to be reversed is determined based on the same criterion applied when training the fourth predictive model. That is, when learning the fourth predictive model, if the left eye image is inverted and the right eye image is not inverted, the server 20 inverts the left eye image and does not invert the right eye image.

If, as described above, in implementing the fourth prediction model, in the case of dualization into the fourth left eye prediction model and the fourth right eye prediction model, the server 20 may not perform left-right reversal processing.

On the other hand, in performing the preprocessing, it has been described that segmentation, masking process, cropping process, resizing process, and left/right reversal process are performed, but the order of each preprocessing can achieve the purpose of the eyelid redness prediction method disclosed by the present application It can be changed within scope.

Input of preprocessed image

The server 20 may input the preprocessed image to the first prediction model (S420).

When the fourth prediction model is implemented as a single model without being binary, the server 20 sequentially inputs the right eye preprocessing image and the left eye preprocessing image inverted to the left into the fourth prediction model.

If, in implementing the fourth prediction model, if the binary is made into a fourth left eye prediction model and a fourth right eye prediction model, the server 20 inputs the left eye preprocessing image to the fourth left eye prediction model, and the right eye preprocessing An image is input to the fourth right eye predictive model. Alternatively, the server 20 inputs the left eye preprocessing image to the fourth left eye prediction model, inputs the horizontally inverted left eye preprocessing image to the fourth right eye prediction model, and inputs the right eye preprocessing image to the fourth right eye prediction model. model, and the left-right inverted right-eye preprocessing image may be input to the fourth left-eye prediction model.

If, in implementing the fourth predictive model, the fourth predictive model is implemented as a single model without being binary and at the same time learned to determine whether conjunctiva is congested without distinguishing between left and right eye images, the server (20) may input the left eye preprocessing image and the right eye preprocessing image to the fourth predictive model without horizontal reversal. Alternatively, the server 20 may input the left eye pre-processing image, the left-right inverted left-eye pre-processing image, the right-eye pre-processing image, and the left-right inverted right eye pre-processing image to the fourth predictive model.

Eyelid redness prediction results

The server 20 may obtain a result value output from the fourth predictive model (S430). The resulting value may be a predicted probability value of redness on the eyelid captured in the image. The server 20 determines that there is redness on the eyelid if the predicted probability value is greater than or equal to the threshold value based on a predetermined threshold value, and if the predicted probability value is less than the threshold value, the eyelid It can be judged that there is no redness in .

The server 20 may obtain both prediction results for the left eye and prediction results for the right eye.

If the server 20 inputs the left eye preprocessing image to the fourth left eye prediction model, inputs the horizontally inverted left eye preprocessing image to the fourth right eye prediction model, and inputs the right eye preprocessing image to the fourth right eye prediction model, When input to the predictive model and the left-right inverted right eye preprocessing image is input to the fourth left eye prediction model, the server 20 inputs the left eye preprocessed image to the fourth left eye prediction model and obtains the result and the A prediction result for the left eye may be obtained by considering all results obtained by inputting the left-right inverted left-eye preprocessing image to the fourth right-eye prediction model. At this time, the server 20 considers both the result obtained by inputting the right eye preprocessing image to the fourth right eye prediction model and the result obtained by inputting the left and right inverted right eye preprocessing image to the fourth left eye prediction model. A prediction result for the right eye may be obtained.

For example, the server 20 is an average value of a result obtained by inputting the left eye preprocessing image to the fourth left eye prediction model and a result obtained by inputting the left and right inverted left eye preprocessing image to the fourth right eye prediction model. A prediction result for the left eye may be obtained based on whether is greater than or equal to the threshold value.

For another example, the server 20 may select a result obtained by inputting the left eye preprocessing image to the fourth left eye prediction model and a result obtained by inputting the left/right inverted left eye preprocessing image to the fourth right eye prediction model. If any one value is equal to or greater than the aforementioned threshold value, it can be predicted that the conjunctiva of the left eye is congested.

As another example, the server 20 inputs the left eye preprocessing image to the fourth left eye prediction model and obtains the obtained result and the left-right inverted left eye preprocessing image to the fourth right eye prediction model. If all of them are above the above-mentioned threshold value, it can be predicted that the conjunctiva of the left eye is congested.

If the server 20 inputs the left eye pre-processing image, the left-right inverted left eye pre-processing image, the right eye pre-processing image, and the left-right inverted right eye pre-processing image to the non-binaryized fourth predictive model, the server ( 20) calculates a prediction result for the left eye by considering both a result obtained by inputting the left eye preprocessing image to the fourth prediction model and a result obtained by inputting the horizontally inverted left eye preprocessing image to the fourth prediction model. can be obtained At this time, the server 20 considers both a result obtained by inputting the right eye pre-processing image to the fourth prediction model and a result obtained by inputting the left-right inverted right eye pre-processing image to the fourth prediction model, and then the right eye A prediction result for can be obtained.

For example, the server 20 calculates an average value of a result obtained by inputting the left eye preprocessing image into the fourth prediction model and a result obtained by inputting the left/right inverted left eye preprocessing image into the fourth prediction model. A prediction result for the left eye may be obtained based on whether the value is greater than or equal to a threshold value.

For another example, the server 20 may select one of a result obtained by inputting the left eye preprocessing image to the fourth prediction model and a result obtained by inputting the left and right inverted left eye preprocessing image to the fourth prediction model. If the value of is greater than or equal to the above-mentioned threshold, it can be predicted that there is redness in the eyelid of the left eye.

As another example, the server 20 may determine both a result obtained by inputting the left eye preprocessing image to the fourth prediction model and a result obtained by inputting the left/right inverted left eye preprocessing image to the fourth prediction model. If it is above the above-mentioned threshold value, it can be predicted that there is redness in the eyelid of the left eye.

The above method can be similarly applied to determining whether the right eye has redness of the eyelid.

9. Eyelid swelling prediction method

The eyelid edema prediction method disclosed by the present application may be performed by the server 20 .

27 is a flowchart for explaining a method for predicting eyelid edema.

Referring to FIG. 27, the server 20 acquires a face image (S500), pre-processes the acquired face image (S510), and converts the pre-processed image to the aforementioned fifth prediction model (eyelid edema prediction model). input (S520), and an output value of the fifth predictive model is obtained (S530).

The eyelid edema prediction method is the same as the eyelid edema prediction method or Since they are very similar, a detailed description will be omitted.

As described above, the pre-processing method of the image used in the eyelid edema prediction method and the eyelid edema prediction method are identical to each other, except that the prediction models into which the pre-processed images are input are different. Therefore, after image pre-processing as described above, the image can be input to different predictive models.

10. Clinical activity score prediction method for thyroid ophthalmopathy

Hereinafter, a method for predicting a clinical activity score related to thyroid ophthalmopathy disclosed by the present application will be described.

28 is a diagram for explaining a method for predicting clinical activity scores related to thyroid ophthalmopathy.

The server 20 may obtain a face image.

The server 20 performs two different types of pre-processing on one face image. The first preprocessing (hereinafter referred to as first preprocessing) includes iris segmentation, eye outline segmentation, first masking, second cropping (eye outline cropping), resizing, and left/right inversion, and the second preprocessing (hereinafter referred to as second preprocessing). includes iris segmentation, eye outline segmentation, second masking, third crop (including eyelid crop), resizing, and horizontal flipping. However, as described in the eyelid redness prediction method, iris segmentation and second masking may be omitted.

The server 20 performs a first pre-processing on the obtained face image to obtain a first pre-processed image, and the first pre-processed image includes a first left eye pre-processed image and a first right eye pre-processed image. At this time, one of the first left eye preprocessing image and the first right eye preprocessing image is a horizontally inverted image. In addition, as already described in detail, since the first preprocessing image is an image obtained using the second crop, the number of pixels corresponding to the eyelid in the first preprocessing image is minimized and the conjunctiva exposed to the outside. and pixels corresponding to the tear hill. In addition, since the first preprocessing image is an image obtained using the first masking, pixel values of pixels corresponding to the iris (or cornea) and eyelids (upper eyelid, lower eyelid) are removed, but the exposed conjunctiva and Pixel values of pixels corresponding to the tear hill are maintained.

In addition, the server 20 obtains a second pre-processed image by performing second pre-processing on the obtained face image, and the second pre-processed image includes a second left eye pre-processed image and a second right eye pre-processed image. At this time, one of the first left eye preprocessing image and the second right eye preprocessing image is a horizontally reversed image. In addition, as already described in detail, since the second pre-processed image is an image obtained by using the third crop, the second pre-processed image sufficiently includes pixels corresponding to the eyelid. In addition, when the second preprocessing image is obtained, if the second masking method is used, pixel values of pixels corresponding to the iris (or cornea) and the eyelids (upper eyelid, lower eyelid) may be removed.

The server 20 sequentially inputs the first preprocessed images (first left eye preprocessed image and first right eye preprocessed image) to the first prediction model. The server 20 obtains a result value (probability value) of the first predictive model for the first pre-processed image of the left eye, and based on it, determines whether or not the conjunctiva of the left eye is congested. In addition, the server 20 obtains a result value (probability value) of the first predictive model for the first right eye preprocessing image, and determines whether or not conjunctival congestion in the right eye is congested based on the result value (probability value).

The server 20 synthesizes the judgment result for the left eye and the judgment result for the right eye, and finally determines whether the conjunctiva is congested in both eyes. For example, when it is determined that conjunctival congestion exists in at least one of the left eye and the right eye, the server 20 finally determines that conjunctival congestion exists.

Subsequently, the server 20 sequentially inputs the first preprocessed images (first left eye preprocessed image and first right eye preprocessed image) to the second predictive model. The server 20 obtains a result value (probability value) of the second predictive model for the first pre-processed image of the left eye, and determines whether or not the conjunctival edema of the left eye is present based on the result value (probability value). In addition, the server 20 obtains a result value (probability value) of the second predictive model for the first right eye pre-processed image, and determines whether or not conjunctival edema is present in the right eye based on the result value (probability value).

The server 20 synthesizes the judgment result for the left eye and the judgment result for the right eye, and finally determines whether the conjunctiva is edema in both eyes. For example, when it is determined that at least one of the left eye and the right eye has conjunctival edema, the server 20 finally determines that there is conjunctival edema.

Subsequently, the server 20 sequentially inputs the first preprocessed images (first left eye preprocessed image and first right eye preprocessed image) to the third predictive model. The server 20 obtains a result value (probability value) of the third predictive model for the first pre-processed image of the left eye, and based on it, determines whether or not the tear hill is edematous for the left eye. In addition, the server 20 obtains a result value (probability value) of the third predictive model for the first right eye pre-processed image, and based on it, determines whether or not the tear hill is edematous for the right eye.

The server 20 synthesizes the judgment result for the left eye and the judgment result for the right eye, and finally determines whether the tear hill is edematous for both eyes. For example, when it is determined that at least one of the left eye and the right eye has edema of the lacrimal hill, the server 20 finally determines that there is edema of the lacrimal hill.

The server 20 sequentially inputs the second preprocessed images (the second left eye preprocessed image and the second right eye preprocessed image) to the fourth predictive model. The server 20 obtains a result value (probability value) of the fourth predictive model for the second left eye pre-processed image, and based on it, determines whether or not eyelid redness is present in the left eye. In addition, the server 20 obtains a result value (probability value) of the fourth predictive model for the second right eye preprocessing image, and based on it, determines whether the right eye has redness of the eyelid.

The server 20 synthesizes the judgment result for the left eye and the judgment result for the right eye, and finally determines whether the eyelids are red for both eyes. For example, when it is determined that there is redness of the eyelids for at least one of the left eye and the right eye, the server 20 finally determines that there is redness of the eyelids.

The server 20 sequentially inputs the second preprocessed images (the second left eye preprocessed image and the second right eye preprocessed image) to the fifth predictive model. The server 20 obtains a result value (probability value) of the fifth predictive model for the second left eye preprocessing image, and determines whether or not eyelid edema for the left eye is present based on the result value (probability value). In addition, the server 20 obtains a result value (probability value) of the fifth predictive model for the second right eye pre-processed image, and determines whether or not eyelid edema for the right eye is based on the result value (probability value).

The server 20 synthesizes the judgment result for the left eye and the judgment result for the right eye, and finally determines whether eyelid edema is present in both eyes. For example, when it is determined that there is eyelid edema in at least one of the left eye and the right eye, the server 20 finally determines that there is eyelid edema.

When it is determined that there is a symptom by the predictive model, the server 20 may assign a predetermined score (eg, 1 point) to the corresponding symptom, and the server determines the results of the five predictive models. Accordingly, scores can be given for each of the five symptoms, and a value obtained by adding all of these scores can be obtained.

It has been described that the method for predicting the thyroid ophthalmopathy clinical activity score disclosed by the above-described present application is performed by the server 20 . However, the above method may also be performed in the user terminal 10 . Alternatively, among the above-described methods, preprocessing may be performed in the user terminal 10 and determination of each symptom may be performed by a server. That is, the above-described steps may be appropriately distributed and implemented in the user terminal 10 and the server 20 .

11. Recommendation method for hospital visit based on continuous monitoring of clinical activity score for thyroid ophthalmopathy

Hereinafter, a method for continuously monitoring a clinical activity score related to thyroid eye disease disclosed by the present application and a method for recommending a hospital visit based thereon will be described.

29 is a diagram for explaining a method for continuously monitoring clinical activity scores related to thyroid eye disease and a method for recommending a hospital visit based on the method disclosed by the present application.

The user terminal 10 may output a guide for obtaining a face image through the display 112 (S600).

The user terminal 10 may output an image captured in real time through the camera 140 (eg, an image reflecting the user's face) through the display 112, and at this time, the guide may be output together. can

The user terminal 10 may obtain a facial image of the user's face through the camera 140 (S610).

The user terminal 10 may transmit the acquired face image to the server 20 (S620).

The user terminal 10 selects spontaneous retrobulbar pain and pain during eye movement (Pain on attempted upward or downward gaze) among a total of seven items considered in determining the clinical activity score for thyroid ophthalmopathy. ) A graphical user interface (GUI) for receiving a user input may be output through the display. Next, the user terminal 10 may receive a user's response to the above two items (S630). The user terminal 10 may assign a predetermined score (eg, 1 point) to each item based on the input user response. For example, if the user inputs that there is voluntary pain in the posterior bulbar region, the user terminal 10 may assign 1 point to the corresponding item, and if the user inputs that there is pain during eyeball movement, the user terminal 10 may assign 1 point to the corresponding item. The user terminal 10 may assign 1 point to the corresponding item.

The user terminal 10, based on the acquired face image, selects conjunctival congestion (redness of conjunctiva) and conjunctival swelling (swelling) among a total of seven items considered in determining the clinical activity score for thyroid eye disease. of conjunctiva), swelling of lacrimal caruncle, redness of eyelids and swelling of eyelids, or a sum score for these received from the server 20 It can be done (S640).

The user terminal 10 calculates the final clinical activity score for thyroid eye disease based on the score determined by the user input and the score received from the server 20 or the score determined based on the judgment result received from the server. It can (S650).

The user terminal 10 determines the time at which the user's face image is obtained, the time at which the final calculated value of the clinical activity score for thyroid eye disease is obtained, or a time corresponding thereto (hereinafter, measurement time, yy/mm/dd, hh:mm) may be stored in the memory 130 together with the calculated clinical activity score. Alternatively, the user terminal 10 may transmit the above-described measurement time and the corresponding clinical activity score to the server 20 . At this time, the server 20 may store the measurement time and clinical activity points corresponding to the user terminal 10 or the user (S660).

Meanwhile, the measurement time includes information on a date. The measurement time may have both information about the date and information about the hour and / or minute, and the measurement time has only information about the date, and the time ( hour or minute information may not be included.

The user terminal 10 may output, through the display 112 , information recommending the user to visit a hospital and undergo a detailed examination based on the calculated clinical activity score (S670).

When the calculated clinical activity score is less than 3 points, the user terminal 10 may output information indicating that there is no risk of thyroid eye disease through the display 112 .

When the calculated clinical activity score is 3 or 4, the user terminal 10 selectively outputs information indicating that there is no risk of thyroid eye disease through the display 112 or prompts the user to visit a hospital and visit a hospital. Information recommending a detailed examination may be output through the display 112 .

The user terminal 10 may output, through the display 112 , information recommending the user to visit a hospital and undergo a detailed examination when the calculated clinical activity score is 5 points or more.

If the calculated clinical activity score is 3 or 4 points, the clinical activity score measured before a predetermined period (eg, 1 week) from the point in time is checked, and the corresponding period (hereinafter referred to as monitoring period) is checked. It can be confirmed whether the clinical activity score has ever been 3 or 4 during the period. At this time, if the number of points 3 or 4 was more than once during the monitoring period, the user terminal 10 outputs information recommending that the user visit a hospital and receive a detailed examination through the display 112, and during the monitoring period If the score has never been 3 or 4, the user terminal 10 outputs information to the effect that there is no risk of thyroid eye disease through the display 112 .

However, when the calculated clinical activity score is 3 points or more, the user terminal 10 provides information recommending the user to visit a hospital and undergo a detailed examination through the display 112 without additional judgment on the past history. can be printed out.

As described above, in outputting information to the user through the user terminal 10, visually outputting the information through the display 112 has been exemplified and described, but in some cases, the information can be heard through a speaker or the like. can be output negatively.

In addition, it has been described that the continuous monitoring method of the clinical activity score related to thyroid eye disease and the method of recommending a hospital visit based thereon disclosed by the above-described present application are performed by the user terminal 10 . However, each step of the above-described method may be properly distributed and implemented in the user terminal 10 and the server 20 . For example, when the measurement time and clinical activity score are transmitted to and stored in the server 20, determining whether there are 3 points or 4 points in the monitoring section may be performed by the server 20.

12. Experiment #1

(1) Preparation of face image

1,020 face images were prepared. Each face image is an image including both the left eye and the right eye, and is an image captured according to a predetermined photographing composition.

(2) Obtaining labeling information for face images

For each of 1,020 facial images, information on conjunctival hyperemia, conjunctival edema, lacrimal edema, eyelid redness, and eyelid edema for the left eye and information on conjunctival hyperemia, conjunctival edema, lacrimal edema, eyelid erythema, and eyelid edema for the right eye. Information was obtained, and these data were used as labeling data.

Of the 1,020 datasets, 714 were used as a training set, 102 as a validation set, and 204 as a test set.

In addition, 1,020 pieces were randomly divided into a learning data set, a validation set, and a verification set 30 times, and accordingly, the first to 30th training data set groups were generated.

(3) Obtaining a first pre-processed image and a second pre-processed image of a face image

For each of the 1,020 face images, the first cropping process (eye outline cropping) was performed on the left eye and the right eye, respectively, to secure a first left eye preprocessing image and a first right eye preprocessing image. In this case, the first right eye preprocessing image used a horizontally inverted image, and the first left eye preprocessing image used an image that was not horizontally inverted. Meanwhile, at this time, both the first left eye preprocessing image and the first right eye preprocessing image are images subjected to the above-described first masking process.

For each of the 1,020 face images, the second cropping process (including eyelid cropping) was performed on the left eye and the right eye in the above-described manner to secure a second left eye preprocessing image and a second right eye preprocessing image. At this time, the left-right inverted image was used as the second right eye preprocessing image, and the left-right reversed image was used as the second left eye preprocessing image. Meanwhile, at this time, both the second left eye preprocessing image and the second right eye preprocessing image are images without masking processing.

(4) Learning of the first to fifth predictive models according to Experimental Example #1

The first to fifth predictive models were trained using the secured first preprocessed images, the secured labeling information, and the secured second preprocessed images and the secured labeling information.

For the prediction model, a model using the aforementioned ViT as a backbone architecture was used, and each prediction model was trained by integrating the left eye prediction model and the right eye prediction model into one model without separating them.

(5) Acquisition of prediction results for each symptom using predictive models

Prediction results were obtained using the verification data sets for the first to fifth predictive models that were learned. At this time, the right-eye image was a left-right inverted preprocessing image, and the left-eye image was a left-right inverted preprocessing image.

(6) Accuracy, Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of the eyelid redness prediction model according to Experimental Example #1

The values shown in [Table 1] are the accuracy, sensitivity, specificity, PPV, and NPV measured for the first to fifth predictive models learned for each of the 30 dataset groups according to Experimental Example #1 described above. are average values.

accuracy(%) responsiveness(%) Specificity (%) PPV (%) NPV (%) conjunctival hyperemia
(First prediction model) 80.80 86.84 76.40 73.58 89.10 conjunctival edema
(Second prediction model) 89.12 42.18 94.82 50.39 93.15 tear hill edema
(Third prediction model) 88.14 55.54 92.52 53.14 93.96 eyelid redness
(The 4th predictive model) 72.42 73.94 71.55 59.10 84.17 eyelid swelling
(Fifth prediction model) 81.29 86.38 53.52 91.16 43.59

13. Experiment #2

(1) Preparation of face image

The face image used in Experimental Example #1 was used as it was.

(2) Obtaining labeling information for face images

The labeling information for the face image used in Experimental Example #1 was utilized as it was.

(3) Obtaining a first pre-processed image and a second pre-processed image of a face image

For each of the 1,020 face images, the first cropping process (eye outline cropping) was performed on each of the left and right eyes in the above-described manner to secure first pre-processed images. Unlike Experimental Example #1, all of the first left eye preprocessing image that is not horizontally inverted, the first left eye preprocessing image that is horizontally inverted, the first right eye preprocessing image that is not horizontally inverted, and the first right eye preprocessing image that is horizontally inverted are all secured, used for learning. In this case, both the first left eye preprocessing image and the first right eye preprocessing image are images subjected to the first masking process.

For each of the 1,020 face images, the second cropping process (cropping including the eyelid) was performed on the left eye and the right eye in the above-described manner to secure second pre-processed images. Unlike Experimental Example #1, a second left eye preprocessing image that is not horizontally inverted, a second left eye preprocessing image that is horizontally inverted, a second right eye preprocessing image that is not horizontally inverted, and a second right eye preprocessing image that is not horizontally inverted are all secured, and these are obtained. used for learning. At this time, both the second left eye preprocessing image and the second right eye preprocessing image are images without masking processing.

(4) Learning of the first to fifth predictive models according to Experimental Example #2

The first to fifth predictive models were trained using the secured first preprocessed images, the secured labeling information, and the secured second preprocessed images and the secured labeling information.

For the prediction model, a model using the aforementioned ViT as a backbone architecture was used, and each prediction model was trained by dividing it into a left-eye prediction model and a right-eye prediction model. In particular, when learning the left eye prediction models, a left eye preprocessing image that is not horizontally inverted and a right eye preprocessing image that is reversed are used. was used.

(5) Acquisition of prediction results for each symptom using predictive models

Prediction results were obtained using the verification data sets for the first to fifth predictive models that were learned. At this time, the prediction result for the right eye was acquired by inputting the right eye preprocessing image that was not horizontally inverted to each right eye prediction model, and the prediction result for the left eye was obtained by inputting the left eye preprocessing image that was not horizontally inverted to each of the left eye prediction models. did

(6) Accuracy, Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of the eyelid redness prediction model according to Experimental Example #2

The values shown in [Table 2] are the accuracy, sensitivity, specificity, PPV, and NPV measured for the first to fifth predictive models learned for each of the 30 dataset groups according to Experimental Example #2 described above. are average values.

accuracy(%) responsiveness(%) Specificity (%) PPV (%) NPV (%) conjunctival hyperemia
(First prediction model) 79.54 87.26 73.99 71.84 89.19 conjunctival edema
(Second prediction model) 89.22 45.04 94.61 52.53 93.48 tear hill edema
(Third prediction model) 88.35 49.51 93.70 55.48 93.20 eyelid redness
(The 4th predictive model) 76.13 70.65 79.08 64.63 83.86 eyelid swelling
(Fifth prediction model) 81.47 86.51 54.44 91.22 44.62

1: system
10: user terminal
20: server

Claims (8)

Prepare a conjunctival congestion prediction model, conjunctival edema prediction model, tear hill edema prediction model, eyelid redness prediction model, and eyelid edema prediction model,
Acquire an image of the target's face,
Obtaining a first processed image and a second processed image from the face image, wherein the first processed image is different from the second processed image;
Obtaining predicted values for conjunctival hyperemia, conjunctival edema, and lacrimal edema, respectively, by inputting the first processed image to the conjunctival congestion prediction model, the conjunctival edema prediction model, and the tear hill edema prediction model,
inputting the second processed image to the eyelid redness prediction model and the eyelid edema prediction model to obtain predicted values for eyelid redness and eyelid edema, respectively;
Based on the predicted value for the conjunctival hyperemia, the predicted value for the conjunctival edema, the predicted value for the tear hill edema, the predicted value for the eyelid redness, and the predicted value for the eyelid edema, it is determined that the subject has thyroid ophthalmopathy. (including determining the possibility of having thyroid eye disease),
The first processed image is based on location information of pixels corresponding to the outline of the eye and location information of pixels corresponding to the outline of the iris included in the eye, and the outside of the outline of the eye and the iris. An image cropped along a first area including the outline of the eye by masking areas corresponding to the inside of the outline of
The second processed image is a first area that extends wider than the first area, based on location information of pixels corresponding to the outline of the eye and location information of pixels corresponding to the outline of the iris included in the eye. 2 image cropped along the region
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 1,
Location information of pixels corresponding to the outline of the eye and location information of pixels corresponding to the outline of the iris included in the eye are obtained by a segmentation model
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 1,
The first processed image includes a first processed left eye image and a first processed right eye image,
The second processed image includes a second processed left eye image and a second processed right eye image.
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 3,
The conjunctival hyperemia prediction model includes a left eye conjunctival hyperemia prediction model and a right eye conjunctival hyperemia prediction model,
The conjunctival edema prediction model includes a left eye conjunctival edema prediction model and a right eye conjunctival edema prediction model,
The lacrimal edema prediction model includes a left eye lacrimal edema prediction model and a right eye lacrimal edema prediction model,
The eyelid redness prediction model includes a left eye eyelid redness prediction model and a right eye eyelid redness prediction model,
The eyelid edema prediction model includes a left eye eyelid edema prediction model and a right eye eyelid edema prediction model
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 4,
The predicted value for the conjunctival congestion is based on a result obtained by inputting the first processed left eye image into the left eye conjunctival congestion model and a result obtained by inputting the first processed right eye image into the right eye conjunctival congestion model determined,
The predicted value for the conjunctival edema is based on a result obtained by inputting the first processed left eye image into the left eye conjunctival edema model and a result obtained by inputting the first processed right eye image into the right eye conjunctival edema model. determined,
The predicted value for the tear hill edema is a result obtained by inputting the first processed left eye image to the left eye tear hill edema model and a result obtained by inputting the first processed right eye image into the right eye tear hill edema model. is determined based on
The predicted value for the eyelid erythema is based on a result obtained by inputting the second processed left eye image into the left eye eyelid erythema model and a result obtained by inputting the second processed right eye image into the right eye eyelid erythema model. is determined,
The predicted value for the eyelid edema is based on a result obtained by inputting the second processed left eye image to the left eye eyelid edema model and a result obtained by inputting the second processed right eye image into the right eye eyelid edema model. determined
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 3,
Left-right inversion processing of one of the first processed left eye image and the first processed right eye image;
Further comprising horizontally inverting one of the second processed left eye image and the second processed right eye image
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 6,
The predicted value for the conjunctival congestion is determined based on result values obtained by inputting the horizontally inverted image and the left-right inverted image into the conjunctival congestion model, respectively,
The predicted value for the conjunctival edema is determined based on result values obtained by inputting the left-right inverted image and the left-right inverted image into the conjunctival edema model, respectively,
The predicted value for the tear hill edema is determined based on result values obtained by inputting the left-right reversed image and the left-right reversed image into the tear hill edema model, respectively;
The predicted value for the eyelid erythema is determined based on result values obtained by inputting the left-right inverted image and the non-left-right inverted image to the eyelid erythema model, respectively,
The predicted value for the eyelid edema is determined based on the result values obtained by inputting the left-right inverted image and the left-right inverted image into the eyelid edema model, respectively.
A computer-implemented method for predicting thyroid ophthalmopathy. According to claim 3,
Resizing the first processed left eye image and the first processed right eye image;
Further comprising resizing the second processed left eye image and the second processed right eye image
A computer-implemented method for predicting thyroid ophthalmopathy.

Images