CN116844687A - Prescription recommendation method and system based on tongue images and knowledge patterns - Google Patents

Abstract

The invention discloses a prescription recommendation method and a system based on tongue images and knowledge maps, wherein the method comprises the following steps: constructing a tongue image prescription data set, dividing a tongue image prescription training set and a testing set, and preprocessing; constructing a tongue diagnosis knowledge graph of the traditional Chinese medicine, and carrying out knowledge representation learning; constructing a neural network model; the neural network model includes: the tongue image feature module, the prescription feature module and the prescription recommendation module are used for carrying out iterative training on the neural network model by utilizing a tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge graph, and testing by utilizing a tongue image prescription testing set to obtain an excellent neural network model; and identifying the tongue image to be processed by adopting the optimal neural network model to obtain a corresponding recommended prescription. The method can realize the auxiliary diagnosis and treatment function of recommending the traditional Chinese medicine prescription according to the tongue image of the patient. The method is developed into a software program without depending on specific medical equipment, and has certain popularization value.

Description

A prescription recommendation method and system based on tongue image and knowledge map

Technical field

The present invention relates to the technical fields of computer image recognition and computer vision, and more specifically to a prescription recommendation method and system based on tongue images and knowledge maps.

Background technique

As the first of the four diagnostics of traditional Chinese medicine (inspection, smelling, questioning, and palpation), inspection is an important part of syndrome differentiation and treatment in traditional Chinese medicine. Inspection is based on the observation of typical mapping areas such as the face, ears, and tongue, and based on these body surface characteristics, the health of the corresponding organs is judged and diseases are diagnosed. Inspection mainly focuses on facial examination and tongue examination. Tongue diagnosis is a method for doctors to judge the health status of the human body by observing the tongue body, tongue texture, tongue coating, sublingual veins and other tongue characteristics. The characteristics of "cold and heat, depth and lightness" are an indispensable reference for TCM syndrome differentiation and treatment, and have an important guiding role in clinical medication.

Traditional Chinese medicine tongue diagnosis relies on the knowledge and diagnostic skills of expert doctors, is easily affected by subjective factors, and faces the pressure of a scarcity of expert doctors. With the rapid development of computer science and artificial intelligence, many researchers have combined artificial intelligence with traditional Chinese medicine diagnosis and treatment. For example, in tongue diagnosis, they have introduced modern computer vision technology to extract signals from tongue coating images and capture their characteristic information. , realize the processing and analysis of tongue coating images to complete medical tasks such as diagnosis, and even surpass the level of top human doctors in some specific tasks. It can be said that the effective application of artificial intelligence in various fields of traditional Chinese medicine can greatly improve the efficiency of traditional Chinese medicine diagnosis and treatment, relieve the pressure of scarce medical resources, and further promote modern traditional Chinese medicine diagnosis and treatment.

Recommendation systems are an important research direction in the computer field, mainly used to deal with problems such as information overload and data noise flooding caused by massive Internet data. The recommendation system realizes personalized recommendations for users. It analyzes the user's behavioral interests and then customizes the user's recommended item list based on the algorithm to achieve the effect of information screening.

In recent years, research on combining information fusion, knowledge graph and other technologies to alleviate the problems of data sparsity and missing key information faced by recommendation systems has also received more and more attention. The personalized customization feature of the recommendation system makes it a solution for the task of prescribing.

Therefore, how to recommend prescriptions based on tongue images based on computer vision technology is an urgent problem that those skilled in the art need to solve.

Contents of the invention

In view of this, and in view of the shortcomings of the existing technology, the present invention provides a prescription recommendation method and system based on tongue images and knowledge maps, which can recommend corresponding TCM prescriptions based on tongue image conditions and provide intelligent assistance for TCM tongue diagnosis.

In order to achieve the above objects, the present invention adopts the following technical solutions:

In a first aspect, embodiments of the present invention provide a prescription recommendation method based on tongue images and knowledge maps, which includes the following steps:

Construct a tongue image prescription data set, divide the tongue image prescription training set and test set and perform preprocessing;

Construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning;

Construct a neural network model; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, used to extract tongue image features and prescription features; and fuse the first multimodal features and the second multimodal features , to obtain prescription recommendations and scores; the first multi-modal feature is the fusion result of the tongue image feature and the corresponding node in the TCM tongue diagnosis knowledge map; the second multi-modal feature is the prescription Fusion results of features and corresponding nodes in the TCM tongue diagnosis knowledge graph;

Using the tongue image prescription training set and the TCM tongue diagnosis knowledge map, the neural network model is iteratively trained, and the tongue image prescription test set is used for testing to obtain an optimal neural network model;

The optimal neural network model is used to identify the collected tongue images to be processed and obtain the corresponding recommended prescription.

Further, the tongue image prescription data set includes multiple tongue images and corresponding prescription information; the prescription information includes: medicinal materials and dosage information.

Further, divide the tongue image prescription training set and test set and perform preprocessing, including:

Divide the tongue image prescription training set and the tongue image prescription test set according to the preset ratio;

The preprocessing of the tongue image prescription training set includes random cropping, size scaling, random horizontal and vertical flipping and normalization operations;

The preprocessing of the tongue image prescription test set includes size scaling, center cropping and normalization operations.

Further, construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning; including:

The medical resource database, TCM medical literature and medical Baidu Encyclopedia are used as the data sources of the TCM tongue diagnosis knowledge map;

Using a top-down construction method, based on the top-level data model of the design knowledge graph, entities, relationships and attribute information are extracted from the data sources to obtain triple knowledge;

Through storage in the graph database Neo4j, the construction of the TCM tongue diagnosis knowledge graph is completed, and knowledge representation learning is performed.

Further, the prescription feature module inputs prescription information, converts it into a one-hot vector, and performs dimensionality reduction through a fully connected layer to obtain a prescription feature representation in text mode.

Further, the processing process of the prescription recommendation module includes:

The tongue image features and the tongue image node features in the TCM tongue diagnosis knowledge map are fused using an additive operation to form tongue image fusion features;

After the prescription characteristics and the Chinese medicinal material node characteristics of the TCM tongue diagnosis knowledge map are fused using an additive operation, the prescription fusion characteristics are formed;

The tongue image fusion features and prescription fusion features are standardized and spliced, input into a multi-layer perceptron machine to learn the internal interaction between the two, and output the recommendation scores predicted by the model; different prescriptions are arranged in descending order according to the recommendation scores. Finally, the recommended prescription and corresponding recommendation score are obtained.

Further, using the tongue image prescription training set and the TCM tongue diagnosis knowledge map, the neural network model is iteratively trained; including:

The tongue image-prescription pair samples in the tongue image prescription training set, and the corresponding tongue image node features and medicinal material node features in the TCM tongue diagnosis knowledge map are used together as training sample data to train the neural network model;

The cross-entropy loss function is used, the optimization algorithm uses the stochastic gradient descent method, and the neural network model is optimized and trained multiple iterations.

In a second aspect, embodiments of the present invention also provide a prescription recommendation system based on tongue images and knowledge graphs, including:

The first building module is used to construct the tongue image prescription data set, divide the tongue image prescription training set and test set and perform preprocessing;

The second building module is used to construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning;

The third building module is used to build a neural network model; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, used to extract tongue image features and prescription features; and fuse the first multi-modal features and the second multi-modal feature to obtain prescription recommendations and scores; the first multi-modal feature is the fusion result of the tongue image feature and the corresponding node in the TCM tongue diagnosis knowledge graph; the second multi-modal feature The modal feature is the fusion result of the prescription feature and the corresponding node in the TCM tongue diagnosis knowledge graph;

The training and testing module uses the tongue image prescription training set, the test set and the TCM tongue diagnosis knowledge map to iteratively train and test the neural network model to obtain the optimal neural network model;

The recommendation module uses the optimal neural network model to identify the collected tongue images to be processed and obtain the corresponding recommended prescriptions.

In a third aspect, embodiments of the present invention provide a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

Memory, used to store computer programs;

The processor, when executing the program stored in the memory, can implement the prescription recommendation method based on the tongue image and the knowledge graph as described in any one of the first aspects.

In a fourth aspect, embodiments of the present invention further provide a storage medium in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the tongue-based method as described in any one of the first aspects. Prescription recommendation method for images and knowledge graphs.

For the description of the second to fourth aspects of the present invention, reference may be made to the detailed description of the first aspect; and for the beneficial effects of the description of the second to third aspects, reference may be made to the analysis of the beneficial effects of the first aspect, which is not included here. Again.

It can be seen from the above technical solutions that compared with the existing technology, the present invention provides a prescription recommendation method based on tongue images and knowledge maps:

1. The method of the present invention adopts the recommendation method of deep learning and is applied in the prescription scenario of tongue diagnosis in traditional Chinese medicine. Through the network model, the characteristics and prescription information of the patient's tongue image can be extracted and mined, and the intrinsic interactive relationship between the two can be learned, that is, whether the current prescription matches the tongue image, and the final recommendation result can be predicted based on the matching degree.

2. The method of the present invention introduces the relevant theoretical knowledge of tongue diagnosis of traditional Chinese medicine in the form of a knowledge map. By constructing a knowledge map of tongue diagnosis of traditional Chinese medicine and performing knowledge representation learning on the knowledge map, a feature representation that integrates knowledge information is obtained, which can provide Auxiliary information in different modalities.

3. The method of the present invention is based on the patient's tongue image and the TCM prescription data set, and applies the recommendation technology of deep learning and the constructed TCM tongue diagnosis knowledge map to the field of traditional TCM diagnosis and treatment, realizing the recommendation of TCM prescriptions based on the patient's tongue image. Auxiliary diagnosis and treatment functions. Compared with traditional Chinese medicine tongue diagnosis, which relies on expert doctors, it reduces repetitive work and labor time costs, and relieves the pressure of scarce medical resources. Moreover, this method does not rely on specific equipment and has certain promotion value.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without exerting creative efforts.

Figure 1 is a flow chart of a prescription recommendation method based on tongue images and knowledge maps provided by the present invention;

Figure 2 is a flow chart for constructing a tongue image prescription data set provided by the present invention;

Figure 3 is a schematic diagram of the architecture of the neural network model provided by the present invention;

Figure 4 is a flow chart of training and testing of the neural network model provided by the present invention;

Figure 5 is a block diagram of a prescription recommendation system based on tongue images and knowledge maps provided by the present invention;

Figure 6 is a block diagram of the computer equipment provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

Example 1:

The embodiment of the present invention discloses a prescription recommendation method based on tongue images and knowledge maps, which includes the following steps:

S10. Construct a tongue image prescription data set, divide the tongue image prescription training set and test set, and perform preprocessing;

S20. Construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning;

S30. Construct a neural network model; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, used to extract tongue image features and prescription features; and fuse the first multimodal features and the second multimodal features. modal features to obtain prescription recommendations and scores; the first multi-modal features are the fusion results of the tongue image features and the corresponding nodes in the TCM tongue diagnosis knowledge graph; the second multi-modal features are the The fusion result of the above prescription characteristics and the corresponding nodes in the TCM tongue diagnosis knowledge graph;

S40. Use the tongue image prescription training set and the TCM tongue diagnosis knowledge map to iteratively train the neural network model, and use the tongue image prescription test set to test to obtain an optimal neural network model;

S50. Use the optimal neural network model to identify the collected tongue images to be processed and obtain the corresponding recommended prescription.

In this embodiment, by constructing a tongue image prescription data set, a TCM tongue diagnosis knowledge graph and a neural network model, the neural network is optimized and trained based on the tongue image prescription data set and the TCM tongue diagnosis knowledge graph; based on the trained neural network model , which can realize the auxiliary diagnosis and treatment function of recommending TCM prescriptions based on the patient's tongue image. A software program developed based on this method does not need to rely on specific medical equipment and has certain promotion value.

Each of the above steps is described in detail below.

In step S10, a tongue image prescription data set is constructed. For example, the data set used in this embodiment contains a total of 26,422 patient tongue images and corresponding 26,422 Chinese medicine prescriptions. The collected types of prescription medicinal materials are 577 categories and corresponding dosage information. The tongue images in this data set come from multiple cooperative medical institutions in Guangzhou City, Guangdong Province. The data collection is completed by trained medical assistants using Android mobile terminals that have deployed tongue image collection systems. The device held by the medical assistant during collection is a mobile camera. During the collection process, the medical assistant will complete the following tasks: collect images of the patient's tongue coating and record the prescription information issued by the expert physician for the patient. This example uses Python language and is based on the PyTorch deep learning framework to implement the code and operation of the embodiment on the Ubuntu system.

As shown in Figure 2, the collected patient tongue image data is divided into a training set and a test set at a ratio of 8:2. The data preprocessing method of the training set is: crop the original image proportionally, scale the cropped image to 224×224 size, then perform random horizontal flipping operations, and normalize and standardize the RGB channels of the image, that is, Traverse each pixel of the image matrix in the data set to obtain the mean μ and variance σ of each channel, and then obtain the standardized value through the z-score formula

x`=(x-μ)/σ

The data preprocessing method of the test set is center cropping, scaling to 224×224 size, and also normalizing and standardizing the image RGB channels respectively. The prescription information in the data set is recorded in the form of a list of medicinal materials.

The data in the data set of this embodiment is in the form of a tongue image-TCM prescription sample pair, and its label is the cosine similarity between the sample prescription and the real prescription (that is: input the tongue image-prescription sample pair, and output the predicted score; the label refers to the real score. ), that is, for a sample pair consisting of a tongue image t _i and a prescription p _j , the real prescription corresponding to the tongue image recorded during data collection is p _i , then the labels of the sample pair are p _j and p _i The higher the similarity, the closer the types of medicinal materials contained between p _j and p _i , and the more suitable p _j is to be recommended to patients with tongue images t _i .

In step S20, construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning;

The data sources of the TCM tongue diagnosis knowledge map constructed in this embodiment are medical resource databases, TCM medical literature and medical Baidu Encyclopedia. It adopts a top-down construction method. For example, the top-level data model of the knowledge map is designed by expert doctors, including a total of 7 There are three entity types and six relationship types. The detailed designs are shown in Table 1 and Table 2 respectively:

Table 1 Entity type table of TCM tongue diagnosis knowledge graph

Entity type Entity example quantity Medicinal materials Ephedra; Atractylodes; Guizhi; Mint 577 effect Reinforcing qi; dispelling wind; reducing fever; relieving external symptoms 260 Disease nature Yang deficiency; Qi stagnation; dampness; heat 33 Syndromes Heart Qi deficiency syndrome; heat disturbing the heart syndrome 331 site of action liver; spleen; kidney; stomach 12 medicinal properties cold; cool; flat; lukewarm 9 tongue image Pale white tongue; cracked tongue; thin coating; moist coating; 27

Table 2 Table of relationship types of TCM tongue diagnosis knowledge map

Relationship type Relationship description quantity The medicinal effect is (medicine, medicinal effect, efficacy) 3404 Governing law relationship (Efficacy, relationship with treatment, syndrome) 2706 Pathological relationship (Syndrome, disease relationship, disease nature) 644 Shown as (Sickness, manifested by tongue signs) 54 The medicinal properties are (medicine, medicinal properties, medicinal properties) 579 Acting on (Medicine, acting on, action site) 1372

After completing the design of the top-level data model of the graph, entity and relationship information is extracted from the specific data to form triplet knowledge, which is stored through the graph database Neo4j and uses the TransR model to facilitate embedding entities and relationships in the knowledge graph into low-dimensional In vector space, the specific operations are:

The triple instance in the knowledge graph is (head, relation, tail). The tail entity tail is the translation of the head entity head through the relationship, expressed as: h+r≈t. By continuously adjusting the vectors of h, r, and t to satisfy the constraint function

Before constraining triples, the TransR model projects the entity vector into the relationship space through the projection matrix M _r :

h _r =hM _r

t _r =tM _r

Substitute the projected entity vectors h _r and t _r into the constraint function to obtain

A new triplet vector is obtained through loss training to represent the knowledge graph:

Among them, S is a set of triples in the constructed knowledge map of traditional Chinese medicine tongue diagnosis, and S` is a negative sample triple combination set that does not belong to S composed of randomly selected entities h`, t` and relationship r, and γ is the threshold. .

In step S30, a neural network model is constructed, the architecture of which is shown in Figure 3; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, which are used to extract tongue image features and prescription features; and fuse the first The multi-modal features and the second multi-modal features are used to obtain prescription recommendations and scores; the first multi-modal features are the fusion results of the tongue image features and the corresponding nodes in the TCM tongue diagnosis knowledge map; The second multi-modal feature is the fusion result of the prescription feature and the corresponding node in the TCM tongue diagnosis knowledge graph.

in:

1) Tongue image feature module, extracts tongue image features based on convolutional neural network;

Specifically, this embodiment can build a convolutional neural network based on the tongue image. The network structure is the residual network ResNet18. First, the input tongue coating image is subjected to 7×7 convolution and 3×3 pooling, and then through four residual module, each module consists of two residual blocks, and the output sizes are 56×56×64, 28×28×128, 14×14×256, and 7×7×512 respectively. After passing through the residual module, an average pooling layer and a fully connected layer are connected to obtain the feature representation of the input tongue image.

2) Prescription feature module, which uses multi-label vectors to represent prescriptions and extract their features;

Specifically, in this embodiment, the prescription input in the form of a list of medicinal materials is converted into a One-Hot vector through one-hot encoding. Each dimension 0/1 in the vector indicates whether the prescription contains the medicinal material. A fully connected layer is used to input the vector. Dimension compression results in a 128-dimensional prescription feature representation.

3) The prescription recommendation module integrates multi-modal feature vectors of tongue images, prescriptions, and atlases, and outputs recommendation scores;

Specifically, this embodiment constructs a traditional Chinese medicine prescription recommendation module, which fuses the information in three different modes: tongue image feature X _t , prescription feature X _p and map feature G respectively, and predicts recommendation results based on the new features after fusion ,Specific steps are as follows:

S301. Compare the tongue image feature representation X _t with the corresponding tongue image node feature representation in the knowledge graph Perform point-to-point addition operation on the vectors, and obtain the feature fusion output based on these two modes as

Among them, g _t,i is the characteristic representation of the i-th tongue image in the knowledge graph, and n _t is the total number of tongue images.

S302. Represent prescription features X _p and corresponding medicinal material node features in the knowledge graph Perform point-to-point addition operation on the vectors, and obtain the feature fusion output based on these two modes as

Among them, g _h,j is the characteristic representation of the jth medicinal material in the prescription in the knowledge graph, and n _p is the total number of medicinal materials in the prescription.

S303. Finally, the tongue image fusion feature e _t and the prescription fusion feature e _p are fused. The features after the fusion of the two are normalized (Normalization) and connected (Concatenate), and then input into the three-layer perceptron. Each layer outputs The dimensions are 128/64/1 respectively. Using the Softmax activation function, the output of the network is a recommendation score of [0,1]. Different prescriptions are sorted in descending order according to the recommendation scores, and finally the prescriptions recommended by the model for the patient are obtained.

In step S40, as shown in Figure 4, the neural network model is iteratively trained using the tongue image prescription training set and the TCM tongue diagnosis knowledge map, and tested using the tongue image prescription test set to obtain an optimal neural network Model.

First, use the tongue image-prescription pair samples in the tongue image prescription training set to train the model;

Specifically, the model training process of this embodiment is as follows: input the tongue image-prescription sample pair of the training set and the TCM tongue diagnosis knowledge graph, the model outputs the recommendation score of the prescription, calculates the loss between it and the real label, and uses the back propagation algorithm , adjust the weight of parameters and iteratively optimize the model. This embodiment uses a stochastic gradient descent optimization algorithm. The loss function is a cross-entropy loss function. The initial learning rate is 0.1. The training batch size (batchsize) is 64. The number of iteration rounds is 200. The number of iterations is 50, 100, and 160. When it drops to the original 0.1, the weight decays to 0.0001.

Secondly, the test samples in the tongue image prescription test set are input into the trained network model, and the model recommends the corresponding prescription.

Specifically, the patient's tongue image in the preprocessed test set is paired with the prescription and input into the trained network model to obtain the predicted recommendation score of the corresponding prescription. Arrange all prescriptions from high to low according to the prediction score. The prescriptions corresponding to the first K items are the recommended prescriptions for the patient; compare the real prescriptions corresponding to the tongue images in the test set with the recommended prescriptions. If the expectations are met, stop Iterative training as the optimal model.

In step S50, the optimal neural network model generated by training is finally used to identify the collected tongue images to be processed, and the corresponding recommended prescription is obtained; that is, the prescription with the highest recommended score.

Example 2:

Based on the same inventive concept, as shown in Figure 5, an embodiment of the present invention also provides a prescription recommendation system based on tongue images and knowledge graphs, including:

The first building module 51 is used to construct the tongue image prescription data set, divide the tongue image prescription training set and the test set and perform preprocessing;

The second building module 52 is used to construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning;

The third building module 53 is used to build a neural network model; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, used to extract tongue image features and prescription features; and fuse the first multi-modal Features and the second multi-modal feature to obtain prescription recommendations and scores; the first multi-modal feature is the fusion result of the tongue image feature and the corresponding node in the TCM tongue diagnosis knowledge map; the second multi-modal feature is the fusion result of the tongue image feature and the corresponding node in the TCM tongue diagnosis knowledge map; Multimodal features are the fusion results of the prescription features and corresponding nodes in the TCM tongue diagnosis knowledge graph;

The training and testing module 54 uses the tongue image prescription training set, the test set and the TCM tongue diagnosis knowledge map to iteratively train and test the neural network model to obtain the optimal neural network model;

The recommendation module 55 uses the optimal neural network model to identify the collected tongue images to be processed and obtain the corresponding recommended prescription.

Further, the tongue image prescription data set in the first building module 51 includes multiple tongue images and corresponding prescription information; the prescription information includes: medicinal materials and dosage information.

Divide the tongue image prescription training set and test set and perform preprocessing, including:

Divide the tongue image prescription training set and the tongue image prescription test set according to the preset ratio;

The preprocessing of the tongue image prescription training set includes random cropping, size scaling, random horizontal and vertical flipping and normalization operations;

The preprocessing of the tongue image prescription test set includes size scaling, center cropping and normalization operations.

Further, the second building module 52 specifically includes:

The medical resource database, TCM medical literature and medical Baidu Encyclopedia are used as the data sources of the TCM tongue diagnosis knowledge map;

Using a top-down construction method, based on the top-level data model of the design knowledge graph, entities, relationships and attribute information are extracted from the data sources to obtain triple knowledge;

Through storage in the graph database Neo4j, the construction of the TCM tongue diagnosis knowledge graph is completed, and knowledge representation learning is performed.

Further, the prescription feature module in the third building module 53 is used to input prescription information, convert it into a one-hot vector, and obtain a prescription feature representation in text mode through dimensionality reduction through a fully connected layer.

The processing process of the prescription recommendation module includes:

The tongue image features and the tongue image node features in the TCM tongue diagnosis knowledge map are fused using an additive operation to form tongue image fusion features;

After the prescription characteristics and the Chinese medicinal material node characteristics of the TCM tongue diagnosis knowledge map are fused using an additive operation, the prescription fusion characteristics are formed;

The tongue image fusion features and prescription fusion features are standardized and spliced, input into a multi-layer perceptron machine to learn the internal interaction between the two, and output the recommendation scores predicted by the model; different prescriptions are arranged in descending order according to the recommendation scores. Finally, the recommended prescription and corresponding recommendation score are obtained.

Further, the training test module 54, the training part includes:

The tongue image-prescription pair samples in the tongue image prescription training set, as well as the corresponding tongue image node characteristics and medicinal material node characteristics in the TCM tongue diagnosis knowledge map, are used together as training sample data to train the neural network model; using Cross-entropy loss function, optimization algorithm uses stochastic gradient descent method, multiple iterations to optimize and train the neural network model.

Example 3:

Based on the same inventive concept, an embodiment of the present invention also provides a computer device, as shown in Figure 6, including a processor 61, a communication interface 62, a memory 63 and a communication bus 64, wherein the processor 61, the communication interface 62 and the memory 63 Complete mutual communication through the communication bus 64;

Memory 63, used to store computer programs;

When the processor 61 is used to execute the program stored on the memory 63, it can implement the prescription recommendation method based on the tongue image and the knowledge map as in Embodiment 1.

Example 4:

An embodiment of the present invention further provides a storage medium in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the prescription recommendation method based on tongue images and knowledge graphs as in Embodiment 1.

Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on its differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For relevant details, please refer to the description in the method section.

The above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be practiced in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A prescription recommendation method based on tongue images and knowledge maps, which is characterized by including the following steps: Construct a tongue image prescription data set, divide the tongue image prescription training set and test set and perform preprocessing; Construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning; Construct a neural network model; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, used to extract tongue image features and prescription features; and fuse the first multimodal features and the second multimodal features , to obtain prescription recommendations and scores; the first multi-modal feature is the fusion result of the tongue image feature and the corresponding node in the TCM tongue diagnosis knowledge map; the second multi-modal feature is the prescription Fusion results of features and corresponding nodes in the TCM tongue diagnosis knowledge graph; Using the tongue image prescription training set and the TCM tongue diagnosis knowledge map, the neural network model is iteratively trained, and the tongue image prescription test set is used for testing to obtain an optimal neural network model; The optimal neural network model is used to identify the collected tongue images to be processed and obtain the corresponding recommended prescription. 2. A prescription recommendation method based on tongue images and knowledge graphs according to claim 1, characterized in that the tongue image prescription data set includes multiple tongue images and corresponding prescription information; the prescription information Includes: Medicinal ingredients and dosage information. 3. A prescription recommendation method based on tongue image and knowledge graph according to claim 1, characterized in that dividing the tongue image prescription training set and test set and performing preprocessing include: Divide the tongue image prescription training set and the tongue image prescription test set according to the preset ratio; The preprocessing of the tongue image prescription training set includes random cropping, size scaling, random horizontal and vertical flipping and normalization operations; The preprocessing of the tongue image prescription test set includes size scaling, center cropping and normalization operations. 4. A prescription recommendation method based on tongue images and knowledge graphs according to claim 1, characterized in that a traditional Chinese medicine tongue diagnosis knowledge graph is constructed and knowledge representation learning is performed; including: The medical resource database, TCM medical literature and medical Baidu Encyclopedia are used as the data sources of the TCM tongue diagnosis knowledge map; Using a top-down construction method, based on the top-level data model of the design knowledge graph, entities, relationships and attribute information are extracted from the data sources to obtain triple knowledge; Through storage in the graph database Neo4j, the construction of the TCM tongue diagnosis knowledge graph is completed, and knowledge representation learning is performed. 5. A prescription recommendation method based on tongue images and knowledge graphs according to claim 1, characterized in that the prescription feature module is input as prescription information, converted into a one-hot vector, and dimensionally reduced through a fully connected layer. Obtain prescription feature representation in text mode. 6. A prescription recommendation method based on tongue images and knowledge graphs according to claim 1, characterized in that the processing process of the prescription recommendation module includes: The tongue image features and the tongue image node features in the TCM tongue diagnosis knowledge map are fused using an additive operation to form tongue image fusion features; After the prescription characteristics and the Chinese medicinal material node characteristics of the TCM tongue diagnosis knowledge map are fused using an additive operation, the prescription fusion characteristics are formed; The tongue image fusion features and prescription fusion features are standardized and spliced, input into a multi-layer perceptron machine to learn the internal interaction between the two, and output the recommendation scores predicted by the model; different prescriptions are arranged in descending order according to the recommendation scores. Finally, the recommended prescription and corresponding recommendation score are obtained. 7. A prescription recommendation method based on tongue image and knowledge map according to claim 1, characterized in that, using the tongue image prescription training set and the traditional Chinese medicine tongue diagnosis knowledge map, the neural network model is Iterative training; includes: The tongue image-prescription pair samples in the tongue image prescription training set, and the corresponding tongue image node features and medicinal material node features in the TCM tongue diagnosis knowledge map are used together as training sample data to train the neural network model; The cross-entropy loss function is used, the optimization algorithm uses the stochastic gradient descent method, and the neural network model is optimized and trained multiple iterations. 8. A prescription recommendation system based on tongue images and knowledge graphs, which is characterized by including: The first building module is used to construct the tongue image prescription data set, divide the tongue image prescription training set and test set and perform preprocessing; The second building module is used to construct a TCM tongue diagnosis knowledge graph and perform knowledge representation learning; The third building module is used to build a neural network model; the neural network model includes: a tongue image feature module, a prescription feature module and a prescription recommendation module, used to extract tongue image features and prescription features; and fuse the first multi-modal features and the second multi-modal feature to obtain prescription recommendations and scores; the first multi-modal feature is the fusion result of the tongue image feature and the corresponding node in the TCM tongue diagnosis knowledge graph; the second multi-modal feature The modal feature is the fusion result of the prescription feature and the corresponding node in the TCM tongue diagnosis knowledge graph; The training and testing module uses the tongue image prescription training set, the test set and the TCM tongue diagnosis knowledge map to iteratively train and test the neural network model to obtain the optimal neural network model; The recommendation module uses the optimal neural network model to identify the collected tongue images to be processed and obtain the corresponding recommended prescriptions. 9. A computer device, characterized in that it includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; Memory, used to store computer programs; The processor, when used to execute the program stored in the memory, can implement the prescription recommendation method based on the tongue image and the knowledge map as described in any one of claims 1 to 7. 10. A storage medium, characterized in that at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the method based on tongue image and speech according to any one of claims 1 to 7. Prescription recommendation method based on knowledge graph.