[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2024171844A1 - Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de traitement d'informations - Google Patents

Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de traitement d'informations Download PDF

Info

Publication number
WO2024171844A1
WO2024171844A1 PCT/JP2024/003422 JP2024003422W WO2024171844A1 WO 2024171844 A1 WO2024171844 A1 WO 2024171844A1 JP 2024003422 W JP2024003422 W JP 2024003422W WO 2024171844 A1 WO2024171844 A1 WO 2024171844A1
Authority
WO
WIPO (PCT)
Prior art keywords
spectrum
fluorescence
autofluorescence
unit
image
Prior art date
Application number
PCT/JP2024/003422
Other languages
English (en)
Japanese (ja)
Inventor
憲治 池田
和博 中川
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2024171844A1 publication Critical patent/WO2024171844A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/62—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light
    • G01N21/63—Systems in which the material investigated is excited whereby it emits light or causes a change in wavelength of the incident light optically excited
    • G01N21/64—Fluorescence; Phosphorescence
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements

Definitions

  • This disclosure relates to an information processing device, a biological sample observation system, and an information processing method.
  • a color separation technique is required to separate dye fluorescence from unintended autofluorescence derived from biological tissue.
  • a color separation technique using a method such as NMF (Non-negative Matrix Factorization) has been developed to spectrally separate autofluorescence and extract the desired dye fluorescence, as described in Patent Document 1.
  • the power ratio of the multiple excitation lights is not necessarily the same on the day the initial spectrum is obtained or on the day the spectrum to be color separated is obtained. For these reasons, it is difficult to sufficiently remove the autofluorescence, and it is difficult to obtain an image with reduced autofluorescence.
  • the information processing device includes a connection unit that connects multiple fluorescence spectra obtained by irradiating a specimen with multiple excitation light beams in the wavelength direction, a color separation unit that separates the connected fluorescence spectrum into spectra for each fluorescent substance, and a spectrum extraction unit that extracts a connected autofluorescence spectrum, which is a combination of multiple autofluorescence spectra, from the connected fluorescence spectrum using jNMF (joint non-negative matrix factorization).
  • jNMF joint non-negative matrix factorization
  • a biological specimen observation system includes an imaging device that acquires an image of a specimen, and an information processing device that processes the image.
  • the information processing device has a linking unit that links multiple fluorescence spectra obtained by irradiating the specimen with multiple excitation light beams in the wavelength direction, a color separation unit that separates the linked fluorescence spectrum into spectra for each fluorescent substance, and a spectrum extraction unit that uses jNMF to extract a linked autofluorescence spectrum, which is a linked autofluorescence spectrum of multiple autofluorescence spectra, from the linked fluorescence spectrum.
  • the information processing method includes an information processing device that concatenates multiple fluorescence spectra obtained by irradiating the specimen with multiple excitation light beams in the wavelength direction, separates the concatenated fluorescence spectrum obtained by concatenating the multiple fluorescence spectra into spectra for each fluorescent substance, and extracts a concatenated autofluorescence spectrum obtained by concatenating multiple autofluorescence spectra from the concatenated fluorescence spectrum using jNMF.
  • FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure.
  • 1 is a flowchart illustrating a flow of an example of basic processing of an information processing device according to an embodiment of the present disclosure.
  • FIG. 2 is a diagram illustrating a first configuration example of an analysis unit according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram for explaining an example of a method for generating a concatenated fluorescence spectrum according to an embodiment of the present disclosure
  • 11A and 11B are diagrams for explaining timings at which the power ratios of multiple excitation light beams differ according to an embodiment of the present disclosure.
  • FIG. 13 is a diagram illustrating a second configuration example of an analysis unit according to an embodiment of the present disclosure.
  • FIGS. 1A to 1C are diagrams for explaining an example of a process for extracting an autofluorescence spectrum according to an embodiment of the present disclosure.
  • 1 is a flowchart illustrating a flow of an example process for extracting an autofluorescence spectrum according to an embodiment of the present disclosure.
  • FIG. 13 is a diagram for explaining jNMF when extracting an autofluorescence spectrum according to an embodiment of the present disclosure.
  • FIG. 13 is a diagram for explaining an NMF when extracting an autofluorescence spectrum according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram for explaining a jNMF according to an embodiment of the present disclosure.
  • FIG. 1A-1C show a wide area of a fluorescence image with jNMF and a fluorescence image with NMF according to an embodiment of the present disclosure.
  • FIG. 1 illustrates a transition of an autofluorescence spectrum according to an embodiment of the present disclosure.
  • FIG. 13 shows multiple magnified fluorescence images of jNMF application according to an embodiment of the present disclosure.
  • FIG. 13 shows multiple magnified fluorescence images of NMF application according to an embodiment of the present disclosure.
  • 1A and 1B are enlarged views of a fluorescence image obtained by applying jNMF and a fluorescence image obtained by applying NMF according to an embodiment of the present disclosure.
  • FIG. 1 illustrates a transition of an autofluorescence spectrum according to an embodiment of the present disclosure.
  • FIG. 13 shows multiple magnified fluorescence images of jNMF application according to an embodiment of the present disclosure.
  • FIG. 13 shows multiple magnified fluorescence images of NMF application according to an embodiment of the present disclosure.
  • 1A and 1B are enlarged views of a fluorescence image obtained by applying jNMF and a fluorescence image obtained by applying NMF according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram for explaining an example of a long-wavelength shift of an autofluorescence spectrum according to an embodiment of the present disclosure.
  • FIG. 13 is a diagram for explaining a process of optimizing the power ratio of multiple excitation light beams when acquiring an initial substance spectrum and an actual sample spectrum according to an embodiment of the present disclosure.
  • FIG. 1 is a flowchart showing a flow of a fluorescence separation process according to an embodiment of the present disclosure.
  • FIG. 13 is a diagram for explaining the flow of a first process in a first loop of the fluorescence separation process according to an embodiment of the present disclosure.
  • FIG. 13 is a diagram for explaining the flow of a second process in the first loop of the fluorescence separation process according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram showing an example of a schematic configuration of a fluorescence observation device.
  • FIG. 2 is a diagram showing an example of a schematic configuration of an observation unit.
  • FIG. 13 is a diagram showing an example of a sample.
  • FIG. 4 is an enlarged view showing an area of the sample illuminated with line illumination.
  • FIG. 1 is a diagram illustrating an overall configuration of a microscope system.
  • FIG. 1 is a diagram illustrating an example of an imaging method.
  • FIG. 1 is a diagram illustrating an example of an imaging method.
  • FIG. 2 is a diagram illustrating an example of a schematic hardware configuration of an information processing device.
  • each embodiment can be implemented independently. However, at least a portion of the following embodiments may be implemented in appropriate combination with at least a portion of the other embodiments. These embodiments may include novel features that are different from one another. Thus, each embodiment may contribute to solving a different purpose or problem, and may provide different effects.
  • Embodiment 1-1 Configuration example of information processing system 1-2.
  • Basic processing example of information processing device 1-3 Configuration example of analysis unit 1-3-1.
  • Second configuration example 1-4 Example of spectrum extraction processing 1-5.
  • jNMF 1-6 Comparative Examples of Fluorescence Images with jNMF and NMF 1-6-1.
  • Comparative Example 2 1-6-3 Example of long wavelength shift of autofluorescence spectrum 1-7.
  • Other embodiments 3.
  • Embodiment ⁇ 1-1 Example of information processing system configuration> An example of the configuration of an information processing system according to this embodiment will be described with reference to Fig. 1.
  • Fig. 1 is a diagram showing an example of the configuration of an information processing system according to this embodiment.
  • the information processing system is an example of a biological sample observation system.
  • the information processing system includes an information processing device 100 and a database 200.
  • the inputs to this information processing system include a fluorescent reagent 10A, a specimen 20A, and a fluorescently stained specimen 30A.
  • the fluorescent reagent 10A is a chemical used to stain the specimen 20A.
  • the fluorescent reagent 10A is, for example, a fluorescent antibody, a fluorescent probe, or a nuclear staining reagent, but the type of the fluorescent reagent 10A is not particularly limited to these.
  • the fluorescent antibody includes, for example, a primary antibody used for direct labeling, or a secondary antibody used for indirect labeling.
  • the fluorescent reagent 10A is managed with identification information that can identify the fluorescent reagent 10A and the manufacturing lot of the fluorescent reagent 10A.
  • the identification information is referred to as "reagent identification information 11A".
  • the reagent identification information 11A is, for example, barcode information such as one-dimensional barcode information or two-dimensional barcode information, but is not limited to this. Even if the fluorescent reagent 10A is the same type of product, the properties of the fluorescent reagent 10A differ for each manufacturing lot depending on the manufacturing method and the state of the cells from which the antibody was obtained. For example, the spectral information, quantum yield, or fluorescent labeling rate of the fluorescent reagent 10A differs for each manufacturing lot.
  • the fluorescent labeling rate is also called "F/P value: Fluorescein/Protein" and refers to the number of fluorescent molecules that label an antibody.
  • the fluorescent reagent 10A is managed for each production lot by being assigned reagent identification information 11A.
  • the reagent information of each fluorescent reagent 10A is managed for each production lot. This allows the information processing device 100 to separate the fluorescent signal and the autofluorescent signal while taking into consideration slight differences in properties that appear for each production lot.
  • the management of the fluorescent reagent 10A for each production lot is merely an example, and the fluorescent reagent 10A may be managed in units finer than the production lot.
  • the specimen 20A is prepared from a specimen or tissue sample taken from a human body for the purpose of pathological diagnosis or clinical testing.
  • the specimen 20A is not particularly limited in terms of the type of tissue used, such as organs or cells, the type of disease to be treated, the attributes of the subject, such as age, sex, blood type, or race, or the lifestyle of the subject, such as diet, exercise, or smoking.
  • the specimens 20A are managed by being given identification information that can identify each specimen 20A.
  • the identification information is referred to as "specimen identification information 21A".
  • the specimen identification information 21A is, for example, barcode information such as one-dimensional barcode information or two-dimensional barcode information, similar to the reagent identification information 11A, but is not limited thereto.
  • the specimen 20A has different properties depending on the type of tissue used, the type of disease to be treated, the attributes of the subject, or the lifestyle of the subject. For example, the measurement channel or spectrum information of the specimen 20A differs depending on the type of tissue to be used. Therefore, in the information processing system according to this embodiment, the specimens 20A are managed individually by being assigned specimen identification information 21A. This enables the information processing device 100 to separate the fluorescent signal and the autofluorescent signal while taking into consideration slight differences in properties that appear for each specimen 20A.
  • the fluorescent stained specimen 30A is created by staining the specimen 20A with a fluorescent reagent 10A.
  • the fluorescent stained specimen 30A is created on the assumption that the specimen 20A is stained with at least one fluorescent reagent 10A, and the number of fluorescent reagents 10A used for staining is not particularly limited.
  • the staining method is determined by the respective combinations of the specimen 20A and the fluorescent reagents 10A, and is not particularly limited.
  • the fluorescent stained specimen 30A is input to the information processing device 100 and an image is taken.
  • the information processing device 100 includes an acquisition unit 110, a storage unit 120, a processing unit 130, a display unit 140, a control unit 150, and an operation unit 160.
  • the acquisition unit 110 is configured to acquire information used for various processes of the information processing device 100. As shown in FIG. 1
  • the information acquiring unit 111 is configured to acquire reagent information and specimen information. More specifically, the information acquiring unit 111 acquires reagent identification information attached to the fluorescent reagent 10A used to generate the fluorescent stained specimen 30A. For example, the information acquiring unit 111 acquires the reagent identification information 11A and the specimen identification information 21A attached to the specimen 20A using a barcode reader or the like. The information acquisition unit 111 acquires the reagent information based on the reagent identification information 11A and the specimen information based on the specimen identification information 21A from the database 200. The information acquisition unit 111 stores the acquired information in the information storage unit 121, which will be described later. Save.
  • the image acquisition unit 112 is configured to acquire image information of the fluorescent stained specimen 30A and the specimen 20A stained with at least one fluorescent reagent 10A. More specifically, the image acquisition unit 112 includes any image sensor such as a CCD or CMOS, and acquires image information by capturing an image of the fluorescent stained specimen 30A using the image sensor.
  • image information is a concept that includes not only the image of the fluorescent stained specimen 30A itself, but also measurement values that are not visualized as an image.
  • the image information may include information on the wavelength spectrum of the fluorescence emitted from the fluorescent stained specimen 30A.
  • the wavelength spectrum of the fluorescence is referred to as the fluorescence spectrum.
  • the image acquisition unit 112 stores the image information in the image information storage unit 122, which will be described later.
  • the storage unit 120 is configured to store information used for various processes of the information processing device 100 or information output by various processes. As shown in FIG. 1 , the storage unit 120 includes an information storage unit 121, an image information storage unit 122, and an analysis result storage unit 123.
  • the information storage unit 121 is configured to store the reagent information and specimen information acquired by the information acquisition unit 111. Note that the analysis process by the analysis unit 131 and the generation process of image information by the image generation unit 132, that is, the image After the information reconstruction process is completed, the information storage unit 121 may increase the available capacity by deleting the reagent information and specimen information used in the process.
  • the image information storage unit 122 is configured to store image information of the fluorescently stained specimen 30A acquired by the image acquisition unit 112. As with the information storage unit 121, after the analysis process by the analysis unit 131 and the image information generation process by the image generation unit 132, i.e., the image information reconstruction process, are completed, the image information storage unit 122 may increase free space by deleting the image information used in the processes.
  • the analysis result storage unit 123 is configured to store the results of the analysis process performed by the analysis unit 131, which will be described later.
  • the analysis result storage unit 123 stores the fluorescent signal of the fluorescent reagent 10A separated by the analysis unit 131.
  • the analysis result storage unit 123 stores the autofluorescence signal of the specimen 20A.
  • the analysis result storage unit 123 provides the results of the analysis process to the database 200 in order to improve the analysis accuracy by machine learning or the like. After providing the analysis results to the database 200, the unit 123 may increase free space by appropriately deleting the analysis results that it has saved.
  • the processing unit 130 has a functional configuration for performing various processes using image information, reagent information, and specimen information. As shown in FIG.
  • the analysis unit 131 is configured to perform various analytical processes using the image information, the specimen information, and the reagent information. For example, the analysis unit 131 performs a process of separating an autofluorescence signal of the specimen 20A, for example an autofluorescence spectrum which is an example of an autofluorescence component, from a fluorescent signal of the fluorescent reagent 10A, for example a dyeing fluorescent spectrum which is an example of a dyeing fluorescent component, from the image information based on the specimen information and the reagent information.
  • an autofluorescence signal of the specimen 20A for example an autofluorescence spectrum which is an example of an autofluorescence component
  • a fluorescent signal of the fluorescent reagent 10A for example a dyeing fluorescent spectrum which is an example of a dyeing fluorescent component
  • the analysis unit 131 recognizes one or more elements that make up the autofluorescence signal based on the measurement channels included in the specimen information. For example, the analysis unit 131 recognizes one or more autofluorescence components that make up the autofluorescence signal. The analysis unit 131 then predicts the autofluorescence signal included in the image information using the spectral information of these autofluorescence components included in the specimen information. The analysis unit 131 then separates the autofluorescence signal and the fluorescent signal from the image information based on the spectral information of the fluorescent components of the fluorescent reagent 10A included in the reagent information and the predicted autofluorescence signal.
  • the analysis unit 131 separates the fluorescent signals of each of these two or more fluorescent reagents 10A from the image information based on the specimen information and the reagent information, or from the fluorescent signal after separation from the autofluorescent signal.
  • the analysis unit 131 uses the spectral information of the fluorescent components of each fluorescent reagent 10A contained in the reagent information to separate the fluorescent signal of each fluorescent reagent 10A from the entire fluorescent signal after separation from the autofluorescent signal.
  • the analysis unit 131 separates the autofluorescence signal of each autofluorescence component from the image information based on the specimen information and the reagent information, or from the autofluorescence signal after separation from the fluorescent signal. For example, the analysis unit 131 uses the spectral information of each autofluorescence component included in the specimen information to separate the autofluorescence signal of each autofluorescence component from the entire autofluorescence signal after separation from the fluorescent signal.
  • the analysis unit 131 performs various processes using these fluorescence signals and autofluorescence signals.
  • the analysis unit 131 may extract a fluorescence signal from the image information of the other specimen 20A by performing a subtraction process on the image information of the other specimen 20A using the autofluorescence signal after separation. This subtraction process is also called "background subtraction process".
  • This subtraction process is also called "background subtraction process”.
  • the similar specimens 20A referred to here include, for example, a tissue section before staining of a tissue section to be stained, a section adjacent to a stained section, a section different from the stained section in the same block, or a section in a different block of the same tissue, or sections taken from different patients.
  • the tissue section will be referred to as a section.
  • the same block is sampled from the same location as the stained section.
  • the different block is sampled from a different location from the stained section.
  • the analysis unit 131 may extract a fluorescence signal from the image information of the other specimen 20A by removing the autofluorescence signal from the image information of the other specimen 20A. Furthermore, when calculating the S/N value using the image information of the other specimen 20A, the analysis unit 131 can improve the S/N value by using the background after removing the autofluorescence signal.
  • the analysis unit 131 can also perform various processes using the separated fluorescent signals or autofluorescent signals. For example, the analysis unit 131 can use these signals to analyze the fixation state of the specimen 20A, or perform segmentation or area division to recognize object regions contained in the image information.
  • Objects include, for example, cells, intracellular structures, or tissues.
  • Intracellular structures include, for example, cytoplasm, cell membranes, and nuclei.
  • Tissues include, for example, tumor areas, non-tumor areas, connective tissue, blood vessels, blood vessel walls, lymphatic vessels, fibrotic structures, necrosis, and the like. The analysis and segmentation of the fixation state of the specimen 20A will be described in detail later.
  • the image generating unit 132 is configured to generate image information based on the fluorescent signal or the autofluorescent signal separated by the analyzing unit 131, that is, to reconstruct the image information.
  • the image generating unit 132 can generate image information including only the fluorescent signal, or image information including only the autofluorescent signal.
  • the image generating unit 132 can generate image information for each component.
  • the analyzing unit 131 performs various processes using the separated fluorescent signal or the autofluorescent signal
  • the image generating unit 132 may generate image information showing the results of those processes.
  • Examples of the various processes include analysis of the fixation state of the specimen 20A, segmentation, and calculation of the S/N value.
  • the distribution information of the fluorescent reagent 10A labeled to the target molecule or the like that is, the two-dimensional spread, intensity, and wavelength of the fluorescence, and the respective positional relationships are visualized, and the visibility of the user, that is, the doctor or researcher, can be improved, especially in the tissue image analysis area where the information of the target substance is complicated.
  • the image generating unit 132 may also generate image information by controlling the image generating unit 132 to distinguish between autofluorescence signals and fluorescent signals based on the fluorescent signals or autofluorescence signals separated by the analyzing unit 131. Specifically, the image generating unit 132 may generate image information by controlling the following: improving the brightness of the fluorescent spectrum of the fluorescent reagent 10A labeled to a target molecule, extracting and discoloring only the fluorescent spectrum of the labeled fluorescent reagent 10A, extracting the fluorescent spectra of two or more fluorescent reagents 10A from a specimen 20A labeled with two or more fluorescent reagents 10A and discoloring each of them to a different color, extracting only the autofluorescence spectrum of the specimen 20A and dividing or subtracting it, improving the dynamic range, etc. This allows the user to clearly distinguish color information derived from the fluorescent reagent bound to the target substance of interest, thereby improving the user's visibility.
  • the display unit 140 is configured to present the image information generated by the image generation unit 132 to the user by displaying it on a display.
  • the type of display used as the display unit 140 is not particularly limited.
  • the image information generated by the image generation unit 132 may be presented to the user by being projected by a projector or printed by a printer. In other words, the method of outputting the image information is not particularly limited.
  • the control unit 150 is a functional configuration that comprehensively controls the overall processing performed by the information processing device 100.
  • the control unit 150 controls the start and end of various processes as described above based on the operation input by the user performed via the operation unit 160.
  • the various processes include, for example, an image capture process of the fluorescent stained specimen 30A, an analysis process, a generation process of image information, and a display process of image information.
  • the generation process of image information includes, for example, a reconstruction process of image information.
  • the control contents of the control unit 150 are not particularly limited.
  • the control unit 150 may control processes generally performed in general-purpose computers, PCs, tablet PCs, etc., such as processes related to an OS (Operating System).
  • OS Operating System
  • the operation unit 160 is configured to receive operation input from a user. More specifically, the operation unit 160 includes various input means such as a keyboard, a mouse, a button, a touch panel, or a microphone, and the user can perform various inputs to the information processing device 100 by operating these input means. Information regarding the operation input performed via the operation unit 160 is provided to the control unit 150.
  • the database 200 is a device that manages specimen information, reagent information, and analysis processing results. More specifically, the database 200 manages specimen identification information 21A and specimen information, and reagent identification information 11A and reagent information, by linking them together. This allows the information acquisition unit 111 to acquire specimen information based on the specimen identification information 21A of the specimen 20A that is the measurement target, and reagent information based on the reagent identification information 11A of the fluorescent reagent 10A, from the database 200.
  • the specimen information managed by the database 200 is information including measurement channel and spectral information specific to the autofluorescence component contained in the specimen 20A.
  • the specimen information may also include target information for each specimen 20A, specifically, for example, the type of tissue used, such as organs, cells, blood, body fluids, ascites, and pleural effusion, the type of disease to be treated, the attributes of the subject, such as age, sex, blood type, or race, or information on the subject's lifestyle, such as diet, exercise habits, or smoking habits, and the information including the measurement channel and spectral information specific to the autofluorescence component contained in the specimen 20A and the target information may be linked to each specimen 20A.
  • tissue to be used is not particularly limited to tissue collected from a subject, but may also include in vivo tissues and cell lines of humans and animals, as well as solutions, solvents, solutes, and materials contained in the subject of measurement.
  • the reagent information managed by the database 200 includes spectral information of the fluorescent reagent 10A, but the reagent information may also include information about the fluorescent reagent 10A, such as the production lot, fluorescent component, antibody, clone, fluorescent labeling rate, quantum yield, fading coefficient, and absorption cross section or molar extinction coefficient.
  • the fading coefficient is information that indicates the ease with which the fluorescence intensity of the fluorescent reagent 10A decreases.
  • the specimen information and reagent information managed by the database 200 may be managed in different configurations, and in particular the information about the reagent may be a reagent database that presents the user with the optimal combination of reagents.
  • the specimen information and reagent information are provided by the manufacturer or are measured independently within the information processing system according to the present disclosure.
  • the manufacturer of the fluorescent reagent 10A often does not measure and provide spectral information or the fluorescent labeling rate for each production lot. Therefore, the accuracy of separating the fluorescent signal and the autofluorescent signal can be improved by measuring and managing this information independently within the information processing system according to the present disclosure.
  • the database 200 may use catalog values published by the manufacturer or literature values described in various documents as specimen information and reagent information, particularly reagent information.
  • actual specimen information and reagent information often differs from catalog values and literature values, so it is more preferable that the specimen information and reagent information be measured and managed independently within the information processing system according to the present disclosure, as described above.
  • the accuracy of analysis processing can be improved by machine learning techniques using the specimen information, reagent information, and analysis processing results managed in the database 200.
  • the entity that performs learning using machine learning techniques there is no particular limitation on the entity that performs learning using machine learning techniques, and in this embodiment, a case where the analysis unit 131 of the information processing device 100 performs learning will be described as an example.
  • the analysis unit 131 uses a neural network to generate a classifier or estimator that has been machine-learned using learning data that links the fluorescent signals and autofluorescence signals after separation to the image information, specimen information, and reagent information used in the separation. Then, when image information, specimen information, and reagent information are newly acquired, the analysis unit 131 can input the information to the classifier or estimator, and predict and output the fluorescent signals and autofluorescence signals contained in the image information.
  • a method may be output in which a similar separation process performed in the past that is more accurate than the predicted fluorescent signal and autofluorescent signal is calculated, the contents of the process in the past are statistically or regressionally analyzed, and the separation process of the fluorescent signal and the autofluorescent signal is improved based on the analysis results.
  • the separation process is, for example, a separation process in which similar image information, specimen information, or reagent information is used.
  • the contents of the process include, for example, information and parameters used in the process.
  • the machine learning method is not limited to the above, and known machine learning techniques may be used.
  • the separation process of the fluorescent signal and the autofluorescent signal may be performed by artificial intelligence.
  • not only the separation process of the fluorescent signal and the autofluorescent signal but also various processes using the fluorescent signal or the autofluorescent signal after separation, such as analysis of the fixation state of the specimen 20A or segmentation, may be improved by machine learning techniques.
  • the above describes an example of the configuration of the information processing system according to this embodiment.
  • the above configuration described with reference to FIG. 1 is merely an example, and the configuration of the information processing system according to this embodiment is not limited to this example.
  • the information processing device 100 does not necessarily have to have all of the functional configuration shown in FIG. 1.
  • the information processing device 100 may have a database 200 built in.
  • the functional configuration of the information processing device 100 can be flexibly modified according to the specifications and operation.
  • the information processing device 100 may also perform processes other than those described above. For example, by including information such as the quantum yield, fluorescent labeling rate, and absorption cross section or molar extinction coefficient for the fluorescent reagent 10A in the reagent information, the information processing device 100 may calculate the number of fluorescent molecules in the image information and the number of antibodies bound to the fluorescent molecules using the image information from which the autofluorescence signal has been removed and the reagent information.
  • FIG. 2 is a flowchart showing the flow of the example of basic processing of the information processing device 100 according to this embodiment.
  • the flow of the basic processing will be described, and norm processing related to the separation accuracy for each pixel in the analysis unit 131 will be described later.
  • step S1000 the user determines the fluorescent reagent 10A and specimen 20A to be used in the analysis.
  • step S1004 the user creates a fluorescently stained specimen 30A by staining the specimen 20A with the fluorescent reagent 10A.
  • step S1008 the image acquisition unit 112 of the information processing device 100 acquires image information (e.g., an image of a fluorescent stained specimen) by capturing an image of the fluorescent stained specimen 30A. Note that in capturing the image, multiple excitation lights (e.g., laser light) are used.
  • step S1012 the information acquisition unit 111 acquires reagent information and specimen information from the database 200 based on the reagent identification information 11A attached to the fluorescent reagent 10A used to generate the fluorescent stained specimen 30A and the specimen identification information 21A attached to the specimen 20A.
  • step S1016 the analysis unit 131 separates the autofluorescence signal (autofluorescence spectrum) of the specimen 20A and the fluorescent signal (fluorescence spectrum) of the fluorescent reagent 10A from the image information based on the specimen information and reagent information. If the fluorescent signal contains signals of multiple fluorescent dyes (Yes in step S1020), the analysis unit 131 separates the fluorescent signals of each fluorescent dye in step S1024. Note that if the fluorescent signal does not contain signals of multiple fluorescent dyes (No in step S1020), the process of separating the fluorescent signals of each fluorescent dye is not performed in step S1024.
  • step S1028 the image generating unit 132 generates image information using the fluorescent signal separated by the analyzing unit 131. For example, the image generating unit 132 generates image information from which the autofluorescent signal has been removed, or generates image information in which the fluorescent signal is displayed for each fluorescent dye.
  • step S1032 the display unit 140 displays the image information generated by the image generating unit 132, thereby completing the series of processes.
  • steps in the flowchart of FIG. 2 do not necessarily have to be processed chronologically in the order in which they are described. In other words, the steps in the flowchart may be processed in an order different from that in which they are described, or may be processed in parallel.
  • the analysis unit 131 may directly separate the fluorescence signal of each fluorescent dye from the image information, rather than separating the fluorescence signal of each fluorescent dye in step S1024. Furthermore, the analysis unit 131 may separate the autofluorescence signal of specimen 20A from the image information after separating the fluorescence signal of each fluorescent dye from the image information.
  • the information processing device 100 may also execute processes not shown in FIG. 2.
  • the analysis unit 131 may not only separate signals, but also perform segmentation based on the separated fluorescent signals or autofluorescent signals, or analyze the fixation state of the specimen 20A.
  • FIG. 3 is a diagram showing a first configuration example of the analysis unit 131 according to the present embodiment.
  • Fig. 4 is a diagram for explaining an example of a method for generating a concatenated fluorescence spectrum according to the present embodiment.
  • Fig. 5 is a diagram for explaining timings at which the power ratios (e.g., laser power ratios) of multiple excitation light beams are different according to the present embodiment.
  • the analysis unit 131 includes a linking unit 1311, a color separation unit 1321, and a spectrum extraction unit 1322.
  • This analysis unit 131 is configured to perform various processes including fluorescence separation processing and spectrum extraction.
  • the analysis unit 131 is configured to link fluorescence spectra as a preprocessing step for the fluorescence separation processing, and to separate the linked fluorescence spectrum into molecules.
  • the connecting unit 1311 is configured to generate a connected fluorescence spectrum by connecting at least a part of the multiple fluorescence spectra acquired by the image acquiring unit 112 in the wavelength direction. For example, the connecting unit 1311 extracts data of a predetermined width (H1 in FIG. 4) from each fluorescence spectrum so as to include the maximum value of the fluorescence intensity in each of three fluorescence spectra (B to D in FIG. 4) out of four fluorescence spectra (A to D in FIG. 4) acquired by the image acquiring unit 112.
  • a predetermined width H1 in FIG. 4
  • the width of the wavelength band from which the connecting unit 1311 extracts data can be determined based on the reagent information, the excitation wavelength, the fluorescence wavelength, or the like, and may be the same or different for each fluorescent substance. In other words, the width of the wavelength band from which the connecting unit 1311 extracts data may be the same or different for each of the fluorescence spectra shown in B to D in FIG. 4. Then, as shown in FIG. 4, the connecting unit 1311 generates one connected fluorescence spectrum by connecting the extracted data to each other in the wavelength direction. It should be noted that since the concatenated fluorescence spectrum is composed of data extracted from a plurality of fluorescence spectra, the wavelengths are not continuous at the boundaries of each concatenated data.
  • connection unit 1311 performs the above connection after aligning the intensity of the excitation light corresponding to each of the multiple fluorescence spectra based on the intensity of each excitation light, in other words, after correcting the multiple fluorescence spectra. More specifically, the connection unit 1311 performs the above connection after aligning the intensity of the excitation light corresponding to each of the multiple fluorescence spectra by dividing each fluorescence spectrum by the excitation power density, which is the intensity of the excitation light. In this way, the fluorescence spectrum when the excitation light of the same intensity is irradiated is obtained.
  • the intensity of the irradiated excitation light differs, the intensity of the spectrum absorbed by the fluorescent stained specimen 30A also differs depending on the intensity.
  • this spectrum is referred to as the "absorption spectrum”. Therefore, as described above, the intensity of the excitation light corresponding to each of the multiple fluorescence spectra is aligned, so that the absorption spectrum can be appropriately evaluated.
  • the intensity of each excitation light which is the basis of the correction, i.e., the power ratio of each excitation light, may differ depending on the timing of acquisition of each fluorescence spectrum. This acquisition timing will be described in detail later.
  • a to D in FIG. 4 are specific examples of fluorescence spectra acquired by the image acquiring unit 112.
  • a to D in FIG. 4 show specific examples of fluorescence spectra acquired when the fluorescent stained specimen 30A contains, for example, four types of fluorescent substances, namely, DAPI, CK/AF488, PgR/AF594, and ER/AF647, and is irradiated with excitation light having excitation wavelengths of 405 [nm] (FIG. 4A), 488 [nm] (FIG. 4B), 532 [nm] (FIG. 4C), and 638 [nm] (FIG. 4D).
  • the fluorescence wavelength is shifted to a longer wavelength side than the excitation wavelength due to the release of energy for fluorescence emission (Stokes shift).
  • the fluorescent substances contained in the fluorescent stained specimen 30A and the excitation wavelength of the irradiated excitation light are not limited to those described above.
  • the linking unit 1311 extracts a fluorescence spectrum SP2 of a predetermined wavelength band from the fluorescence spectrum shown in FIG. 4B, extracts a fluorescence spectrum SP3 of a predetermined wavelength band from the fluorescence spectrum shown in FIG. 4C, and extracts a fluorescence spectrum SP4 of a predetermined wavelength band from the fluorescence spectrum shown in FIG. 4D.
  • the linking unit 1311 corrects, for example, the intensity of the fluorescence spectrum SP2 to 1.2 times and the wavelength resolution to 8 nm, corrects the intensity of the fluorescence spectrum SP3 to 1.5 times (no wavelength resolution correction), and corrects the intensity of the fluorescence spectrum SP4 to 4.0 times and the wavelength resolution to 4 nm.
  • the linking unit 1311 then links the corrected fluorescence spectra SP2 to SP4 in order to generate a linked fluorescence spectrum as shown in FIG. 4.
  • FIG. 4 shows a case where the linking unit 1311 extracts and links fluorescence spectra SP2 to SP4 of a predetermined bandwidth (e.g., 200 nm width) from the excitation wavelength when each fluorescence spectrum was acquired, but the bandwidth of the fluorescence spectrum extracted by the linking unit 1311 does not need to be the same for each fluorescence spectrum and may be different.
  • the region extracted by the linking unit 1311 from each fluorescence spectrum only needs to be a region that includes the peak wavelength of each fluorescence spectrum, and the wavelength band and bandwidth may be changed as appropriate.
  • the shift in spectral wavelength due to the Stokes shift may be taken into consideration. In this way, by narrowing down the wavelength band to be extracted, it is possible to reduce the amount of data, and therefore it is possible to perform the fluorescence separation process more quickly.
  • the intensity of the excitation light in this description may be the excitation power or the excitation power density, as described above.
  • the excitation power or the excitation power density may be the power or the power density obtained by actually measuring the excitation light emitted from the light source, or may be the power or the power density obtained from the driving voltage applied to the light source.
  • the intensity of the excitation light in this description may be a value obtained by correcting the above-mentioned excitation power density with the absorption rate of the slice to be observed for each excitation light, or the amplification rate of the detection signal in the detection system that detects the fluorescence emitted from the slice, for example, the image acquisition unit 112.
  • the intensity of the excitation light in this description may be the power density of the excitation light that actually contributed to the excitation of the fluorescent substance, or a value obtained by correcting the power density with the amplification rate of the detection system, etc.
  • the absorption rate and the amplification rate it is possible to appropriately correct the intensity of the excitation light that changes depending on the change in the machine state, the environment, etc., and therefore it is possible to generate a concatenated fluorescence spectrum that enables color separation with higher accuracy.
  • the correction value based on the intensity of the excitation light for each fluorescence spectrum is not limited to a value for aligning the intensity of the excitation light corresponding to each of the multiple fluorescence spectra, and may be modified in various ways.
  • the above correction value is also called an intensity correction value.
  • the signal intensity of a fluorescence spectrum having an intensity peak on the long wavelength side tends to be lower than the signal intensity of a fluorescence spectrum having an intensity peak on the short wavelength side.
  • a concatenated fluorescence spectrum includes both a fluorescence spectrum having an intensity peak on the long wavelength side and a fluorescence spectrum having an intensity peak on the short wavelength side
  • the fluorescence spectrum having an intensity peak on the long wavelength side is hardly taken into account, and only the fluorescence spectrum having an intensity peak on the short wavelength side may be extracted.
  • by setting a larger intensity correction value for the fluorescence spectrum having an intensity peak on the long wavelength side it is possible to improve the separation accuracy of the fluorescence spectrum having an intensity peak on the short wavelength side.
  • the fluorescence spectra are linked, but the intensity of each excitation light that is the basis for correction, i.e., the power ratio of each excitation light, may differ depending on the timing of acquisition of each fluorescence spectrum.
  • the power ratio of the multiple excitation lights does not match when the initial substance spectrum is acquired (initial substance spectrum acquisition time (1)), when the dye standard spectrum is acquired (dye standard spectrum acquisition time (2)), when the actual sample spectrum is acquired (actual sample spectrum acquisition time (3)), etc.
  • the intensity ratio when each fluorescence spectrum is linked i.e., the packing ratio (linking ratio) of each fluorescence spectrum, may differ.
  • the packing ratio of each fluorescence spectrum is, for example, the ratio of the individual areas of the intensity waveforms of each fluorescence spectrum having a predetermined bandwidth, and in the example of FIG. 4, it is the area ratio of each fluorescence spectrum SP2 to SP4.
  • the packing ratio of the fluorescence spectrum SP2, the fluorescence spectrum SP3, and the fluorescence spectrum SP4 is 1:2:0.5.
  • the packing ratios of each of the fluorescence spectra SP2 to SP4 may differ depending on the timing at which the packing ratios are acquired.
  • the packing ratio of each fluorescence spectrum when the initial substance spectrum is acquired (1) may not match the packing ratio of each fluorescence spectrum when the actual sample spectrum is acquired (3). Also, the packing ratio of each fluorescence spectrum when the initial substance spectrum is acquired (1) may not match the packing ratio of each fluorescence spectrum when the dye standard spectrum is acquired (2). Also, the packing ratio of each fluorescence spectrum when the dye standard spectrum is acquired (2) may not match the packing ratio of each fluorescence spectrum when the actual sample spectrum is acquired (3).
  • a spectrum is acquired without an actual sample.
  • the actual sample spectrum acquisition for example, a spectrum is acquired for an unstained actual sample and a spectrum is acquired for a stained actual sample.
  • the color separation unit 1321 includes, for example, a first color separation unit 1321a and a second color separation unit 1321b, and separates the combined fluorescence spectrum of the stained slices input from the connection unit 1311 into molecules by color.
  • a stained section is also called a stained sample.
  • the first color separation unit 1321a separates the concatenated fluorescence spectrum into spectra for each molecule by performing a color separation process on the concatenated fluorescence spectrum of the stained sample input from the connection unit 1311, using the concatenated fluorescence reference spectrum included in the reagent information and the concatenated autofluorescence reference spectrum included in the specimen information input from the information storage unit 121.
  • the color separation process may use least squares method (LSM), weighted least squares method (WLSM), non-negative matrix factorization (NMF), Restricted-NMF, NMF using a Gram matrix t AA, jNMF, Restricted-jNMF, or the like.
  • the second color separation unit 1321b separates the concatenated fluorescence spectrum into spectra for each molecule by performing a color separation process on the concatenated fluorescence spectrum of the stained sample input from the connection unit 1311 using the adjusted concatenated autofluorescence reference spectrum input from the spectrum extraction unit 1322.
  • a color separation process for example, LSM, WLSM, NMF, Restricted-NMF, NMF using a Gram matrix t AA, jNMF, Restricted-jNMF, etc. may be used, as in the first color separation unit 1321a.
  • LSM calculates the color mixing rate by, for example, fitting the concatenated fluorescence spectrum generated by the connection unit 1311 to a reference spectrum. Furthermore, in WLSM, weighting is applied so as to emphasize errors at low signal levels, taking advantage of the fact that the noise in the concatenated fluorescence spectrum (Signal), which is the measured value, has a Poisson distribution. However, the upper limit value at which weighting is not applied in WLSM is set as the Offset value. The Offset value is determined by the characteristics of the sensor used for measurement, and requires separate optimization when an image sensor is used as the sensor. Other various NMFs and various jNMFs will be described in detail later.
  • FIG. 3 illustrates an example in which the combined autofluorescence reference spectrum is adjusted once, but this is not limiting.
  • the color separation result by the second color separation unit 1321b may be input to the spectrum extraction unit 1322, and the spectrum extraction unit 1322 may repeat the process of adjusting the combined autofluorescence reference spectrum again one or more times to obtain the final color separation result.
  • the first color separation unit 1321a and the second color separation unit 1321b perform the fluorescence separation process using reference spectra linked in the wavelength direction (linked autofluorescence reference spectrum and linked fluorescence reference spectrum), and can output a unique spectrum as the separation result.
  • the separation results do not differ for each excitation wavelength. Therefore, the practitioner can more easily obtain the correct spectrum.
  • the reference spectrum for the autofluorescence used in the separation (linked autofluorescence reference spectrum) is automatically obtained and the fluorescence separation process is performed, eliminating the need for the practitioner to extract a spectrum equivalent to the autofluorescence from an appropriate space in the unstained section.
  • the spectrum extraction unit 1322 is configured to improve the combined autofluorescence reference spectrum so as to obtain a more accurate color separation result, and adjusts the combined autofluorescence reference spectrum included in the specimen information input from the information storage unit 121 based on the color separation result by the color separation unit 1321 so as to obtain a more accurate color separation result.
  • the spectrum extraction unit 1322 executes a spectrum extraction process using the color separation result input from the first color separation unit 1321a on the combined autofluorescence reference spectrum input from the information storage unit 121, and adjusts the combined autofluorescence reference spectrum based on the result, thereby improving the combined autofluorescence reference spectrum to obtain a more accurate color separation result.
  • jNMF joint non-negative matrix factorization
  • restricted-jNMF restricted-jNMF
  • LSM etc.
  • Second configuration example> A second configuration example of the analysis unit 131 according to this embodiment will be described with reference to Fig. 6.
  • Fig. 6 is a diagram showing the second configuration example of the analysis unit 131 according to this embodiment. In the second configuration example, color separation calculation of a stained image is performed using an autofluorescence spectrum extracted from an unstained image.
  • the analysis unit 131 performs the fluorescence separation process using an actually measured concatenated autofluorescence reference spectrum, i.e., the concatenated fluorescence spectrum of an unstained sample, as shown in FIG. 6. More specifically, the spectrum extraction unit 1322 extracts a concatenated autofluorescence reference spectrum for each autofluorescent substance from at least a portion of a plurality of autofluorescence spectra obtained by irradiating a specimen identical to or similar to the specimen 20A with a plurality of excitation lights having different excitation wavelengths, concatenated in the wavelength direction. The second color separation unit 1321b then performs the fluorescence separation process using the extracted concatenated autofluorescence reference spectrum and concatenated fluorescence reference spectrum, i.e., the same spectrum as in the first processing example, as reference spectra.
  • the analysis unit 131 basically has the same configuration as the analysis unit 131 described with reference to FIG. 3.
  • the spectrum extraction unit 1322 of the analysis unit 131 receives the concatenated fluorescence spectrum of the unstained section input from the connection unit 1311 instead of the concatenated autofluorescence reference spectrum included in the specimen information.
  • the unstained section is also called an unstained sample, and the concatenated fluorescence spectrum is also called a concatenated autofluorescence spectrum.
  • the spectrum extraction unit 1322 executes a spectrum extraction process using the color separation result input from the first color separation unit 1321a on the concatenated autofluorescence spectrum of the unstained sample input from the connection unit 1311, and adjusts the concatenated autofluorescence reference spectrum based on the result, thereby improving the concatenated autofluorescence reference spectrum to obtain a more accurate color separation result.
  • jNMF JNMF
  • Restricted-jNMF LSM, etc.
  • an unstained or stained section can be used as the same or similar section as specimen 20A used to extract the combined autofluorescence reference spectrum.
  • a section before staining used as the stained section a section adjacent to the stained section, a section different from the stained section in the same block, or a section in a different block in the same tissue can be used.
  • the same block is sampled from the same location as the stained section.
  • a different block is sampled from a different location than the stained section.
  • Fig. 7 is a diagram for explaining an example of processing for autofluorescence spectrum extraction according to this embodiment.
  • Fig. 8 is a flowchart showing the flow of an example of processing for autofluorescence spectrum extraction according to this embodiment.
  • Fig. 9 is a diagram for explaining jNMF when extracting autofluorescence spectrum according to this embodiment.
  • Fig. 10 is a diagram for explaining NMF when extracting autofluorescence spectrum according to this embodiment.
  • the spectrum extraction unit 1322 applies jNMF to autofluorescence spectrum extraction (e.g., autofluorescence reference spectrum extraction) in order to optimize the power ratio of each excitation light when acquiring the initial substance spectrum (1) and when acquiring the dye standard spectrum (2).
  • autofluorescence spectrum extraction e.g., autofluorescence reference spectrum extraction
  • step S11 the concatenation unit 1311 concatenates at least a portion of the multiple fluorescence spectra of the unstained sample in the wavelength direction to generate a concatenated fluorescence spectrum of the unstained sample.
  • step S12 the first color separation unit 1321a performs color separation processing (for example, NMF using the Gram matrix t AA ) on the concatenated fluorescence spectrum of the unstained sample.
  • color separation processing for example, NMF using the Gram matrix t AA
  • step S13 the spectrum extraction unit 1322 performs a spectrum extraction process using jNMF on the concatenated fluorescence spectrum of the unstained sample, using the color separation result input from the first color separation unit 1321a, i.e., the initial value (initial coefficient) of the concatenated autofluorescence spectrum.
  • a concatenated autofluorescence spectrum is extracted from the concatenated fluorescence spectrum of an unstained sample using jNMF, and the concatenated autofluorescence spectrum (e.g., concatenated autofluorescence reference spectrum) is adjusted based on the concatenated autofluorescence spectrum.
  • the concatenated autofluorescence spectrum e.g., concatenated autofluorescence reference spectrum
  • each autofluorescence spectrum corresponding to each excitation light is treated as a single autofluorescence spectrum (concatenated autofluorescence spectrum), but each autofluorescence spectrum is optimized independently.
  • the concatenated autofluorescence spectrum (e.g., concatenated autofluorescence reference spectrum) is updated while allowing some degree of freedom between each of the autofluorescence spectra contained in the concatenated autofluorescence spectrum.
  • the initial value of the combined autofluorescence spectrum is obtained in advance, for example, at the time of acquiring the initial substance spectrum (1), by a color separation process (e.g., NMF using the Gram matrix t ) performed by the first color separation unit 1321a or a spectrum extraction process (e.g., LSM) performed by the spectrum extraction unit 1322.
  • a color separation process e.g., NMF using the Gram matrix t
  • a spectrum extraction process e.g., LSM
  • step S14 the concatenation unit 1311 concatenates at least a portion of the multiple fluorescence spectra of the stained sample in the wavelength direction to generate a concatenated fluorescence spectrum of the stained sample.
  • step S15 the second color separation unit 1321b performs color separation processing (e.g., LSM) on the combined fluorescence spectrum of the stained sample using the adjusted combined autofluorescence reference spectrum input from the spectrum extraction unit 1322.
  • color separation processing e.g., LSM
  • jNMF is used in the spectrum extraction process, and we will compare the use of jNMF with the use of NMF.
  • Figure 9 shows an overview of the use of jNMF to extract the autofluorescence spectrum
  • Figure 10 shows an overview of the use of NMF to extract the autofluorescence spectrum.
  • jNMF is able to prevent the solution from settling into a local optimum solution depending on the initial value, and can adequately remove autofluorescence.
  • the area near the boundary between the fluorescence spectra for each excitation wavelength is not affected by adjacent fluorescence spectra, so autofluorescence can be adequately removed.
  • the spectrum extraction unit 1322 uses jNMF when extracting the autofluorescence spectrum with multiple fluorescence spectra linked in the wavelength direction.
  • the spectrum extraction unit 1322 prepares a matrix for each excitation wavelength so that the power ratio of each excitation wave is also subject to spectrum optimization, and performs spectrum optimization using jNMF.
  • the autofluorescence spectrum settles into a spectrum closer to the global optimal solution, enabling appropriate autofluorescence removal.
  • the use of jNMF makes it possible to optimize the autofluorescence spectrum by providing freedom between excitation wavelengths, and autofluorescence can be sufficiently removed.
  • jNMF The jNMF used by the spectrum extraction unit 1322 according to this embodiment to extract the autofluorescence spectrum and/or the fluorescence spectrum will be described with reference to Fig. 11.
  • Fig. 11 is a diagram for explaining the jNMF according to this embodiment. be.
  • jNMF joint NMF
  • NMF Non-negative Matrix Factorization
  • This jNMF can target multiple matrices, enabling integrated analysis of multi-omics data.
  • the NMF that is the basis of jNMF targets a single matrix, and therefore decomposes a matrix into two smaller matrices.
  • a matrix is an N x M matrix A that can be expressed as the product of matrices W and H.
  • Matrix A corresponds to the matrix X mentioned above.
  • matrices W and H are determined so that the mean square residual D between matrix A and the product (W*H) of matrix W and matrix H is minimized.
  • k is the number of clustering.
  • the mean square residual D is expressed by the formula in FIG. 11. Note that "norm(D, 'fro')" refers to the Frobenius norm of the mean square residual D.
  • matrix A corresponds to the spectrum (N is the number of pixels and M is the number of wavelength channels) before the autofluorescence spectrum (or autofluorescence reference spectrum) is extracted
  • matrix H corresponds to the extracted autofluorescence spectrum (or autofluorescence reference spectrum).
  • the number of autofluorescence spectra is the number of autofluorescent substances.
  • k is the number of autofluorescence spectra (or autofluorescence reference spectra)
  • M is the number of wavelength channels.
  • NMF Factorization in NMF uses an iterative method that starts with random initial values for matrices W and H, for example. While a value of k is required in NMF, the initial values of matrices W and H are not mandatory and can be set as an option, and once the initial values of matrices W and H are set, the solution is essentially constant. NMF can, for example, highlight the associations between matrix elements by decomposing latent elements rather than explicit clustering, and is also a method that is suitable for capturing outliers such as mutations and overexpression.
  • spectra similar in the wavelength direction or intensity direction are classified into the same class.
  • an image with a smaller number of pixels than the stained image is generated, and it is possible to reduce the size of the matrix that uses this image as input.
  • FIG. 12 is a diagram showing a wide area of a fluorescence image G1 with jNMF applied and a fluorescence image G2 with NMF applied according to this embodiment.
  • FIG. 13 is a diagram showing the transition of the autofluorescence spectrum according to this embodiment.
  • FIG. 14 is a diagram showing an enlarged view of multiple fluorescence images G11-G19 with jNMF applied according to this embodiment.
  • FIG. 15 is a diagram showing an enlarged view of multiple fluorescence images G21-G29 with NMF applied according to this embodiment.
  • FIG. 16 is a diagram showing an enlarged view of a fluorescence image G13 with jNMF applied and a fluorescence image G23 with NMF applied according to this embodiment.
  • a wide area of fluorescence image G1 with jNMF applied and a wide area of fluorescence image G2 with NMF applied are shown.
  • fluorescence image G1 with jNMF applied jNMF is used to extract the autofluorescence spectrum
  • fluorescence image G2 with NMF applied NMF is used to extract the autofluorescence spectrum. It can be seen that the degree of fluorescence in fluorescence image G2 with NMF applied is stronger than the degree of fluorescence in fluorescence image G1 with jNMF applied.
  • the upper part of FIG. 13 shows the initial autofluorescence spectrum (autofluorescence spectra A11 to A15), the middle part of FIG. 13 shows the autofluorescence spectrum when jNMF is applied (autofluorescence spectra B11 to B14), and the lower part of FIG. 13 shows the autofluorescence spectrum when NMF is applied (autofluorescence spectra C11 to A14).
  • autofluorescent substances include arachidonic acid, elastin, FAD (flavin adenine dinucleotide), PLD, and protoporphyrin.
  • fluorescence image G11 is an image corresponding to AF488/CD7.
  • AF488 is one of the dyes
  • CD7 is one of the antibody channels (and so on).
  • Fluorescence image G12 is an image corresponding to AF555/CD3
  • fluorescence image G13 is an image corresponding to eF615/CD20
  • fluorescence image G14 is an image corresponding to AF647/CD5.
  • Fluorescence image G15 is an image corresponding to DIFF. DIFF is one of the staining methods.
  • Each of the fluorescence images G16 to G19 is an image corresponding to each of AutoF1 to F4, respectively. These AutoF1 to F4 correspond to the process of applying jNMF to extract the autofluorescence spectrum.
  • fluorescence image G21 is an image corresponding to AF488/CD7
  • fluorescence image G22 is an image corresponding to AF555/CD3
  • fluorescence image G23 is an image corresponding to eF615/CD20
  • fluorescence image G24 is an image corresponding to AF647/CD5
  • fluorescence image G25 is an image corresponding to DIFF.
  • Each of the fluorescence images G26 to G29 is an image corresponding to each of AutoF1 to F4. These AutoF1 to F4 correspond to the process of applying NMF to extract the autofluorescence spectrum.
  • fluorescence image G13 with jNMF applied was selected and are shown in Figure 16.
  • Fluorescence image G13 with jNMF applied is shown in the upper part of Figure 16
  • fluorescence image G23 with NMF applied is shown in the lower part of Figure 16.
  • the fluorescence image G13 in which jNMF was applied it can be seen that the part corresponding to the autofluorescence has been removed while leaving the original eF615/CD20 signal.
  • the fluorescence image G13 in which jNMF was applied is an image in which autofluorescence has been highly effectively removed, and autofluorescence has been sufficiently removed from the fluorescence image G13.
  • This fluorescence image G13 will also be an important image in the cell analysis in the subsequent process.
  • Comparative Example 2 Comparative Example 2 between the jNMF-applied fluorescence images G31 to G39 and the NMF-applied fluorescence images G41 to G49 according to the present embodiment will be described with reference to Fig. 17 to Fig. 20.
  • Comparative Example 2 a sample different from that in Comparative Example 1 is used, and processing is performed on that sample.
  • FIG. 17 is a diagram showing the transition of the autofluorescence spectrum according to this embodiment.
  • FIG. 18 is a diagram showing an enlarged view of multiple fluorescence images G31 to G39 using jNMF according to this embodiment.
  • FIG. 19 is a diagram showing an enlarged view of multiple fluorescence images G41 to G49 using NMF according to this embodiment.
  • FIG. 20 is a diagram showing an enlarged view of fluorescence image G33 using jNMF and fluorescence image G43 using NMF according to this embodiment.
  • the upper part of FIG. 17 shows the initial autofluorescence spectra (autofluorescence spectra A21 to A25), the middle part of FIG. 17 shows the autofluorescence spectra when jNMF is applied (autofluorescence spectra B21 to B24), and the lower part of FIG. 17 shows the autofluorescence spectra when NMF is applied (autofluorescence spectra C21 to A24).
  • the autofluorescent substances are, for example, the same as those in the example of FIG. 13.
  • an autofluorescence spectrum B21 of AutoF1 with jNMF applied is obtained.
  • NMF is applied to the initial autofluorescence spectrum A21
  • an autofluorescence spectrum C21 of AutoF1 with NMF applied is obtained.
  • fluorescence image G31 is an image corresponding to AF488/CD7
  • fluorescence image G32 is an image corresponding to AF555/CD3
  • fluorescence image G33 is an image corresponding to eF615/CD20
  • fluorescence image G34 is an image corresponding to AF647/CD5
  • fluorescence image G35 is an image corresponding to DIFF.
  • Each of the fluorescence images G36 to G39 is an image corresponding to each of AutoF1 to F4. These AutoF1 to F4 correspond to the process of applying jNMF to extract the autofluorescence spectrum.
  • fluorescence image G41 is an image corresponding to AF488/CD7
  • fluorescence image G42 is an image corresponding to AF555/CD3
  • fluorescence image G43 is an image corresponding to eF615/CD20
  • fluorescence image G44 is an image corresponding to AF647/CD5
  • fluorescence image G45 is an image corresponding to DIFF.
  • Each of the fluorescence images G46 to G49 is an image corresponding to each of AutoF1 to F4. These AutoF1 to F4 correspond to the process of applying NMF to extract the autofluorescence spectrum.
  • fluorescence image G33 with jNMF applied and fluorescence image G43 with NMF applied were selected and are shown in Figure 20.
  • Fluorescence image G33 with jNMF applied (eF615/CD20) is shown in the upper part of Figure 20
  • fluorescence image G43 with NMF applied is shown in the lower part of Figure 20.
  • the fluorescence image G33 in which jNMF was applied it can be seen that the part corresponding to the autofluorescence has been removed while leaving the original eF615/CD20 signal.
  • the fluorescence image G33 in which jNMF was applied is an image in which autofluorescence has been highly effectively removed, and autofluorescence has been sufficiently removed from the fluorescence image G33.
  • This fluorescence image G33 will also be an important image in the cell analysis in the subsequent process.
  • Fig. 21 is a diagram for explaining an example of the long wavelength shift of the autofluorescence spectrum according to this embodiment.
  • Autofluorescence spectrum D1 in Figure 21 is the autofluorescence spectrum when NMF is applied
  • autofluorescence spectrum D2 in Figure 21 is the autofluorescence spectrum when jNMF is applied.
  • the fluorescence spectrum ratio corresponding to each excitation light changes significantly due to the long wavelength shift.
  • the fluorescence spectrum ratio corresponding to each excitation light does not change as much as in autofluorescence spectrum D1. This shows that spectrum optimization according to the degree of fibrosis is possible.
  • This fluorescence separation process is a process for optimizing the power ratio (e.g., laser power ratio) of each excitation light when the initial substance spectrum is acquired (1) and when the actual sample spectrum is acquired (3).
  • the fluorescence separation process is executed by the color separation unit 1321 or the spectrum extraction unit 1322.
  • FIG. 22 is a diagram for explaining the process of optimizing the power ratio of each excitation light when acquiring an initial substance spectrum (1) and when acquiring an actual sample spectrum (3) according to this embodiment.
  • FIG. 23 is a flowchart showing the flow of the fluorescence separation process according to this embodiment.
  • FIG. 24 is a diagram for explaining the flow of the first process in the first loop of the fluorescence separation process according to this embodiment.
  • FIG. 25 is a diagram for explaining the flow of the second process in the first loop of the fluorescence separation process according to this embodiment.
  • the power ratio of each excitation light may differ at multiple times.
  • the optimization of the power ratio of each excitation light when the initial substance spectrum is acquired (1) and when the dye standard spectrum is acquired (2) can be achieved by the aforementioned jNMF (the aforementioned spectrum extraction process), but as shown in FIG. 22, the optimization of the power ratio of each excitation light when the initial substance spectrum is acquired (1) and when the actual sample spectrum is acquired (3) can be achieved by Restricted-jNMF (hereinafter referred to as R-jNMF).
  • R-jNMF is basically the same as jNMF, but it is a jNMF with restrictions. This restriction means, for example, replacing the dye portion (dye spectrum) corresponding to the dye fluorescence spectrum with a constant value. The dye portion is replaced with a constant value, for example, within a loop (for example, a for loop).
  • (1) is a method in which the dye fluorescence spectrum (dye part) is fixed including the power ratio of each excitation light, and only the autofluorescence spectrum is optimized including the power ratio of each excitation light.
  • the second method, (2) is a method in which only the power ratio of each excitation light of the dye fluorescence spectrum (dye part) is optimized, and the autofluorescence spectrum is optimized including the power ratio of each excitation light. Note that the power ratio of each excitation light can be calculated from the area ratio of the individual fluorescence spectra of each excitation light, as mentioned above.
  • the variable i is reset to zero (step S401).
  • the variable i indicates the number of times factorization in the R-jNMF has been repeated. Therefore, the matrix H 0 shown in E1 of Fig. 24 or F1 of Fig. 25 corresponds to the initial value of the matrix H. Note that, in this example, for clarification, the position of the dye fluorescence spectrum in the matrix H is the top row, but this is not limited thereto and can be changed in various ways, such as the bottom row or an intermediate row.
  • a non-negative N row and M column (N ⁇ M) matrix A is divided by a non-negative N row and k column (N ⁇ k) matrix W i to obtain a non-negative k row and M column (k ⁇ M) matrix H i+1 (step S402).
  • a matrix H 1 shown in E2 of FIG. 24 or F2 of FIG. 25 is obtained.
  • the rows of the fluorescent dye spectra in the matrix H i+1 obtained in step S402 are replaced with the initial values of the fluorescent dye spectra, i.e., the rows of the dye fluorescent spectra in the matrix H 0 (step S403). That is, in this embodiment, the fluorescent dye spectra in the matrix H are fixed to their initial values. For example, in the first loop, as shown in E3 of Fig. 24 or F3 of Fig. 25, it is possible to fix the dye fluorescent spectra by replacing the top row in the matrix H 1 with the top row in the matrix H 0 .
  • the dye fluorescence spectrum is fixed including the power ratio of each excitation light.
  • the dye fluorescence spectrum is fixed without including the power ratio of each excitation light.
  • the matrix A is divided by the matrix H i+1 obtained in step S403 to obtain the matrix W i+1 (step S404).
  • step S405 it is determined whether or not the mean square residual D satisfies a predetermined branching condition (step S405), and if the predetermined branching condition is satisfied (Yes in step S405), the finally obtained matrices H i+1 and W i+1 are taken as the solution, and the R-NMF is terminated.
  • the predetermined branching condition is not satisfied (No in step S405), the variable i is incremented by 1 (step S406), and then the process returns to step S402, and the next loop is executed.
  • the information processing device 100 includes a connection unit 1311 that connects multiple fluorescence spectra obtained by irradiating a specimen with multiple excitation light beams in the wavelength direction, a color separation unit 1321 that separates the connected fluorescence spectrum obtained by connecting the multiple fluorescence spectra into spectra for each fluorescent substance, and a spectrum extraction unit 1322 that extracts a connected autofluorescence spectrum obtained by connecting multiple autofluorescence spectra from the connected fluorescence spectrum using jNMF.
  • the spectrum extraction unit 1322 may also prepare a matrix for each wavelength of excitation light and execute jNMF. This makes it possible to extract an appropriate concatenated autofluorescence spectrum and reliably obtain an image with reduced autofluorescence.
  • the spectrum extraction unit 1322 may also prepare a matrix for each wavelength to include the power ratio of each excitation light. This makes it possible to reliably subject the power ratio of each excitation wave to spectrum optimization, so that an appropriate combined autofluorescence spectrum can be extracted and an image with reduced autofluorescence can be reliably obtained.
  • the color separation unit 1321 or the spectrum extraction unit 1322 may also use Restricted-jNMF to separate the concatenated fluorescence spectrum into spectra for each fluorescent substance. This makes it possible to reliably obtain an image with reduced autofluorescence.
  • the color separation unit 1321 or the spectrum extraction unit 1322 may fix the power ratio of each excitation light in the pigment portion of the dye fluorescence spectrum and execute Restricted-jNMF. This makes it possible to reliably obtain an image with reduced autofluorescence.
  • the color separation unit 1321 or the spectrum extraction unit 1322 may execute Restricted-jNMF, including the power ratio of each excitation light in the pigment portion of the dye fluorescence spectrum. This makes it possible to reliably obtain an image with reduced autofluorescence.
  • the color separation unit 1321 or the spectrum extraction unit 1322 may also multiply the initial value of the dye fluorescence spectrum by the power ratio of each excitation light and execute Restricted-jNMF. This makes it possible to reliably obtain an image with reduced autofluorescence.
  • the power ratio of each excitation light may be determined from the combined autofluorescence spectrum. This ensures that an image with reduced autofluorescence can be obtained.
  • a fluorescence observation device 500 which is an example of a microscope system.
  • a configuration example of the applicable fluorescence observation device 500 will be described with reference to Figs. 26 and 27.
  • Fig. 26 is a diagram showing an example of a schematic configuration of the fluorescence observation device 500 according to this embodiment.
  • Fig. 27 is a diagram showing an example of a schematic configuration of the observation unit 1 according to this embodiment.
  • the fluorescence observation device 500 has an observation unit 1, a processing unit 2, and a display unit 3.
  • the observation unit 1 includes an excitation section (irradiation section) 10, a stage 20, a spectroscopic imaging section 30, an observation optical system 40, a scanning mechanism 50, a focusing mechanism 60, and a non-fluorescent observation section 70.
  • the excitation unit 10 irradiates the observation object with multiple illumination lights of different wavelengths.
  • the excitation unit 10 irradiates the observation object, i.e., a pathological specimen, with multiple line illumination lights of different wavelengths arranged parallel to different axes.
  • the stage 20 is a platform that supports the pathological specimen, and is configured to be movable by the scanning mechanism 50 in a direction perpendicular to the direction of the line light from the line illumination.
  • the spectroscopic imaging unit 30 includes a spectroscope, and acquires the fluorescence spectrum of the pathological specimen excited in a line shape by the line illumination, i.e., spectroscopic data.
  • the observation unit 1 functions as a line spectroscope that acquires spectral data corresponding to the line illumination.
  • the observation unit 1 also functions as an imaging device that captures multiple fluorescent images generated by the pathological specimen, which is the imaging target, for each of multiple fluorescent wavelengths, line by line, and acquires data on the captured multiple fluorescent images in the order of the lines.
  • different axes but parallel means that the multiple line lights are on different axes and parallel.
  • “Different axes” means that they are not on the same axis, and there is no particular restriction on the distance between the axes.
  • "Parallel” is not limited to “parallel” in the strict sense, but also includes a state where the lights are nearly parallel. For example, there may be deviations from the parallel state due to distortion from the optical system such as lenses, or manufacturing tolerances, and in this case they are still considered to be parallel.
  • the excitation section 10 and the spectral imaging section 30 are connected to the stage 20 via the observation optical system 40.
  • the observation optical system 40 has the function of tracking the optimal focus using a focus mechanism 60.
  • a non-fluorescence observation section 70 for performing dark-field observation, bright-field observation, etc. may be connected to the observation optical system 40.
  • a control section 80 that controls the excitation section 10, the spectral imaging section 30, the scanning mechanism 50, the focus mechanism 60, the non-fluorescence observation section 70, etc. may be connected to the observation unit 1.
  • the processing unit 2 includes a memory section 21, a data calibration section 22, and an image formation section 23. Based on the fluorescence spectrum of the pathological specimen acquired by the observation unit 1, the processing unit 2 typically forms an image of the pathological specimen, or outputs the distribution of the fluorescence spectrum.
  • the pathological specimen is also referred to as the sample S.
  • the image here refers to the composition ratio of the pigments that make up the spectrum and the autofluorescence derived from the sample, the waveform converted into RGB (red, green, blue) colors, the luminance distribution of a specific wavelength band, etc.
  • the storage unit 21 includes a non-volatile storage medium such as a hard disk drive or flash memory, and a storage control unit that controls the writing and reading of data to the storage medium.
  • the storage unit 21 stores spectroscopic data that indicates the correlation between each wavelength of light emitted by each of the multiple line lights included in the excitation unit 10 and the fluorescence received by the camera of the spectral imaging unit 30.
  • the storage unit 21 also stores in advance information indicating the standard spectrum of the autofluorescence for the sample (pathological specimen) to be observed, and information indicating the standard spectrum of the dye alone that stains the sample.
  • the data calibration unit 22 constructs the spectroscopic data stored in the memory unit 21 based on the captured image captured by the camera of the spectroscopic imaging unit 30.
  • the image formation unit 23 forms a fluorescent image of the sample based on the spectroscopic data and the interval ⁇ y between the multiple line lights irradiated by the excitation unit 10.
  • the processing unit 2 including the data calibration unit 22 and the image formation unit 23 is realized by hardware elements used in computers such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory) and necessary programs (software).
  • a PLD Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • the display unit 3 displays various information such as an image based on the fluorescent image formed by the image forming unit 23.
  • the display unit 3 may be configured as a monitor integrally attached to the processing unit 2, or may be a display device connected to the processing unit 2.
  • the display unit 3 includes a display element such as a liquid crystal device or an organic EL device, and a touch sensor, and is configured as a UI (User Interface) that displays input settings for shooting conditions, captured images, etc.
  • UI User Interface
  • the excitation unit 10 includes two line illuminators Ex1 and Ex2, each of which emits light at two wavelengths.
  • the line illuminator Ex1 emits light with wavelengths of 405 nm and light with wavelengths of 561 nm
  • the line illuminator Ex2 emits light with wavelengths of 488 nm and light with wavelengths of 645 nm.
  • the excitation unit 10 has multiple collimator lenses 11, multiple laser line filters 12, multiple dichroic mirrors 13a, 13b, and 13c, a homogenizer 14, a condenser lens 15, and an entrance slit 16 to correspond to each of the excitation light sources L1 to L4.
  • the laser light emitted from excitation light source L1 and the laser light emitted from excitation light source L3 are each collimated by collimator lens 11, then pass through laser line filter 12 to cut the base of each wavelength band, and are made coaxial by dichroic mirror 13a.
  • the two coaxial laser lights are further beam-shaped by homogenizer 14, such as a fly-eye lens, and condenser lens 15 to become line illumination Ex1.
  • the laser light emitted from excitation light source L2 and the laser light emitted from excitation light source L4 are similarly made coaxial by dichroic mirrors 13b and 13c, and are converted into line illumination to become line illumination Ex2 having a different axis from line illumination Ex1.
  • Line illumination Ex1 and Ex2 form different-axis line illuminations, i.e., primary images, separated by a distance ⁇ y at entrance slit 16, which has multiple slit sections through which each of the line illuminations can pass.
  • the primary image is irradiated onto the sample S on the stage 20 via the observation optical system 40.
  • the observation optical system 40 has a condenser lens 41, dichroic mirrors 42 and 43, an objective lens 44, a bandpass filter 45, and a condenser lens 46.
  • the condenser lens 46 is an example of an imaging lens.
  • the line illumination Ex1 and Ex2 are collimated by the condenser lens 41 paired with the objective lens 44, reflected by the dichroic mirrors 42 and 43, transmitted through the objective lens 44, and irradiated onto the sample S on the stage 20.
  • FIG. 28 is a diagram showing an example of a sample S according to this embodiment.
  • FIG. 28 shows the sample S as viewed from the irradiation direction of line illumination Ex1 and Ex2, which are excitation light.
  • the sample S is typically composed of a slide including an observation object Sa, such as a tissue slice as shown in FIG. 28, but of course it can be something else.
  • the observation object Sa is, for example, a biological sample such as nucleic acid, cells, proteins, bacteria, or viruses.
  • the sample S, i.e., the observation object Sa is stained with multiple fluorescent dyes.
  • the observation unit 1 observes the sample S by magnifying it to a desired magnification.
  • FIG. 29 is an enlarged view of area A where line illumination Ex1 and Ex2 are irradiated on sample S according to this embodiment.
  • two line illuminations Ex1 and Ex2 are arranged in area A, and the shooting areas R1 and R2 of the spectral imaging unit 30 are arranged so as to overlap with the respective line illuminations Ex1 and Ex2.
  • the two line illuminations Ex1 and Ex2 are each parallel to the Z-axis direction and are arranged a predetermined distance ⁇ y apart in the Y-axis direction.
  • line illuminations Ex1 and Ex2 are formed as shown in FIG. 29. Fluorescence excited in sample S by these line illuminations Ex1 and Ex2 is collected by objective lens 44 as shown in FIG. 27, reflected by dichroic mirror 43, transmitted through dichroic mirror 42 and bandpass filter 45 that cuts excitation light, collected again by condenser lens 46, and enters spectroscopic imaging unit 30.
  • the spectral imaging unit 30 has an observation slit 31, an image sensor 32, a first prism 33, a mirror 34, a diffraction grating 35, and a second prism 36.
  • the observation slit 31 is an opening.
  • the diffraction grating 35 is, for example, a wavelength dispersion element.
  • the image sensor 32 includes two image sensors 32a and 32b.
  • the image sensor 32 receives a plurality of light beams, such as fluorescent light, that are wavelength-dispersed by a diffraction grating 35.
  • the image sensor 32 may be, for example, a two-dimensional imager such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).
  • the observation slit 31 is placed at the focal point of the condenser lens 46, and has the same number of slits as the number of excitation lines, two in this example.
  • the fluorescence spectra from the two excitation lines that pass through the observation slit 31 are separated by the first prism 33, and are further separated into fluorescence spectra of each excitation wavelength by being reflected by the grating surface of the diffraction grating 35 via the mirror 34.
  • the four separated fluorescence spectra are incident on the image sensors 32a and 32b via the mirror 34 and the second prism 36, and are expanded as spectral data into spectral data (x, ⁇ ) expressed by the position x in the line direction and the wavelength ⁇ .
  • the spectral data (x, ⁇ ) is the pixel value of the pixel at the position x in the row direction and the wavelength ⁇ in the column direction among the pixels included in the image sensor 32. Note that the spectral data (x, ⁇ ) is sometimes simply described as spectral data.
  • the pixel size (nm/pixel) of the image sensors 32a and 32b is not particularly limited, and is set to, for example, 2 (nm/pixel) or more and 20 (nm/pixel) or less.
  • This dispersion value may be realized optically by the pitch of the diffraction grating 35, or may be realized by using hardware binning of the image sensors 32a and 32b.
  • a dichroic mirror 42 and a bandpass filter 45 are inserted in the optical path to prevent the excitation light, i.e., the line illumination Ex1 and Ex2, from reaching the image sensor 32.
  • Each of the line illuminations Ex1 and Ex2 is not limited to being composed of a single wavelength, and may each be composed of multiple wavelengths.
  • the fluorescence excited by these also contains multiple spectra.
  • the spectral imaging unit 30 has a wavelength dispersion element for separating the fluorescence into spectra derived from the excitation wavelengths.
  • the wavelength dispersion element is composed of a diffraction grating, a prism, or the like, and is typically placed on the optical path between the observation slit 31 and the image sensor 32.
  • the stage 20 and the scanning mechanism 50 form an XY stage, and the sample S is moved in the X-axis and Y-axis directions to obtain a fluorescent image of the sample S.
  • WSI Whole Slide Imaging
  • the sample S is scanned in the Y-axis direction, then moved in the X-axis direction, and then scanned again in the Y-axis direction.
  • dye spectra i.e., fluorescence spectra
  • ⁇ y on the sample S
  • the scanning mechanism 50 changes the position on the sample S where the illumination light is irradiated over time. For example, the scanning mechanism 50 scans the stage 20 in the Y-axis direction.
  • This scanning mechanism 50 can scan the stage 20 with the multiple line illuminations Ex1 and Ex2 in the Y-axis direction, that is, in the arrangement direction of each line illumination Ex1 and Ex2. This is not limited to this example, and the multiple line illuminations Ex1 and Ex2 may be scanned in the Y-axis direction by a galvanometer mirror arranged midway through the optical system.
  • Data derived from each line illumination Ex1 and Ex2 becomes data whose coordinates are shifted by a distance ⁇ y on the Y-axis, and is corrected and output based on the distance ⁇ y stored in advance or the value of the distance ⁇ y calculated from the output of the image sensor 32.
  • the non-fluorescent observation section 70 is composed of a light source 71, a dichroic mirror 43, an objective lens 44, a condenser lens 72, an image sensor 73, etc.
  • the non-fluorescent observation section 70 shows an observation system using dark-field illumination.
  • the light source 71 is disposed on the side of the stage 20 opposite the objective lens 44, and irradiates the sample S on the stage 20 with illumination light from the side opposite the line illumination Ex1 and Ex2.
  • the light source 71 illuminates from outside the NA (numerical aperture) of the objective lens 44, and the light diffracted by the sample S (dark-field image) is captured by the image sensor 73 via the objective lens 44, dichroic mirror 43, and condenser lens 72.
  • dark-field illumination even seemingly transparent samples such as fluorescently stained samples can be observed with contrast.
  • the dark-field image may be observed simultaneously with the fluorescence and used for real-time focusing.
  • the illumination wavelength may be selected so that it does not affect the fluorescence observation.
  • the non-fluorescence observation unit 70 is not limited to an observation system that acquires a dark-field image, but may be configured with an observation system that can acquire non-fluorescence images such as bright-field images, phase-contrast images, phase images, and in-line hologram images.
  • various observation methods such as the Schlieren method, phase-contrast method, polarized observation method, and epi-illumination method can be used as a method for acquiring non-fluorescence images.
  • the position of the illumination light source is not limited to below the stage 20, but may be above the stage 20 or around the objective lens 44.
  • other methods such as a pre-focus map method in which the focus coordinate (Z coordinate) is recorded in advance may also be used.
  • the line illumination as excitation light is composed of two line illuminations Ex1 and Ex2, but this is not limited to this and may be three, four, five or more.
  • Each line illumination may also include multiple excitation wavelengths selected so as to minimize degradation of color separation performance. Even if there is only one line illumination, if the excitation light source is composed of multiple excitation wavelengths and each excitation wavelength is linked to and recorded with the data obtained by the image sensor 32, a multi-color spectrum can be obtained, although the separation ability is not as good as that of different parallel axes.
  • the above describes an example of application of the technology disclosed herein to the fluorescence observation device 500.
  • the above configuration described with reference to Figs. 26 and 27 is merely one example, and the configuration of the fluorescence observation device 500 according to this embodiment is not limited to this example.
  • the fluorescence observation device 500 does not necessarily have to include all of the configurations shown in Figs. 26 and 27, and may include configurations that are not shown in Figs. 26 and 27.
  • a microscope device 5100 which is a part of the microscope system 5000, functions as an imaging device.
  • the microscope system 5000 shown in FIG. 30 includes a microscope device 5100, a control unit 5110, and an information processing unit 5120.
  • the microscope device 5100 includes a light irradiation unit 5101, an optical unit 5102, and a signal acquisition unit 5103.
  • the microscope device 5100 may further include a sample mounting unit 5104 on which a biological sample S is placed.
  • the configuration of the microscope device is not limited to that shown in FIG. 30, and for example, the light irradiation unit 5101 may be present outside the microscope device 5100, and for example, a light source not included in the microscope device 5100 may be used as the light irradiation unit 5101.
  • the light irradiation unit 5101 may be arranged so that the sample mounting unit 5104 is sandwiched between the light irradiation unit 5101 and the optical unit 5102, and may be arranged on the side where the optical unit 5102 is present, for example.
  • the microscope device 5100 may be configured to perform one or more of bright-field observation, phase-contrast observation, differential interference observation, polarized observation, fluorescent observation, and dark-field observation.
  • the microscope system 5000 may be configured as a so-called WSI (Whole Slide Imaging) system or a digital pathology imaging system, and may be used for pathological diagnosis.
  • the microscope system 5000 may also be configured as a fluorescence imaging system, in particular a multiplexed fluorescence imaging system.
  • the microscope system 5000 may be used to perform intraoperative pathological diagnosis or remote pathological diagnosis.
  • the microscope device 5100 may acquire data of a biological sample S obtained from a subject of the surgery while the surgery is being performed, and transmit the data to the information processing unit 5120.
  • the microscope device 5100 may transmit data of the acquired biological sample S to the information processing unit 5120 located in a location away from the microscope device 5100 (such as a different room or building).
  • the information processing unit 5120 receives and outputs the data.
  • a user of the information processing unit 5120 may perform a pathological diagnosis based on the output data.
  • the biological sample S may be a sample containing a biological component.
  • the biological component may be a biological tissue, a cell, a liquid component of a biological body (such as blood or urine), a culture, or a living cell (such as a cardiac muscle cell, a neuron, or a fertilized egg).
  • the biological sample may be a solid, and may be a specimen fixed with a fixing agent such as paraffin or a solid formed by freezing.
  • the biological sample may be a slice of the solid.
  • a specific example of the biological sample may be a slice of a biopsy sample.
  • the biological sample may have been subjected to a process such as staining or labeling.
  • the process may be staining to show the morphology of the biological component or to show substances (such as surface antigens) contained in the biological component, and examples of such staining include HE (Hematoxylin-Eosin) staining and immunohistochemistry staining.
  • the biological sample may have been subjected to the process using one or more reagents, and the reagents may be fluorescent dyes, color-developing reagents, fluorescent proteins, or fluorescently labeled antibodies.
  • the specimen may be prepared from a tissue sample for the purpose of pathological diagnosis or clinical testing.
  • the specimen may be derived not only from the human body, but also from animals, plants, or other materials.
  • the characteristics of the specimen vary depending on the type of tissue (e.g., organs or cells) used, the type of disease being treated, the attributes of the subject (e.g., age, sex, blood type, or race), or the lifestyle of the subject (e.g., diet, exercise, or smoking habits).
  • the specimens may be managed with identification information (e.g., barcodes or QR codes (registered trademarks)) that can identify each specimen.
  • the light irradiation unit 5101 is a light source for illuminating the biological sample S, and an optical unit for guiding the light irradiated from the light source to the specimen.
  • the light source can irradiate the biological sample with visible light, ultraviolet light, or infrared light, or a combination of these.
  • the light source can be one or more of a halogen light source, a laser light source, an LED light source, a mercury light source, and a xenon light source.
  • the type and/or wavelength of the light source in the fluorescence observation may be multiple and may be appropriately selected by a person skilled in the art.
  • the light irradiation unit can have a transmissive, reflective, or epi-illumination (coaxial epi-illumination or lateral illumination) configuration.
  • the optical unit 5102 is configured to guide light from the biological sample S to the signal acquisition unit 5103.
  • the optical unit can be configured to enable the microscope device 5100 to observe or image the biological sample S.
  • the optical unit 5102 may include an objective lens. The type of objective lens may be appropriately selected by a person skilled in the art depending on the observation method.
  • the optical unit may also include a relay lens for relaying an image magnified by the objective lens to the signal acquisition unit.
  • the optical unit may further include optical components other than the objective lens and the relay lens, such as an eyepiece lens, a phase plate, and a condenser lens.
  • the optical unit 5102 may further include a wavelength separation unit configured to separate light having a predetermined wavelength from the light from the biological sample S.
  • the wavelength separation unit may be configured to selectively allow light of a predetermined wavelength or wavelength range to reach the signal acquisition unit.
  • the wavelength separation unit may include, for example, one or more of a filter that selectively transmits light, a polarizing plate, a prism (Wollaston prism), and a diffraction grating.
  • the optical components included in the wavelength separation unit may be disposed, for example, on the optical path from the objective lens to the signal acquisition unit.
  • the wavelength separation unit is provided in the microscope device when fluorescence observation is performed, particularly when the microscope device includes an excitation light irradiation unit.
  • the wavelength separation unit may be configured to separate fluorescent light from each other or to separate white light from fluorescent light.
  • the signal acquisition unit 5103 may be configured to receive light from the biological sample S and convert the light into an electrical signal, particularly a digital electrical signal.
  • the signal acquisition unit may be configured to acquire data related to the biological sample S based on the electrical signal.
  • the signal acquisition unit may be configured to acquire data of an image (image, particularly a still image, a time lapse image, or a moving image) of the biological sample S, particularly configured to acquire data of an image enlarged by the optical unit.
  • the signal acquisition unit includes one or more imaging elements, such as a CMOS or a CCD, having a plurality of pixels arranged in one or two dimensions.
  • the signal acquisition unit may include an imaging element for acquiring a low-resolution image and an imaging element for acquiring a high-resolution image, or may include an imaging element for sensing for AF or the like and an imaging element for outputting an image for observation or the like.
  • the imaging element may include a signal processing unit (including one or more of a CPU, a DSP, and a memory) that performs signal processing using pixel signals from each pixel, and an output control unit that controls the output of image data generated from the pixel signals and processed data generated by the signal processing unit.
  • the imaging element including the plurality of pixels, the signal processing unit, and the output control unit may be preferably configured as a one-chip semiconductor device.
  • the microscope system 5000 may further include an event detection sensor.
  • the event detection sensor may include a pixel that photoelectrically converts incident light, and may be configured to detect an event when a luminance change of the pixel exceeds a predetermined threshold.
  • the event detection sensor may be particularly of an asynchronous type.
  • the control unit 5110 controls the imaging by the microscope device 5100.
  • the control unit may drive the movement of the optical unit 5102 and/or the sample placement unit 5104 to adjust the positional relationship between the optical unit and the sample placement unit.
  • the control unit 5110 may move the optical unit and/or the sample placement unit in a direction approaching or moving away from each other (for example, in the optical axis direction of the objective lens).
  • the control unit may also move the optical unit and/or the sample placement unit in any direction in a plane perpendicular to the optical axis direction.
  • the control unit may control the light irradiation unit 5101 and/or the signal acquisition unit 5103.
  • the sample placement unit 5104 may be configured to fix the position of the biological sample on the sample placement unit, and may be a so-called stage.
  • the sample placement unit 5104 may be configured to move the position of the biological sample in the optical axis direction of the objective lens and/or in a direction perpendicular to the optical axis direction.
  • the information processing unit 5120 may acquire data (such as imaging data) acquired by the microscope device 5100 from the microscope device 5100.
  • the information processing unit may execute image processing on the imaging data.
  • the image processing may include an unmixing process, particularly a spectral unmixing process.
  • the unmixing process may include a process of extracting data of a light component of a predetermined wavelength or wavelength range from the imaging data to generate image data, or a process of removing data of a light component of a predetermined wavelength or wavelength range from the imaging data.
  • the image processing may also include an autofluorescence separation process that separates autofluorescence components and pigment components of a tissue slice, or a fluorescence separation process that separates wavelengths between pigments having different fluorescence wavelengths.
  • an autofluorescence signal extracted from one of the multiple specimens having the same or similar properties may be used to remove the autofluorescence component from image information of the other specimen.
  • the information processing unit 5120 may transmit data for imaging control to the control unit 5110, and the control unit 5110 may receive the data and control imaging by the microscope device 5100 in accordance with the data.
  • the information processing unit 5120 may be configured as an information processing device such as a general-purpose computer, and may include a CPU, RAM, and ROM.
  • the information processing unit may be included within the housing of the microscope device 5100, or may be outside the housing.
  • various processes or functions performed by the information processing unit may be realized by a server computer or cloud connected via a network.
  • the method of imaging the biological sample S using the microscope device 5100 may be appropriately selected by a person skilled in the art depending on the type of biological sample and the purpose of imaging. Examples of such imaging methods are described below.
  • the microscope device may first identify an imaging target region.
  • the imaging target region may be identified so as to cover the entire region in which the biological sample is present, or may be identified so as to cover a target portion of the biological sample (a portion in which a target tissue slice, a target cell, or a target lesion is present).
  • the microscope device divides the imaging target region into a plurality of divided regions of a predetermined size, and the microscope device sequentially images each divided region. In this way, an image of each divided region is acquired.
  • the microscope apparatus identifies an imaging target region R that covers the entire biological sample S.
  • the microscope apparatus then divides the imaging target region R into 16 divided regions.
  • the microscope apparatus then images the divided region R1, and may then image any region included in the imaging target region R, such as a region adjacent to the divided region R1.
  • the divided regions are then imaged until there are no divided regions that have yet to be imaged. Regions other than the imaging target region R may also be imaged based on the captured image information of the divided regions.
  • the positional relationship between the microscope device and the sample mounting part is adjusted. The adjustment may be performed by moving the microscope device, the sample mounting part, or both.
  • the imaging device that images each divided region may be a two-dimensional imaging element (area sensor) or a one-dimensional imaging element (line sensor).
  • the signal acquisition part may image each divided region via the optical part.
  • imaging of each divided region may be performed continuously while moving the microscope device and/or the sample mounting part, or the movement of the microscope device and/or the sample mounting part may be stopped when imaging each divided region.
  • the imaging target region may be divided so that parts of each divided region overlap, or the imaging target region may be divided so that parts of each divided region do not overlap.
  • Each divided region may be imaged multiple times by changing imaging conditions such as focal length and/or exposure time.
  • the information processing device may also stitch adjacent divided regions together to generate image data of a larger region. By performing the stitching process over the entire imaging target region, an image of a larger region can be obtained for the imaging target region. Image data with a lower resolution may also be generated from the images of the divided regions or the images that have been subjected to the stitching process.
  • the microscope device may first specify an imaging target area.
  • the imaging target area may be specified to cover the entire area in which the biological sample is present, or may be specified to cover a target portion of the biological sample (a portion in which a target tissue slice or a target cell is present).
  • the microscope device scans and images a part of the imaging target area (also called a "divided scan area") in one direction (also called a "scan direction”) in a plane perpendicular to the optical axis.
  • the microscope device After completing the scan of the divided scan area, the microscope device then scans a divided scan area adjacent to the scan area. These scanning operations are repeated until the entire imaging target area is imaged.
  • the microscope apparatus identifies an area (gray portion) in the biological sample S where a tissue slice exists as the imaging target area Sa.
  • the microscope apparatus then scans a divided scan area Rs in the imaging target area Sa in the Y-axis direction.
  • the microscope apparatus scans the adjacent divided scan area in the X-axis direction. This operation is repeated until scanning of the entire imaging target area Sa is completed.
  • the positional relationship between the microscope device and the sample placement unit is adjusted for scanning each divided scan area and for imaging the next divided scan area after imaging a certain divided scan area. The adjustment may be performed by moving the microscope device, the sample placement unit, or both.
  • the imaging device that images each divided scan area may be a one-dimensional imaging element (line sensor) or a two-dimensional imaging element (area sensor).
  • the signal acquisition unit may image each divided area via a magnifying optical system.
  • imaging of each divided scan area may be performed continuously while moving the microscope device and/or the sample placement unit.
  • the imaging target area may be divided so that parts of each divided scan area overlap, or may be divided so that they do not overlap.
  • Each divided scan area may be imaged multiple times by changing imaging conditions such as focal length and/or exposure time.
  • the information processing device may also stitch adjacent divided scan areas together to generate image data of a larger area. By performing the stitching process over the entire imaging target area, an image of a larger area of the imaging target area can be acquired. Also, image data with a lower resolution may be generated from the images of the divided scan areas or from the images that have been stitched.
  • FIG. 33 is a block diagram showing an example of a schematic configuration of the hardware of the information processing device 100.
  • Various processes by the information processing device 100 are realized, for example, by cooperation between software and the hardware described below.
  • the CPU 901 functions as an arithmetic processing device and control device, and controls the overall operation within the information processing device 100 in accordance with various programs.
  • the CPU 901 may also be a microprocessor.
  • the ROM 902 stores programs and arithmetic parameters used by the CPU 901.
  • the RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution.
  • the CPU 901 may embody, for example, at least the processing unit 130 and control unit 150 of the information processing device 100.
  • the CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a, which includes a CPU bus.
  • the host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect/Interface) bus, via a bridge 904.
  • an external bus 904b such as a PCI (Peripheral Component Interconnect/Interface) bus
  • PCI Peripheral Component Interconnect/Interface
  • the input device 906 is realized by a device into which the implementer inputs information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever.
  • the input device 906 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device such as a mobile phone or PDA that supports the operation of the information processing device 100.
  • the input device 906 may also include, for example, an input control circuit that generates an input signal based on information input by the implementer using the above-mentioned input means and outputs it to the CPU 901. By operating this input device 906, the implementer can input various data to the information processing device 100 and instruct processing operations.
  • the input device 906 may embody at least the operation unit 160 of the information processing device 100, for example.
  • the output device 907 is formed of a device capable of visually or audibly notifying the user of acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, and lamps, audio output devices such as speakers and headphones, and printer devices.
  • the output device 907 may embody at least the display unit 140 of the information processing device 100, for example.
  • the storage device 908 is a device for storing data.
  • the storage device 908 is realized, for example, by a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, or a magneto-optical storage device.
  • the storage device 908 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium.
  • This storage device 908 stores the programs and various data executed by the CPU 901, as well as various data acquired from the outside.
  • the storage device 908 may embody at least the storage unit 120 of the information processing device 100, for example.
  • the drive 909 is a reader/writer for storage media, and is built into the information processing device 100 or attached externally.
  • the drive 909 reads information recorded on a removable storage medium, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903.
  • the drive 909 can also write information to the removable storage medium.
  • the communication device 913 is, for example, a communication interface formed by a communication device for connecting to the network 920.
  • the communication device 913 is, for example, a communication card for a wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB).
  • the communication device 913 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.
  • This communication device 913 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP/IP.
  • the network 920 is a wired or wireless transmission path for information transmitted from devices connected to the network 920.
  • the network 920 may include public line networks such as the Internet, telephone line networks, and satellite communication networks, as well as various LANs (Local Area Networks) and WANs (Wide Area Networks) including Ethernet (registered trademark).
  • the network 920 may also include dedicated line networks such as an IP-VPN (Internet Protocol-Virtual Private Network).
  • the above shows an example of a hardware configuration capable of realizing the functions of the information processing device 100.
  • Each of the above components may be realized using general-purpose components, or may be realized by hardware specialized for the function of each component. Therefore, it is possible to change the hardware configuration used as appropriate depending on the technical level at the time when this disclosure is implemented.
  • the present technology can also be configured as follows. (1) a connector that connects, in a wavelength direction, a plurality of fluorescence spectra obtained by irradiating a sample with a plurality of excitation lights; a color separation unit that separates a combined fluorescence spectrum obtained by combining the plurality of fluorescence spectra into spectra for each fluorescent substance; a spectrum extraction unit that extracts a concatenated autofluorescence spectrum obtained by concatenating a plurality of autofluorescence spectra from the concatenated fluorescence spectrum using jNMF; An information processing device comprising: (2) the spectrum extraction unit prepares a matrix for each wavelength of the excitation light and executes the jNMF; The information processing device according to (1).
  • the spectrum extraction unit prepares the matrix for each of the wavelengths so as to include a power ratio of the plurality of excitation lights.
  • the spectrum extraction unit updates the combined autofluorescence spectrum while allowing a degree of freedom between two or more autofluorescence spectra included in the combined autofluorescence spectrum.
  • the color separation unit or the spectrum extraction unit separates the concatenated fluorescence spectrum into the spectra for each of the fluorescent substances using Restricted-jNMF.
  • the information processing device according to any one of (1) to (4).
  • the color separation unit or the spectrum extraction unit fixes a power ratio of the plurality of excitation lights in a dye portion of the dye fluorescence spectrum and executes the Restricted-jNMF.
  • the information processing device according to (5) The color separation unit or the spectrum extraction unit fixes a power ratio of the plurality of excitation lights using an initial value of the dye fluorescence spectrum, and executes the Restricted-jNMF.
  • the color separation unit or the spectrum extraction unit executes the Restricted-jNMF by including a power ratio of the plurality of excitation lights in a dye portion of the dye fluorescence spectrum.
  • the information processing device according to (5) The color separation unit or the spectrum extraction unit multiplies the initial value of the dye fluorescence spectrum by a power ratio of the plurality of excitation lights and executes the Restricted-jNMF.
  • the power ratio is determined from the combined autofluorescence spectrum.
  • an imaging device for acquiring an image of the specimen; an information processing device for processing the image; Equipped with The information processing device includes: a connector that connects, in a wavelength direction, a plurality of fluorescence spectra obtained by irradiating the specimen with a plurality of excitation lights; a color separation unit that separates a combined fluorescence spectrum obtained by combining the plurality of fluorescence spectra into spectra for each fluorescent substance; a spectrum extraction unit that extracts a combined autofluorescence spectrum by combining a plurality of autofluorescence spectra from the combined fluorescence spectrum using jNMF (joint non-negative matrix factorization);
  • a biological sample observation system comprising: (12) An information processing device, Concatenating a plurality of fluorescence spectra obtained by irradiating a sample with a plurality of excitation lights in a wavelength direction; Separating a combined fluorescence spectrum obtained by combining the plurality of fluorescence spectra into spectra for each fluorescent substance;

Landscapes

  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)

Abstract

Un mode de réalisation de la présente divulgation concerne un dispositif de traitement d'informations comprenant : une partie de concaténation qui concatène, dans une direction de longueur d'onde, une pluralité de spectres de fluorescence obtenus par irradiation d'un échantillon avec une pluralité de faisceaux d'excitation ; une partie de séparation de couleurs qui sépare le spectre de fluorescence concaténé, obtenu par concaténation de la pluralité de spectres de fluorescence, en un spectre pour chaque substance fluorescente ; et une partie d'extraction de spectre qui extrait un spectre d'autofluorescence concaténé, obtenu par concaténation d'une pluralité de spectres d'autofluorescence, à partir du spectre de fluorescence concaténé, au moyen d'une factorisation conjointe en matrices non négatives (jNMF).
PCT/JP2024/003422 2023-02-15 2024-02-02 Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de traitement d'informations WO2024171844A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023021393 2023-02-15
JP2023-021393 2023-02-15

Publications (1)

Publication Number Publication Date
WO2024171844A1 true WO2024171844A1 (fr) 2024-08-22

Family

ID=92421823

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/003422 WO2024171844A1 (fr) 2023-02-15 2024-02-02 Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de traitement d'informations

Country Status (1)

Country Link
WO (1) WO2024171844A1 (fr)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020138205A1 (en) * 1998-04-16 2002-09-26 Northeastern University Expert system for analysis of DNA sequencing electropherograms
WO2019049442A1 (fr) * 2017-09-08 2019-03-14 ソニー株式会社 Dispositif de mesure de microparticules, dispositif de traitement d'informations et procédé de traitement d'informations
WO2020179586A1 (fr) * 2019-03-04 2020-09-10 ソニー株式会社 Dispositif de traitement d'informations et système de microscope
WO2021153192A1 (fr) * 2020-01-31 2021-08-05 ソニーグループ株式会社 Dispositif de traitement d'informations, dispositif d'analyse de particules, procédé de traitement d'informations et programme
WO2021157397A1 (fr) * 2020-02-06 2021-08-12 ソニーグループ株式会社 Appareil et système de traitement d'information
WO2021200411A1 (fr) * 2020-03-31 2021-10-07 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, programme, et système de mesure optique
WO2022004500A1 (fr) * 2020-06-30 2022-01-06 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, programme, système de microscope et système d'analyse
WO2022075040A1 (fr) * 2020-10-09 2022-04-14 ソニーグループ株式会社 Système de génération d'image, système de microscope et procédé de génération d'image
WO2022249583A1 (fr) * 2021-05-26 2022-12-01 ソニーグループ株式会社 Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
WO2022264539A1 (fr) * 2021-06-17 2022-12-22 ソニーグループ株式会社 Système de traitement d'informations, procédé de traitement d'informations et structure de substance fluorescente
WO2023276219A1 (fr) * 2021-06-29 2023-01-05 ソニーグループ株式会社 Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
WO2023276254A1 (fr) * 2021-06-29 2023-01-05 ソニーグループ株式会社 Dispositif de traitement d'informations, système de microscope, et procédé de traitement d'informations

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020138205A1 (en) * 1998-04-16 2002-09-26 Northeastern University Expert system for analysis of DNA sequencing electropherograms
WO2019049442A1 (fr) * 2017-09-08 2019-03-14 ソニー株式会社 Dispositif de mesure de microparticules, dispositif de traitement d'informations et procédé de traitement d'informations
WO2020179586A1 (fr) * 2019-03-04 2020-09-10 ソニー株式会社 Dispositif de traitement d'informations et système de microscope
WO2021153192A1 (fr) * 2020-01-31 2021-08-05 ソニーグループ株式会社 Dispositif de traitement d'informations, dispositif d'analyse de particules, procédé de traitement d'informations et programme
WO2021157397A1 (fr) * 2020-02-06 2021-08-12 ソニーグループ株式会社 Appareil et système de traitement d'information
WO2021200411A1 (fr) * 2020-03-31 2021-10-07 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, programme, et système de mesure optique
WO2022004500A1 (fr) * 2020-06-30 2022-01-06 ソニーグループ株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations, programme, système de microscope et système d'analyse
WO2022075040A1 (fr) * 2020-10-09 2022-04-14 ソニーグループ株式会社 Système de génération d'image, système de microscope et procédé de génération d'image
WO2022249583A1 (fr) * 2021-05-26 2022-12-01 ソニーグループ株式会社 Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
WO2022264539A1 (fr) * 2021-06-17 2022-12-22 ソニーグループ株式会社 Système de traitement d'informations, procédé de traitement d'informations et structure de substance fluorescente
WO2023276219A1 (fr) * 2021-06-29 2023-01-05 ソニーグループ株式会社 Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
WO2023276254A1 (fr) * 2021-06-29 2023-01-05 ソニーグループ株式会社 Dispositif de traitement d'informations, système de microscope, et procédé de traitement d'informations

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
GARBARINO FEDERICO, SCELFO DANIEL, PAULONE GABRIELE, PAGANELLI ALESSIA, ULRICI ALESSANDRO, MAGNONI CRISTINA, PASQUALI LUCA: "Comparative study of in‐vitro autofluorescence of normal versus non‐melanoma‐skin‐cancer cells at different excitation wavelengths", JOURNAL OF BIOPHOTONICS, WILEY - VCH VERLAG, DE, vol. 16, no. 5, 1 May 2023 (2023-05-01), DE , XP093201034, ISSN: 1864-063X, DOI: 10.1002/jbio.202200361 *
HEASTER TIFFANY M., WALSH ALEX J., ZHAO YUE, HIEBERT SCOTT W., SKALA MELISSA C.: "Autofluorescence imaging identifies tumor cell‐cycle status on a single‐cell level", JOURNAL OF BIOPHOTONICS, WILEY - VCH VERLAG, DE, vol. 11, no. 1, 1 January 2018 (2018-01-01), DE , XP093201033, ISSN: 1864-063X, DOI: 10.1002/jbio.201600276 *
LAFI SANA; KHALFALLAH ALI; BOUZID MOHAMED; BOUCHOT ANDRE; BOUHLEL MED SALIM: "A spectroscopic approach based on Non-negative Matrix Factorization method for multispectral fluorescence images unmixing", 2016 2ND INTERNATIONAL CONFERENCE ON ADVANCED TECHNOLOGIES FOR SIGNAL AND IMAGE PROCESSING (ATSIP), IEEE, 21 March 2016 (2016-03-21), pages 375 - 380, XP032930750, DOI: 10.1109/ATSIP.2016.7523109 *
ZHANG SHIHUA, LIU CHUN-CHI, LI WENYUAN, SHEN HUI, LAIRD PETER W., ZHOU XIANGHONG JASMINE: "Discovery of multi-dimensional modules by integrative analysis of cancer genomic data", NUCLEIC ACIDS RESEARCH, OXFORD UNIVERSITY PRESS, GB, vol. 40, no. 19, 1 October 2012 (2012-10-01), GB , pages 9379 - 9391, XP093201036, ISSN: 0305-1048, DOI: 10.1093/nar/gks725 *

Similar Documents

Publication Publication Date Title
US11879843B2 (en) Multispectral sample imaging
US10824847B2 (en) Generating virtually stained images of unstained samples
US10580128B2 (en) Whole slide multispectral imaging systems and methods
AU2010307140B2 (en) Multi-modality contrast and brightfield context rendering for enhanced pathology determination and multi-analyte detection in tissue
US11776124B1 (en) Transforming multispectral images to enhanced resolution images enabled by machine learning
WO2022004500A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations, programme, système de microscope et système d'analyse
US20230095577A1 (en) Microscope device, image acquisition system, and image acquisition method
WO2023276219A1 (fr) Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
US20240169534A1 (en) Medical image analysis device, medical image analysis method, and medical image analysis system
WO2022075040A1 (fr) Système de génération d'image, système de microscope et procédé de génération d'image
US11842555B2 (en) Signal acquisition apparatus, signal acquisition system, and signal acquisition method
WO2022249583A1 (fr) Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
WO2024171844A1 (fr) Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de traitement d'informations
WO2023157755A1 (fr) Dispositif de traitement d'informations, système d'analyse d'échantillon biologique et procédé d'analyse d'échantillon biologique
WO2023157756A1 (fr) Dispositif de traitement d'informations, système d'analyse d'échantillon biologique et procédé d'analyse d'échantillon biologique
WO2023149296A1 (fr) Dispositif de traitement d'informations, système d'observation d'échantillon biologique et procédé de production d'image
US20240053267A1 (en) Data generation method, fluorescence observation system, and information processing apparatus
WO2023189393A1 (fr) Système d'observation d'échantillon biologique, dispositif de traitement d'informations et procédé de génération d'image
WO2024185434A1 (fr) Dispositif de traitement d'informations, système d'analyse d'échantillon biologique et procédé d'analyse d'échantillon biologique
JP7090171B2 (ja) 画像処理装置の作動方法、画像処理装置、及び画像処理装置の作動プログラム
WO2022259647A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et système de microscope

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24756694

Country of ref document: EP

Kind code of ref document: A1