JP7514673B2 - Medical image processing device and medical image processing system - Google Patents

Description

The embodiments disclosed in this specification and the drawings relate to a medical image processing device and a medical image processing system.

In the medical field, ultrasound diagnostic devices are used to image the inside of a subject using ultrasound generated by multiple transducers (piezoelectric transducers) in an ultrasound probe. The ultrasound diagnostic device transmits ultrasound into the subject from an ultrasound probe connected to the ultrasound diagnostic device, generates echo signals based on the reflected waves, and obtains the desired ultrasound image through image processing.

Although interventional treatment using catheters has been performed on the coronary arteries of the heart, attention is now being paid to interventional treatment for organic heart diseases such as valvular heart disease. Interventional treatment for valvular heart disease is called "Structural Heart Disease (SHD) interventional treatment." In recent years, when performing SHD interventional treatment, there are an increasing number of cases in which not only fluoroscopic image data obtained from X-ray imaging devices (e.g., X-ray circulatory system (Angio)) but also ultrasound image data obtained from ultrasound diagnostic devices are used. As a result, there is a growing need for multi-modality in SHD interventional treatment.

Specifically, a procedure called Mitra Clip (registered trademark) is used to treat the valve by inserting a medical device such as a clip through a catheter inserted into the body of a patient with mitral regurgitation (MR). In this procedure, under guidance from fluoroscopic image data, the surgeon places the medical device while grasping the positional relationship between the device and the cardiac tissue and checking the blood flow conditions with ultrasonic transesophageal echocardiography. Therefore, this procedure can be performed without opening the patient's chest, taking into consideration the patient's physical strength and age.

In addition, a surgical procedure is used in which a medical device such as the WATCHMAN (registered trademark) device is inserted through a catheter inserted into the body of a patient with non-valvular atrial fibrillation and placed in the left atrial appendage. Once the device is placed, the surgeon uses ultrasound images to check that it has been placed in the appropriate position in the left atrial appendage, and then releases the medical device.

JP 2014-76331 A

One of the problems that the embodiments disclosed in this specification and the drawings attempt to solve is to generate image data that can assist an operator such as a surgeon in an interventional treatment. However, the problems that the embodiments disclosed in this specification and the drawings attempt to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems.

The medical image processing device according to the embodiment has a first image acquisition unit, a second image acquisition unit, a third image acquisition unit, a first alignment unit, a second alignment unit, and an image generation unit. The first image acquisition unit acquires an ultrasound image of the subject as a first medical image. The second image acquisition unit acquires an X-ray image of the subject as a second medical image. The third image acquisition unit acquires a third medical image of the subject, which is different from the ultrasound image and the X-ray image. The first alignment unit aligns the X-ray image with the ultrasound image. The second alignment unit aligns the third medical image with the ultrasound image. The image generation unit aligns the X-ray image with the third medical image based on the alignment result by the first alignment unit and the alignment result by the second alignment unit, and generates an image in which the third medical image is aligned with the X-ray image.

FIG. 1 is a schematic diagram showing an example of the configuration of a medical image processing system including a medical image processing apparatus according to an embodiment. FIG. 2 is a diagram showing the appearance of a medical image processing system including a medical image processing apparatus according to an embodiment. FIG. 3 is a schematic diagram showing an example of the arrangement of an X-ray imaging apparatus in a medical image processing system including a medical image processing apparatus according to an embodiment. FIG. 4 is a block diagram showing an example of functions of a medical image processing system including a medical image processing apparatus according to an embodiment. FIG. 5 is a flowchart showing an example of the operation of a medical image processing system including the medical image processing apparatus according to the embodiment. FIG. 6 is a flowchart showing an example of the operation of the medical image processing system including the medical image processing apparatus according to the embodiment. FIG. 7 is a diagram showing an example of a superimposed image based on superimposed image data in which ultrasound image data is superimposed on X-ray fluoroscopic image data in a medical image processing system including a medical image processing apparatus according to an embodiment. Figure 8 is a diagram showing an example of a superimposed image based on superimposed image data in which CT image data including the entire left atrium is superimposed on X-ray fluoroscopic image data in a medical image processing system including a medical image processing device according to an embodiment. Figure 9 is a diagram showing an example of a superimposed image based on superimposed image data in which CT image data including the entire left atrium and ultrasound image data including part of the heart are superimposed on X-ray fluoroscopy image data in a medical image processing system having a medical image processing device according to an embodiment. FIG. 10 is a diagram showing an example of a superimposed image based on superimposed image data in which CT image data including blood vessels is superimposed on X-ray fluoroscopic image data in a medical image processing system including a medical image processing apparatus according to an embodiment. FIG. 11 is a block diagram showing a modified example of the medical image processing system shown in FIG. 3 in a medical image processing system including the medical image processing apparatus according to the embodiment.

Below, we will explain in detail the embodiments of the medical image processing device and medical image processing system with reference to the drawings.

Figure 1 is a schematic diagram showing an example of the configuration of a medical image processing system that includes a medical image processing device according to an embodiment. Figure 2 is a diagram showing the external appearance of a medical image processing system that includes a medical image processing device according to an embodiment.

Figure 1 shows a medical image processing system 1. The medical image processing system 1 includes an ultrasound diagnostic device 10 as a first medical image generating device (also called a "modality"), an ultrasound probe 20, a position sensor 30, an X-ray imaging device (e.g., an X-ray circulatory system (Angio)) 40 as a second medical image generating device, and a medical image providing device 50. Note that an apparatus in which at least one of the ultrasound probe 20 and the position sensor 30 is added to the ultrasound diagnostic device 10 may also be referred to as an ultrasound diagnostic device. In the following explanation, a case in which the ultrasound probe 20 and the position sensor 30 are provided outside the ultrasound diagnostic device will be described.

The ultrasound diagnostic device 10 includes a transmission/reception circuit 11, a B-mode processing circuit 12, a Doppler processing circuit 13, an image generation circuit 14, an image memory 15, a network interface 16, a medical image processing device 17 according to this embodiment, an input interface 18, and a display 19. The circuits 11 to 14 are configured by application specific integrated circuits (ASICs) or the like. However, this is not limited to this case, and all or part of the functions of the circuits 11 to 14 may be realized by the processing circuit 171 of the medical image processing device 17 executing a program. The ultrasound diagnostic device 10 may also include at least one of the input interface 18 and the display 19 externally. In the following description, a case in which both the input interface 18 and the display 19 are provided inside the ultrasound diagnostic device will be described.

The transmission/reception circuit 11 has a transmission circuit and a reception circuit (not shown). The transmission/reception circuit 11 controls the transmission directivity and reception directivity in transmitting and receiving ultrasound under the control of the processing circuit 171. Note that, although a case where the transmission/reception circuit 11 is provided in the ultrasound diagnostic device 10 will be described, the transmission/reception circuit 11 may be provided in the ultrasound probe 20, or may be provided in both the ultrasound diagnostic device 10 and the ultrasound probe 20. Note that the transmission/reception circuit 11 is an example of a transmission/reception unit.

The transmission circuit has a pulse generation circuit, a transmission delay circuit, a pulsar circuit, etc., and supplies a drive signal to the ultrasonic transducer. The pulse generation circuit repeatedly generates rate pulses to form a transmission ultrasonic wave at a predetermined rate frequency. The transmission delay circuit provides each rate pulse generated by the pulse generation circuit with a delay time for each piezoelectric transducer required to focus the ultrasonic waves generated from the ultrasonic transducers of the ultrasonic probe 20 into a beam and determine the transmission directivity. In addition, the pulsar circuit applies a drive pulse to the ultrasonic transducer at a timing based on the rate pulse. The transmission delay circuit changes the delay time provided to each rate pulse to arbitrarily adjust the transmission direction of the ultrasonic beam transmitted from the piezoelectric transducer surface.

The receiving circuit has an amplifier circuit, an A/D (Analog to Digital) converter, an adder, etc., and receives the echo signal received by the ultrasonic transducer and performs various processes on this echo signal to generate echo data. The amplifier circuit amplifies the echo signal for each channel and performs gain correction processing. The A/D converter A/D converts the gain-corrected echo signal and gives the digital data a delay time required to determine the reception directivity. The adder performs addition processing of the echo signal processed by the A/D converter to generate echo data. The addition processing of the adder emphasizes the reflected components from the direction corresponding to the reception directivity of the echo signal.

Under the control of the processing circuit 171, the B-mode processing circuit 12 receives echo data from the receiving circuit, and performs logarithmic amplification and envelope detection processing, etc. to generate data (two-dimensional or three-dimensional data) in which the signal strength is expressed as luminance brightness. This data is generally called B-mode data. The B-mode processing circuit 12 is an example of a B-mode processing unit.

The B-mode processing circuit 12 can change the frequency band to be visualized by changing the detection frequency through filter processing. By using the filter processing function of the B-mode processing circuit 12, it is possible to perform harmonic imaging such as contrast harmonic imaging (CHI) and tissue harmonic imaging (THI). That is, the B-mode processing circuit 12 can separate reflected wave data (harmonic data or sub-frequency data) of harmonic components whose reflection source is the contrast agent (microbubbles, bubbles) from reflected wave data of a subject into which a contrast agent has been injected, and reflected wave data (fundamental wave data) of fundamental components whose reflection source is tissue in the subject, from reflected wave data of the harmonic component (received signal). The B-mode processing circuit 12 can also generate B-mode data for generating contrast image data from reflected wave data of the harmonic component (received signal), and can also generate B-mode data for generating fundamental image data from reflected wave data of the fundamental component (received signal).

In addition, in THI, by using the filter processing function of the B-mode processing circuit 12, harmonic data or sub-frequency data, which is reflected wave data (received signal) of harmonic components, can be separated from the reflected wave data of the subject.The B-mode processing circuit 12 can then generate B-mode data for generating tissue image data from which noise components have been removed, from the reflected wave data (received signal) of harmonic components.

Furthermore, when performing harmonic imaging of CHI or THI, the B-mode processing circuit 12 can extract harmonic components by a method different from the above-mentioned method using filter processing. In harmonic imaging, an imaging method called the AMPM method, which is a combination of the amplitude modulation (AM) method, the phase modulation (PM) method, and the AM method and PM method, is performed. In the AM method, the PM method, and the AMPM method, ultrasonic waves with different amplitudes and phases are transmitted multiple times for the same scanning line. As a result, the transmission/reception circuit 11 generates and outputs multiple reflected wave data (received signals) for each scanning line. Then, the B-mode processing circuit 12 extracts harmonic components by performing addition and subtraction processing on the multiple reflected wave data (received signals) for each scanning line according to the modulation method. Then, the B-mode processing circuit 12 performs envelope detection processing or the like on the reflected wave data (received signals) of the harmonic components to generate B-mode data.

For example, when the PM method is performed, the transmission/reception circuitry 11 transmits ultrasound waves of the same amplitude with inverted phase polarity, for example (-1, 1), twice on each scan line according to the scan sequence set by the processing circuitry 171. The transmission/reception circuitry 11 then generates a reception signal resulting from the transmission of "-1" and a reception signal resulting from the transmission of "1", and the B-mode processing circuitry 12 adds these two reception signals together. This removes the fundamental wave component, generating a signal in which the second harmonic component remains as the main component. The B-mode processing circuitry 12 then performs envelope detection processing on this signal, generating B-mode data for THI and B-mode data for CHI.

Alternatively, for example, in THI, a method of imaging using the second harmonic component and the difference frequency component contained in the received signal has been put to practical use. In the imaging method using the difference frequency component, for example, a transmission ultrasound having a composite waveform obtained by combining a first fundamental wave having a center frequency of "f1" and a second fundamental wave having a center frequency of "f2" greater than "f1" is transmitted from the ultrasound probe 20. This composite waveform is a waveform obtained by combining the waveform of the first fundamental wave and the waveform of the second fundamental wave, the phases of which are adjusted so that a difference frequency component having the same polarity as the second harmonic component is generated. The transmission/reception circuit 11 transmits the transmission ultrasound having the composite waveform, for example, twice while inverting the phase. In such a case, for example, the B-mode processing circuit 12 adds two received signals to remove the fundamental wave component, extracts the harmonic components in which the difference frequency component and the second harmonic component mainly remain, and then performs envelope detection processing, etc.

Under the control of the processing circuit 171, the Doppler processing circuit 13 performs frequency analysis of the velocity information from the echo data from the receiving circuit, and generates data (two-dimensional or three-dimensional data) that extracts the movement information of the moving object, such as the average velocity, variance, and power, for multiple points. This data is generally called Doppler data. Examples of moving objects include blood flow, tissues such as the heart wall, and contrast agents. The Doppler processing circuit 13 is an example of a Doppler processing unit.

Under the control of the processing circuit 171, the image generation circuit 14 generates ultrasound image data expressed in a predetermined brightness range based on the echo signal received by the ultrasound probe 20. For example, the image generation circuit 14 generates, as ultrasound image data, B-mode image data in which the intensity of the reflected wave is expressed as brightness from the two-dimensional B-mode data generated by the B-mode processing circuit 12. The image generation circuit 14 also generates, as ultrasound image data, average velocity image data, variance image data, power image data, or color Doppler image data as image data of a combination of these, which represent the locomotion information from the two-dimensional Doppler data generated by the Doppler processing circuit 13. The image generation circuit 14 is an example of an image generation unit.

Here, the image generating circuit 14 generally converts (scan converts) the scan line signal sequence of the ultrasound scan into a scan line signal sequence of a video format, such as that of a television, to generate ultrasound image data for display. Specifically, the image generating circuit 14 generates ultrasound image data for display by performing coordinate conversion according to the ultrasound scanning form of the ultrasound probe 20. In addition to scan conversion, the image generating circuit 14 also performs various image processing, such as image processing (smoothing processing) that regenerates an average brightness image using multiple image frames after scan conversion, and image processing (edge enhancement processing) that uses a differential filter within the image. The image generating circuit 14 also combines text information of various parameters, scales, body marks, etc., with the ultrasound image data.

That is, the B-mode data and Doppler data are ultrasound image data before the scan conversion process, and the data generated by the image generation circuit 14 is ultrasound image data for display after the scan conversion process. Note that the B-mode data and Doppler data are also called raw data. The image generation circuit 14 generates two-dimensional ultrasound image data for display from the two-dimensional ultrasound image data before the scan conversion process.

Furthermore, the image generation circuitry 14 generates three-dimensional B-mode image data by performing coordinate transformation on the three-dimensional B-mode data generated by the B-mode processing circuitry 12. The image generation circuitry 14 also generates three-dimensional Doppler image data by performing coordinate transformation on the three-dimensional Doppler data generated by the Doppler processing circuitry 13. The image generation circuitry 14 generates "three-dimensional B-mode image data or three-dimensional Doppler image data" as "three-dimensional ultrasound image data (volume data)."

Then, the image generation circuitry 14 performs rendering processing on the volume data to generate various types of two-dimensional image data for displaying the volume data stored in the three-dimensional memory on the display 19. As the rendering processing, the image generation circuitry 14 performs, for example, multi-planar reconstruction (MPR) to generate MPR image data from the volume data. As the rendering processing, the image generation circuitry 14 also performs, for example, volume rendering (VR) processing to generate two-dimensional image data reflecting three-dimensional information.

The image memory 15 includes a processor-readable recording medium, such as a magnetic recording medium, an optical recording medium, or a semiconductor memory. The image memory 15 stores ultrasound image data generated by the image generation circuitry 14 under the control of the processing circuitry 171. The image memory 15 is an example of a storage unit.

The network interface 16 implements various information communication protocols according to the type of network. In accordance with these various protocols, the network interface 16 connects the ultrasound diagnostic apparatus 10 to other devices such as an external X-ray imaging apparatus 40 and a medical image providing apparatus 50. This connection can be an electrical connection via an electronic network. Here, the electronic network refers to a general information communication network that uses electrical communication technology, and includes not only wireless/wired hospital-based LANs (Local Area Networks) and Internet networks, but also telephone communication line networks, optical fiber communication networks, cable communication networks, and satellite communication networks.

The network interface 16 may also implement various protocols for non-contact wireless communication. In this case, the ultrasound diagnostic device 10 can transmit and receive data directly to, for example, the ultrasound probe 20 without going through a network. The network interface 16 is an example of a network connection unit.

The medical image processing device 17 includes a processing circuit 171 and a main memory 172.

The processing circuit 171 refers to a dedicated or general-purpose CPU (central processing unit), MPU (micro processor unit), or GPU (graphics processing unit), as well as an ASIC and a programmable logic device. Examples of programmable logic devices include a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA).

The processing circuit 171 may be configured as a single circuit, or may be configured as a combination of multiple independent circuit elements. In the latter case, the main memory 172 may be provided separately for each circuit element, or a single main memory 172 may store programs corresponding to the functions of multiple circuit elements. The processing circuit 171 is an example of a processing unit.

The main memory 172 is composed of semiconductor memory elements such as RAM (random access memory) and flash memory, a hard disk, an optical disk, etc. The main memory 172 may be composed of portable media such as a universal serial bus (USB) memory and a digital video disk (DVD). The main memory 172 stores various processing programs (including application programs and an operating system (OS)) used in the processing circuit 171 and data required for executing the programs. The OS can also include a graphical user interface (GUI) that makes extensive use of graphics to display information to the operator on the display 19 and allows basic operations to be performed by the input interface 18. The main memory 172 is an example of a storage unit.

The input interface 18 includes an input device that can be operated by the operator, and an input circuit that inputs a signal from the input device. The input device can be realized by a trackball, a switch, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touchscreen that integrates the display screen and touchpad, a non-contact input device that uses an optical sensor, and a voice input device. In addition, when the ultrasound probe 20 is a TEE probe, the input device can also be realized by a controller that can operate the TEE probe. When the input device is operated by the operator, the input circuit generates a signal corresponding to the operation and outputs it to the processing circuit 171. The input interface 18 is an example of an input unit.

The display 19 is configured with a general display output device such as a liquid crystal display or an OLED (Organic Light Emitting Diode) display. The display 19 displays various information according to the control of the processing circuit 171. The display 19 is an example of a display unit.

The ultrasound probe 20 has multiple tiny transducers (piezoelectric elements) on the front surface, and transmits and receives ultrasound waves to and from an area including a target, such as an area including the left atrium of the heart. Each transducer is an electroacoustic conversion element, and has the function of converting an electrical pulse into an ultrasonic pulse when transmitting, and converting a reflected wave into an electrical signal (received signal) when receiving. The ultrasound probe 20 is small and lightweight, and is connected to the ultrasound diagnostic device 10 via a cable (or wireless communication).

The ultrasonic probe 20 is divided into types such as linear type, convex type, and sector type depending on the scanning method. Also, the ultrasonic probe 20 is divided into types such as 1D array probes in which multiple transducers are arranged one-dimensionally (1D) in the azimuth direction and 2D array probes in which multiple transducers are arranged two-dimensionally (2D) in both the azimuth direction and the elevation direction depending on the array arrangement dimension. Note that 1D array probes include probes in which a small number of transducers are arranged in the elevation direction.

Here, when 3D imaging, i.e., volume imaging (e.g., volume scan), is performed, a 2D array probe equipped with a scanning method such as linear type, convex type, or sector type is used as the ultrasound probe 20. Alternatively, when volume imaging is performed, a 1D probe equipped with a scanning method such as linear type, convex type, or sector type and equipped with a mechanism for mechanically oscillating in the elevation direction is used as the ultrasound probe 20. The latter probe is also called a mechanical 4D probe.

Here, the ultrasound probe 20 is an ultrasound probe (i.e., a body surface probe) that is operated on the body surface of the subject, for example, patient P. For example, the ultrasound probe 20 is a transthoracic echocardiography (TTE) probe that is operated on the body surface of the patient P's chest and can scan the left ventricle, left atrium, aorta, mitral valve, aortic valve, etc. Alternatively, the ultrasound probe 20 is an ultrasound probe (i.e., an intracavity probe) that is inserted into the body cavity of the patient P and operated within the body cavity. For example, the ultrasound probe 20 is a transesophageal echocardiography (TEE) probe that is operated in the esophagus and can scan the left ventricle, left atrium, aorta, mitral valve, aortic valve, etc. Note that the TEE probe can acquire three-dimensional ultrasound image data of the heart, etc. in approximately real time.

The position sensor 30 detects multiple pieces of position information of the ultrasound probe 20 in time series and outputs them to the ultrasound diagnostic device 10. There are two types of position sensor 30: a sensor that is attached to the ultrasound probe 20, and a sensor that is provided separately from the body surface probe that serves as the ultrasound probe 20. The latter type of sensor is an optical sensor that captures images of the characteristic points of the body surface probe, which is the measurement target, from multiple positions and detects each position of the body surface probe using the principle of triangulation. Below, we will explain the case where the position sensor 30 is the former type of sensor.

The position sensor 30 is attached to the ultrasonic probe 20, detects its own position information, and outputs it to the ultrasonic diagnostic device 10. The position information of the position sensor 30 can also be considered as the position information of the ultrasonic probe 20. The position information of the ultrasonic probe 20 includes the position and attitude (tilt angle) of the ultrasonic probe 20. For example, the attitude of the ultrasonic probe 20 can be detected by a magnetic field transmitter (not shown) sequentially transmitting three-axis magnetic fields and the position sensor 30 sequentially receiving the magnetic fields. The position sensor 30 may also be a so-called nine-axis sensor that includes at least one of a three-axis gyro sensor that detects three-axis angular velocity in three-dimensional space, a three-axis acceleration sensor that detects three-axis acceleration in three-dimensional space, and a three-axis geomagnetic sensor that detects three-axis geomagnetism in three-dimensional space.

The X-ray imaging device 40 is connected to the ultrasound diagnostic device 10 and the medical image providing device 50 via the network N so that they can communicate with each other.

Figure 3 is a schematic diagram showing an example of the configuration of the X-ray imaging device 40 of the medical image processing system 1.

The X-ray imaging device 40 includes a high-voltage supply device 41, an X-ray irradiation device 42, an X-ray detection device 43, an input interface 44, a display 45, a network interface 46, a medical image processing device 47, a C-arm 48 (shown only in FIG. 2), and a bed 49 (shown only in FIG. 2).

The high voltage supply device 41 supplies high voltage power to the X-ray tube of the X-ray irradiation device 42 under the control of the processing circuit 471.

The X-ray irradiation device 42 is provided at one end of the C-arm 48. The X-ray irradiation device 42 is provided with an X-ray tube (X-ray source) and a movable diaphragm device. The X-ray tube receives high-voltage power from the high-voltage supply device 41 and generates X-rays according to the conditions of the high-voltage power. Under the control of the processing circuit 471, the movable diaphragm device movably supports diaphragm blades made of a material that blocks X-rays at the X-ray irradiation port of the X-ray tube. In addition, a radiation quality adjustment filter (not shown) that adjusts the radiation quality of the X-rays generated by the X-ray tube may be provided on the front side of the X-ray tube.

The X-ray detection device 43 is provided at the other end of the C-arm 48 so as to face the X-ray irradiation device 42. Under the control of the processing circuit 471, the X-ray detection device 43 can move along the SID (Source-Image Distance) direction, i.e., can move back and forth. Also, under the control of the processing circuit 471, the X-ray detection device 43 can move along a rotational direction centered on the SID direction, i.e., can perform a rotational movement.

The input interface 44 has the same configuration as the input interface 18. When the input interface 44 is operated by an operator D (technologist D1, ultrasound technician D2, assistant, etc.) in the treatment room, an operation signal is sent to the processing circuit 471.

The display 45 has a configuration equivalent to that of the display 19. The display 45 displays X-ray fluoroscopic image data as an X-ray fluoroscopic image during the interventional treatment procedure. The display 45 displays superimposed image data in which third medical image data (e.g., CT image data) is superimposed on the X-ray fluoroscopic image data as a superimposed image during the interventional treatment procedure.

Network interface 46 has the same configuration as network interface 16.

The medical image processing device 47 includes a processing circuit 471 and a memory 472.

The processing circuit 471 has a configuration equivalent to that of the processing circuit 171. The memory 472 has a configuration equivalent to that of the main memory 172.

The C-arm 48 (shown in FIG. 2) supports the X-ray irradiator 42 and the X-ray detector 43 so that they are arranged opposite to each other. The C-arm 48 can rotate in an arc direction, i.e., in the direction of the CRA (Cranial View) and in the direction of the CAU (Caudal View), under the control of the processing circuit 471 or in accordance with manual operation. The C-arm 48 can also rotate around a fulcrum, i.e., in the direction of the LAO (Left Anterior Oblique View) and in the direction of the RAO (Right Anterior Oblique View), under the control of the processing circuit 471 or in accordance with manual operation. The C-arm 48 may have a configuration in which the rotation in the arc direction corresponds to the rotation in the LAO direction and the rotation in the RAO direction, and the rotation around the fulcrum corresponds to the rotation in the CRA direction and the rotation in the CAU direction.

In addition, in FIG. 2, the C-arm structure of the X-ray imaging device 40 is shown as an under-table structure in which the X-ray irradiator 42 is located below the top plate of the bed 49. However, this is not limited to this case, and the X-ray irradiator 42 may be an over-table structure in which the X-ray irradiator 42 is located above the top plate. In addition, the C-arm 48 may be replaced by an Ω arm, or may be combined with an Ω arm.

The bed 49 is equipped with a tabletop (not shown) on which the patient P can be placed. The tabletop can move along the X-axis direction, i.e., slide left and right, under the control of the processing circuit 471. The tabletop can move along the Y-axis direction, i.e., slide up and down, under the control of the processing circuit 471. The tabletop can move along the Z-axis direction, i.e., slide head-to-foot, under the control of the processing circuit 471. The tabletop can also perform rolling and tilting movements under the control of the processing circuit 471.

Returning to the explanation of Figures 1 and 2, the medical image providing device 50 is connected to the ultrasound diagnostic device 10 and the X-ray imaging device 40 via the network N so as to be able to communicate with each other. For example, the medical image providing device 50 is a medical image management device (also called an "image server") that collects and stores third medical image data from a third medical image generating device that generates third medical image data of a different type from both ultrasound image data and X-ray image data. The medical image management device as the medical image providing device 50 is, for example, a DICOM (Digital Imaging and Communications in Medicine) server, and is connected to devices such as the ultrasound diagnostic device 10 so as to be able to communicate with each other via the network N. The medical image management device manages the third medical image data as a DICOM file.

Alternatively, the medical image providing device 50 is a third medical image generating device itself that generates the third medical image data. For example, the medical image providing device 50 is an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, or the like. Below, a case will be described where the medical image providing device 50 is an X-ray CT (Computed Tomography) device. However, the present invention is not limited to this case.

Next, we will explain the functions of the medical image processing system 1.

Figure 4 is a block diagram showing an example of the functions of the medical image processing system 1.

The processing circuitry 171 of the ultrasound diagnostic device 10 reads and executes a computer program (e.g., an image processing program) stored in the main memory 172 or a non-transitory recording medium such as a memory within the processing circuitry 171, thereby realizing a first image acquisition function F1, a second image acquisition function F2, a third image acquisition function F3, a first alignment function F4, a second alignment function F5, an image generation function F6, and an output control function F7. Below, an example will be described in which functions F1 to F7 are realized by a computer program, but all or part of functions F1 to F7 may be provided in the ultrasound diagnostic device 10 as functions of a circuit such as an ASIC.

The medical image processing device 17 of the ultrasound diagnostic device 10 controls the transmission/reception circuit 11, the B-mode processing circuit 12, the Doppler processing circuit 13, the image generation circuit 14, etc. to perform ultrasound imaging using the ultrasound probe 20 and generate ultrasound image data as first medical image data. The ultrasound image data is stored in the image memory 15.

The first image acquisition function F1 includes a function of acquiring ultrasound image data stored in the image memory 15. Specifically, the image acquisition function F1 acquires B-mode image data or the like as ultrasound image data. The first image acquisition function F1 is an example of a first image acquisition unit.

The second image acquisition function F2 includes a function of acquiring X-ray image data as second medical image data from an X-ray imaging device 40 that generates X-ray image data via the network N. The second image acquisition function F2 is an example of a second image acquisition unit.

Although multi-fusion (fusion of image output) between X-ray image data and ultrasound image data has been realized, it is difficult to image the entire left atrium (or the entire heart) using ultrasound image data. Even if ultrasound image data and markers are output from the ultrasound diagnostic device 10 to the X-ray imaging device 40, it is difficult for the operator to grasp the overall position information of the left atrium on the display 45 of the X-ray imaging device 40. For example, when performing atrial septal puncture, it is difficult to see the puncture position when displaying an ultrasound image alone using a TEE probe as the ultrasound probe 20, and the current situation is that the puncture position and angle are seen using a preoperative CT scan. Also, even if one wants to confirm where the left atrium of the heart is located and how large it is, the left atrium is too close to the TEE probe 20 to image the entire left atrium, so the entire left atrium cannot be confirmed.

The medical image processing device 17 has the following functions F3 to F7, and aligns the second medical image data with the third medical image data via the ultrasound image data, and generates superimposed image data by fusing (superimposing) the third medical image data on the second medical image data. As an example, we will assume that the superimposed image data is generated on the ultrasound diagnostic device 10 side, output to the X-ray imaging device 40, and displayed on the display 45 of the X-ray imaging device 40.

The third image acquisition function F3 includes a function of acquiring third medical image data different from ultrasound image data and X-ray image data from the medical image providing device 50 via the network N. For example, the third image acquisition function F3 acquires image data of a type different from ultrasound image data and X-ray image data as third medical image data different from ultrasound image data and X-ray image data. Examples of image data of a type different from ultrasound image data and X-ray image data include CT image data, MRI image data, image data including a predetermined anatomical feature (e.g., left atrium, heart, etc.), or an image including information indicating a catheter path (e.g., blood vessels). In addition, for example, the third image acquisition function F3 acquires ultrasound image data (e.g., past ultrasound image data) of the same type as the ultrasound image data but acquired in a different time phase as the third medical image data different from the ultrasound image data and X-ray image data. Below, a case where the third image acquisition function F3 acquires CT image data will be described. Note that the third image acquisition function F3 is an example of a third image acquisition unit.

The first alignment function F4 includes a function of aligning the X-ray image data acquired by the second image acquisition function F2 with the ultrasound image data acquired by the first image acquisition function F1. Alignment means spatial synchronization of both image data. The first alignment function F4 either generates new image data after aligning the X-ray image data with the ultrasound image data, or generates data regarding the amount of movement of each part of the X-ray image data after aligning the X-ray image data with the ultrasound image data. The first alignment function F4 is an example of a first alignment unit.

The second alignment function F5 includes a function of aligning the CT image data acquired by the third image acquisition function F3 with the ultrasound image data acquired by the first image acquisition function F1. The second alignment function F5 either generates new image data after aligning the CT image data with respect to the ultrasound image data, or generates data regarding the amount of movement of each part of the CT image data after aligning the CT image data with respect to the ultrasound image data. The second alignment function F5 is an example of a second alignment unit.

The image generation function F6 includes a function of aligning the X-ray image data and the CT image data based on the alignment result by the first alignment function F4 and the alignment result by the second alignment function F5, and generating image data in which the CT image data is aligned with the X-ray image data. For example, the image generation function F6 generates superimposed image data in which the aligned CT image data is superimposed on the X-ray image data as image data in which the CT image data is aligned with the X-ray image data. The following describes an example in which the image generation function F6 generates superimposed image data. Note that the image generation function F6 is an example of an image generation unit.

The output control function F7 includes a function for outputting the superimposed image data generated by the image generation function F6 to an external X-ray imaging device 40 via the network N. The output control function F7 may also include a function for displaying the superimposed image data on the display 19. The output control function F7 is an example of an output control unit.

The processing circuit 471 of the X-ray imaging device 40 realizes the display control function F8 by reading and executing a computer program (e.g., an image processing program) stored in a non-transient recording medium such as the memory 472 or a memory in the processing circuit 471. Below, an example will be described in which the function F8 is realized by a computer program, but all or part of the function F8 may be provided in the X-ray imaging device 40 as a function of a circuit such as an ASIC.

The medical image processing device 47 of the X-ray imaging device 40 controls the high voltage supply device 41, the X-ray irradiation device 42, the X-ray detection device 43, etc. to perform X-ray imaging and generate X-ray image data as second medical image data. The X-ray image data is stored in the memory 472.

The display control function F8 includes a function for displaying the superimposed image data output from the ultrasound diagnostic device 10 on the display 45 as a superimposed image. The output of the superimposed image data by the output control function F7 of the ultrasound diagnostic device 10 allows the display 45 of the X-ray imaging device 40 to display the superimposed image data as a superimposed image in approximately real time. The display control function F8 also includes a function for acquiring the X-ray image data stored in the memory 472 and displaying the acquired X-ray image data as an X-ray image on the display 45 in approximately real time. The output of the superimposed image data by the output control function F7 allows the display 45 of the X-ray imaging device 40 to display the superimposed image data as a superimposed image in approximately real time. Note that "approximately real time" includes a case where the X-ray image data or superimposed image data is generated (or displayed) at the same time as the detection of the X-ray, as well as a case where there is a time lag corresponding to the processing time from the detection of the X-ray to the generation (or display) of the X-ray image data or superimposed image data.

Details of functions F1 to F8 are explained using Figures 5 to 10.

Next, we will explain the operation of the medical image processing system 1.

Figures 5 and 6 are diagrams showing an example of the operation of the medical image processing system 1 as a flowchart. In Figures 5 and 6, the letters "ST" followed by a number indicate each step of the flowchart. Note that in the explanation of Figures 5 and 6, the ultrasound probe 20 is a TEE probe, but the present invention is not limited to this case. The ultrasound probe 20 may also be a body surface probe.

First, FIG. 5 will be described. The third image acquisition function F3 acquires CT image data (e.g., volume data) from the medical image providing device 50 as third medical image data relating to a specific patient P who is the subject of interventional treatment (step ST1). The CT image data includes at least an image of the entire left atrium that needs to be observed in order to place a medical device in the mitral valve.

Next, in the interventional treatment of the mitral valve of patient P using a medical device, the medical image processing system 1 generates X-ray fluoroscopic image data using the X-ray imaging device 40, and acquires ultrasound image data using the ultrasound diagnostic device 10 using the TEE probe 20 inserted into the esophagus of patient P. Then, the medical image processing system 1 superimposes third medical image data on the X-ray fluoroscopic image data aligned based on the ultrasound image data, and generates superimposed image data in which the third medical image data is superimposed on the X-ray fluoroscopic image data in approximately real time (steps ST2 to ST13 described below). An operator such as surgeon D1 performing the procedure advances the medical device while observing the superimposed image data displayed in approximately real time, and the medical device reaches a predetermined position.

The X-ray imaging device 40 starts X-ray fluoroscopy (step ST2). The X-ray imaging device 40 controls the high-voltage supply device 41, the X-ray irradiation device 42, and the X-ray detection device 43 to perform X-ray imaging, thereby generating a predetermined frame of X-ray image data (e.g., X-ray fluoroscopy image data with a low X-ray dose) for the patient P, and displays the X-ray fluoroscopy image data on the display 45 of the X-ray imaging device 40 (step ST3).

The first image acquisition function F1 determines whether or not to start ultrasound imaging (e.g., volume imaging) by the ultrasound diagnostic device 10 using the TEE probe 20 inserted into the esophagus of the patient P (step ST4). If the determination in step ST4 is NO, i.e., if it is determined that ultrasound imaging by the ultrasound diagnostic device 10 is not to be started, the X-ray imaging device 40 generates and displays X-ray fluoroscopic image data for the next frame (step ST3). While the X-ray fluoroscopic image data is being displayed for multiple frames, the surgeon D1 continues the procedure of advancing the catheter inside the patient P's body.

On the other hand, if the determination in step ST4 is YES, i.e., if it is determined that ultrasound imaging by the ultrasound diagnostic device 10 is to be started, the second image acquisition function F2 requests the X-ray imaging device 40 to output X-ray fluoroscopic image data (step ST4). The X-ray imaging device 40 generates X-ray fluoroscopic image data for the next frame (step ST7) and outputs the generated X-ray fluoroscopic image data to the ultrasound diagnostic device 10 via the network N.

The second image generation function F2 acquires X-ray fluoroscopic image data from the X-ray imaging device 40 (step ST8). The ultrasound diagnostic device 10 controls the transmission/reception circuitry 11, the B-mode processing circuitry 12 (or the Doppler processing circuitry 13), and the image generation circuitry 14 to perform ultrasound imaging, thereby generating a predetermined frame of ultrasound image data for the patient P. The first image acquisition function F1 acquires ultrasound image data from the image memory 15 (step ST8).

The first alignment function F4 aligns the X-ray fluoroscopy image data acquired in step ST7 with the ultrasound image data acquired in step ST8 (step ST9). For example, the first alignment function F4 aligns the X-ray fluoroscopy image data with the ultrasound image data by detecting the position of the TEE probe 20 with the position sensor 30, or by detecting the position of the TEE probe 20 from the X-ray fluoroscopy image data.

The second alignment function F5 also aligns the CT image data including the entire left atrium acquired in step ST1 with the ultrasound image data acquired in step ST8 (step ST10). Here, the second alignment function F5 aligns the CT image data with the ultrasound image data using a method different from the method of aligning the X-ray fluoroscopic image data with the ultrasound image data by the first alignment function F4. For example, the second alignment function F5 aligns the CT image data with the ultrasound image data based on characteristic areas depicted in the CT image data and the ultrasound image data (e.g., the heart, left atrium, blood vessels, etc.).

Moving on to the explanation of FIG. 6, the image generation function F6 generates superimposed image data (step ST11) in which CT image data based on the result of the alignment in step ST10 is superimposed on the X-ray fluoroscopic image data based on the result of the alignment in step ST9. This makes it possible to generate superimposed image data that can assist an operator such as surgeon D1 in interventional treatment.

The output control function F7 outputs the superimposed image data generated in step ST11 to the X-ray imaging device 40 via the network N (step ST12). The display control function F8 of the X-ray imaging device 40 displays the superimposed image data output in step ST12 as a superimposed image on the display 45 (step ST13). As a result, in interventional treatment, the superimposed image data on which the aligned CT image data is superimposed is displayed on the display 45, thereby appropriately supporting the procedure of an operator such as the surgeon D1.

Figure 7 shows an example of a superimposed image based on superimposed image data in which ultrasound image data is superimposed on X-ray fluoroscopic image data.

The superimposed image shown in FIG. 7 includes an X-ray fluoroscopic image IX obtained by X-raying an area including the entire left atrium, and an ultrasound image IU superimposed thereon. As described above, the position of the left atrium (left atrium image LA) is too close to the TEE probe 20 (TEE probe image Q) so that the entire left atrium cannot be imaged.

The CT image data, which is volume data like the ultrasound image data and can be aligned three-dimensionally and includes the entire left atrium, is aligned with the ultrasound image data as a reference. The image generation function F6 superimposes the aligned CT image data on the X-ray fluoroscopy image data that has also been aligned with the ultrasound image data as a reference.

Figure 8 shows an example of a superimposed image based on superimposed image data in which CT image data including the entire left atrium is superimposed on X-ray fluoroscopic image data.

The superimposed image shown in FIG. 8 comprises an X-ray fluoroscopic image IX obtained by X-raying an area including the entire left atrium, and a CT image IC including the entire left atrium superimposed thereon. Even if the position of the left atrium is too close to the TEE probe 20 (TEE probe image Q) to image the entire left atrium, an operator such as surgeon D1 can visually confirm the entire left atrium from the left atrium (left atrium image LA) depicted in an appropriately aligned CT image. In addition, this display makes it unnecessary to check the puncture position and angle in advance using a CT image with a large view when performing atrial septal puncture, and can assist the operator in determining the puncture position and angle.

In addition, the display screen may show the image area IU' of the ultrasound image that was used as the reference for alignment as a frame on the superimposed image.

Returning to the explanation of FIG. 6, the first image acquisition function F1 determines whether or not to end the ultrasound imaging (step ST14). When the placement and release of the medical device in the specified position is completed, the surgeon D1 operates the input interface 44 of the X-ray imaging device 40 to instruct the ultrasound diagnostic device 10 to end the ultrasound imaging. Alternatively, the ultrasound technician D2 operates the input interface 18 of the ultrasound diagnostic device 10 to instruct the ultrasound imaging to end.

If the determination in step ST14 is YES, i.e., if it is determined that the ultrasound imaging is to be terminated, the first image acquisition function F1 requests the X-ray imaging device 40 to terminate the output of the X-ray fluoroscopic image data (step ST15). In response to the request, the X-ray imaging device 40 terminates the display of the superimposed image data on the display 45 and returns to displaying the X-ray fluoroscopic image data alone (step ST3).

On the other hand, if the determination in step ST14 is NO, i.e., if it is determined that ultrasound imaging is not to be terminated, the X-ray imaging device 40 is not requested to terminate the output of X-ray fluoroscopic image data. Therefore, the X-ray imaging device 40 generates X-ray fluoroscopic image data for the next frame (step ST6) and outputs the X-ray fluoroscopic image data for the next frame to the ultrasound diagnostic device 10 (step ST7), thereby maintaining the updated display of the superimposed image data on the display 45.

5 and 6, the image generation function F6 aligns the X-ray image data (e.g., X-ray fluoroscopic image data) and the CT image data for all consecutive frames based on the alignment result by the first alignment function F4 and the alignment result by the second alignment function F5, and generates image data in which the CT image data is aligned with the X-ray fluoroscopic image data for all consecutive frames, for example, superimposed image data. Then, the display control function F8 displays the superimposed image data on the display 45 as a moving image in approximately real time. Note that this is not limited to this case. The image generation function F6 may align the X-ray fluoroscopic image data and the CT image data for a predetermined frame out of the consecutive frames based on the alignment result by the first alignment function F4 and the alignment result by the second alignment function F5, and generate image data in which the CT image data is aligned with the X-ray fluoroscopic image for the predetermined frame, for example, superimposed image data. In this case, the display control function F8 displays the X-ray fluoroscopic image data as an element of the superimposed image data on the display 45 as a moving image and the CT image data as a still image.

As described above, according to the medical image processing system 1, when performing interventional treatment, superimposed image data on which aligned CT image data including the entire left atrium is superimposed is displayed on the display 45, allowing an operator such as surgeon D1 to easily grasp the overall positional information of the left atrium.

(First Modification)
In Fig. 8, a case has been described in which the superimposed image data is generated by superimposing CT image data including the entire left atrium on the X-ray fluoroscopic image data, but the present invention is not limited to this case. For example, the image generating function F6 can generate superimposed image data in which the CT image data including the entire left atrium and the ultrasound image data used as the reference for alignment are superimposed on the X-ray fluoroscopic image data. The ultrasound image data partially includes an image of the left atrium. This allows the display control function F8 to display the superimposed image data as a superimposed image on the display 45 of the X-ray imaging device 40.

Figure 9 shows an example of a superimposed image based on superimposed image data in which CT image data including the entire left atrium and ultrasound image data including part of the heart are superimposed on X-ray fluoroscopy image data.

As shown in FIG. 9, the left atrium LA on the ultrasound image is further superimposed on the left atrium LA on the CT image shown in FIG. 8. This allows the display of a portion of the past CT image including the entire left atrium to be replaced with an ultrasound image showing the current state. The display control function F8 can switch from displaying the superimposed image shown in FIG. 8 to displaying the superimposed image shown in FIG. 9 based on the position of the catheter appearing on the X-ray fluoroscopic image or an operation via the input interface 44. The display control function F8 is not limited to displaying the superimposed image data superimposed with the ultrasound image data that was the reference for alignment, and may display the superimposed image data and the ultrasound image data that was the reference for alignment in parallel on the display 45.

In addition to the above effects, the first modified example allows the display of a portion of a past CT image that includes the entire left atrium to be replaced with an ultrasound image showing the current state, allowing an operator such as surgeon D1 to easily grasp the overall positional information of the left atrium.

(Second Modification)
In Fig. 8, a case where superimposed image data is generated by superimposing CT image data including the entire left atrium on X-ray fluoroscopic image data has been described, and in Fig. 9, a case where superimposed image data is generated by superimposing CT image data including the entire left atrium and ultrasound image data including a part of the heart on X-ray fluoroscopic image data has been described, but the present invention is not limited to this case. For example, the third image acquisition function F3 acquires CT image data including blood vessels, and the image generation function F6 can generate superimposed image data in which the CT image data including blood vessels is superimposed on the X-ray fluoroscopic image data. This allows the display control function F8 to display the superimposed image data as a superimposed image on the display 45 of the X-ray imaging device 40.

Figure 10 shows an example of a superimposed image based on superimposed image data in which CT image data including blood vessels is superimposed on X-ray fluoroscopic image data.

The superimposed image shown in FIG. 10 comprises an X-ray fluoroscopic image IX obtained by X-raying an area including the periphery of the left atrium, and a CT image IC including the catheter path up to the mitral valve, i.e., blood vessels, superimposed thereon. Even if the position of the left atrium is too close to the TEE probe 20 (TEE probe image Q) to image the periphery of the left atrium, an operator such as surgeon D1 can visually confirm the catheter path around the left atrium from the blood vessels (blood vessel image B) depicted in an appropriately aligned CT image.

According to the second modified example, when performing interventional treatment, superimposed image data on which aligned CT image data including blood vessels is superimposed is displayed on the display 45, allowing an operator such as surgeon D1 to easily understand the catheter path.

(Third Modification)
4, the medical image processing device 17 of the ultrasound diagnostic device 10 has the functions F1 to F7, and the medical image processing device 47 of the X-ray imaging device 40 has the function F8, but the present invention is not limited to this case. In the medical image processing system 1, multiple devices can cooperate to generate superimposed image data. In other words, in the medical image processing system 1, as long as the X-ray imaging device 40 has the function F8, any of the functions F1 to F7 may be provided. For example, the ultrasound diagnostic device 10 may have the functions F1 to F5 and F7, and the X-ray imaging device 40 may have the functions F6 and F8. This case will be described with reference to FIG. 11.

Figure 11 is a block diagram showing a modified example of the medical image processing system 1 shown in Figure 3.

The processing circuitry 171 of the ultrasound diagnostic device 10 reads and executes a computer program (e.g., an image processing program) stored in the main memory 172 or a non-transitory recording medium such as a memory within the processing circuitry 171, thereby realizing a first image acquisition function F1, a second image acquisition function F2, a third image acquisition function F3, a first alignment function F4, a second alignment function F5, and an output control function F7.

The first image acquisition function F1, as described above, includes a function of acquiring ultrasound image data as the first medical image data. The second image acquisition function F2, as described above, includes a function of acquiring X-ray image data as the second medical image data from an X-ray imaging device 40 that generates X-ray image data via a network N. The third image acquisition function F3, as described above, includes a function of acquiring third medical image data from a medical image provision device 50 via a network N.

As described above, the first alignment function F4 includes a function for aligning the X-ray image data acquired by the second image acquisition function F2 with the ultrasound image data acquired by the first image acquisition function F1. As described above, the second alignment function F5 includes a function for aligning the CT image data acquired by the third image acquisition function F3 with the ultrasound image data acquired by the first image acquisition function F1.

The output control function F7 includes a function to output the aligned X-ray image data generated by the first alignment function F4 and the aligned CT image data generated by the second alignment function F5 to an external X-ray imaging device 40 via the network N.

The image generation function F6 of the X-ray imaging device 40 includes a function for generating superimposed image data in which CT image data based on the result of alignment by the second alignment function F5 is superimposed on X-ray image data based on the result of alignment by the first alignment function F4, which is output from the output control function F7.

The display control function F8 includes a function for displaying the superimposed image data generated by the image generation function F6 on the display 45. By outputting the superimposed image data by the ultrasound diagnostic device 10 output control function F7, the display 45 of the X-ray imaging device 40 can display the superimposed image data as a superimposed image in approximately real time.

For example, the X-ray imaging device 40 may have function F8, and a third medical image processing device that is neither the ultrasound diagnostic device 10 nor the X-ray imaging device 40 and is connected via the network N may have functions F1 to F7. For example, the third medical image processing device may be a workstation that performs various image processing on medical image data, or a portable information processing terminal such as a tablet terminal.

According to the third modified example, the above effects can be obtained without introducing new functions F1 to F7 into the existing devices 10 and 40.

According to at least one of the embodiments described above, it is possible to generate image data that can assist an operator, such as a surgeon, during interventional treatment.

The first image acquisition function F1 is an example of a first image acquisition section. The second image acquisition function F2 is an example of a second image acquisition section. The third image acquisition function F3 is an example of a third image acquisition section. The first alignment function F4 is an example of a first alignment section. The second alignment function F5 is an example of a second alignment section. The image generation function F6 is an example of an image generation section. The output control function F7 is an example of an output control section. The display control function F8 is an example of a display control section.

Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, combinations of embodiments, and combinations of embodiments with one or more modified examples can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims.

1 Medical image processing system 10 Ultrasound diagnostic device 171, 471 Processing circuit 40 X-ray imaging device 45 Display 50 Medical image providing device F1 First image acquisition function F2 Second image acquisition function F3 Third image acquisition function F4 First alignment function F5 Second alignment function F6 Image generation function F7 Output control function F8 Display control function

Claims (12)

a first image acquisition unit that acquires an ultrasound image of a subject as a first medical image;
a second image acquisition unit that acquires an X-ray image of the subject as a second medical image;
a third image acquisition unit that acquires a third medical image of the subject, the third medical image being different from the ultrasound image and the X-ray image;
a first registration unit that registers the X-ray image with the ultrasound image;
a second registration unit that registers the third medical image with the ultrasound image;
an image generating unit that aligns the X-ray image and the third medical image based on a result of the alignment by the first alignment unit and a result of the alignment by the second alignment unit, and generates an image in which the third medical image is aligned with the X-ray image;
a display control unit that causes a display unit to display an image in which the third medical image is aligned with the X-ray image;
having
the display control unit aligns the ultrasound image that is the reference for alignment and further displays it on the display unit, and switches from displaying an image in which the third medical image is aligned with respect to the X-ray image to displaying an image in which the third medical image and the ultrasound image are aligned with respect to the X-ray image based on a position of the catheter that appears in the X-ray image.
Medical imaging equipment . the second alignment unit aligns the third medical image with the ultrasound image by a method different from a method of aligning the X-ray image with the ultrasound image by the first alignment unit;
The medical image processing device according to claim 1 . the first alignment unit detects a position of an ultrasound probe from the X-ray image to align the X-ray image with the ultrasound image;
The second registration unit performs registration of the third medical image with the ultrasound image based on characteristic parts depicted in the third medical image and the ultrasound image.
The medical image processing device according to claim 2 . the third image acquisition unit acquires, as the third medical image, a CT image or an MRI image of the subject, which is a type different from the ultrasound image and the X-ray image;
The medical image processing device according to claim 1 . The third image acquisition unit acquires an image including a predetermined anatomical feature as the third medical image.
The medical image processing apparatus according to claim 1 . the third image acquisition unit acquires, as the third medical image, an image including information indicating a path of the catheter;
The medical image processing apparatus according to claim 1 . A medical image processing system including an ultrasonic diagnostic device that generates ultrasonic images and an X-ray imaging device that generates X-ray images,
a first image acquisition unit that acquires an ultrasound image of a subject as a first medical image;
a second image acquisition unit that acquires an X-ray image of the subject as a second medical image;
a third image acquisition unit that acquires a third medical image of the subject, the third medical image being different from the ultrasound image and the X-ray image;
a first registration unit that registers the X-ray image with the ultrasound image;
a second registration unit that registers the third medical image with the ultrasound image;
an image generating unit that aligns the X-ray image and the third medical image based on a result of the alignment by the first alignment unit and a result of the alignment by the second alignment unit, and generates an image in which the third medical image is aligned with the X-ray image;
having
a display control unit for causing a display unit to display an image in which the third medical image is aligned with the X-ray image,
the display control unit aligns the ultrasound image that is the reference for alignment and further displays it on the display unit, and switches from displaying an image in which the third medical image is aligned with respect to the X-ray image to displaying an image in which the third medical image and the ultrasound image are aligned with respect to the X-ray image based on a position of the catheter that appears in the X-ray image.
Medical image processing system . The first alignment unit is provided in the ultrasonic diagnostic apparatus.
The medical image processing system according to claim 7 . The image generating unit is provided in the X-ray imaging device.
9. The medical image processing system according to claim 7 or 8 . the display control unit causes the display unit to display, in parallel, an image in which the third medical image is registered with respect to the X-ray image and the ultrasound image used as a reference for the registration.
The medical image processing system according to any one of claims 7 to 9 . the image generating unit aligns the X-ray images and the third medical image for all consecutive frames based on the alignment result by the first alignment unit and the alignment result by the second alignment unit, and generates a plurality of images in which the third medical image is aligned with the X-ray images for all consecutive frames;
The display control unit causes the display unit to display the plurality of images as a moving image in approximately real time.
The medical image processing system according to any one of claims 7 to 10 . the image generating unit aligns an X-ray image relating to a predetermined frame among a plurality of consecutive frames with the third medical image based on a result of the alignment by the first alignment unit and a result of the alignment by the second alignment unit, and generates an image in which the third medical image is aligned with the X-ray image relating to the predetermined frame;
the display control unit causes the display unit to display the X-ray image as an element of an image in which the third medical image is aligned with the X-ray image as a moving image and the third medical image as the element as a still image.
The medical image processing system according to any one of claims 7 to 10 .