JP5987640B2 - Method and apparatus for three-dimensional restoration of subject using ultrasound - Google Patents

Description

3.1 Industrial Applicability The present invention relates to an ultrasonic diagnostic imaging technique for restoring a living tissue three-dimensionally. Specifically, according to the present invention, a tubular structure can be three-dimensionally restored from a series of two-dimensional images using a conventional linear array transducer. It does not matter whether a 3D (or matrix) transducer is present. The 3D transducer can be mounted on a medical image processing apparatus with or without a position tracking device for the transducer. As a specific use of the present invention, it can be used to create a 3D model of the carotid artery for the diagnosis of atherosclerosis and the evaluation of plaque. The methods described below are used not only for carotid arteries but also for tubular tissue structures.

3.2 Background Art and Prior Art 3.2.1 Ultrasound Imaging System An ultrasound imaging system is useful when a doctor observes an organ of a human body. A typical ultrasound diagnostic imaging system includes a transducer (also known as a probe), a data acquisition unit, a data processing unit, a data display unit, and a user interface for connecting a human and a machine.

  A signal having a specific frequency and energy is used to excite the array elements of the probe and generate an acoustic pressure wave. The acoustic pressure wave passes through the medium and is directed to the subject to be observed. Furthermore, the acoustic pressure wave is reflected or absorbed by various physical features of the subject. The reflected wave bounces back to the probe, and the array element receives the reflected wave. Information on biological tissue is acquired from the reflected wave and processed to generate an ultrasound image.

  The ultrasonic diagnostic imaging apparatus is a “real-time” tomographic imaging apparatus and is one of the most widely used and versatile imaging modalities in the medical field. Compared to other modalities such as magnetic resonance (MR) and computed tomography (CT), the device is relatively inexpensive and easy to carry. Moreover, the risk to the patient by the utilization of the ultrasonic diagnostic imaging apparatus in medical treatment is not confirmed now. These advantages are the reason why an ultrasonic diagnostic imaging apparatus is used as a main diagnostic imaging modality.

  One of the most common types of ultrasound images is a B-mode image. The B-mode image is a gray scale image of the lower tissue generated from the received ultrasonic signal. The brighter region of the image corresponds to a portion of the biological tissue that reflects the ultrasound signal well, such as a boundary between organs. The darker region corresponds to a part of the biological tissue that does not reflect ultrasonic waves much, such as a hollow structure filled with fluid. The B-mode image shows the morphological structure of the scanned biological tissue. The scanned biological tissue can be used by a doctor for diagnosis. Alternatively, it is used when an important feature is acquired by an automatic algorithm for automatic evaluation of a living tissue.

  The advantages of an ultrasound diagnostic imaging device compared to other diagnostic imaging modalities include high frame rate, patient safety and ease of use. Other modalities include, for example, MRI, CT, mammography and the like. Since MRI and CT have a long scanning time, an image can be generated only in a relatively stationary state as compared with an ultrasonic diagnostic imaging apparatus. Although the patient is irradiated with X-rays when using mammography, frequent use of X-rays is not preferable. On the other hand, a major drawback of the ultrasonic diagnostic imaging apparatus is that the resolution of the image is lower than that of other diagnostic imaging modalities. Nevertheless, the ultrasonic diagnostic imaging apparatus is the most practical modality used for diagnosing a living tissue having a lot of movement such as the heart system.

  Many attempts have been made to improve the quality of an ultrasonic diagnostic imaging apparatus by a method for improving resolution and improving image quality, or a method for extracting an important feature of an ultrasonic image used for diagnosis. In group screening, which does not necessarily have the support of doctors, in order to perform ultrasonic image diagnosis of a larger number of people in particular, the manufacturer of an ultrasonic image diagnosis apparatus automatically extracts features of biological tissue. Tend to manufacture.

  Previous ultrasonic diagnostic imaging apparatuses can only generate 2D images using linearly arranged ultrasonic transducers. Due to the performance of the hardware, only a limited number of transducer elements could be used and the elements could only be arranged linearly. Recent advances in hardware and software have made it possible to use matrix ultrasonic transducers. An ultrasonic signal is generated from the components arranged in a matrix by the matrix ultrasonic transducer, and a large number of reception signals are generated, so that a 3D image of the scanned biological tissue is obtained. However, a wider range of usage and general usage of the ultrasonic diagnostic imaging apparatus in the clinic, miniaturization of the ultrasonic transducer for enabling imaging of each living tissue, and lower price of the ultrasonic diagnostic imaging apparatus, etc. In order to meet the demand, it is necessary to further develop an improved product of such an ultrasonic diagnostic imaging system including a matrix transducer.

  In many fields, 3D information ensures a more in-depth diagnosis. Some developers study whether it is possible to perform 3D imaging using current systems with only 2D functionality if a system with 3D imaging functionality that is readily available is not available. This can be achieved by combining a plurality of 2D images and creating a 3D model of the scanned biological tissue.

3.2.2 Prior Art A method for creating a 3D model of a scanned biological tissue by combining a plurality of 2D images has been invented.

  Some prior art attempts to create a 3D model of an artery by combining a series of 2D slice images, such as [2]. A 3D model is obtained by classifying and extracting coronary artery portions in a series of 2D images and combining the series of images. An interpolation method is used to acquire missing data between frames. The method uses morphological images of arteries and does not use any parameter information. Further, there is no guarantee that the position information of the scanning plane is correctly estimated, and there is no index for measuring the image quality of the extracted artery image.

  Another prior art [3] is also aimed at creating a 3D volume of blood vessels from a series of 2D images. In the prior art, since a series of 2D images are registered using the central axis of the blood vessel, an assumed error can be corrected. The images are combined to obtain a 3D model. As with the prior art described above, there is no guarantee of the accuracy of the location information and there is no index for measuring the quality of the registration step.

  Similarly, the prior art [4] aims to create a 3D model by combining a series of 2D images. Prior art [4] describes a system that 3D displays various features of tubular structures by combining a series of 2D images, such as midline, intra image, axial image, sagittal image, coronal image, etc. Yes.

  Prior art [5] also describes the creation of a 3D model by combining multiple 2D slice images. The 3D model may be available in advance, and the prior art focuses on a method for detecting the exact position of the 2D image in such a 3D model in order to display various anatomical structures of the scanned biological tissue. . Use display methods such as texture mapping to improve display (image) quality.

  Prior arts such as [6] and [7] describe systems that capture a series of 2D images using a position sensor. Since the position sensor is mounted on the ultrasonic probe as a position translator, the position of the probe may be derived from the exact position of the position translator. These prior arts focus on a system implementation method, not a method of creating a 3D model by combining 2D images.

  Some other prior arts attempt to create a 3D model of a certain structure using 2D images acquired from multi-views, such as [1]. Instead of capturing a series of 2D images parallel to each other, the prior art captures a series of 2D images while rotating the linear ultrasound probe at various angles. Using prior knowledge of the hollow shape, the boundary of the left ventricle of the heart is determined, and a 3D model of the left ventricle is created by combining 2D images acquired from various angles.

  In summary, some prior arts reconstruct a 3D model using 2D images, some use position sensors, and others reconstruct 3D from just two 2D images. There is no prior art with an index. Some prior art uses a predetermined model of scanned biological tissue based on expertise to improve the quality of the 3D model, but such a model is not applicable to all patients. Also, there is no prior art that utilizes multiple scan views in 3D reconstruction of tubular structures. Therefore, there is a shortage of data necessary to reliably obtain a high-quality 3D model.

US Pat. No. 5,871,019, “Fast cardiac boundary imaging” U.S. Pat. No. 8,105,239, “Method and apparatus to visualize the coronary arteries using ultra sound” US Pat. No. 7,480,398, “Method of registering a sequence of 2D image data with 3D image data” International Publication No. 1998/032371, “System for two-dimensional and three-dimensional imaging of tubular structures in human body” US Patent Application Publication No. 2005/0018892, US Patent Application Publication No. 2007/0253610, International Publication No. 1997/024697, US Patent Application Publication No. 2002/0191822, US Patent Application Publication No. 2005. No./0058327, International Publication No. WO 1996/041567, “Analytical visualization and measurement system” European Patent No. 1,653,861, U.S. Patent No. 6,996,432, U.S. Patent Application Publication No. 2004/0133105, International Publication No. 2005/018460, "Automated longitudinal position translator for" ultrasonic imaging probes, and methods of using same " US Pat. No. 6,193,736, US Pat. No. 5,361,768, US Pat. No. 5,485,846, US Pat. No. 5,592,942, US Pat. No. 5,759,153, US Pat. No. 6,013,030, US Pat. No. 6,409,672, US Pat. No. 6,623,433, US Patent Application Publication 2001/0021841, U.S. Patent Application Publication No. 2002/0143255, “Automated longitudinal position translator for ultrasonic imaging probes, and methods of using same”

3.3 Challenges The main challenges of the prior art are the lack of a method to ensure the quality of 3D models reconstructed using acquired 2D images, and multiple scan views used to reconstruct 3D models of tubular structures Lack of.

  The 2D image may be used as it is as the morphological image of the tubular structure, and the outline of the tubular structure in the image may be extracted as a parametric image. However, there is no prior art that shows how to quantify the quality of these images.

  For 3D reconstruction of biological tissue from 2D images, the exact position and orientation of the 2D image (information about it) is required. However, many prior arts do not provide a way to guarantee position and orientation accuracy.

  In addition, errors are caused by using only a series of 2D images from a particular scan view. Examples of errors include bad data due to freehand scanning, missing or unusable frames due to errors in the collected data, low-quality parts in the image due to limitations of scanning modalities, and influences due to movement in living tissue such as blood vessel pulsation. .

  An envisaged approach to overcoming the above problem is to use scan data obtained from multiple views of the structure. Some prior art uses data from more than one scan angle, but does not explain how to combine multiple views to overcome the limitations of each scan view.

3.4 Solution There is a need to provide a method for overcoming the above problems. The present invention provides a quality indicator and teaches a method for guaranteeing the quality of a reconstructed 3D model using multiple 2D images by using multi-view 2D images.

  The quality index is calculated using criteria such as gray scale change, consistency of gray scale change, arterial pulsation phase, and the like in order to indicate a boundary of living tissue. Note that the reference example is not limited to the above example.

  Multi-view images of the tubular structure of living tissue. The multiview may be classified into a short axis image and a long axis image. Multiple views are combined by defining one or more reference views. The contours of the biological tissue in these reference views may be adjusted by incorporating a quality index of the contour as a guide for adjustment using other views.

  The adjustment process is performed by minimizing a cost function composed of various parameters for adjusting the characteristics of the final 3D model. Different weights are given to parameters for adjusting the fitness with the 3D model for different factors, such as the contour in 3D space for each view and the smoothness of the 3D model.

3.6 Effects of the Invention The method described in the present invention may be used to improve the quality of a 3D model of a tubular structure. In one example, the method is validated using carotid phantom data. The result is shown in FIG.

  A large variation in the diameter of the tubular structure is seen in the 3D model before adjustment. After adjustment, this variation is greatly reduced.

Explanation of 4 drawings
FIG. 1 is a diagram illustrating the 3D reconstruction method of the present invention. FIG. 2 is a diagram illustrating another method of the 3D reconstruction method of the present invention using model vertex reliability calculation. FIG. 3 is a diagram showing another method of the 3D reconstruction method of the present invention using position information acquisition. FIG. 4 is a diagram illustrating another method of the 3D reconstruction method of the present invention using model vertex reliability calculation and position information acquisition. FIG. 5 is a diagram showing the vascular wall and intima-media thickness of the carotid artery. FIG. 6 is a contour diagram. FIG. 7 is a diagram illustrating a short-axis image and a long-axis image. FIG. 8 is a diagram showing a short-axis contour. FIG. 9 is a diagram showing a long-axis contour. FIG. 10 is a diagram illustrating an example of a preferred embodiment of the present invention in which position information is not acquired and reliability is not calculated. FIG. 11 is a diagram illustrating an example of a preferred embodiment of the present invention in which reliability is calculated without acquiring position information. FIG. 12 is a diagram illustrating an example of a preferred embodiment of the present invention that acquires position information and calculates reliability. FIG. 13 is a diagram illustrating an example of a preferred embodiment of the present invention using position information acquisition, reliability calculation, and navigation. FIG. 14 is a diagram illustrating an example of a preferred embodiment of the present invention using position information acquisition, reliability calculation, navigation, and contour selection. FIG. 15 is a diagram of intersections. FIG. 16 is a diagram of adjusted vertices. FIG. 17 is a diagram of an individual 3D model. FIG. 18 is a diagram showing the effect of the present invention.

3.5 Embodiments The method of the present invention aims to provide a method for reconstructing a 3D model of a tubular tissue structure from a multi-view 2D image. FIG. 1 is a schematic block diagram of the present invention. The following embodiments merely illustrate various inventive principles. Variations on the details described herein will be apparent to those skilled in the art. Accordingly, the present invention is limited only by the following claims and is not limited by the specific embodiments described herein.

  The medical ultrasonic (image diagnosis) apparatus may be configured by a plurality of elements or all elements of a signal transmitter / receiver, a data acquisition unit, a data processing unit, and a display unit. A high-frequency ultrasonic pulse may be sent to the scanned biological tissue by a transmitter. The receiver receives the reflection and scattering of the ultrasonic pulse. The data acquisition unit may convert the received data into an appropriate format for the ultrasound system along with other types of data required for the function of the ultrasound system. The data processing unit of the ultrasonic apparatus extracts the substructure of the scanned biological tissue and its characteristics as necessary, and displays the structure as an ultrasonic image. The present invention teaches a method to be executed in a data acquisition unit, a data processing unit and / or a display unit of a medical ultrasonic apparatus. When using the present invention, other components of the medical imaging device may be used.

  To illustrate the steps and effects of this embodiment, a linear array probe (linear probe) may be used in the method described herein. However, such description is for illustrative purposes only and linear array probes are one of the most widely used probes for medical imaging devices. 1.5D probes and 2D (matrix) probes or mechanical linear probes are also within the scope of the present invention.

  The description herein may be applied to scan data of any image modality that can provide a 2D scan image or a 3D scan image. Examples of image modalities include CT and MRI, but are not limited to these. In the following sections and paragraphs, the description will be given using ultrasound data for illustrative purposes, but only for ease of understanding. Such implementation of the methods described herein using other image modalities is also within the scope of the present invention.

  The concept of contour is shown in FIG. For illustration, two views of the carotid artery are shown. The same view can be used for more scan views of other tubular structures, and such illustrations are within the scope of the present invention.

  FIG. 7 shows an image acquired by the method described in FIG. FIG. 7 shows two scan views which are a short-axis direction view and a super-axis direction view. An ultrasonic transducer is placed relatively perpendicular to the centerline of the artery and a short axis image is acquired. In addition, a long axis image is acquired by placing the ultrasonic transducer relatively parallel to the center line of the artery.

  FIG. 8 shows an example of the contour in the minor axis direction. Extract the short-axis contour from the image using an algorithm such as an active contour model using the gradient of the image corresponding to the change from the blood vessel lumen (dark) to the blood vessel wall and surrounding biological tissue (bright) as an evaluation value. Also good. The short axis contour consists of a plurality of control points called vertices. In the case of a tubular structure through which a fluid flows, the short axis contour may be extracted using the Doppler method. The vertices may be arranged according to an angle with respect to the vertical direction.

  To image the entire artery, the transducer is moved from the common carotid artery (CCA) toward the internal carotid artery (ICA) / external carotid artery (ECA), and a series of 2D images in a short axis image are captured. Separately, position information and orientation information corresponding to each image may be acquired. Further, such information may be estimated by pattern matching the short axis image to the long axis image using the long axis image as a guide. Further, when a mechanical 3D probe is used, such information may be calculated from the position of the probe. Further, when a matrix probe is used for acquiring a 2D image, such information may be calculated from the position of a component of a transducer used at the time of image acquisition.

  Examples of apparatuses that may be used to acquire position information and orientation information include an optical camera system, a magnetic sensor, an accelerometer, and a gyroscope. However, the apparatus is not limited to these apparatuses. A magnetic sensor, accelerometer, or gyroscope may be physically attached to the ultrasound probe (may be mounted). Alternatively, an optical camera system may be used to track optical markers attached to the ultrasound probe. An optical image of one or a plurality of markers attached to a predetermined position of the ultrasonic probe may be acquired by one or a plurality of cameras. Such a marker may be used as a mark when detecting the position and orientation of the ultrasonic probe using an image processing algorithm for the acquired optical image.

  FIG. 9 shows an example of a long axis contour. The long axis contour may be extracted from the image by using an algorithm such as dynamic programming or pattern matching, or by using the gradient of the image on the blood vessel wall as an index. In the case of a tubular structure through which a fluid flows, the long axis contour may be extracted using the Doppler method. Further, the long axis contour may have a control point (vertex). Further, the long axis contour may be a curve without a control point.

  In order to image the entire artery, a scan process similar to the process for the short axis image may be performed on the long axis image. On the other hand, more than one view in the long axis direction may be obtained from various angles to make the model of the tubular structure more complete.

  For the sake of simplicity, the vertices of the curve, the control points, and all the contour points are hereinafter collectively referred to as vertices.

3.5.1 Summary of the Invention The main embodiment of the present invention is shown in FIG. The present invention includes the main blocks of data acquisition (100), contour extraction (101), individual 3D position calculation (102), contour reliability calculation (103), and 3D contour adjustment (104).

  Data acquisition (100) records a scanned image according to the requirements of the present application. As an example, the data acquisition unit (100) may acquire a scan image from a view along the short axis (lateral) direction and a view along the long axis (longitudinal) direction, which are two views of the carotid artery. . In another example, the position and orientation of the scan plane are acquired separately. In yet another example, a long axis image is acquired from two or more angles.

  In contour extraction (101), a tissue boundary (contour) of a tubular tissue structure is extracted using a scanned image. In one example, a short-axis contour is extracted from the short-axis image of the carotid artery, and a long-axis contour is extracted from the long-axis image of the carotid artery. For illustrative purposes, the contour may be the vessel wall and may be the adventitia-media boundary and the intima-lumen boundary for measuring intima-media complex thickness (IMT). In FIG. 5, the intima-media complex thickness (IMT) is shown in more detail. However, the contour concept can be used for any tubular structure and such applications are within the scope of the present invention.

  The individual 3D position calculation (102) calculates the position and orientation of the scanned image of each view in the 3D space. In one example, the individual 3D position calculation (102) uses a different device to acquire position information and orientation information of the scan plane. Alternatively, the information is calculated from the scanned image using a pattern matching method. Alternatively, the information is calculated from the position and orientation of the mechanical probe. When a matrix probe is used, the above information is calculated from the positions and orientations of the transducer components used during image acquisition. The above information is used for positioning the scan plane in the 3D space. In another example, individual 3D models of scanned tissue (eg, carotid artery) can be created by combining scan images of specific views for registration or visualization of individual models. In another example, a method of estimating position and / or orientation from an image is used when position information and / or orientation information is not available. For example, manual, semi-automatic or automatic image alignment by pattern matching can be mentioned.

  In the contour reliability calculation (103), the reliability of the vertex or control point of the contour in each scan view is calculated. The reliability determines the reliability of the vertex and determines whether the vertex is reliable for use in reconstructing the final 3D model of the tubular structure. Hereinafter, this reliability is referred to as contour vertex reliability.

  The 3D contour adjustment (104) considers the contour vertex reliability, and creates individual 3D models of tubular structures by combining individual contours or models extracted from each view in the 3D space. The 3D model also has vertices. Hereinafter, the process of calculating the final 3D model is referred to as an optimization process. The final 3D model is referred to as an optimized 3D model.

  FIG. 2 shows another embodiment of the present invention. The data acquisition (200), contour extraction (201), individual 3D position calculation (202), contour reliability calculation (203), and 3D contour adjustment (204) shown in FIG. Similar to the corresponding block.

  Model vertex reliability calculation (205) is an additional optional step for calculating the vertex reliability of the optimized 3D model. Hereinafter, this reliability is referred to as vertex reliability. This model vertex reliability is used when the 3D model has already been optimized and a new view is acquired. In consideration of the previous model vertex reliability and the new model vertex reliability, a new optimized 3D model may be calculated using the previously optimized 3D model and the newly captured contour.

  FIG. 3 shows another embodiment of the present invention. The data acquisition (300), contour extraction (301), individual 3D position calculation (302), contour reliability calculation (303), and 3D contour adjustment (304) shown in FIG. Similar to the corresponding block.

  In the position information acquisition (305), the position and orientation of the acquired data used in the individual 3D position calculation (302) are calculated. Each image may have position information and orientation information acquired using an individual device. Further, such information may be estimated using a long axis scan as an index for pattern matching. Further, when a mechanical 3D probe is used, such information may be calculated from the position of the probe. Further, when a matrix probe is used for acquiring a 2D image, such information may be calculated from the position of a component of a transducer used at the time of image acquisition.

  FIG. 4 shows another embodiment of the present invention. This embodiment is a combination of the above two embodiments shown in FIGS.

  Details of each processing block are described in the following paragraphs.

3.5.2 Data Acquisition Data acquisition (100, 200, 300, 400) is a process of acquiring scan data of tubular structures of various views. The tubular structure may be the carotid artery. Where the carotid artery is described below, the description is for purposes of illustration only. The methods described herein can be used with any tubular structure, such as the abdominal aorta, and such embodiments are within the scope of the invention.

  The description herein may be applied to scan data of any imaging modality that can provide 2D or 3D scan images. Examples of imaging modalities include CT and MRI, but are not limited to these. Where ultrasound data is described below, the description is for purposes of illustration only. Implementation of the methods described herein using other imaging modalities is within the scope of the present invention.

  In the tubular structure, the overall shape of the tubular structure is expressed by the shape of the center line. A short-axis image may be defined as an image acquired on a scan plane that is relatively perpendicular to the center line. In this image, the intersection between the tubular structure and the scan plane is a relatively circular and closed contour. The long axis image may be an image acquired on a scan plane relatively parallel to the center line. In this image, the intersection of the tubular structure and the scan plane has a relatively elongated profile (which may be closed or open).

  The scan view is a set of scan planes near a specific direction of the ultrasonic probe. Such a scan plane may be acquired by scanning (moving) the linear array ultrasonic probe along one direction while keeping the orientation of the probe constant. Such a scan plane may be acquired by fixing a linear array ultrasonic probe that automatically performs three-dimensional scanning at a specific position. The data is obtained as the probe moves from one end to the other. Such a scan plane may be obtained by fixing a matrix (2D) probe in a specific position and actuating a linear set of parallel transducer components. Thereby, the acquired plurality of scan planes are parallel to each other.

  In one embodiment, a multi-view of the tubular structure is acquired along the lateral direction to acquire a short-axis image, and a multi-view of the tubular structure is acquired along the vertical direction to acquire a long-axis image. A short axis image is acquired by scanning the transducer from one end of the tubular structure to the other. Each image of cine data corresponds to one cross-sectional image at a specific position on the tubular structure. A long axis image may be obtained using a similar technique by scanning the transducer from one end of the tubular structure to the other.

  In another embodiment, the tubular structure may be serpentine or bent. The tubular structure may be subdivided into a plurality of portions that are relatively straight. In each portion, a long-axis scan image and / or a short-axis scan image may be acquired individually. These pieces of data in different parts may be used as individual scan views.

  In another embodiment, the long axis image may be acquired by fixing the transducer at a specific position and angle to acquire a maximum split plane image of the long axis image, ie, an image including the centerline of the tubular structure. .

  In another embodiment, in order to more efficiently capture the features of the longitudinal tubular structure, it may be captured at more than one tilt angle by selecting different scan views.

  In another embodiment, for example, if there are multiple tributaries in a tubular structure, such as the common carotid artery, the external carotid artery, and the internal carotid artery, to obtain a complete long axis image of each tributary, An axial image may be selected. Each view matches each tributary. More than one tilt angle may be used to capture more accurately the features of the tubular structure.

  In another embodiment, in the case of a human blood vessel, the image may be further filtered so that only images with a particular beating state are selected for subsequent processing. For example, when the carotid artery is imaged for measuring the intima-media thickness, only the image of the smallest arterial diameter in the systole is selected.

  The output of data acquisition (100, 200, 300, 400) is a multi-view image.

3.5.3 Acquisition of position information Acquisition of position information (305, 405) is processing for acquiring position information of an acquired image obtained from data acquisition (300, 400). This block is an arbitrary block, and an embodiment in which processing of this block is not performed is also within the scope of the present invention.

  In one embodiment, the position and orientation of each scan plane may be acquired using another device, such as in synchronization with the acquisition of scan data. This information is used to indicate the 3D position of the scan plane.

  Examples of other devices that may be used to acquire position information and orientation information include optical camera systems, magnetic sensors, accelerometers, and gyroscopes. However, the present invention is not limited to these devices. The magnetic sensor, accelerometer, and gyroscope may be physically attached to the ultrasound probe. Alternatively, the optical camera system may be used to track optical markers attached to the ultrasound probe. An optical image of one or a plurality of markers attached to a predetermined position of the ultrasonic probe may be acquired by one or a plurality of cameras. Such a marker may be used as a mark when detecting the position and orientation of the ultrasonic probe using an image processing algorithm for the acquired optical image.

  Details of such an apparatus are outside the scope of the present invention and will not be described further.

  In another embodiment, when the linear probe is mechanically scanned for data acquisition, the acquired position and orientation of the scan plane are determined from the position and orientation of the linear probe. The acquired position information and orientation information on the scan plane can be acquired by an ultrasonic device. Details of such an ultrasonic device are beyond the scope of the present invention and will not be described further.

  In another embodiment, the position and orientation of the acquired scan plane may be determined by the position and configuration of the transducer components used in the matrix (2D) probe during image acquisition. The matrix probe may be used to acquire 3D data, or may be used to acquire 2D data when only a linear set of transducer components is in operation. The position and orientation of the scan plane relative to such a linear set is obtained from the position of the active transducer components and the steering angle of the ultrasound beam. Details of such calculation are beyond the scope of the present invention and will not be described further.

  In another embodiment, if a device that obtains position information and orientation information is not available, such information may be estimated using pattern matching of contours between scan views. In order to perform accurate estimation, scan views may be acquired at angles orthogonal to each other. For example, a short-axis image may be acquired perpendicularly to the center line of the tubular structure, and a long-axis image may be acquired parallel to the center line of the tubular structure.

  The output of position information acquisition (305, 405) is the position information of the scan plane as obtained by data acquisition (300, 400).

3.5.4 Contour Extraction Contour extraction (101, 201, 301, 401) is processing for determining the boundary of a tubular structure in a scanned image. Hereinafter, such a boundary will be collectively referred to as an outline. These contours contain control points or vertices. Such control points and vertices are collectively referred to as vertices.

  These contours in the image are characterized by grayscale changes between regions in the image. FIG. 5 shows an example of a gray scale change in the case of a blood vessel. The lumen is generally darker than the surrounding biological tissue. Also, since the middle layer is generally darker than the inner and outer membranes, it becomes a thin and dark region between two brighter regions. An appropriate matching filter may be used to characterize the lumen, intima, media, and adventitia using the characteristics of gray scale change. The description here is an example in which the characteristics of a specific gray scale change are used to extract the boundary of a living tissue. Other characteristics of gray scale changes may be used for other types of living tissue, and this is also within the scope of the present invention.

  When utilizing gray scale changes, various techniques can be applied, which can detect gray scale changes and continuity of points where the gray scale changes. The contour may be extracted using a Doppler method (for example, a color Doppler method or a power Doppler method).

  All views can be classified into two types, short axis images and long axis images. A short axis image may be defined as a view in which the contour is closed and such a contour surrounds the center point. A long-axis image may be defined as a view in which a contour is open and such a contour surrounds the center line.

  FIG. 7 shows an example of a short axis image and a long axis image. FIG. 8 shows an example of a short axis contour, and FIG. 9 shows an example of a long axis contour.

  A contour extraction method is established in the prior art.

  An example of the contour extraction algorithm is edge detection using image processing. Edge detection processing is widely introduced in the prior art. The actual implementation of the edge detection algorithm is outside the scope of the present invention and will not be described further here.

  Another example of the contour extraction algorithm is an active contour algorithm. In the active contour model, the energy of the contour including many vertices is minimized. The total energy includes external energy and internal energy. External energy is measured from image features at the contours. Internal energy is measured from features of a specific shape of the contour. It can be seen from the characteristics of the image that the external energy is a tensile force. For example, a gray scale gradient with respect to a blood vessel wall or an edge of an image. The internal energy is the elastic force between the vertices and becomes the curvature energy of the spline. Active contour models are widely introduced in the prior art. Details of the active contour model are described in prior art documents. The actual implementation of the active contour is outside the scope of the present invention and will not be described further here.

  Another example of a contour extraction algorithm is the detection of grayscale in partial segments of an image using a matching filter. A matching filter having a predetermined profile exhibiting a desired gray scale change characteristic is used for the partial segment. Such a partial segment may be a scan line of an image. As a result of the filtering process, a best match (optimal combination) at a position following the profile of the matching filter is obtained. Of a series of curves passing through all the partial segments, a curve having the maximum accumulated cost of matching in the partial segments is extracted as a contour.

  The above extraction algorithm is expected to improve accuracy by narrowing the region of interest including the contour before applying the contour extraction algorithm. In many of the captured images, there may be multiple regions in the image that are similar to the structure of interest (eg, the short axis image is a closed circular outline, and the long axis image is an elongated outline). However, in most cases, the correct area is only one of these areas. Such narrowing may be performed using a pattern matching algorithm.

  Another example of the contour extraction algorithm uses the Doppler method. For tubular structures where fluids flow through structural walls (such as blood vessels), the Doppler method may be used to detect the areas where fluids flow, and the boundary between these regions is the boundary between the fluid region and the structural wall. is there. The inner wall of the structure may be extracted by detecting the boundary from the Doppler image.

  In another example of the contour extraction algorithm, multiple contours may be extracted, particularly in blood vessels. An example of such multiple contours is the lumen-intima boundary (or intima-media inner contour) and the media-outer membrane boundary (or intima-media outer contour). In order to diagnose the carotid artery such as measurement of intima-media complex thickness (IMT) or detection of plaque, both layers may be extracted and IMT may be calculated to detect plaque. The application of the outer contour to the diagnosis of the aorta is more important. The contour used depends on the respective field of application. Such applications are also within the scope of the present invention.

  The output of contour extraction is the vertex of the contour on the scan plane.

3.5.5 Individual 3D position calculation In the individual 3D position calculation (102, 202, 302, 402), all 2D calculated contour vertex coordinates in each scan plane are converted into 3D space coordinates. Based on the corresponding position information and orientation information, the above-described conversion is performed by moving and rotating the scan plane.

In the embodiment in which the position information and the orientation information of the scan plane can be obtained by acquiring the position information (305, 405), the available information may be used as it is to move and rotate.
In embodiments where the scan plane position information and orientation information are not available, such information may be estimated from the scanned image using a pattern matching method. For use in the estimation process, scan views may be acquired at angles orthogonal to each other.

  In embodiments where the position and orientation of the scan plane is not available, the scan planes of a particular view may be parallel to each other and the scan plane spacing in that view may be constant. Even if the position of each scan plane is calculated from the scan time (for example, the time required for the probe to move from one end of the scan area to the other end), the frame rate of the data acquisition unit, the size of the scan area, etc. Good.

  As an additional embodiment to overcome unexpected errors in the data acquisition process, such as missing frames and low quality images that lead to missing contours, using the contours of other scan views within the same 3D region as a guide Such a missing contour may be interpolated from adjacent contours of adjacent frames in the same scan view. Further, an isolated contour may exist due to the influence of the contour extraction error. An isolated contour may differ significantly from nearby contours. These isolated contours may be removed, and the contour position of the corresponding frame may be interpolated between adjacent frames.

  In an additional embodiment for preparing the next step of combining the contours of multiple scan views, one scan view is selected as the reference scan view. In one embodiment, such a scan image is a short axis scan image. For each of the scan planes of this reference scan view, the intersections of all the contours of this scan plane and other scan views are calculated. The 3D coordinates of these intersection points are converted into corresponding 2D coordinates of the scan plane in the scan view.

  In other embodiments, manual, semi-automatic, or fully automatic alignment may be performed to interpolate errors in position and orientation information. Such alignment may be performed using the center line of the individual model. Moreover, you may perform using the feature point matching of B mode.

  The output of this block is an individual 3D model, the position of the intersection in 2D. FIG. 17 shows an example of an individual 3D model of the carotid artery. Two models are shown, a model created from a short-axis scan image and a model created from a long-axis scan image.

  The short-axis contour is in the form of a circular ring, and the linear probe moves along the long-axis direction of the artery (the length of the artery), so the short-axis contour is photographed on a relatively parallel surface. All contours are shown in the same coordinate system, and a model as shown in the left panel of FIG. 17 is created.

  The long-axis contour is an open gentle curve, and the linear probe moves along the short-axis direction of the artery (the width of the artery), so the long-axis contour is photographed on a relatively parallel surface. Only the contour with the best rendering state is used, and in this example, the contour with the highest quality corresponds to the scan plane corresponding to the largest split surface of the artery.

  FIG. 15 shows an example of the scan plane and the intersection. In FIG. 15, the reference plane is a short axis plane, and the carotid phantom is scanned. Since the artery appears as a circular ring in the short axis plane, the short axis contour also appears as a circular ring. The major axis (direction) scan is performed a plurality of times at an angle relatively perpendicular to the minor axis surface. The intersection of the long axis contour and the short axis surface is detected. These intersections are used to adjust the short axis contour.

3.5.6 Contour Reliability Calculation The contour reliability calculation (103, 203, 303, 403) calculates the reliability of all the contours, the vertexes of the contour on a specific scan plane, and / or the intersection of the contours. The input of this block is the position of the B-mode image, the contour point and / or the intersection of a specific scan plane.

  The reliability of the vertices of the entire contour or of individual contours is calculated from the reliability index of the reliability parameter and various reliability values. For example, the range of reliability values may be set between 0 and 1. One or less reliability parameters are shown below.

  When the reliability index is the maximum value, the reliability value is 1.

  When the reliability index is the minimum value, the reliability value is 0.

  • Linear mapping is applied when the reliability index is between the minimum and maximum values.

  In one embodiment of the contour reliability calculation for a blood vessel, the reliability parameter is the pulsation time phase and the reliability index is the blood vessel diameter. The blood vessel diameter becomes maximum during the period when the blood vessel expands, and the blood vessel diameter becomes minimum during the period when the blood vessel shrinks. The blood vessel diameter may be obtained from a distance between relatively parallel long-axis contours in the long-axis scan, or from a line segment from one end of the short-axis contour to the other end passing through the center point of the contour. As the influence of pulsation, the reliability of the contour vertex is high during the period when the blood vessel shrinks, and the reliability of the contour vertex is low during the period when the blood vessel expands. Therefore, the reliability value may be set to 1 in the case of the minimum blood vessel diameter, and the reliability value may be set to 0 in the case of the maximum blood vessel diameter. In this embodiment, the same confidence value is applied to all vertices of the contour.

  In another example of contour reliability calculation for a blood vessel, the reliability parameter is the pulsatile phase and the reliability index is an averaged power Doppler value in the corresponding blood vessel. Since the blood flow is greatest during the period when the blood vessel is expanded, the power Doppler value is maximized. In addition, since the blood flow is minimized during the period when the blood vessel shrinks, the power Doppler value is minimized. As the influence of pulsation, the reliability of the contour vertex is high during the period when the blood vessel is reduced, and the reliability of the contour vertex is low during the period when the blood vessel is enlarged. Therefore, the reliability value may be set to 1 when the averaged power Doppler value is minimum, and the reliability value may be set to 0 when the averaged power Doppler value is maximum. In this embodiment, the same confidence value is applied to all vertices of the contour.

  In another embodiment of the contour confidence calculation for any tubular structure, the confidence parameter is the proximity of the boundary of the tubular structure to the contour. The reliability index is a gray scale change of the B-mode image at the contour vertex or intersection. If the gradient change is large at the position of the vertex or intersection, the reliability is high. Further, when the change in the gradient is small at the position of the vertex or intersection, the reliability is low. The gradient change is calculated assuming that the gray scale value of the inner region of the tubular structure is low and the gray scale value of the outer region is high. Thus, the largest change in grayscale indicates the boundary between these two regions. Based on this assumption, the gray scale change at all vertices of the extracted contour is calculated in the direction from the inside to the outside of the tubular structure with respect to the specific scan plane. The maximum and minimum grayscale changes may be calculated, the confidence value may be set to 1 if the grayscale change is maximum, and the confidence value is set to 0 if the grayscale change is minimum. May be. In the present embodiment, individual reliability values are calculated for individual vertices or intersections.

  In another embodiment of the contour confidence calculation, the confidence parameter is a series of gray scale changes along the contour, and the confidence measure is the difference in gray scale change near the calculated contour vertex or intersection. If similar grayscale changes continue for a long time in the vicinity of the vertex or intersection, the confidence level is high. In addition, the reliability decreases under the opposite condition. The assumed method for detecting the continuity of the grayscale change is a method for checking the difference in the grayscale change of adjacent vertices or intersections of the contour. When this difference is large, the reliability is low. When this difference is small, the reliability is high. The maximum difference and the minimum difference may be calculated for all vertices or intersections of the contour. For the smallest difference, the confidence value may be set to 1. In the case of the maximum difference, the reliability value may be set to zero. In the present embodiment, individual reliability values are calculated for individual vertices or intersections.

  In another embodiment of contour confidence calculation, the confidence parameter is the consistency of the gray scale change over the whole contour and the confidence measure is the standard deviation of the gray scale change at all vertices or intersections. For more uniform grayscale changes, the confidence is high and the standard deviation is low. In the opposite case, it can be said that the reliability is low and the standard deviation is high. The standard deviation of the gray scale change is calculated for a specific contour. The maximum standard deviation value and the minimum standard deviation value can be obtained for all the contours of the same scan view. If the grayscale standard deviation is the smallest, the contour reliability value may be set to 1. When the standard deviation of the gray scale is the maximum, the contour reliability value may be set to zero. In this embodiment, the same confidence value is applied to all vertices of the contour.

  In another embodiment of contour reliability calculation, there are two or more reliability values calculated for the contour or the vertex of the contour. In such a case, a specific gravity may be placed on a more important reliability parameter, and a plurality of reliability values may be weighted and summed.

  Also, different combinations of reliability indices can be used for the short-axis image and the long-axis image. In another embodiment of the contour reliability calculation, the contour reliability value in the long-axis scan image may be set higher than the contour reliability value in the short-axis scan image.

  In another embodiment of the contour reliability calculation, the contour reliability value of the short-axis scan image may be adjusted based on the contour reliability value of the nearby long-axis scan image. For example, as the reliability value of the contour of the long-axis scan image increases, the reliability value of the short-axis scan image is decreased.

  In one example, in a tubular structure such as a blood vessel, the blood vessel wall moves as the fluid pulsates. By using a method used to acquire a short-axis scan image, for example, a method of scanning a probe from one end to the other end of a tubular structure, different scan planes may correspond to different pulsation time phases. Thereby, inconsistency in the 3D model created from these scan planes appears as irregularities on the blood vessel wall. On the other hand, the long-axis scan image acquired by selecting a specific scan position and angle has the same pulsation time phase as a whole. In such a case, a higher reliability value may be given to the long-axis scan image than to the short-axis scan image.

  Also, the edge of the tubular structure of the short axis scan image is unclear on the side of the tubular structure. Therefore, the contour extracted from these edges may not be reliable. The position and orientation of the long-axis scan image may be determined so that these unclear edges can be photographed from different angles. In such a case, a higher reliability value may be given to the long-axis scan image than to the short-axis scan image.

  The output of the contour reliability calculation is a reliability value of the contour or the vertex of the contour on all the scan planes used in the next 3D contour adjustment step.

3.5.7 Contour Adjustment in 3D 3D contour adjustment (103, 204, 304, 404) is a 3D model that integrates individual 3D models of different scan views of the same tubular structure according to confidence values. It is a process to improve the accuracy of the model.

  In the individual 3D position calculation, the reference scan view may be selected. The reference scan-view image may be based on the fact that the scan view includes the entire tubular structure, or that the outline of the scan view indicates the shape of the entire tubular structure. There may be two or more reference scan views to be selected.

  Other scan views may not be complete views due to possible reasons such as the scan angle can only capture a specific position, or the scan probe is not long enough to capture the entire length of the tubular structure. These incomplete scan views may not be selected as reference scan views. Instead, these imperfect scan views may be used to adjust the reference scan view and improve quality. These scan views may be referred to as “non-reference views”.

  The contour vertices used for adjustment are the vertices of the reference view and the intersections of the contours of the non-reference view and the scan plane of the reference view. The contours and vertices of the reference view are called reference contours and reference vertices. Vertices derived from non-reference views are called intersection vertices.

  In one embodiment of 3D contour adjustment, the contour in each scan plane in the reference view is optimized using the original contour vertices in the scan plane, the intersection of other scan views in the scan plane, and the corresponding confidence value This adjustment may be performed.

  The optimization process may be performed continuously on the scan plane of the reference view. The optimization process may be performed on a plurality of scan planes in one operation. The result of the optimization process is called “new contour”.

  Optimization is performed by minimizing a cost function that includes at least all of the following elements or a subset of the following elements:

-Distance from new contour to reference contour weighted by reliability (contour vertex cost)
・ Distance from new contour to intersection weighted by reliability (intersection cost)
-Difference in radius between adjacent contour points in the new contour (smoothness cost of contour in the frame)
The distance between the new contour vertex and the corresponding contour vertex of the adjacent scan plane of the reference view (contour smoothness cost between frames)
・ Distance between new center point and adjacent center point (smoothness cost of center line)

  Each element adjusts the parameters that the optimized 3D model fits. The weight factor for that element adjusts the fitness. Here, the weighting factor is a relative value. For example, when the absolute values of the weighting factors are all equal and large, the fitness for each parameter is the same. On the other hand, if the weighting factor is a relatively different value, the fitness for each parameter is adjusted according to the value of the weighting factor. More specifically, the entire cost function may be described using the following equation. The importance of each element is adjusted by weighting each term of the cost function with a weighting factor. A higher weight factor means that the corresponding element is more important. A lower weight factor means that the corresponding element is less important.

  In another embodiment, the element weight associated with the long axis scan image is set higher than the element weight associated with the short axis scan image.

  In one example, a tubular structure, such as a blood vessel, has a moving wall that is generated by vibrating a fluid flow. By using a method used to acquire a short-axis scan image, for example, a method of scanning a probe from one end to the other end of a tubular structure, different scan planes may correspond to different pulsation time phases. Thereby, the mismatch in the 3D model created from these scan planes appears as a waveform on the structural wall. On the other hand, a long-axis scan image may be acquired by selecting a specific scan position and angle. In the long-axis scan image, the entire scan image corresponds to a specific pulsation time phase. In such a case, a higher weight may be given to the cost factor related to the long-axis scan image than to the cost factor related to the short-axis scan image.

  In another example, the edge of the tubular structure of the short-axis scan image is unclear on the side of the tubular structure. Therefore, the contour extracted from these edges may not be reliable. The position and orientation of the long-axis scan image may be determined so that these unclear edges can be photographed from different angles. In such a case, a higher weight may be given to the element related to the long-axis scan image than to the short-axis scan image.

  In the following equation, the new vertex is called newVtx. The new contour formed by these vertices is called newContours. The vertex is called vtx. The intersection vertex is called intsec.

  The total cost function may be calculated using the following formula.

  The contour vertex cost may be calculated using the following formula.

Here, i indicates a new vertex index. vtxReliability _i indicates the reliability of the reference vertex. vtx _i indicates a reference vertex. newVtx _i indicates a newly sampled vertex.

  The intersection vertex cost may be calculated using the following formula.

Here, j indicates an index of a new vertex having a nearby intersection vertex. k represents an index of all intersecting vertices in the vicinity of the vertex j. intsecReliability _{k, j} indicates the reliability of the intersection vertex. intsec _{k, j} indicates the intersection vertex. newVtx _j indicates a newly sampled vertex.

  You may calculate the contour smoothness cost in a flame | frame using a following formula.

Here, i indicates an index of all the vertices. k indicates an index of all vertices adjacent to a specific vertex. newVtx _i indicates a newly sampled vertex. “adjacentVtx _{k, i”} indicates a vertex adjacent to the vertex i.

  The contour smoothness cost between frames may be calculated using the following equation.

i indicates an index of a new vertex. p indicates an index of all corresponding vertices of the adjacent surface. vtxReliability _p indicates the reliability of the adjacent vertex. newVtx _i indicates a newly sampled vertex. adjPlaneVtx _{i, p} indicates the vertex of the adjacent surface corresponding to the vertex i.

  The smoothness cost of the center line may be calculated using the following formula.

  Here, p indicates an index of all center points adjacent to the current scan plane and including the current scan plane. newCenterPt indicates the center point of the new vertex. AdjacentCenterpt indicates the original center point in all adjacent faces including the current face.

  As a cost function minimizing method composed of a large number of parameters, a statistical optimization algorithm such as an annealing method can be used. Such embodiments are outside the scope of the present invention and will not be described further here.

  The output of the 3D contour adjustment is an optimized contour of the reference view or a 3D model of a tubular structure including all optimized vertices in all scan planes of the reference view.

  FIG. 16 shows an example of the optimization vertex.

  In another embodiment of 3D contour adjustment, the adjustment may be made by averaging the contour coordinates of the corresponding reference contour and non-reference contour. A confidence value may be used as a weighting factor for the averaging process.

  In another embodiment of 3D contour adjustment, the adjusted contour may be detected by solving a least squares problem that minimizes the difference between the adjusted contour and the reference / non-reference contour. Such a problem may be solved using a pseudo inverse matrix method. The implementation of such an algorithm is described in detail in the prior art and its detailed implementation is outside the scope of the present invention and will not be described further here.

3.5.8 Model Vertex Reliability Calculation As an additional optional step of the present invention, model vertex reliability calculation (205, 406) is a reliability value for all optimized vertices in an optimized 3D model of a tubular structure. Is calculated.

  In each scan view, the reliability value calculated by the contour reliability calculation is assigned to all vertices or contours. These values may be incorporated into one confidence value for the corresponding vertex of the optimized 3D model of the tubular structure, as calculated by 3D contour adjustment. This confidence value indicates the overall confidence of the optimization vertex.

  In one embodiment, by summing each confidence value of the corresponding reference vertex with the optimization vertex, the corresponding reference model vertex, and the intersection used in the optimization process weighted by the distance between the intersections, Calculate model vertex reliability.

3.5.9 Optional Step of 3D Contour Adjustment This step is an optional step, and embodiments without this step are also within the scope of the present invention.

  The output of the 3D contour adjustment is an optimized 3D model of the tubular structure as calculated from the individual 3D model of the scan view and the confidence value. Also, after optimization, the quality of this optimized 3D model may be further improved by incorporating additional scan views into this calculation. During the first optimization process, additional scan views may not be available.

  In one embodiment, if a new scan view is added to the data set, the model vertex confidence may be used to re-optimize the 3D model of the tubular structure using similar techniques. The cost function used for 3D contour adjustment (104) may include another cost element called model cost. Calculate the difference between the new vertex and the previously optimized vertex. This cost factor is also incorporated into the weighting factor to adjust the importance of the previously optimized 3D model.

  The total cost function is shown below.

Since the remaining elements other than the model cost Cost _model are used in the first optimization process, the weighting factor may be reduced to avoid overfitting (overlap fitting).

The same method of calculating Cost _vtx , Cost _intsec , Cost _{intra_smooth} , Cost _{inter_smooth} , and Cost _{center_smooth} as described in _3.5.7 may be applied to the above equation. In such a case, Cost _vtx and Cost _intsec may be calculated as additional scan views.

  The model cost may be calculated using the following formula.

Here, i indicates a new vertex index. vtxReliability _i indicates the vertex reliability / intersection vertex reliability of the new scan view. modelReliability _i indicates the reliability of the optimized vertex. newVtx _i indicates a newly sampled vertex. vtx _i indicates the vertex / intersection of the new scan view.

3.5.10 Preferred Embodiments of the Invention This section describes preferred embodiments of the invention. The embodiment described below is an implementation example of the present invention in a system and apparatus having a specific configuration, and the scope of the present invention is not limited by the following description. Embodiments different from those described below are also within the scope of the present invention.

Details are described in the following sections.

3.5.10.1 3D Model Reconstruction without Reliability and Acquisition Location FIG. 10 describes a preferred embodiment of the present invention.
In this embodiment, a short axis image and a long axis image are acquired using an ultrasonic probe without using a means for acquiring the position and orientation of the scan plane. Reliability is not calculated for contours and contour vertices.

Scanning of short-axis image and long-axis image In this embodiment, the short-axis image and the long-axis image may be simultaneously acquired before the 3D model is reconstructed.

  There may be one set of short axis images corresponding to one short axis scan image. Such a short axis image may be acquired by moving an ultrasonic probe arranged perpendicular to the center line from one end of the tubular structure to the other end.

  There may be one or more long-axis image sets corresponding to one or more long-axis scan images. The ultrasonic probe arranged in parallel to the center line is fixed at a specific position and scanning angle, or the ultrasonic probe arranged in parallel to the center line is scanned from one end to the other end of the tubular structure. Thus, such a long axis image may be acquired.

  The optimal position and orientation may be selected so that the long axis scan image passes through the largest cross-sectional area of the tubular structure and the long axis scan image may be acquired from more than one angle. For example, two orthogonal long axis images may be selected as a target plane for acquiring a long axis scan image.

  While the ultrasound probe scans along the tubular structure, the scan view may be acquired so that the position of the tubular structure is kept at the same position in the captured image. In order to surely perform the above photographing, a B-mode image on the ultrasonic scanner may be confirmed in real time during image photographing.

  A scan view may be acquired so that the tubular structure and its boundaries are clearly visible in the scanned image. In order to reliably perform the above photographing, a live B-mode image may be confirmed on an ultrasonic scanner in real time during image photographing.

  Since the reliability value is not calculated, it may be necessary to ensure that the image quality of the captured image is sufficient to extract the contour. The image quality index may be a condition that the lumen region is significantly darker than the surrounding region.

FIG. 7 shows examples of good short-axis scan images and long-axis scan images.

Contour Extraction from Short Axis Image and Long Axis Image Based on the contour extraction in 3.5.4, the short axis contour and the long axis contour may be extracted from the scan images of the short axis scan image and the long axis scan image. In the present embodiment, reliability values are not calculated for the contour and the contour vertex. Therefore, all contours and contour vertices have the same reliability value 1.

Calculation of 3D position of individual contour The 3D position of each contour in the short-axis scan image may be estimated based on the scanning length, the scanning time, the imaging frame rate, and the like. It may be assumed that the probe is moved along a straight line.

  Assuming that the long axis image is acquired in the largest cross-sectional area of the tubular structure, the 3D position of each contour in the long axis scan image may be estimated. Automatic or semi-automatic pattern matching may be performed between the short-axis scan image and the long-axis scan image, and the scan views may be arranged in 3D space.

  In this embodiment, the short-axis scan image may be selected as a reference view, and the long-axis scan image may be selected as a non-reference view. An intersection vertex between the long axis contour and the short axis scan plane may be extracted and used in the three model adjustment step.

Correction of the 3D position of the short axis contour by referencing the long axis contour For each short axis scan plane, by minimizing the cost function as described in 35.7 3D contour adjustment, An optimized contour may be calculated.

  By reproducing the contour vertices in 3D space, an optimized 3D model of the tubular structure may be formed with the optimized contour of the short-axis scan plane.

3.5.10.2 Reconstruction of 3D model with confidence and without acquisition position FIG. 11 describes a preferred embodiment of the present invention. In this embodiment, a short-axis image and a long-axis image are acquired using an ultrasonic probe without using a means for acquiring the position and orientation of the scan plane. The reliability is calculated for the contour and the contour vertex.

Scanning the short axis image and the long axis image In the present embodiment, the short axis image and the long axis image may be simultaneously acquired before the reconstruction of the 3D model.

  There may be one set of short-axis images corresponding to one short-axis scan image. Such a short axis image may be acquired by scanning an ultrasonic probe arranged relatively perpendicular to the center line from one end of the tubular structure to the other end.

  There may be one or more long-axis image sets corresponding to one or more long-axis scan images. By fixing the ultrasonic probe arranged parallel to the center line at a specific position and scan angle, or from one end of the tubular structure to the other end of the ultrasonic probe arranged relatively parallel to the center line Such a long axis image may be obtained by scanning.

  The optimal position and orientation may be selected so that the long axis scan image passes through the largest cross-sectional area of the tubular structure and the long axis scan image may be acquired from more than one angle. For example, two orthogonal long axis images may be selected as a target plane for acquiring a long axis scan image.

  While the ultrasonic probe scans along the tubular structure, the scan view may be acquired so that the position of the tubular structure in the captured image is kept at the same position. In order to reliably perform the above photographing, the B mode image on the ultrasonic scanner may be confirmed in real time during image photographing.

Contour Extraction from Short Axis Image and Long Axis Image Based on the contour extraction in 3.5.4, the short axis contour and the long axis contour may be extracted from the scan images of the short axis scan image and the long axis scan image.

Calculation of 3D position of each contour The 3D position of each contour in the short-axis scan image may be estimated based on the assumption of the scan length, scan time, and imaging frame rate. It may be assumed that the probe is moved along a straight line.

  Assuming that the long axis image is acquired in the largest cross-sectional area of the tubular structure, the 3D position of each contour in the long axis scan image may be estimated. Automatic or semi-automatic pattern matching may be performed between the short-axis scan image and the long-axis scan image, and the scan views may be arranged in 3D space.

  In this embodiment, the short-axis scan image may be selected as a reference view, and the long-axis scan image may be selected as a non-reference view. An intersection vertex between the long axis contour and the short axis scan plane may be extracted and used in the three model adjustment step.

Calculation of reliability of individual contours The reliability of contours or contour vertices may be calculated based on the contour reliability calculation of 3.5.6 and the model vertex reliability calculation of 3.5.8. By setting the threshold value, the extracted low-quality contour or contour vertex portion may be removed using the calculated reliability value. Contours or contour vertices with confidence values below the threshold may be removed from the 3D model reconstruction step.

Correction of the 3D position of the short axis contour by referencing the long axis contour For each short axis scan plane, by minimizing the cost function as described in 35.7 3D contour adjustment, An optimized contour may be calculated. By reproducing the contour vertices in 3D space, an optimized 3D model of the tubular structure may be formed with the optimized contour of the short-axis scan plane.

3.5.10.3 3D Model Reconstruction Using Reliability and Acquisition Location FIG. 12 describes a preferred embodiment of the present invention. In the present embodiment, a short axis image and a long axis image are acquired using an ultrasonic probe using means for acquiring the position and orientation of the scan plane. The reliability is calculated for the contour and the contour vertex.

Scanning of short-axis image and long-axis image having position information In this embodiment, a short-axis image and a long-axis image may be simultaneously acquired before reconstructing the 3D model.

  There may be one set of short-axis images corresponding to one short-axis scan image. Such a short axis image may be acquired by scanning an ultrasonic probe arranged relatively perpendicular to the center line from one end of the tubular structure to the other end.

  There may be one or more long-axis image sets corresponding to one or more long-axis scan images. Fix the ultrasonic probe placed relatively parallel to the center line at a specific position and scan angle, or change the ultrasonic probe placed parallel to the center line from one end of the tubular structure to the other. Such a long axis image may be acquired by scanning to the end.

  The optimal position and orientation may be selected so that the long axis scan image passes through the largest cross-sectional area of the tubular structure and the long axis scan image may be acquired from more than one angle. For example, two orthogonal long axis images may be selected as a target plane for acquiring a long axis scan image.

  Based on the acquisition of position information in 3.5.3, the position information and orientation information of the scan plane may be acquired simultaneously with the acquisition of the scan image.

Contour Extraction from Short Axis Image and Long Axis Image Based on the contour extraction in 3.5.4, the short axis contour and the long axis contour may be extracted from the scan images of the short axis scan image and the long axis scan image.

3D position calculation of individual contours Since 3D position information of the scan plane is available, 3D position information of individual contours and contour vertices may be calculated based on the individual 3D position calculation of 3.5.5. . In this embodiment, the short-axis scan image may be selected as a reference view, and the long-axis scan image may be selected as a non-reference view. An intersection vertex between the long axis contour and the short axis scan plane may be extracted and used in the 3D model adjustment step.

Calculation of reliability of individual contours The reliability of contours and contour vertices may be calculated based on the contour reliability calculation in 3.5.6 and the model vertex reliability calculation in 3.5.8. By setting the threshold value, the extracted low-quality contour or contour vertex portion may be removed using the calculated reliability value. Contours or contour vertices with confidence values below the threshold may be removed from the 3D model reconstruction step.

Correction of the 3D position of the short axis contour by referencing the long axis contour For each short axis scan plane, by minimizing the cost function as described in 35.7 3D contour adjustment, An optimized contour may be calculated. By reproducing the contour vertices in 3D space, an optimized 3D model of the tubular structure may be formed with the optimized contour of the short-axis scan plane.

3.5.10.4 Reconstruction of 3D model using reliability, acquisition position and navigation FIG. 13 describes a preferred embodiment of the present invention. In the present embodiment, a short axis image and a long axis image are acquired using an ultrasonic probe using means for acquiring the position and orientation of the scan plane. The reliability is calculated for the contour and the contour vertex. The navigation step includes displaying a temporary 3D model of the tubular structure and displaying the 3D position of the ultrasound probe relative to the temporary 3D model for the purpose of obtaining further acquired data.

Scanning a short axis image having position information In this embodiment, a short axis image may be acquired first. There may be one set of short-axis images corresponding to one short-axis scan image. Such a short axis image may be acquired by scanning an ultrasonic probe arranged relatively perpendicular to the center line from one end of the tubular structure to the other end. Based on the acquisition of position information in 3.5.3, the position information and orientation information of the scan plane may be acquired simultaneously with the acquisition of the scan image.

Contour Extraction from Short Axis Image Based on the contour extraction in 3.5.4, a short axis contour may be extracted from a scan image of a short axis scan image.

Creation of 3D volume of blood vessel contour extracted from short axis image Using available short axis contours or contour vertices and their position information and orientation information, based on individual 3D position calculation of 3.5.5, A temporary 3D model may be created. A temporary 3D model may be displayed for navigation purposes.

Determination of target position and angle for scanning a long axis image A temporary 3D model of a tubular structure may be used to determine the optimal position and orientation for further taking a scan view such as a long axis scan image. Such an optimal position and orientation can be selected so that the long-axis scan image passes through the largest cross-sectional area of the tubular structure, and the long-axis scan image may be acquired from more than one angle. Also good. For example, two orthogonal long axis images may be selected as a target plane for acquiring a long axis scan image.

  Further, the method for selecting the optimal scan plane as described above may be applied to an embodiment in which the navigation step is not performed.

Display of 3D volume using superimposed target scan plane The position and orientation of the scan plane may be acquired in real time by acquiring the position information of 3.5.3. The selected position and angle may be confirmed by superimposing the scan plane on the temporary 3D model using such information. This step may be called a navigation step.

Scanning a long axis image with position information A long axis image may be acquired at a selected target position and angle using a navigation step. There may be one or more long-axis image sets corresponding to one or more long-axis scan images. Based on the acquisition of position information in 3.5.3, the position information and orientation information of the scan plane may be acquired simultaneously with the acquisition of the scan image.

Contour Extraction from Long Axis Image Based on the contour extraction in 3.5.4, a long axis contour may be extracted from the scan image of the long axis scan image.

3D position calculation of individual contours Since 3D position information of the scan plane is available, 3D position information of individual contours and contour vertices may be calculated based on the individual 3D position calculation of 3.5.5. . In this embodiment, the short-axis scan image may be selected as a reference view, and the long-axis scan image may be selected as a non-reference view. An intersection vertex between the long axis contour and the short axis scan plane may be extracted and used in the 3D model adjustment step.

Calculation of reliability of individual contours The reliability of contours or contour vertices may be calculated based on the contour reliability calculation in 3.5.6 and the model vertex reliability calculation in 3.5.8. By setting the threshold value, the extracted low-quality contour or contour vertex portion may be removed using the calculated reliability value. Contours or contour vertices with confidence values below the threshold may be removed from the 3D model reconstruction step.

Correction of the 3D position of the short axis contour by referencing the long axis contour For each short axis scan plane, by minimizing the cost function as described in 35.7 3D contour adjustment, An optimized contour may be calculated. By reproducing the contour vertices in 3D space, an optimized 3D model of the tubular structure may be formed with the optimized contour of the short-axis scan plane.

3.5.10.5 3D Model Reconstruction Using Reliability, Acquisition Location, Navigation, and Selected Contours FIG. 14 describes a preferred embodiment of the present invention.

  In the present embodiment, a short axis image and a long axis image are acquired using an ultrasonic probe using means for acquiring the position and orientation of the scan plane. The reliability is calculated for the contour and the contour vertex. The navigation step includes displaying a temporary 3D model of the tubular structure and displaying the 3D position of the ultrasound probe relative to the temporary 3D model for the purpose of obtaining further acquired data. The contour selection step relates to the selection of appropriate contour segments for use in adjusting the 3D model.

  For example, 3.5.10.1 (3D model reconstruction without reliability and acquisition position), 3.5.10.2 (3D model reconstruction without reliability and acquisition position), and 3. This selection step may be used, such as in the embodiment described in 5.10.3 (3D model reconstruction using confidence level and acquisition position).

Scanning a short axis image having position information In this embodiment, a short axis image may be acquired first. There may be one set of short-axis images corresponding to one short-axis scan image. Such a short axis image may be acquired by scanning an ultrasonic probe arranged relatively perpendicular to the center line from one end of the tubular structure to the other end. Based on the acquisition of position information in 3.5.3, the position information and orientation information of the scan plane may be acquired simultaneously with the acquisition of the scan image.

Contour Extraction from Short Axis Image Based on the contour extraction in 3.5.4, a short axis contour may be extracted from a scan image of a short axis scan image.

Creation of 3D volume of blood vessel contour extracted from short-axis image Using available short-axis contours or contour vertices and their position information and orientation information, the temporary structure of the tubular structure is calculated based on the individual 3D position calculation of 3.5.5. A 3D model may be created. A temporary 3D model may be displayed for navigation purposes.

Determination of target position and angle for scanning a long-axis image A temporary 3D model of a tubular structure may be used to determine an optimal position and orientation for further acquiring a scan image such as a long-axis scan image. Such an optimal position and orientation can be selected so that the long-axis scan image passes through the largest cross-sectional area of the tubular structure, and the long-axis scan image may be acquired from more than one angle. Also good. For example, two orthogonal long axis images may be selected as a target plane for acquiring a long axis scan image.

  Further, the method for selecting the optimum scan plane as described above may be applied to these embodiments in which the navigation step is not performed.

Display of 3D volume of superimposed target scanning plane Position information and orientation information of the scanning plane may be acquired in real time by acquiring position information of 3.5.3. The selected position and angle may be confirmed by superimposing the scan plane on the temporary 3D model using such information. This step may be referred to as a navigation step.

Scanning a long axis image with position information A long axis image may be acquired at a selected target position and angle using a navigation step. There may be one or more long-axis image sets corresponding to one or more long-axis scan images. Based on the acquisition of position information in 3.5.3, the position information and orientation information of the scan plane may be acquired simultaneously with the acquisition of the scan image.

Contour Extraction from Long Axis Image Based on the contour extraction in 3.5.4, a long axis contour may be extracted from the scan image of the long axis scan image.

Calculation of 3D position of individual contour Since 3D position information of the scan plane can be used, 3D position information of individual contours and contour vertices may be calculated based on the individual 3D position calculation of 3.5.5. In this embodiment, the short-axis scan image may be selected as a reference view, and the long-axis scan image may be selected as a non-reference view. An intersection vertex between the long axis contour and the short axis scan plane may be extracted and used in the 3D model adjustment step.

Selection of an appropriate contour segment After the position of the contour in the 3D space is specified, the type of the contour segment may be confirmed. For example, the carotid artery may be classified into a common carotid artery, an internal carotid artery, and an external carotid artery based on the position and orientation of the contour. The type of carotid artery may be selected as appropriate.

  For example, 3.5.10.1 (3D model reconstruction without reliability and acquisition position), 3.5.10.2 (3D model reconstruction without reliability and acquisition position), and 3. This contour selection step may be used, such as in the embodiment described in 5.10.3 (3D model reconstruction using confidence level and acquisition position).

Calculation of reliability of individual contours The reliability of contours or contour vertices may be calculated based on the contour reliability calculation in 3.5.6 and the model vertex reliability calculation in 3.5.8. By setting the threshold value, the extracted low-quality contour or contour vertex portion may be removed using the calculated reliability value. Contours or contour vertices with confidence values lower than the threshold may be removed from the 3 model reconstruction step.

Correction of the 3D position of the short axis contour by referencing the long axis contour For each short axis scan plane, by minimizing the cost function as described in 35.7 3D contour adjustment, An optimized contour may be calculated. By reproducing the contour vertices in 3D space, an optimized 3D model of the tubular structure may be formed with the optimized contour of the short-axis scan plane.

Claims (28)

A method of reconstructing a 3D model of a subject using a plurality of 2D images,
a. A plurality of 2D images of a plurality of scan views, acquiring a plurality of 2D image is data obtained by scanning the subject,
b. extracting the contour of the subject in the plurality of 2D images acquired in a ;
c. Selecting one or more reference scan views from the plurality of scan views;
Calling the remaining scan views non-reference scan views;
d. Reconstructing the outline of the scan view in 3D space;
e. Calculating a relative position and orientation of the non-reference scan view with respect to the reference scan view;
f. Placing the non-reference scan view based on the calculated relative position and orientation;
g. Modifying the position of the contour of the reference scan view based on the contour of the non-reference scan view in the vicinity thereof.
Method. Further comprising obtaining position information and orientation information of the 2D image,
The method of claim 1. The one or more scan views are short-axis images of the tubular structure;
The method of claim 1. One or more of the scan views is a long-axis image of a tubular structure;
The method of claim 1. Further comprising calculating a reliability of the contour of the scan view.
The method of claim 1. When the tubular structure is a blood vessel, the reliability is calculated from the pulsation phase.
The method of claim 5. Calculating the reliability from the grayscale change of the image at the contour or contour vertex,
The method of claim 5. From the difference in gray scale change of the image near the contour or contour vertex, the reliability is calculated.
The method of claim 5. Such as the standard deviation of the gray scale transition from the consistency of the gray-scale variation for the entire contour, and calculates the reliability,
The method of claim 5. The reliability value of the short axis image is lower than the reliability value of the long axis image.
The method of claim 5. Weighted sum of multiple confidence values,
The method of claim 5. Selecting a contour segment with the highest confidence value based on a predetermined threshold;
The method of claim 5. a. Displaying an outline of the reference view in 3D space on a user interface;
b. Selecting a desired scan position and orientation from the display;
c. Superimposing the current position and orientation of the ultrasound probe in 3D space on the user interface;
d. Obtaining an image of the non-reference scan view at the desired scan position and orientation.
The method of claim 1. a. Calculating vertex reliability for all vertices of the modified contour;
b. Wherein the stearyl-up to get the object of the scanned image of the one or more additional views,
c. Extracting the contour of the subject in the additional view;
d. Calculating contour reliability using the image and contour of the one or more additional views;
e. Further modifying the reference view contour based on an additional view contour near the reference view contour;
The method of claim 1. The reference scan view is the short-axis scan image of a tubular structure;
The method of claim 1. The non-reference scan view is the long-axis scan image of a tubular structure;
The method of claim 1. The calculating step of calculating the relative position and orientation of the non-reference scan view with respect to the reference scan view further includes a best match (optimal combination) between the contour of the non-reference view and the shape of the 3D model of the reference view in 3D space. )
The method of claim 1. The correcting step of correcting the contour position of the reference scan view adjusts the position of the contour of the reference scan view, and the position of the contour of the reference scan view and the position of the contour of the non-reference scan view nearby. Reduce the difference between
The method of claim 1. The cost function is optimized to correct the position of the contour of the reference scan view, and the cost function includes the following elements:
a. The distance from the new contour to the contour of the reference scan view,
b. The distance from the new contour to the contour of the unreferenced scan view,
c. The difference in radius between adjacent contour points of the new contour,
d. The distance between the new contour and the corresponding contour of the adjacent scan plane of the reference view;
e. The method of claim 1, comprising at least all or a subset of a distance between a new center point of a contour and a center point of an adjacent contour of the adjacent scan plane of the reference view. Multiplying each of the elements by a weight factor to determine the importance of each of the elements;
The method of claim 19. Weight each of the elements with the confidence level of the corresponding contour of the corresponding scan view.
The method of claim 19. Calculating the position of the modified contour by averaging the coordinates of the corresponding contour of the reference scan view and the non-reference scan view;
The method of claim 1. Weighting the coordinates of the corresponding contour by corresponding reliability values;
The method of claim 22. An apparatus for reconstructing a 3D model of a subject using a plurality of 2D images,
a. A plurality of 2D images of a plurality of scan views, means for obtaining a plurality of 2D image is data obtained by scanning the subject,
b. means for extracting contours of the subject in the plurality of 2D images acquired in a ;
c. Selecting one or more reference scan views from the plurality of scan views;
The remaining scan views are referred to as non-reference scan views;
d. Means for reconstructing the outline of the scan view in 3D space;
e. Means for calculating a relative position and orientation of the non-reference scan view with respect to the reference scan view;
f. Means for arranging the non-reference scan view based on the calculated relative position and orientation;
g. Means for correcting the position of the contour of the reference scan view based on the contour of the non-reference scan view in the vicinity thereof. Means for obtaining position information and orientation information of the 2D image;
25. The device according to claim 24. Means for calculating a reliability of the contour of the scan view;
25. The device according to claim 24. a. Means for displaying a contour of the reference view in 3D space on a user interface;
b. Means for selecting a desired scan position and orientation from the display;
c. Means for superimposing on the user interface the current position and orientation of the ultrasound probe in 3D space;
d. Means for acquiring an image of the non-reference scan view at the desired scan position and orientation;
25. The device according to claim 24. a. Means for calculating vertex reliability for all vertices of the modified contour;
b. Means for obtaining a scanned image of the subject in one or more additional views;
c. Means for extracting the contour of the subject of the additional view;
d. Means for calculating contour reliability using images and contours of the one or more additional views;
e. Means for further modifying the contour of the reference view based on the contour of the additional view near the contour of the reference view.
25. The device according to claim 24.