Fetal Ultrasound Image Segmentation for Automatic Head Circumference Biometry Using Deeply Supervised Attention-Gated V-Net

Deeply Supervised V-Net

To design an end-to-end segmentation method for fetal ultrasound images, we were inspired by the V-Net architecture [15]. The V-Net model is an excellent three-dimensional CNN, which is commonly used in biomedical image processing. The high-resolution features of the encoder in V-Net can provide accurate localization of the target. The skip connection structure connects the high-resolution features with the output features sampled on the decoder, making the final prediction results more accurate; in particular, V-Net can overcome the serious imbalance between the number of foreground pixels and background pixels [15]. On this basis, we improved the architecture, dimensions, and details of V-Net, by incorporating attention modules and deeply supervised strategy [19] with hybrid loss function. Figure 3 shows the architecture of the proposed DAG V-Net, consisting of an encoder on the left and a decoder on the right.

Hybrid Loss Function

Dice [15] was selected as the loss function for training. Dice is a common loss function and quality evaluation index in the field of image segmentation. Dice and Dice similarity coefficient (DSC) are defined as:

where $Are{a}_G$ is the region manually annotated by radiologists as the gold standard, and $Are{a}_P$ is the segmented region predicted by the network model.

Refer to Fig. 3. The output of our proposed network had five different scales of segmentation results. We calculated the Dices of $los{s}_1$, $los{s}_2$, $los{s}_3$, and $los{s}_4$ from the first four scales and added them as $mix\_loss$. In order to control the influence of these four Dices on the final segmentation results, a weight factor $\alpha$ (with an initial value of 1) was multiplied to control the proportion:

Our experiments showed that it was best to decay $\alpha$ by 0.9 times for each epoch. As the training progressed, the weight of the low-level feature contribution decreased. It should be noted that the low-level features were not directly output, but they were restored through the up-sampling process. Finally, the Dice calculated by the final mixed output layer, $loss\_all$, was defined as:

Note that we did not use the loss coefficient as the index to evaluate the segmentation performance of our method, because the loss function of the proposed DAG V-Net was calculated by the sum of the losses of all five scales, totally different from the loss function defined in other related networks.

Attention-Gated Module

The attention mechanism was initially applied in the field of natural language processing [24] and is now widely used in a variety of tasks, including image segmentation. Rich spatial information is the key for semantic segmentation. U-Net has achieved good results in the field of medical image segmentation, but the skip connection module in the encoder and decoder of U-Net tends to use unnecessary information and computational sources, where likable low-level encoder features are repeatedly used at multiple scales [39], which may affect the further improvement of image detail segmentation. For fetal ultrasound image segmentation, the shallow feature map not only contains the details of the fetal head but also includes the non-fetal-head region. The deep feature map can capture the highly semantic information to describe the location, but it may lose the details of the head boundary. Hence, there is a need for a method to guide the network to carry out accurate localization learning.

In order to improve the accuracy of segmentation, some current segmentation frameworks [25, 26] added object localization models in feature extraction, which undoubtedly complicated the overall model and consumed a lot of computing resources. Inspired by the work of Schlemper et al. [27] and Zhang et al. [28], we added attention-gated modules to the segmentation model for training (Fig. 5), which suppressed the features irrelevant to learning and tasks, while strengthening the features related to learning and tasks. In order to refine the features of each layer, we introduced the deep attention module. The proposed DAG V-Net used attention blocks instead of skip connection modules to capture low-level spatial and high-level context features. Through filtering on low-resolution and high-resolution feature maps, spatial information was recovered, and structure information was fused from different resolution levels, while the feature response of unrelated background regions was gradually suppressed. Attention blocks further extended the network performance by fusion with the deeply supervised V-Net to guide the network in learning the most significant features.

Attention modules were very important in our method, which made the whole network performance further optimized. In particular, the attention block was used to highlight the foreground and reduce the influence of background pixels, which could effectively solve the problem that when the area of the structure similar to the fetal head in the image, such as the interface of the uterine wall and amniotic fluid, was very large, some regions outside the head were mistakenly classified as fetal heads. Refer to Fig. 5. Let us take the first attention block as an example. Specifically, it consists of the following six steps.

Step 1. The attention block had two input gates: one was the gating signal g in the low-resolution layer, and the other was the input signal x in the high-resolution layer. Before inputted to the attention module, the signal x underwent channel transform to obtain the same feature channel as the signal g, and then the feature map x' was obtained.

Step 2. Feature mapping of x' and g was conducted, in order to obtain the same size as the input signal x. The linear transformation with a 1 × 1 convolution kernel was conducted. The strides of the gating signal g were set at 2. Then, the feature map g' was obtained.

Step 3. The residual connection $\oplus$ between the feature maps x' and g' was conducted, which was called the attention based on vector connection [29], where the connected features were linearly mapped to a potential space. Then, the activated feature map $ad{d}_{\mathit{xg}}$ was obtained by ReLU on the connected feature:

where ${\sigma }_1(x)=max(0,x)$; $W_x^T$, $W_g^T$, and ${\psi }^T$ are convolution weights; $b_x$, $b_g$, and $b_{\psi }$ are bias terms.

Step 4. The feature map $ad{d}_{\mathit{xg}}$ was convolved with a 1 × 1 × 1 convolution kernel. The purpose was to reduce the depth of these combined feature maps to 1. Then, the attention coefficient was normalized by the sigmoid function ${\sigma }_2(x)$. In this way, each pixel value of the attention map was rescaled to [0, 1], and the attention coefficient ${\alpha }_{\mathit{psi}}$ was obtained:

where $W_{\mathit{psi}}^T$ is a convolution weight and $b_{\mathit{psi}}$ is a bias term. Next, ${\alpha }_{\mathit{psi}}$ was restored to the original size by a factor of 2 to obtain the attention coefficient ${\alpha }_{\mathit{psi}}^{{}^{\prime }}$.

Step 5. The attention coefficient ${\alpha }_{\mathit{psi}}^{{}^{\prime }}$ was multiplied by the matrix of the gating signal g to recalibrate the relative activation of each feature. The channel was mapped back to the original size by the 1 × 1 convolution kernel, followed by the GN with 16 channels, further accelerating the training convergence to obtain the attention-guided feature map $ga{t}_{\mathit{out}}$:

where $W_{\mathit{gat}}^T$ is a convolution weight and $b_{\mathit{gat}}$ is a bias term.

Step 6. The last step was to connect the result of the previous steps with the original input signal x. Before that, the input signal needed to be deconvolved to improve the resolution to obtain the signal x″, where a 3 × 3 convolution kernel was used.

It is worth noting that our attention modules are very simple and can be directly inserted into the pipeline of similar encoder structures. They do not add too many parameters and can encode abundant low-level spatial information at low computational cost.

Ellipse Fitting

In clinical diagnosis, the measurement plane is the thalamus plane (the same as the measurement plane of BPD) when the fetal head is scanned. There are two commonly used estimation methods for fetal HC, as shown in Fig. 6. Method 1 (Fig. 6a) is based on occipitofrontal diameter (OFD) and BPD:

Method 1 has few measurement parameters and is easy to calculate. It is more accurate in the 12th to 28th week of pregnancy. However, in the early stage of pregnancy, there will be large deviation due to factors such as irregular shapes of gestational sacs and blurred skulls around the head. Therefore, at present, the ellipse fitting method, i.e., method 2 (Fig. 6b), is an important method for clinical measurement of fetal HC. The fitting position and size were determined according to the long axis (semi_axis_a), short axis (semi_axis_b), ellipse center point (center_x, center_y), and central angle (angle) of the fitted ellipse, as denoted in Fig. 6b. In method 2, HC was calculated by

In this paper, method 2 was used for HC biometric measurement.

Dataset

The dataset used in this work is the open-access HC18 Challenge dataset collected from the obstetric database of Radboud University Nijmegen Medical Center, the Netherlands, without any abnormal growth. The ultrasound images were acquired from 551 pregnant women who received a routine ultrasound screening exam between May 2014 and May 2015. Images were acquired by experienced sonographers using either the Voluson E8 or the Voluson 730 ultrasound device (General Electric, Austria) [12]. As shown in Table 1, there were 1354 two-dimensional ultrasound images collected from 551 pregnant women at different semesters of pregnancy. There were 999 images in the training set and 355 images in the testing set. The training set also contained manually annotated HC images by experienced radiologists, which were used as gold standard for image segmentation. However, the testing set did not provide gold standard for segmentation. Each image was 800 × 540 pixels in size, and the pixel resolution was from 0.052 to 0.326 mm. This big change in pixel resolution resulted from adjusting the ultrasound scanners (depth setting and zoom) according to different fetal sizes. For the HC18 Challenge (https://hc18.grand-challenge.org/), the results of the participant’s algorithm on the testing set should be uploaded to HC18 to obtain the final performance.

Augmentation

The number of images in the training set of HC18 (n = 999) is far from satisfying a deep network model for learning. In order to improve the network robustness, prevent the overfitting of training data, and improve the generalization ability, we expanded the dataset by data augmentation (Fig. 2). Especially in the process of ultrasound fetal scanning, the size, shape, and location of fetuses have significant differences among individuals. Data enhancement can help to introduce more such training data diversity and balanced samples. We used rotation and flip transforms to generate 20 images for each image in the training set. For data enhancement, we transformed each image by flipping the image horizontally and vertically, using a fixed rotation angle from − 20° to 20°. The scaling ratio of the image was [0.85, 1.15]. We generated a total of 19,980 enhanced images for the augmented set. The images in the augmented set were randomly divided into 80% training dataset and 20% verification dataset. Note that we did nothing on the testing set provided by HC18, all of which was directly used to test the performance of the deep learning model.

Training

Our experiments were performed on a graphics workstation, with Intel (R) Xeon (R) CPU e5-2620 V4 @ 2.10 GHz, NVIDIA GeForce GTX 1080ti 11G, and 64G RAM. The popular Tensorflow and Keras were selected for the deep learning framework. We tried several models to test the effectiveness of the attention module and the deep supervision (DS) strategy respectively, namely, U-Net, V-Net, Attention V-Net, DS U-Net, DS V-Net, and DAG V-Net. Some key hyperparameter settings have been described previously in this paper. Then, we initialized the network parameters and trained 20 epochs each time. The training time of each epoch was about 85 min, and the total training time was about 30 h. The input size of model training was 768 × 512, and we found that the precision of training at this scale would be higher. The optimizer used Adam et al.’s method [30]. The initial learning rate was 0.001, the batch size was 2, and the dropout ratio in all steps was 0.5.

Evaluation

In order to analyze the model quantitatively and evaluate the performance of the model, we used indices including DSC, Hausdorff distance (HD), HC difference (DF), and HC absolute difference (ADF) as the performance indices of the model. DF and ADF are defined as:

where $H{C}_G$ is the gold standard HC, and $H{C}_P$ is the HC predicted by deep learning network models.

HD is defined by:

where

where $P=\{p_1,p_2,...,p_n\}$ represents the pixels in the segmentation results, and $G=\{g_1,g_2,...,g_n\}$ represents the pixels from the gold standard.

Significance of this study

In this paper, a new improved deep learning method was proposed for automatic fetal ultrasound image segmentation and HC biometric measurement. The advanced three-dimensional V-Net was transformed into two-dimensional network models, which then incorporated the attention-gated module. The attention mechanism adaptively integrated local features and their global dependencies by capturing rich context information. At the same time, multi-scale loss function was introduced to carry out deep supervision, so as to effectively address the problems of blurred boundary and edge missing by using multi-level features to refine the segmentation. The problems including head boundary missing and interference from structures similar to fetal head (e.g., the interface of the uterine wall and amniotic fluid was very large) were effectively solved by the proposed DAG V-Net method. As a result, the fetal head was accurately segmented by the proposed method, and the segmentation and biometry performance was improved. For the HC18 Challenge, the DSC by DAG V-Net on the training set was 98.29%; on the testing set, the DSC was 97.93%, and the AD was 1.77 ± 1.69 mm. Currently, the proposed DAG V-Net method ranks fifth among the participants in the HC18 Challenge.

Contribution of Residual Mechanism

Compared with U-Net, V-Net showed better segmentation performance (Table 2), which is largely due to the residual mechanism introduced in convolutional blocks of each layer. The residual mechanism can make the residual information have semantic significance in the shallow and deep layers of the network. It can learn from the edge (shallow layer) to the very complex features (deep layer) in different abstract levels, optimizing the gradient derivation process of reverse training.

Contribution of Attention Mechanism

When combining attention mechanism with the V-Net model, the convergence speed of the resulted network (Attention V-Net) was significantly improved; the DSC on the testing set was 97.91 ± 1.24%, and the AD was reduced to 1.85 ± 1.71 mm, with improved performance in comparison with V-Net. Conventional U-Net and V-Net models were not able to segment the images more accurately and effectively, especially for the problem of blurred boundary, resulting in a large error in pixel classification, which greatly affected the quality of subsequent HC ellipse fitting. However, after model fusion, the attention mechanism can capture the specific position of the fetal head and can further guide the model to learn the edge details with more attention resources given. Local features with their global dependencies are adaptively integrated, so as to guide the network to learn how to use the multi-level features to refine the segmentation, which is the key for learning. The attention module can effectively solve the problem that the skip connection module in the U-Net encoder architecture cannot effectively use the structure information, which may affect the image segmentation performance.

Contribution of Deep Supervision

After incorporation of deep supervision, the convergence speed and segmentation accuracy were improved by the proposed DAG V-Net model. The deep supervision strategy can accelerate the training convergence of fetal ultrasound image segmentation based on deep learning, with a high accuracy. We introduced the multi-scale loss function for deep supervision, making full use of the complementary information encoded by each layer of CNNs. Multi-level features were fused to refine the features of each individual layer, gradually suppressing the non-fetal noise in the shallow layer of CNNs, while highlighting the foreground information and putting more fetal details into the deep features. As a result, it is possible to recover spatial information and fuse structure information from different resolution levels.

Comparison Between Different Trimesters of Pregnancy

The fetal skull is not fully developed in the first trimester of pregnancy, so it is challenging for fetal head segmentation from ultrasound images in this trimester. The proposed method achieved generally better segmentation performance for the second and third trimesters of pregnancy. From experiments, we found two challenges for the proposed method. First, the DSC of the proposed method in the early pregnancy was relatively lower, as shown in Fig. 10a. From the training set images, we found that part of the fetal skull in the early pregnancy has not developed, and the skull contour is small, with noise which may affect image segmentation performance. The second was that the AD was generally larger in the third trimester of pregnancy, as shown in Fig. 10b. We considered two possible reasons. (i) Because the fetal skull grows increasingly at late pregnancy, it may be difficult to capture a complete structure on an ultrasound imaging plane, resulting in partial loss of the skull at the two ends, so the fitted ellipse may be significantly larger. (ii) The edge structure of fetal skull is obviously thicker at the third semester, meaning that there are more, brighter pixels at the skull with high echogenicity. Some of these pixels on the thick skull may cause misclassification, as the computer-detected fetal head boundary is a thinner ellipse curve (Fig. 10b).

Comparison with Published State-of-the-Art Methods

Table 4 compares the proposed method with published state-of-the-art methods for fetal head segmentation and HC biometry from ultrasound images. The testing ultrasound images can be classified into 3 groups. One is the testing set provided by the HC18 Challenge [12], containing 355 testing ultrasound images; the data distribution over different semesters of pregnancy is uniform. The second is a dataset of International Symposium on Biomedical Imaging (ISBI) 2012 [4], containing 90 testing ultrasound images; the data distribution over different semesters of pregnancy is less uniform. Note that ISBI 2012 is a subset of HC18. The third is the private dataset. Compared with published state-of-the-art methods, the proposed method yielded better performance in DF and AD. For DSC, Liu et al. [32] had the highest mean DSC (98.05 mm), but their standard deviation of DSC is large (4.02 mm); our method (97.93 ± 1.25 mm) is comparable with their method in DSC. For HD, both Liu et al. [32] and our method had a lowest mean HD (1.27 mm), while Liu et al. [32] had a slightly smaller standard deviation of HD (0.77 mm vs. 0.80 mm). Overall, our method is better than or comparable with published state-of-the-art methods.

However, the algorithm execution time was rarely described in the above-published literature. We compared the proposed method with some state-of-the-art methods (U-Net [14], SCSE U-Net [40], DeepLabV3 + [41], PSPNet [42]) using the same computer and image datasets, in terms of mean AD, DSC, and algorithm execution time. These comparisons are shown in Table 5. It can be seen that the proposed method outperforms these methods with respect to AD and DSC, while the mean algorithm execution time of the proposed method does not increase much as compared with U-Net [14] (6.3 vs. 5.3 ms).

Potential Applications of the Proposed Method

Fetal sonography is a relatively low-cost examination and can be taught to those whose specialty is not fetal imaging. This would make the proposed automated HC measurement method more valuable for “medically underserved” areas where ultrasound expertise may be lacking or in short supply. Use in point-of-care situations is another potential application of the proposed method in which a person who is not a trained ultrasound technologist can get this necessary measurement in a bedside or remote location.

Limitations and Future Work

This study has some limitations. Firstly, the number of cases is limited. Secondly, the segmentation accuracy for the first trimester of pregnancy is lower. For the third trimester of pregnancy, the AD is larger. These limitations may be overcome in future work.

Conflict of Interest

The authors declare that they have no competing interests.

Ethical Approval

This retrospective study used fetal ultrasound images provided by the HC18 Challenge, which were collected from the database of the Department of Obstetrics of the Radboud University Medical Center, Nijmegen, the Netherlands, in accordance with the local ethics committee (CMO Arnhem-Nijmegen). All data were anonymized according to the tenets of the Declaration of Helsinki.

Informed Consent

This study used retrospective data provided by the HC18 Challenge, so informed consent was waived.