Abstract
Microscopy image analysis often requires the segmentation of objects, but training data for this task is typically scarce and hard to obtain. Here we propose DenoiSeg, a new method that can be trained end-to-end on only a few annotated ground truth segmentations. We achieve this by extending Noise2Void, a self-supervised denoising scheme that can be trained on noisy images alone, to also predict dense 3-class segmentations. The reason for the success of our method is that segmentation can profit from denoising, especially when performed jointly within the same network. The network becomes a denoising expert by seeing all available raw data, while co-learning to segment, even if only a few segmentation labels are available. This hypothesis is additionally fueled by our observation that the best segmentation results on high quality (very low noise) raw data are obtained when moderate amounts of synthetic noise are added. This renders the denoising-task non-trivial and unleashes the desired co-learning effect. We believe that DenoiSeg offers a viable way to circumvent the tremendous hunger for high quality training data and effectively enables learning of dense segmentations when only very limited amounts of segmentation labels are available.
T.-O. Buchholz and M. Prakash—Equal contribution (alphabetical order).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Batson, J., Royer, L.: Noise2Self: blind denoising by self-supervision (2019)
Buchholz, T.O., Jordan, M., Pigino, G., Jug, F.: Cryo-CARE: Content-aware image restoration for cryo-transmission electron microscopy data. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 502–506. IEEE (2019)
Caicedo, J.C., et al.: Evaluation of deep learning strategies for nucleus segmentation in fluorescence images. Cytometry Part A 95(9), 952–965 (2019)
Chen, H., Qi, X., Yu, L., Heng, P.A.: DCAN: deep contour-aware networks for accurate gland segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2487–2496 (2016)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)
Guerrero-Pena, F.A., Fernandez, P.D.M., Ren, T.I., Yui, M., Rothenberg, E., Cunha, A.: Multiclass weighted loss for instance segmentation of cluttered cells. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 2451–2455. IEEE (2018)
Hirsch, P., Mais, L., Kainmueller, D.: Patchperpix for instance segmentation. arXiv preprint arXiv:2001.07626 (2020)
Ihle, S.J., et al.: Unsupervised data to content transformation with histogram-matching cycle-consistent generative adversarial networks. Nat. Mach. Intell. 1(10), 461–470 (2019)
Jug, F., Pietzsch, T., Preibisch, S., Tomancak, P.: Bioimage informatics in the context of drosophila research. Methods 68(1), 60–73 (2014)
Krull, A., Buchholz, T.O., Jug, F.: Noise2Void-learning denoising from single noisy images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2129–2137 (2019)
Krull, A., Vicar, T., Jug, F.: Probabilistic Noise2Void: unsupervised content-aware denoising (2019). https://www.frontiersin.org/articles/10.3389/fcomp.2020.00005/full
Lehtinen, J., et al.: Noise2Noise: learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018)
Liu, D., Wen, B., Liu, X., Wang, Z., Huang, T.S.: When image denoising meets high-level vision tasks: a deep learning approach. arXiv preprint arXiv:1706.04284 (2017)
Moen, E., Bannon, D., Kudo, T., Graf, W., Covert, M., Van Valen, D.: Deep learning for cellular image analysis. Nat. Methods 16, 1–14 (2019)
Osokin, A., Chessel, A., Carazo Salas, R.E., Vaggi, F.: GANs for biological image synthesis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2233–2242 (2017)
Prakash, M., Buchholz, T.O., Lalit, M., Tomancak, P., Jug, F., Krull, A.: Leveraging self-supervised denoising for image segmentation (2019)
Prakash, M., Lalit, M., Tomancak, P., Krull, A., Jug, F.: Fully unsupervised probabilistic Noise2Void (2019). https://ieeexplore.ieee.org/document/9098612
Razzak, M.I., Naz, S., Zaib, A.: Deep learning for medical image processing: overview, challenges and the future. In: Dey, N., Ashour, A.S., Borra, S. (eds.) Classification in BioApps. LNCVB, vol. 26, pp. 323–350. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-65981-7_12
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Sandfort, V., Yan, K., Pickhardt, P.J., Summers, R.M.: Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks. Sci. rep. 9(1), 1–9 (2019)
Schindelin, J., et al.: Fiji: an open-source platform for biological-image analysis. Nat. Methods 9(7), 676–682 (2012)
Schmidt, U., Weigert, M., Broaddus, C., Myers, G.: Cell detection with star-convex polygons. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 265–273. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_30
Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 60 (2019)
Stringer, C., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. bioRxiv (2020)
Ulman, V., et al.: An objective comparison of cell-tracking algorithms. Nat. Methods 14(12), 1141 (2017)
Wang, S., Wen, B., Wu, J., Tao, D., Wang, Z.: Segmentation-aware image denoising without knowing true segmentation. arXiv preprint arXiv:1905.08965 (2019)
Weigert, M., et al.: Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15(12), 1090–1097 (2018)
Zhao, A., Balakrishnan, G., Durand, F., Guttag, J.V., Dalca, A.V.: Data augmentation using learned transformations for one-shot medical image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8543–8553 (2019)
Zheng, H., et al.: Biomedical image segmentation via representative annotation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5901–5908 (2019)
Acknowledgment
The authors would like to acknowledge Romina Piscitello and Suzanne Eaton from MPI-CBG for fly wing data and Diana Afonso and Jacqueline Tabler from MPI-CBG for mouse nuclei data. We also acknowledge the Scientific Computing Facility at MPI-CBG for giving us access to their HPC cluster.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendices
A DSB Results with Increasingly Many GT labels
The DSB dataset we have used offers the possibility to run DenoiSeg with arbitrary many segmentation GT labels. While DenoiSeg is intended in cases where the amount of such labels is very limited, in Fig. 8 we plot the segmentation results of DenoiSeg, the sequential baseline, as well as the baseline as defined in the main text.
Extended version of Fig. 4. Results for DSB n0, n10 and n20, evaluated with Average Precision (AP) [22] and SEG-Score [25]. DenoiSeg outperforms both baseline methods, mainly when only limited segmentation ground truth is available. Note that the advantage of our proposed method for this dataset is at least partially compromised when the image data is not noisy (row 3).
As expected, With additional labels, the advantage of also seeing noisy images decreases, leading to similarly good results for all compared methods. It is still reassuring to see that the performance of DenoiSeg is still essentially on par with the results of a vanilla U-Net that does not perform the joint training we propose.
B Our Baseline vs Vanilla 3-class U-Net
The baseline method we used in this work is, as explained in the main text, a DenoiSeg network with \(\alpha \) being set to 0. This is, in fact, very similar to using a vanilla 3-class U-Net. While we are still feeding noisy images, we are not backpropagating any denoising loss, meaning that only the data for which segmentation labels exist will contribute to the training. The one difference is, that some of the hyperparameters (number of epochs, adaptation of learning rate, etc.) will slightly diverge in these two baseline setups. Figure 9 shows that these subtle differences are in fact not making any practical differences.
Comparison of vanilla U-Net with our DenoiSeg \(\alpha =0\) baseline for DSB datasets. Our DenoiSeg \(\alpha =0\) baseline is at least as good or better than the vanilla U-net baseline both in terms of Average Precision (AP) [22] and SEG-Score [25] metrics. Hence, we establish a stronger baseline with DenoiSeg \(\alpha =0\) and measure our performance against this baseline (see Fig. 8, Fig. 3 and Fig. 5.)
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Buchholz, TO., Prakash, M., Schmidt, D., Krull, A., Jug, F. (2020). DenoiSeg: Joint Denoising and Segmentation. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12535. Springer, Cham. https://doi.org/10.1007/978-3-030-66415-2_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-66415-2_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66414-5
Online ISBN: 978-3-030-66415-2
eBook Packages: Computer ScienceComputer Science (R0)