[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

Non-Expert Markings of Active Chronic Graft-Versus-Host Disease Photographs: Optimal Metrics of Training Effects

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Lack of reliable measures of cutaneous chronic graft-versus-host disease (cGVHD) remains a significant challenge. Non-expert assistance in marking photographs of active disease could aid the development of automated segmentation algorithms, but validated metrics to evaluate training effects are lacking. We studied absolute and relative error of marked body surface area (BSA), redness, and the Dice index as potential metrics of non-expert improvement. Three non-experts underwent an extensive training program led by a board-certified dermatologist to mark cGVHD in photographs. At the end of the 4-month training, the dermatologist confirmed that each trainee had learned to accurately mark cGVHD. The trainees’ inter- and intra-rater intraclass correlation coefficient estimates were “substantial” to “almost perfect” for both BSA and total redness. For fifteen 3D photos of patients with cGVHD, the trainees’ median absolute (relative) BSA error compared to expert marking dropped from 20 cm2 (29%) pre-training to 14 cm2 (24%) post-training. Total redness error decreased from 122 a*·cm2 (26%) to 95 a*·cm2 (21%). By contrast, median Dice index did not reflect improvement (0.76 to 0.75). Both absolute and relative BSA and redness errors similarly and stably reflected improvements from this training program, which the Dice index failed to capture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Lee SJ, Klein JP, Barrett AJ, Ringden O, Antin JH, Cahn JY, Carabasi MH, Gale RP, Giralt S, Hale GA, Ilhan O, McCarthy PL, Socie G, Verdonck LF, Weisdorf DJ, Horowitz MM. Severity of chronic graft-versus-host disease: Association with treatment-related mortality and relapse. Blood. 2002;100(2):406–14.

    Article  CAS  PubMed  Google Scholar 

  2. Gandelman JS, Zic J, Dewan AK, Lee SJ, Flowers M, Cutler C, Pidala J, Chen H, Jagasia MH, Tkaczyk ER. The Anatomic Distribution of Skin Involvement in Patients with Incident Chronic Graft-versus-Host Disease. Biol Blood Marrow Transplant. 2019 Feb 1;25(2):279–86.

    Article  PubMed  Google Scholar 

  3. Jacobsohn DA, Kurland BF, Pidala J, Inamoto Y, Chai X, Palmer JM, Arai S, Arora M, Jagasia M, Cutler C, Weisdorf D, Martin PJ, Pavletic SZ, Vogelsang G, Lee SJ, Flowers MED. Correlation between NIH composite skin score, patient-reported skin score, and outcome: Results from the chronic GVHD Consortium. Blood. 2012 Sep 27 [cited 2020 Jul 6];120(13):2545–52.

  4. Baker LX, Byrne M, Martin PJ, Lee S, Chen H, Jagasia M, Tkaczyk E. Association of skin response in erythema and sclerosis with survival in chronic graft-versus-host disease [abstract]. J Invest Dermatol. 2020;140(7):S57.

    Article  Google Scholar 

  5. Curtis LM, Grkovic L, Mitchell SA, Steinberg SM, Cowen EW, Datiles MB, Mays J, Bassim C, Joe G, Comis LE, Berger A, Avila D, Taylor T, Pulanic D, Cole K, Baruffaldi J, Fowler DH, Gress RE, Pavletic SZ. NIH response criteria measures are associated with important parameters of disease severity in patients with chronic GVHD. Bone Marrow Transplant. 2014;49(12):1513–20.

  6. Mitchell SA, Jacobsohn D, Thormann Powers KE, Carpenter PA, Flowers MED, Cowen EW, Schubert M, Turner ML, Lee SJ, Martin P, Bishop MR, Baird K, Bolaños-Meade J, Boyd K, Fall-Dickson JM, Gerber LH, Guadagnini JP, Imanguli M, Krumlauf MC, Lawley L, Li L, Reeve BB, Clayton JA, Vogelsang GB, Pavletic SZ. A multicenter pilot evaluation of the national institutes of health chronic graft-versus-host disease (cGVHD) therapeutic response measures: Feasibility, interrater reliability, and minimum detectable change. Biol Blood Marrow Transplant. 2011 Nov 1;17(11):1619–29.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159.

    Article  CAS  PubMed  Google Scholar 

  8. Wang J, Chen F, Dellalana LE, Jagasia MH, Dawant BM, Tkaczyk ER. Segmentation of skin lesions in chronic graft versus host disease photographs with fully convolutional networks. In: 10575 Medical Imaging 2018: Computer-Aided Diagnosis. 2018. p. 105750N-1–105750N – 7.

  9. Liu X, Parks K, Saknite I, Reasat T, Cronin AD, Wheless LE, Dawant BM, Tkaczyk ER. Baseline Photos and Confident Annotation Improve Automated Detection of Cutaneous Graft-Versus-Host Disease. Clin Hematol Int. 2021;3(3):108–15.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Esteva, Andre, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, no. 7639 (2017): 115-118.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Tschandl P, Codella N, Akay BN, Argenziano G, Braun RP, Cabo H, Gutman D, Halpern A, Helba B, Hofmann-Wellenhof R, Lallas A, Lapins J, Longo C, Malvehy J, Marchetti MA, Marghoob A, Menzies S, Oakley A, Paoli J, Puig S, Rinner C, Rosendahl C, Scope A, Sinz C, Soyer HP, Thomas L, Zalaudek I, Kittler H. Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study. Lancet Oncol. 2019 Jul;20(7):938–947.

    Article  PubMed  PubMed Central  Google Scholar 

  12. McNeil A, Parks K, Liu X, Saknite I, Chen F, Reasat T, Cronin A, Wheless L, Dawant BM, Tkaczyk ER. Artificial intelligence recognition of cutaneous chronic graft-versus-host disease by a deep learning neural network. Br J Haematol. 2022;74(March):1–4.

    Google Scholar 

  13. Tkaczyk ER, Chen F, Wang J, Gandelman JS, Saknite I, Dellalana LE, Jagasia MH, Dawant BM. Overcoming human disagreement assessing erythematous lesion severity on 3D photos of chronic graft-versus-host disease. Bone Marrow Transplantation. 2018;53:1356–1358.

  14. Cotliar JA. Clinical Presentation of Acute Cutaneous Graft-Versus-Host Disease. In: Atlas of Graft-versus-Host Disease. Springer; 2017. p. 21–8.

  15. Carpenter PA. How I conduct a comprehensive chronic graft-versus-host disease assessment. Blood. 2011;118(10):2679–87.

    Article  CAS  PubMed  Google Scholar 

  16. Lee SJ, Cook EF, Soiffer R, Antin JH. Development and validation of a scale to measure symptoms of chronic graft-versus-host disease. Biol Blood Marrow Transplant. 2002;8(8):444–52.

    Article  PubMed  Google Scholar 

  17. Lee SJ, Wolff D, Kitko C, Koreth J, Inamoto Y, Jagasia M, Pidala J, Olivieri A, Martin PJ, Przepiorka D, Pusic I, Dignan F, Mitchell SA, Lawitschka A, Jacobsohn D, Hall AM, Flowers MED, Schultz KR, Vogelsang G, Pavletic S. Measuring Therapeutic Response in Chronic Graft-versus-Host Disease. National Institutes of Health Consensus Development Project on Criteria for Clinical Trials in Chronic Graft-versus-Host Disease: IV. The 2014 Response Criteria Working Group Report. Biol Blood Marrow Transplant. 2015;21(6):984–99.

  18. Jagasia MH, Greinix HT, Arora M, Williams KM, Wolff D, Cowen EW, Palmer J, Weisdorf D, Treister NS, Cheng GS, Kerr H, Stratton P, Duarte RF, McDonald GB, Inamoto Y, Vigorito A, Arai S, Datiles MB, Jacobsohn D, Heller T, Kitko CL, Mitchell SA, Martin PJ, Shulman H, Wu RS, Cutler CS, Vogelsang GB, Lee SJ, Pavletic SZ, Flowers MED. National Institutes of Health Consensus Development Project on Criteria for Clinical Trials in Chronic Graft-versus-Host Disease: I. The 2014 Diagnosis and Staging Working Group Report. Biol Blood Marrow Transplant. 2015;21(3):389–401.e1. Available from: https://doi.org/10.1016/j.bbmt.2014.12.001

  19. Hymes SR, Alousi AM, Cowen EW. Graft-versus-host disease: Part I. Pathogenesis and clinical manifestations of graft-versus-host disease. J Am Acad Dermatol. 2012;66(4):515.e1–515.e18.

  20. Matias AR, Ferreira M, Costa P, Neto P. Skin colour, skin redness and melanin biometric measurements: comparison study between Antera ® 3D, Mexameter ® and Colorimeter ®. Ski Res Technol [Internet]. 2015 Aug 1 [cited 2020 Sep 2];21(3):346–62. Available from: https://doi.org/10.1111/srt.12199

  21. Zijdenbos AP, Dawant BM, Margolin RA, Palmer AC. Morphometric Analysis of White Matter Lesions in MR Images: Method and Validation. IEEE Trans Med Imaging. 1994;13(4):716–24.

    Article  CAS  PubMed  Google Scholar 

  22. Eliasziw M, Young SL, Woodbury MG, Fryday-Field K. Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther [Internet]. 1994 Aug;74(8):777–88. Available from: http://view.ncbi.nlm.nih.gov/pubmed/8047565

  23. Bland JM, Altman DG. Transformations, means, and confidence intervals. BMJ Br Med J. 1996;312(7038):1079.

    Article  CAS  Google Scholar 

  24. Codella NCF, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza SW, Kalloo A, Liopyris K, Mishra N, Kittler H, Halpern A. Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). Proc - Int Symp Biomed Imaging. 2018;2018-April(Isbi):168–72.

  25. Yuan Y, Chao M, Lo Y-C. Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance. IEEE Trans Med Imaging. 2017;36(9):1876–86.

    Article  PubMed  Google Scholar 

  26. Wilmer EN, Gustafson CJ, Davis SA, Feldman SR, Huang WW. Most common dermatologic conditions encountered by dermatologists and nondermatologists. Cutis. 2014;94(6).

Download references

Funding

This work was supported by Career Development Award Number IK2 CX001785 from the United States Department of Veterans Affairs Clinical Sciences R&D (CSRD) Service to ERT, the National Institutes of Health Grants K12 CA090625 and R21 AR074589, and the European Regional Development Fund (1.1.1.2/VIAA/4/20/665) to IS.

Author information

Authors and Affiliations

  1. Dermatology Service and Research Service, Department of Veterans Affairs, Tennessee Valley Healthcare System, Nashville, TN, USA

    Kelsey Parks, Xiaoqi Liu, Zain Khera, Laura X. Baker & Eric R. Tkaczyk

  2. Department of Dermatology, Vanderbilt University Medical Center, Nashville, TN, USA

    Kelsey Parks, Xiaoqi Liu, Zain Khera, Laura X. Baker, Inga Saknite & Eric R. Tkaczyk

  3. Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA

    Xiaoqi Liu, Tahsin Reasat & Benoit M. Dawant

  4. Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA

    Heidi Chen

  5. Biophotonics Laboratory, Institute of Atomic Physics and Spectroscopy, University of Latvia, Riga, Latvia

    Inga Saknite

  6. Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, USA

    Eric R. Tkaczyk

Authors
  1. Kelsey Parks
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoqi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tahsin Reasat
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zain Khera
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Laura X. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Heidi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Benoit M. Dawant
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Inga Saknite
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Eric R. Tkaczyk
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Eric R. Tkaczyk conceptualized and designed the study. Data collection and analysis were performed by Kelsey Parks, Xiaoqi Liu, Tahsin Reasat, Zain Khera, and Laura X. Baker. Statistical analyses were done by Heidi Chen. Manuscript was written and revised by Kelsey Parks, Inga Saknite, and Eric R. Tkaczyk. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eric R. Tkaczyk.

Ethics declarations

Ethics Approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Vanderbilt Institutional Review Board (Date: 4/22/2022 / #170456).

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 23 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Parks, K., Liu, X., Reasat, T. et al. Non-Expert Markings of Active Chronic Graft-Versus-Host Disease Photographs: Optimal Metrics of Training Effects. J Digit Imaging 36, 373–378 (2023). https://doi.org/10.1007/s10278-022-00730-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-022-00730-8

Keywords

Search

Navigation