Submit to this Journal Review for this Journal Propose a Special Issue

Article Menu

Share Help Cite Discuss in SciProfiles

...

Endorse

...

Comment

Order Article Reprints

Font Type:

Font Size:

Line Spacing:

Column Width:

Background:

Open AccessArticle

An Extensive Quality Control and Quality Assurance (QC/QA) Program Significantly Improves Inter-Laboratory Concordance Rates of Flow-Cytometric Minimal Residual Disease Assessment in Acute Lymphoblastic Leukemia: An I-BFM-FLOW-Network Report

Margarita Maurer-Granofszky

Angela Schumich

¹,

Barbara Buldini

²,

Giuseppe Gaipa

Janos Kappelmayer

⁴,

Ester Mejstrikova

⁵,

Leonid Karawajew

⁶,

Jorge Rossi

⁷,

Adın Çınar Suzan

⁸,

Evangelina Agriello

⁹,

Theodora Anastasiou-Grenzelia

¹⁰,

Virna Barcala

¹¹,

Gábor Barna

¹²

Drago Batinić

¹³,

Jean-Pierre Bourquin

¹⁴,

Monika Brüggemann

¹⁵,

Karolina Bukowska-Strakova

¹⁶

Hasan Burnusuzov

¹⁷

Daniela Carelli

¹⁸,

Günnur Deniz

⁸,

Klara Dubravčić

¹³,

Tamar Feuerstein

¹⁹,

Marie Isabel Gaillard

²⁰,

Adriana Galeano

²¹,

Hugo Giordano

²²,

Alejandro Gonzalez

²³,

Stefanie Groeneveld-Krentz

⁶,

Zsuzsanna Hevessy

⁴

Ondrej Hrusak

⁵,

Maria Belen Iarossi

²⁴,

Pál Jáksó

²⁵,

Veronika Kloboves Prevodnik

^26,27

Saskia Kohlscheen

¹⁵,

Elena Kreminska

²⁸,

Oscar Maglia

³,

Cecilia Malusardi

²⁹,

Neda Marinov

³⁰

Bibiana Maria Martin

³¹,

Claudia Möller

¹⁴,

Sergey Nikulshin

³²

Jorge Palazzi

³³,

Georgios Paterakis

³⁴

Alexander Popov

³⁵,

Richard Ratei

³⁶,

Cecilia Rodríguez

³⁷,

Elisa Olga Sajaroff

⁷,

Simona Sala

³,

Gordana Samardzija

³⁸,

Mary Sartor

³⁹,

Pamela Scarparo

²,

Łukasz Sędek

⁴⁰

Bojana Slavkovic

³⁸,

Liliana Solari

⁴¹,

Peter Svec

⁴²

Tomasz Szczepanski

⁴³

Anna Taparkou

⁴⁴,

Montserrat Torrebadell

⁴⁵,

Marianna Tzanoudaki

⁴⁶,

Elena Varotto

²,

Helly Vernitsky

⁴⁷,

Andishe Attarbaschi

⁴⁸

Martin Schrappe

⁴⁹,

Valentino Conter

⁵⁰,

Andrea Biondi

⁵⁰,

Marisa Felice

⁵¹

Myriam Campbell

³⁰,

Csongor Kiss

⁵²

Giuseppe Basso

^2,†,

Michael N. Dworzak

^1,48,*

and

on behalf of I-BFM-FLOW-Network

^‡ Show full author list Hide full author list

Children’s Cancer Research Institute, Medical University of Vienna, 1090 Vienna, Austria

Pediatric Hematology, Oncology and Stem Cell Transplant Division, Maternal and Child Health Department, University of Padova, 35122 Padova, Italy

M. Tettamanti Foundation Research Center, Department of Pediatrics, University of Milano-Bicocca, 20900 Monza, Italy

⁴

Department of Laboratory Medicine, University of Debrecen, 4032 Debrecen, Hungary

⁵

Department of Paediatric Haematology and Oncology, University Hospital Motol, 150 06 Prague, Czech Republic

⁶

Department of Pediatric Oncology and Hematology, Charité Berlin, 10117 Berlin, Germany

⁷

Cellular Immunology Laboratory, Hospital de Pediatria “Dr. Juan P. Garrahan”, Buenos Aires C1245, Argentina

⁸

Department of Immunology, Aziz Sancar Institute of Experimental Medicine, Istanbul University, 34452 Istanbul, Turkey

⁹

LEB Laboratorio, Servicio de Hematologia Hospital Penna, Bahia Blanca B8000, Argentina

¹⁰

P.&A. Kyriakou Children’s Hospital of Athens, 115 27 Athens, Greece

¹¹

Laboratory—Flow Cytometry, Citomlab, Buenos Aires C1406AWK, Argentina

¹²

1st Department of Pathology and Experimental Cancer Research, Semmelweis University, 1085 Budapest, Hungary

¹³

Division of Laboratory Immunology, Department of Laboratory Diagnostics, University Hospital Centre Zagreb & School of Medicine, 10000 Zagreb, Croatia

¹⁴

Department of Oncology and Children’s Cancer Research Center, University Children’s Hospital, 8032 Zurich, Switzerland

¹⁵

Department of Hematology, University Hospital Schleswig-Holstein, 24105 Kiel, Germany

¹⁶

Department of Clinical Immunology, Institute of Pediatrics, Jagiellonian University Medical College, 31-008 Krakow, Poland

¹⁷

Center of Competence “PERIMED”, Department of Pediatrics, Department of Microbiology and Clinical Immunology, Medical University Plovdiv, 4002 Plovdiv, Bulgaria

¹⁸

Hospital Guillermo Rawson, San Juan J5400, Argentina

¹⁹

The Rina Zaizov Division of Pediatric Hematology-Oncology, Schneider’s Children’s Medical Center, Petah Tikva 4920235, Israel

²⁰

Bioquimica, Inmunologia, Hospital de Ninos Rocardo Gutierrez, Buenos Aires C1425EFD, Argentina

²¹

Flow Cytometry Laboratory, FUNDALEU, Buenos Aires C1114, Argentina

²²

Fundación Pérez Scremini, Pediatric Hematology-Oncology Service, Pereira Rossell Hospital, Montevideo 11600, Uruguay

²³

PAPSI Laboratorio Mendoza, Rosario S2000, Argentina

²⁴

Flow Cytometry Laboratory, Provincial Histocompatibility Reference Centre, CUCAIBA, Buenos Aires C1114, Argentina

²⁵

Flow Cytometry Laboratory, Department of Pathology, Clinical Centre, University of Pécs, 7622 Pécs, Hungary

²⁶

Department of Cytopathology, Institute of Oncology, 1000 Ljubljana, Slovenia

²⁷

Faculty of Medicine, University of Ljubljana, 1000 Ljubljana, Slovenia

²⁸

Clinical Laboratory Diagnostics and Metrology of NCSH “OHMATDYT”, Ministry of Heath of Ukraine, 01601 Kiev, Ukraine

²⁹

Hospital de Clinica Jose de San Martin, Buenos Aires C1120, Argentina

³⁰

PINDA, Chilean National Pediatric Oncology Group, Hospital Roberto del Rio, Universidad de Chile, Santiago 8380418, Chile

³¹

Laboratorio CEBAC SRL, Posadas N3300, Argentina

³²

Hematopathology and Flow Cytometry Division, Children’s Clinical University Hospital, LV-1004 Riga, Latvia

³³

IICT Labs, Rosario S2000, Argentina

³⁴

Georgios Gennimatas General Hospital of Athens, 115 27 Athens, Greece

³⁵

Laboratory of Leukemia Immunophenotyping, Dmitry Rogachev National Medical Research Center of Pediatric Hematology, Oncology and Immunology, 117997 Moscow, Russia

³⁶

Clinic for Hematology and Tumor Immunology, HELIOS Klinikum Berlin-Buch, 13125 Berlin, Germany

³⁷

Hospital Nacional de Clínicas, Universidad Nacional de Córdoba, Cordoba X5000HUA, Argentina

³⁸

Laboratory for Flow Cytometry and Immunology, Institute for Health and Protection of Mother and Child of Serbia, 11070 Belgrade, Serbia

³⁹

The Children’s Hospital at Westmead, Sydney, NSW 2145, Australia

⁴⁰

Department of Microbiology and Immunology, Medical University of Silesia, 40-055 Katowice, Poland

⁴¹

Servicio de Bioquimica, Hospital Posadas, Buenos Aires B1684, Argentina

⁴²

National Institute of Children’s Diseases, 831 01 Bratislava, Slovakia

⁴³

Department of Pediatric Hematology and Oncology, Zabrze, Medical University of Silesia, 40-055 Katowice, Poland

⁴⁴

Department of Pediatric Oncology Hippokration General Hospital, 546 42 Thessaloniki, Greece

⁴⁵

Pediatric Hematology and Oncology, Hospital Sant Joan de Deu, 08950 Barcelona, Spain

⁴⁶

Department of Immunology & Histocompatibility, “Agia Sophia” Children’s Hospital, 115 27 Athens, Greece

⁴⁷

Hematology Lab, Sheba Medical Center, Ramat Gan 52621, Israel

⁴⁸

St. Anna Children’s Hospital, Department of Pediatrics, Medical University of Vienna, 1090 Vienna, Austria

⁴⁹

Department of Pediatrics, University Medical Center SchleswigHolstein, Christian-Albrechts-University of Kiel, 24118 Kiel, Germany

⁵⁰

Clinica Pediatrica University degli Studi di Milano Biococca, Fondazione MBBM, 20900 Monza, Italy

⁵¹

Department of Hematology and Oncology, Hospital de Pediatria “Dr. Juan P. Garrahan”, Buenos Aires C1245, Argentina

⁵²

Division of Pediatric Hematology-Oncology, Department of Pediatrics, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary

Show full affiliation list

Hide full affiliation list

Author to whom correspondence should be addressed.

^†

The author have already passed away.

^‡

Membership of I-BFM-FLOW-Network is provided in the Acknowledgements.

Cancers 2021, 13(23), 6148; https://doi.org/10.3390/cancers13236148

Submission received: 30 September 2021 / Revised: 25 November 2021 / Accepted: 30 November 2021 / Published: 6 December 2021

(This article belongs to the Special Issue Recent Advances in Pediatric Acute Leukemia)

Download

Browse Figures

Figure 1
Twinning maturation program. (A) Number of failures by all 22 laboratories in the main sample set (samples 1–25) (B) Number of failures of the 16 laboratories that finished within a range of a maximum of 31 samples (31 samples = mean number of samples needed for maturation). "> Figure 2
Correlation of the local MRD estimates of UK NEQAS samples analyzed by I-BFM-Flow laboratories. The target value represents the median of all submitted results. Red dots indicate outliers (i.e., MRD results outside the concordance margin, n = 78). Blue dots indicate concordant results (n = 1601). Positivity was defined as ≥10 MRD events among 3 × 105 nucleated events, i.e., >0.003% (pos, positive; neg, negative). "> Figure 3
Regular ring trial tests using the list-mode file sent around reveal a significant performance improvement. LMD Files from 20 B-ALL cases at day 78 or after relapse therapies were sent out to all participating laboratories once a year. Laboratories, which reported false MRD values in >10%–≤25% of cases were flagged with a warning (yellow circle) and reports with >25% false MRD values were considered to be critical (red flag). Total contributing I-BFM labs contained AIEOP-BFM, ALL IC-BFM, and other associated laboratories. Statistical analysis was performed using a 3 × 2 contingency table and two-sided Fisher’s test. "> Figure 4
Plot from a representative list-mode file ring trial. RT 2020 comprised 20 regenerative bone marrow samples with either negative or varying levels of MRD. LMD files were sent out to 41 centers, all of which reported results. AIEOP-BFM n = 12, ALL IC-BFM = 19, and other I-BFM cooperating centers = 10. The target MRD value was calculated as a median from six experienced and matured AIEOP-BFM reference laboratories (Vienna, Monza, Padova, Prague, Kiel and Berlin). Two opportunities for educational intervention can be seen in this figure: (i) similar errors by four and six centers, respectively, in samples 14 and sample 4 representing a systematic error; and (ii) an accumulation of errors in single centers (see center 13 and 14) representing lack of experience. "> Figure 5
Homogeneity of B-ALL risk classification data of well-trained and matured centers within the AIEOP-BFM ALL 2009 trial. In a survey including nine AIEOP-BFM FLOW reference laboratories from eight countries, independent data of FCM-MRD-based risk classification of B-ALL on day 15 were collected and compared. Statistical analysis was performed using GraphPad Prism 8.3.0. FHR, Flow high risk; FLR, Flow low risk; FMR, Flow medium risk; INS, contains combined data from the two reference laboratories in Israel. ">

Versions Notes

Abstract

Simple Summary

Standardization of flow-cytometric assessment of minimal residual disease in acute lymphoid leukemia (ALL) is necessary to allow concordant multicentric application of the methodology. This is a prerequisite for internationally collaborative trials, such as the AIEOP-BFM-ALL and the ALL IC-BFM trial. We developed and applied a comprehensive training and quality control program involving a large number of international laboratories within the I-BFM consortium to complement standardization of the methodology with an educational component as well as with persistent quality control measures to allow large ALL treatment trials which use multi-laboratory FCM-MRD assessments for risk stratification of pediatric patients with ALL.

Abstract

Monitoring of minimal residual disease (MRD) by flow cytometry (FCM) is a powerful prognostic tool for predicting outcomes in acute lymphoblastic leukemia (ALL). To apply FCM-MRD in large, collaborative trials, dedicated laboratory staff must be educated to concordantly high levels of expertise and their performance quality should be continuously monitored. We sought to install a unique and comprehensive training and quality control (QC) program involving a large number of reference laboratories within the international Berlin-Frankfurt-Münster (I-BFM) consortium, in order to complement the standardization of the methodology with an educational component and persistent quality control measures. Our QC and quality assurance (QA) program is based on four major cornerstones: (i) a twinning maturation program, (ii) obligatory participation in external QA programs (spiked sample send around, United Kingdom National External Quality Assessment Service (UK NEQAS)), (iii) regular participation in list-mode-data (LMD) file ring trials (FCM data file send arounds), and (iv) surveys of independent data derived from trial results. We demonstrate that the training of laboratories using experienced twinning partners, along with continuous educational feedback significantly improves the performance of laboratories in detecting and quantifying MRD in pediatric ALL patients. Overall, our extensive education and quality control program improved inter-laboratory concordance rates of FCM-MRD assessments and ultimately led to a very high conformity of risk estimates in independent patient cohorts.

Keywords:

acute lymphoblastic leukemia; minimal residual disease; flow cytometry; quality control program

1. Introduction

The application of flow cytometric minimal residual disease measurement (FCM-MRD) has become a routine clinical practice in most current study protocols worldwide, in order to assign patients to different risk groups and to tailor treatment intensity [1,2,3,4,5,6]. FCM-MRD is applicable in almost all (>98%) patients with lymphoblastic leukemia and can reach sensitivities similar to a quantitative real-time polymerase chain reaction (RQ-PCR) (≥10⁻⁵) if it is ensured that the laboratories involved in MRD testing are able to reliably detect and quantify leukemic cells, even in the low MRD range [1]. Several national centers and international consortia previously worked on standardizing the FCM-MRD methodology in multi-centric settings to improve inter-laboratory concordance rates, and to assure sensitive and accurate MRD assessments [5,7,8,9,10,11]. The allied Associazione Italiana di Ematologia Oncologia Pediatrica (AIEOP) and the Berlin-Frankfurt-Münster (BFM) FCM-MRD study groups, which have collaborated since May 2000, initiated a standardization program based on education of personnel, technical alignment, and on continued quality control (QC) [9]. This study, involving four experienced national reference laboratories, proved that FCM-MRD can be standardized for multi-centric application in large trials. Such a multi-facetted QC/quality assurance (QA) program included the standardization of the methodology, continuous staff training, performance reviews through sample and list-mode data (LMD) exchange, as well as the assessment of concordance of “on trial” MRD results from independent patient cohorts [9]. Similarly, an educational component complementing the standardization of the FCM-MRD methodology was necessary to warrant concordant quality of MRD assessment in a study involving eight experienced centers in North America [12].

In the current study, we performed a unique and comprehensive training and QC/QA program involving a total of 41 laboratories collaborating in the international (I)-BFM consortium. Of note, many associated laboratories of the network were inexperienced when entering the program. Our approach consists of four major building blocks: (i) a twinning maturation program, where well-trained expert laboratories work closely with more inexperienced laboratories to improve their performance in individual feedback sessions; (ii) the obligatory participation in external QA programs (spiked sample send around), e.g., United Kingdom National External Quality Assessment Service (UK NEQAS); (iii) regular participation in the analysis of LMD file ring trials (data file send arounds) with an educational component, where difficult and discordant cases are jointly discussed; and (iv) surveys of independent data derived from trial results, where the agreement of different centers, with respect to MRD results from their locally recruited patient cohorts, were tested.

We demonstrate that the training of laboratories using experienced twinning partners along with continuous educational feedback, significantly improves the performance of laboratories to detect and quantify MRD in pediatric ALL patients. In addition to a standardized methodology, educational feedback and quality oversight by a coordinating structure is required to ensure the sufficient quality of MRD assessment for clinical routine.

2. Materials and Methods

2.1. Samples

MRD investigations were approved as part of the international trial by the institutional ethical committees and were conducted according to informed consent guidelines. For list-mode file ring trials, we used bone marrow (BM) samples from pediatric patients (0–≤18 years of age) recruited in the AIEOP-BFM-ALL clinical trial in Vienna (AUT).

In total, data from 101 BM samples from regenerative post-induction time-points of pediatric patients with B-ALL were used for the LMD file ring trials (“B-cell precursor (BCP) Regeneration”) during 2016–2020. These regenerative bone marrows are usually characterized by the appearance of BCP subsets at different stages of (BCP 1–3) [13].

Of the 101 BM samples, 53 were FCM-MRD negative, and 48 samples were FCM-MRD positive (range 0.004–18.16%). If available, RT-PCR data served as an independent confirmation of the FCM-MRD target votes. The target MRD values were calculated as a median from six experienced and matured AIEOP-BFM reference laboratories (Vienna, Monza, Padova, Prague, Kiel and Berlin).

Sample Preparation

Detailed standard operating procedures (SOP) from the AIEOP-BFM (Supplementary Materials File S1 “SOP FLOW-MRD 1.5 AEIOP-BFM”) as well as from the intercontinental ALL- IC reference centers can be found in the Supplementary Materials (Supplementary Materials File S2 and S3 “SOP ALL-IC 2009” and “Amendment to SOP ALL-IC-BFM_2014”). Samples for the generation of LMD files were prepared at the coordinating center in Vienna briefly as follows: whole BM samples were stained either with a two-tube approach (7- and 8-color panel, 2016–2017, Table S1) or with a single, dried, 10-color format tube (ReALB, DuraClone^TM, Beckman Coulter, Brea, CA, USA), Table S1) (2018–2020) for 15′ and lysed with a fixing lysis reagent (BD Lyse, Becton Dickinson, Franklin Lanes, Evansville, EL, USA) for another 15′. Samples were washed once with 1× phosphate-buffered saline (PBS, Gibco, Thermo Fisher Scientific Inc., Waltham, MA, USA), resuspended in 200 µL 1×PBS and measured immediately. Samples were acquired using an LSRII flow cytometer (Becton Dickinson) and analyzed at the reference center using DIVA software (Becton Dickinson). Syto41 staining was used to exclude non-nucleated events such as erythrocytes, platelets, and debris. For detailed information on the gating strategy, see Supplementary Materials S1–S3. MRD was reported as a percentage of Syto41-positive cells with exclusion of doublets using FSC-H/FSC-A gating. Minimal residual disease positivity was defined as an accumulation of ≥10 cells with a leukemic phenotype [9].

2.2. I-BFM-FLOW Twinning and Maturation Program

A total number of 29 laboratories participated in the twinning and maturation program of which 22 were ALL IC-BFM (Acute Lymphoblastic Leukemia Intercontinental Berlin-Frankfurt-Münster) trainee laboratories. Seven experienced and already matured AIEOP-BFM laboratories served as twinning and training partners. “Maturation” is granted to a center according to specific criteria and is defined as the permission to release MRD results for clinical use without further control by the partner laboratory (“twinning partner”). “Maturation” can be granted for either B-lineage ALL-MRD at day 15/d33 or at regenerative time-points, and for T-ALL-MRD (independent of time-point). “Maturation” is granted after completion of a series of at least 25 different locally and consecutively recruited patient sample pairs (diagnosis and MRD time-points) per lineage, the analysis of which, along with the results, was promptly submitted to the twinning partner for review. Harmonization and technical alignment of the FCM-MRD technology including machine settings, panel composition, sample preparation as well as staining was an essential part of the twinning maturation program and took place before the test series was started. During the initial phase of the program, an intensive interaction and discussion between the trainee laboratory and its twinning partner following each case ensured a rapid and effective training progress. Therefore, the first half of the samples serves, for the most part, an intensive training, the immediate success of which should and must be proven with the second half of the samples.

In the second half of the sample series, each gross error then requires a further three MRD assessments of consecutive cases without errors to achieve maturation. Gross errors are defined as inadequacy of staining and analysis, errors in blast population, or gross misinterpretation or misdocumentation on a predefined report sheet. Provided that no more gross errors in identifying and/or quantifying MRD are detected in the most recent half (i.e., 12 patients) of the sample series, maturation is granted. For a detailed description of the guidelines, see Supplementary Materials Files S4 and S5 “I-BFM FLOW Twinning Maturation” and “Maturation Certificate”. Feedback by training laboratories was mainly provided via email and/or phone or during training visits and included recommendations regarding further optimization of technical issues (e.g., compensation, etc.) as well as the gating strategy or explanations of phenotypic properties of normal and aberrant cells, respectively.

2.3. External Quality Assessment (EQA)

UK NEQAS (https://ukneqas.org.uk/ (accessed on 30 November 2021) and [14]) provides external quality assessment/proficiency testing for all major aspects of clinical laboratory testing including “Minimal Residual Disease for ALL by flow cytometry”. UK NEQAS ships stabilized blood samples spiked with B-ALL blasts to participating centers. Individual results are compared to the robust mean of all the submitted values. UK NEQAS sends out two samples with different MRD values per trial and usually issues four wet trials per year. This program was used by the I-BFM network for EQA. The local results of all I-BFM centers submitting results were evaluated in a centralized manner by the I-BFM coordinating center in Vienna.

2.4. List-Mode-Data File Ring Trials (LMD Ring Trial)

Anonymized list-mode files were prepared and sent out by the I-BFM FLOW coordinating center in Vienna. A set of at least 20 files containing data of regenerating bone marrow samples with either negative MRD or different positive levels of MRD was sent to the participating laboratories once a year for blinded LMD file interpretation. The participating laboratories were members of the AIEOP-BFM or ALL-IC BFM consortia or cooperating laboratories. Reported MRD values that were either more than three times or less than one-third of the target MRD value (concordance margin, see [9]) were considered as false assessments. The target MRD value was calculated as a median from six experienced and matured AIEOP-BFM reference laboratories (Vienna, Monza (ITA), Padova (ITA), Prague (CZE), Kiel (GER) and Berlin (GER)). Laboratories, which reported false MRD values in >10%–≤25% of cases were flagged with a warning and reports with >25% false MRD values were considered to be critical.

2.5. Independent Data Survey

We conducted a survey among eight AIEOP-BFM reference laboratories to collect FCM-MRD-based risk classification data within the AIEOP-BFM-ALL 2009 trial. The following reference laboratories participated in this survey: Berlin, Monza, Padova, Prague, Vienna, Zurich (CH), Sydney (AUS) and the two reference laboratories of Israel (ISR). Details of all requested data can be found in Supplementary Materials File S6, “Data query”. According to the AIEOP-BFM ALL 2009 study protocol, the criteria for assignment of patients to either flow low risk (FLR), flow medium risk (FMR) or flow high risk (FHR) were defined as follows: percentage of blasts in the BM at day 15 of induction therapy ≤0.1% FLR, >0.1% <10% FMR, ≥10% FHR.

2.6. Statistical Analysis

The concordance of independent flow-risk estimate data was assessed using GraphPad Prism ver. 8.3.0 (GraphPad Software, San Diego, CA, USA) and Two-Way ANOVA (Tukey test for multiple comparison correction). We used http://www.analyticscalculators.com (accessed on 30 November 2021) for computation of exact test probabilities for 2 × 2 and 2 × 3 Contingency Tables [15,16].

3. Results

3.1. Twinning Maturation Program

In total, 682 samples were processed and analyzed by the 22 trainee laboratories using diagnostic methodology based on central standard operating procedures (SOPs) (see Supplementary Materials File S1–S3), and the data were sent to the training laboratories for review (Table 1).

According to the guidelines, a minimum of 25 locally and consecutively recruited samples per lineage (B- or T-ALL, Table S1) had to be assessed by the trainees without gross errors in the second half of the samples (i.e., in the last 12 samples). For each gross error, three additional samples had to be processed, analyzed and sent again for review. Overall, out of the 682 analyzed B-ALL samples, MRD assessment was erroneous in 11.6% of samples (Table 2). In total, laboratories required a median number of 28 (range 25–57) and a mean number of 31 ± 8.8 samples to reach maturation status (Table 1). The difference in the number or errors in the first (Series 1: sample 1–13) and second half (Series 2: sample 14–25) of the sample set was marginally not statistically significant (14.7% vs. 9.5%), but reached significance when considering the entire sample set (Series 1: sample 1–13 vs. Series 3: sample 14-pX) (14.7% vs. 9.3%; p < 0.05) (Table 2 and Figure 1). Sixteen laboratories achieved maturation status within thirty-one samples (which is the mean number of samples needed for maturation). In those 16 laboratories, there was a significant performance improvement between sample series 1 and 2 (11.1% vs. 3.1%, p < 0.01; Table S2 and Figure 1). Overall, the initial performance of the training laboratories was variable, and six laboratories required more than 31 samples to achieve maturation status. The continuous educational feedback during the twinning maturation process significantly improved the ability of the trainee laboratories to correctly detect and quantify MRD in B- (Table 2) and T-ALL (data not shown) samples.

3.2. External Quality Assessment

The United Kingdom National External Quality Assessment Scheme (UK NEQAS) for leukocyte immunophenotyping provides external quality assessment (EQA)/proficiency testing (PT) programs for minimal residual disease for ALL by FCM.

We collected performance data from 29 issued UK NEQAS ring trials (containing 2 samples/ring trial) from up to 38 participating I-BFM-FLOW laboratories, and monitored their performance over time by correlating local results of I-BFM centers with target votes (Figure 2). Target votes were defined as the median of all submitted I-BFM center results. Reported MRD values within the concordance margin were considered as concordant (i.e., either more than three times or less than 1/3 of the target MRD value [9]), whereas MRD values outside the concordance margin were defined as outliers. In total, results of 1679 samples have been submitted and compared, of which 487 samples were submitted by AIEOP-BFM laboratories (n = 9) and 1192 by ALL IC-BFM and other associated laboratories (n = 29). Generally, AIEOP-BFM laboratories had a significantly lower proportion of outliers (2.3%) than ALL IC-BFM/other associated laboratories (5.6%) (p = 0.0043), which is in concordance with the higher number of long-term experienced centers within the AIEOP-BFM group. The overall discordance rate was 4.6% (Table 3 and Figure 2). We also assessed the concordance of local versus target votes regarding MRD positivity or negativity, where positivity was defined as ≥10 MRD events among 3 × 10⁵ nucleated events. Here, the concordance rate was 97%, specificity was 81% (141/175) and sensitivity was 99% (1488/1504) (Figure 2).

3.3. List Mode Data File Ring Trials (LMD Ring Trial)

For further continuous quality assessment and improvement, the performance of the I-BFM-FLOW laboratories was monitored regularly via LMD ring trials consisting of data files with either negative or varying levels of MRD from B- or T-ALL cases, or of data files from diagnosis for testing the ability of the reference laboratories for the correct lineage assessment of AL by immunophenotyping (see Table S3). The staining panel setup is shown in Table S1 (B- and T-ALL).

Since the detection and quantification of MRD by FCM is particularly challenging in B-ALL in periods of heavy hematopoietic regeneration, while it is relatively straight forward at early timepoints of therapy (i.e., d15), a special focus was on MRD training at regenerative timepoints.

A series of list mode data ring trials of B-ALL at regenerative time points (e.g., day 78 or after relapse therapies) is shown in Figure 3. LMD ring trials (RT) were organized once a year by the I-BFM-FLOW coordinating center in Vienna. The total number of participating laboratories (AIEOP-BFM, ALL IC-BFM as well as other associated laboratories) varied from 31 in the first RT 2016 up to 41 laboratories in RT 2020. In the first LMD ring trial, 19 out of 31 laboratories (61%) reported MRD values consistent with the target value, while four laboratories (13%) submitted >10% discordant values and eight laboratories (26%) were even considered critical with >25% discordant values (Figure 3, lower panel).

Samples that were incorrectly assessed by several centers were subjected to error analysis (n = 24) (Table S4). Most of the repeatedly misinterpreted samples were regenerative BM with a particularly high number of early B cell progenitors (hematogones) (18 out of 24 discordant samples; 75%) resulting mostly in false positive assessments in 13 out of these 18 samples (72%). Additionally, false negative assessments (5/18) as well as inaccurate gating leading to lower or higher MRD levels (5/18) occurred in this group of samples. Along with these considerations, common reasons for failure were an insufficient theoretical separation of blasts from hematogones by classical leukemia-associated immunophenotype (LAIP) markers or maker combinations (7/24), the absence of mature B-cells that could serve as a normal reference population (6/24), negative or only dim expression of CD34 on early BCP1 in burst regeneration (4/24), the co-existence of BCP1 and blasts with only a few aberrations of the latter (4/24), a rare and/or heterogeneous phenotype of the blasts (2/24), or the presence of only very few MRD cells (2/24) (Table S4). The most common errors, together with solutions and suggestions for improvements, were presented and discussed at regular joint meetings and educational material was disseminated to all participants (Supplementary Materials File S7 “Educational Slides”). During these educational feedback meetings, guidelines, and suggestions for better distinguishing hematogones from aberrant blasts were provided.

This educational feedback proved to be efficient and sustainable. In the second LMD ring trial (RT 2017), the overall performance of the laboratories significantly improved (p = 0.005). No laboratory submitted >25% discordant results while 8/32 laboratories (25%) were still assigned a warning flag (Figure 3, lower panel). In the last LMD ring trial (RT 2020), 93% of all laboratories succeeded, while no laboratory was assigned a critical status, and only three laboratories (7%) were assigned a warning flag.

Of the ALL IC-BFM laboratories, only 54% passed successfully in RT 2016, whereas this number increased to 80% in RT 2017 and remained at high levels (range 68–84%) in RT 2018–2020 (Figure 3, middle panel and table). There was no significant improvement when considering only the AIEOP-BFM laboratories, although the percentage of laboratories successfully passing the LMD ring trial improved from 70% in 2016 (7 out of 10) to 100% in 2020 (Figure 3, upper panel).

When assessing only those laboratories which were active from the beginning, results of the improvement status over time did not differ from the assessment of all participating laboratories (Figure S1).

A representative example of an LMD ring trial summary with data files from B-ALL samples of regenerative timepoints is illustrated in Figure 4.

3.4. Independent Data Concordance: A Retrospective Quality Control Measure

We tested the agreement of eight matured and highly experienced I-BFM centers regarding the distribution of patients in different FCM-based risk groups within the AIEOP-BFM-ALL 2009 trial (details on all requested data of the survey can be found in Supplementary Materials File S6 “Data query”). The percentage of ALL patients assigned to either the low-risk (FLR), medium-risk (FMR) or high-risk (FHR) group by FCM were highly concordant and there were no significant differences between the independent patient cohorts from the eight centers (Figure 5 (B-ALL) and Figure S2 (T-ALL)).

4. Discussion

This I-BFM-FLOW network report summarizes the comprehensive QC/QA program on FCM-MRD involving an unprecedented, intercontinental effort of two of the world’s largest pediatric ALL trial groups. As of November 2020, twenty-two trainee and seven expert trainer laboratories participated in a twinning maturation program and all trainee laboratories achieved maturation status, enabling them to report FCM-MRD results for clinical decision making without further control by partner laboratories according to the study group’s guidelines. Forty-one I-BFM-FLOW laboratories submitted results in the most recent LMD file send around (RT2020) and data on 1679 results of 29 issued external QA ring trials (UK NEQAS) of up to 38 laboratories participating per ring trial were collected. As shown in this report, these exercises were successfully developed, and maintained and controlled the proficiency of the many FCM laboratories involved in AIEOP-BFM and ALL IC-BFM trial conduct.

The harmonization and alignment of the technical aspects of the FCM-MRD methodology, including machine settings, panel composition, sample preparation, as well as sample staining between different centers, are important prerequisites for concordant multicenter applications [9]. Several previous studies showed that the harmonization of technical and methodological aspects is feasible, even in an international context [5,9,10,12]. This is particularly important for large, internationally collaborative trials, such as the AIEOP-BFM-ALL and the ALL IC-BFM trial, where the results of the different laboratories must be concordant for the correct stratification and risk-adapted treatment of patients. While the technical aspects of the FCM-MRD method can be well harmonized, the human factor, i.e., the interpretation of the electronic files of the acquired sample, is the actual Achilles heel of the methodology and strongly relies on operator skills and expert knowledge [9,12,17]. Hence, discrepancies in MRD values between laboratories can be due to wrong interpretations of the data [9,12]. This becomes especially apparent in the case of discrepancies in the results of FCM data file exchanges, where preanalytical and acquisition issues do not play a role.

Hence, we based the continuous quality control on two major cornerstones: firstly, the LMD file ring trials, where we focused on the assessment of the analysis performance of each laboratory; and, secondly, on the external UK NEQAS quality control program that allows the performance of the whole methodology to be assessed. Our LMD file ring trials also contain an educational component, where problematic cases are discussed in joint meetings. As part of such meetings, suggestions and solutions for the proper identification of the aberrant cells are provided. In the LMD ring trials, strong BCP regeneration in the bone marrow was found to be the main cause of incorrect MRD assessment. In most cases, these were false positive assessments because normal B cell progenitors (hematogones) were wrongly mistaken for leukemic cells. Therefore, we specifically selected cases with different stages of regeneration and different MRD levels for the LMD ring trials to train the laboratories, especially on such challenging samples. Although BCP regeneration does normally not occur at day 15 or at day 33, it becomes relevant at day 78 of BFM-type induction or during later follow-up time points [4]. While most previous studies were conducted using four-color techniques, panels with ≥6 colors significantly improved the specificity and sensitivity of MRD analysis [18,19,20] and the use of a single-tube with the addition of markers such as CD38, frequently allowed for a reliable differentiation between hematogones and leukemic cells [19,20].

Our approach has some limitations. There are situations where standard panels do not allow for a correct identification of MRD events, even by the experts themselves, unless additional markers, which are useful for such rare cases, are deployed for a better distinction of aberrant cells [21]. In addition, time and staff constraints form a bottleneck for the continued QC and training necessities in a large consortium such as the I-BFM FLOW network. Therefore, approaches with the aim of supporting the operator by using an objective, automated, FCM-analysis tool based on machine learning techniques are necessary and are already being developed [17,22,23,24]. Such a tool may not only relevantly decrease technical variability inherent in manual gating strategies but may also serve as a virtual training partner in twinning programs, such as those presented here to reduce the need for the laborious and costly face-to-face trainings or remote educational feedback loops of trainees with experts. With respect to the survey of independent on-trial data, it does not allow for short-term quality control and immediate intervention and improvements, however, can be used as retrospective evidence of the reliability of the data.

5. Conclusions

Our extensive training and quality assurance program contribute significantly to improving and continuously ensuring the performance of individual laboratories. Of note, the structure based on a coordinating center responsible for quality oversight and feedback proved particularly effective and relevant. Overall, this led to a very high inter-laboratory concordance rate of FCM-MRD assessments and to a high conformity of risk estimates in independent patient cohorts. This is essential for conducting large ALL treatment trials which use multi-laboratory FCM-MRD assessments for stratification.

“The I-BFM FLOW-network is open to further national or trial-based reference laboratories to join the quality program for providing excellent, quality-assured diagnostics to young patients with acute leukemia and for promoting collaborative research.”

Supplementary Materials

The following data are available online at https://www.mdpi.com/article/10.3390/cancers13236148/s1, Figure S1: Results of regular ring trial tests of core laboratories that were involved from the beginning of the program. Figure S2: Homogeneity of T-ALL risk classification data of well trained and matured centers within the AIEOP-BFM 2009 trial, Table S1: Antibodies used for MRD assessment in samples from pediatric patients with B-ALL, Table S2: Twinning-maturation program, Table S3: LMD Ring Trials performed by the I-BFM FLOW network, Table S4: Table summarizing the most common problems causing incorrect MRD assessments in LMD File ring trials, Supplemental File Sl: “SOP FLOW-MRD 1.5 AIEOP-BFM”, Supplemental File S2: “SOP ALL-IC 2009”, Supplemental File S3: “Amendment to SOP ALL-IC_2014”, Supplemental File S4: “I-BFM FLOW Twinning Maturation”, Supplemental File S5: “Maturation Certificate”, Supplemental File S6: “Data Query”, Supplemental File S7: “Educational Slides”.

Author Contributions

Conceptualization, G.B. (Giuseppo Basso) and M.N.D.; Data curation, A.S., E.M., A.Ç.S., E.A., T.A.-G., V.B., G.B. (Gabor Barna), D.B., K.B.-S., H.B., D.C., G.D., K.D., T.F., M.I.G., A.G. (Adriana Galeano), H.G., A.G. (Alejandro Gonzalez), S.G.-K., Z.H., M.B.I., P.J., V.K.P., S.K., E.K., O.M., C.M. (Cecilia Malusardi), N.M., B.M.M., C.M. (Claudia Möller), S.N., J.P., G.P., R.R., C.R., E.O.S., S.S., G.S., M.S., P.S. (Pamela Scarparo), Ł.S. (Łukasz Sędek), B.S., L.S. (Liliana Solari), P.S. (Peter Svec), T.S., A.T., M.T. (Montserrat Torrebadell), M.T. (Marianna Tzanoudaki), E.V. and H.V.; Formal analysis, M.M.-G., S.N. and M.N.D.; Funding acquisition, M.N.D.; Project administration, M.N.D.; Resources, B.B., G.G., J.K., L.K., J.R., J.-P.B., M.B., O.H., A.P., A.A., M.S., V.C., A.B., M.F., M.C., C.K., G.B. (Giuseppe Basso) and M.N.D.; Validation, M.N.D.; Writing—original draft, M.M.-G.; Writing—review and editing, M.N.D. All authors have read and agreed to the published version of the manuscript.

Funding

The study was supported by The European Union’s H2020 Research and Innovation Program through Grant number 825749 “CLOSER: Childhood Leukemia: Overcoming Distance between South America and Europe Regions” and the Polish Ministry of Health program through Grant number 26.01.2019/42/772 “CLOSER: Childhood Leukemia: Overcoming Distance between South America and Europe Regions “Quality control program in the diagnosis of acute leukemia in children for the years 2019–2021”.

Institutional Review Board Statement

Sampling was approved along with the international trials by the institutional ethical committees of the coordinating center in Vienna as well as of the other institutions (Ethics vote No. 1627/2018).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

For original data, please contact [email protected].

Acknowledgments

This study was initiated and promoted by the international BFM study group and is dedicated to the respectful commemoration of late Giuseppe “Beppe” Basso, co-founder of the I-BFM-FLOW network, expert, and teacher to many in the field, as well as a generous host and dear friend. The authors thank all doctors, nurses, and technicians of the participants of the BFM, AEIOP, and ALL IC-BFM study groups for their close collaboration in providing samples and data from their patients. The authors also thank Diem Markus and Kowarsch Florian (Technical University, Vienna) for help with figure preparation, Suhendra Susanne, Scharner Daniela, and Printz Dieter (CCRI, Vienna) for excellent technical assistance. We also thank Novakova Michaela (University Hospital Motol, Prague, Czech Republic), Iommi María Paula and Torreguitart Federico (Servicio de Hematologia Hospital Panna, Bahia Blanca, Argentina), Kodman Yona (Schneider’s Children’s Medical Center, Petah Tikva, Israel), Roffe Georgina (Hospital de Pediatria “Dr. Juan P. Garrahan”, Buenos Aires, Argentina), Kolenova Alexandra and Sestakova Zuzana (National Institute of Children’s Diseases, Bratislava, Slovakia), Marzulli Teresa (The Children’s Hospital at Westmead, Sydney, Australia), Surman Marta (University Children’s Hospital of Kracow, Krakow, Poland), Brožič Andreja and Buček Simon (Department of Cytopathology, Ljubljana, Slovenia), Isola Ignacio (SJD Hospital, Barcelona, Spain), and Pont Sandra (Institut de Recerca SJD, Barcelona, Spain).

Conflicts of Interest

MND received payments for invited lectures and travel expenses from Beckman-Coulter. Beckman-Coulter, Exbio, and Becton Dickinson supported the study with respect to privileged material acquisition. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

Gaipa, G.; Buracchi, C.; Biondi, A. Flow cytometry for minimal residual disease testing in acute leukemia: Opportunities and challenges. Expert Rev. Mol. Diagn. 2018, 18, 775–787. [Google Scholar] [CrossRef] [PubMed]
Coustan-Smith, E.; Behm, F.G.; Sanchez, J.; Boyett, J.M.; Hancock, M.L.; Raimondi, S.C.; E Rubnitz, J.; Rivera, G.K.; Sandlund, J.T.; Pui, C.-H.; et al. Immunological detection of minimal residual disease in children with acute lymphoblastic leukaemia. Lancet 1998, 351, 550–554. [Google Scholar] [CrossRef]
Borowitz, M.J.; Devidas, M.; Hunger, S.P.; Bowman, W.P.; Carroll, A.J.; Carroll, W.L.; Linda, S.; Martin, P.L.; Pullen, D.J.; Viswanatha, D.; et al. Clinical significance of minimal residual disease in childhood acute lymphoblastic leukemia and its relationship to other prognostic factors: A Children’s Oncology Group study. Blood 2008, 111, 5477–5485. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Dworzak, M.N.; Fröschl, G.; Printz, D.; Mann, G.; Pötschger, U.; Mühlegger, N.; Fritsch, G.; Gadner, H.G. Austrian Berlin-Frankfurt-Munster Study, Prognostic significance and modalities of flow cytometric minimal residual disease detection in childhood acute lymphoblastic leukemia. Blood 2002, 99, 1952–1958. [Google Scholar] [CrossRef] [Green Version]
Brüggemann, M.; Schrauder, A.; Raff, T.; Pfeifer, H.; Dworzak, M.; Ottmann, O.G.; Asnafi, V.; Baruchel, A.; Bassan, R.; Benoit, Y.; et al. Standardized MRD quantification in European ALL trials. In Proceedings of the Second International Symposium on MRD Assessment, Kiel, Germany, 18–20 September 2008. [Google Scholar]
Basso, G.; Veltroni, M.; Valsecchi, M.G.; Dworzak, M.N.; Ratei, R.; Silvestri, D.; Benetello, A.; Buldini, B.; Maglia, O.; Masera, G.; et al. Risk of Relapse of Childhood Acute Lymphoblastic Leukemia Is Predicted By Flow Cytometric Measurement of Residual Disease on Day 15 Bone Marrow. J. Clin. Oncol. 2009, 27, 5168–5174. [Google Scholar] [CrossRef]
Irving, J.; Jesson, J.; Virgo, P.; Case, M.; Minto, L.; Eyre, L.; Noel, N.; Johansson, U.; Macey, M.; Knotts, L.; et al. Establishment and validation of a standard protocol for the detection of minimal residual disease in B lineage childhood acute lymphoblastic leukemia by flow cytometry in a multi-center setting. Haematologica 2009, 94, 870–874. [Google Scholar] [CrossRef] [Green Version]
Björklund, E.; Matinlauri, I.; Tierens, A.; Axelsson, S.; Forestier, E.; Jacobsson, S.; Ahlberg, A.J.; Kauric, G.; Mäntymaa, P.; Osnes, L.; et al. Quality Control of Flow Cytometry Data Analysis for Evaluation of Minimal Residual Disease in Bone Marrow from Acute Leukemia Patients during Treatment. J. Pediatr. Hematol. Oncol. 2009, 31, 406–415. [Google Scholar] [CrossRef]
Dworzak, M.N.; Gaipa, G.; Ratei, R.; Veltroni, M.; Schumich, A.; Maglia, O.; Karawajew, L.; Benetello, A.; Pötschger, U.; Husak, Z.; et al. Standardization of flow cytometric minimal residual disease evaluation in acute lymphoblastic leukemia: Multicentric assessment is feasible. Cytom. Part B Clin. Cytom. 2008, 74, 331–340. [Google Scholar] [CrossRef]
Theunissen, P.; Mejstrikova, E.; Sędek, L.; Van Der Sluijs-Gelling, A.J.; Gaipa, G.; Bartels, M.; da Costa, E.S.; Kotrová, M.; Novakova, M.; Sonneveld, E.; et al. Standardized flow cytometry for highly sensitive MRD measurements in B-cell acute lymphoblastic leukemia. Blood 2017, 129, 347–357. [Google Scholar] [CrossRef]
Lorenzana, R.; Coustan-Smith, E.; Antillon, F.; Ribeiro, R.; Campana, D. Simple methods for the rapid exchange of flow cytometric data between remote centers. Leukemia 2000, 14, 336–337. [Google Scholar] [CrossRef] [Green Version]
Keeney, M.; Wood, B.L.; Hedley, B.D.; DiGiuseppe, J.A.; Stetler-Stevenson, M.; Paietta, E.; Lozanski, G.; Seegmiller, A.C.; Greig, B.W.; Shaver, A.C.; et al. A QA Program for MRD Testing Demonstrates That Systematic Education Can Reduce Discordance Among Experienced Interpreters. Cytom. Part B Clin. Cytom. 2018, 94, 239–249. [Google Scholar] [CrossRef] [Green Version]
Dworzak, M.N.; Fritsch, G.; Fleischer, C.; Printz, D.; Fröschl, G.; Buchinger, P.; Mann, G.; Gadner, H. Multiparameter phenotype mapping of normal and post-chemotherapy B lymphopoiesis in pediatric bone marrow. Leukemia 1997, 11, 1266–1273. [Google Scholar] [CrossRef] [Green Version]
Bainbridge, J.; Rountree, W.; Louzao, R.; Wong, J.; Whitby, L.; Denny, T.N.; Barnett, D. Laboratory Accuracy Improvement in the UK NEQAS Leucocyte Immunophenotyping Immune Monitoring Program: An Eleven-Year Review via Longitudinal Mixed Effects Modeling. Cytom. Part B Clin. Cytom. 2017, 94, 250–256. [Google Scholar] [CrossRef] [Green Version]
Freeman, G.H.; Halton, J.H. Note on an Exact Treatment of Contingency, Goodness of Fit and Other Problems of Significance. Biometrika 1951, 38, 141–149. [Google Scholar] [CrossRef]
Soper, D. 2 × 3 Contingency Table Exact Test Calculator. 2020. Available online: http://www.analyticscalculators.com (accessed on 20 November 2021).
Reiter, M.; Diem, M.; Schumich, A.; Maurer-Granofszky, M.; Karawajew, L.; Rossi, J.G.; Ratei, R.; Groeneveld-Krentz, S.; Sajaroff, E.O.; Suhendra, S.; et al. Automated Flow Cytometric MRD Assessment in Childhood Acute B-Lymphoblastic Leukemia Using Supervised Machine Learning. Cytom. Part A 2019, 95, 966–975. [Google Scholar] [CrossRef]
Denys, B.; Van Der Sluijs-Gelling, A.J.; Homburg, C.; Van Der Schoot, C.E.; De Haas, V.; Philippé, J.; Pieters, R.; Van Dongen, J.; Van Der Velden, V.H.J. Improved flow cytometric detection of minimal residual disease in childhood acute lymphoblastic leukemia. Leukemia 2012, 27, 635–641. [Google Scholar] [CrossRef]
Karawajew, L.; Dworzak, M.; Ratei, R.; Rhein, P.; Gaipa, G.; Buldini, B.; Basso, G.; Hrusak, O.; Ludwig, W.-D.; Henze, G.; et al. Minimal residual disease analysis by eight-color flow cytometry in relapsed childhood acute lymphoblastic leukemia. Haematologica 2015, 100, 935–944. [Google Scholar] [CrossRef]
Bouriche, L.; Bernot, D.; Nivaggioni, V.; Arnoux, I.; Loosveld, M. Detection of Minimal Residual Disease in B Cell Acute Lymphoblastic Leukemia Using an Eight-Color Tube with Dried Antibody Reagents. Cytom. Part B Clin. Cytom. 2019, 96, 158–163. [Google Scholar] [CrossRef]
Tembhare, P.R.; Pg, P.G.S.; Ghogale, S.; Chatterjee, G.; Patkar, N.; Gupta, A.; Shukla, R.; Badrinath, Y.; Deshpande, N.; Narula, G.; et al. A High-Sensitivity 10-Color Flow Cytometric Minimal Residual Disease Assay in B-Lymphoblastic Leukemia/Lymphoma Can Easily Achieve the Sensitivity of 2-in-10 6 and Is Superior to Standard Minimal Residual Disease Assay: A Study of 622 Patients. Cytom. Part B Clin. Cytom. 2019, 98, 57–67. [Google Scholar] [CrossRef]
Van Gassen, S.; Callebaut, B.; Van Helden, M.J.; Lambrecht, B.N.; Demeester, P.; Dhaene, T.; Saeys, Y. FlowSOM: Using self-organizing maps for visualization and interpretation of cytometry data. Cytom. Part A 2015, 87, 636–645. [Google Scholar] [CrossRef]
Ni, W.; Hu, B.; Zheng, C.; Tong, Y.; Wang, L.; Li, Q.-Q.; Tong, X.; Han, Y. Automated analysis of acute myeloid leukemia minimal residual disease using a support vector machine. Oncotarget 2016, 7, 71915–71921. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Conrad, V.K.; Dubay, C.J.; Malek, M.; Brinkman, R.R.; Koguchi, Y.; Redmond, W.L. Implementation and Validation of an Automated Flow Cytometry Analysis Pipeline for Human Immune Profiling. Cytom. Part A 2019, 95, 183–191. [Google Scholar] [CrossRef] [PubMed] [Green Version]

Figure 1. Twinning maturation program. (A) Number of failures by all 22 laboratories in the main sample set (samples 1–25) (B) Number of failures of the 16 laboratories that finished within a range of a maximum of 31 samples (31 samples = mean number of samples needed for maturation).

Figure 2. Correlation of the local MRD estimates of UK NEQAS samples analyzed by I-BFM-Flow laboratories. The target value represents the median of all submitted results. Red dots indicate outliers (i.e., MRD results outside the concordance margin, n = 78). Blue dots indicate concordant results (n = 1601). Positivity was defined as ≥10 MRD events among 3 × 10⁵ nucleated events, i.e., >0.003% (pos, positive; neg, negative).

Figure 3. Regular ring trial tests using the list-mode file sent around reveal a significant performance improvement. LMD Files from 20 B-ALL cases at day 78 or after relapse therapies were sent out to all participating laboratories once a year. Laboratories, which reported false MRD values in >10%–≤25% of cases were flagged with a warning (yellow circle) and reports with >25% false MRD values were considered to be critical (red flag). Total contributing I-BFM labs contained AIEOP-BFM, ALL IC-BFM, and other associated laboratories. Statistical analysis was performed using a 3 × 2 contingency table and two-sided Fisher’s test.

Figure 4. Plot from a representative list-mode file ring trial. RT 2020 comprised 20 regenerative bone marrow samples with either negative or varying levels of MRD. LMD files were sent out to 41 centers, all of which reported results. AIEOP-BFM n = 12, ALL IC-BFM = 19, and other I-BFM cooperating centers = 10. The target MRD value was calculated as a median from six experienced and matured AIEOP-BFM reference laboratories (Vienna, Monza, Padova, Prague, Kiel and Berlin). Two opportunities for educational intervention can be seen in this figure: (i) similar errors by four and six centers, respectively, in samples 14 and sample 4 representing a systematic error; and (ii) an accumulation of errors in single centers (see center 13 and 14) representing lack of experience.

Figure 5. Homogeneity of B-ALL risk classification data of well-trained and matured centers within the AIEOP-BFM ALL 2009 trial. In a survey including nine AIEOP-BFM FLOW reference laboratories from eight countries, independent data of FCM-MRD-based risk classification of B-ALL on day 15 were collected and compared. Statistical analysis was performed using GraphPad Prism 8.3.0. FHR, Flow high risk; FLR, Flow low risk; FMR, Flow medium risk; INS, contains combined data from the two reference laboratories in Israel.

Table 1. Overview of the I-BFM FLOW twinning program.

Overview
Nº of trainee labs ^§	22
Nº of expert trainer labs ^*	7
Nº of samples reviewed	682
Nº of samples required for maturation/lab	25
Median needed [range]	28 (25–57)
Mean needed [SD]	31 (8.8)

^§ Laboratories of the ALL IC-BFM FLOW network; * Laboratories of the i-BFM FLOW network.

Table 2. Twinning maturation program. The twinning maturation program significantly reduces the number of wrong MRD assessments after a first series of training samples.

All 22 Trainee Laboratories
Main Sample Set (Sample 1–25)		Series 1 (Sample 1–13)	Series 2 (Sample 14–25)	Series 3 (Sample 14–pX) ^§
Total Nº of reported results	550	286	264	396
Nº of failures	67	42	25	37
Failures [%]	12.2	14.7	9.5	9.3
All samples (sample 1–57)
Total Nº of reported results	682
Nº of failures	79
Failures [%]	11.6

^§ pX = last sample (sample57 latest).

Table 3. Table summarizing participation and performance of AIEOP-BFM and ALL IC-BFM laboratories in UK NEQAS “Minimal residual disease for ALL by Flow cytometry” ring trials. Data evaluation was completed with each single ring trial by the I-BFM FLOW coordinating center in Vienna. In case of outliers or missing results, the local/individual center was contacted, and corrective measures were discussed. Note that the № of possible results is higher than the number of submitted results, because not all laboratories submitted all results.

Participation
Nº of issued trials ^a	29
Nº of participating laboratories	29–38
Nº of possible results	1925
Nº of submitted results	1679
Performance
	AIEOP-BFM	ALL IC-BFM/others	Total
Nº of submitted results	487	1192	1679
Nº of Outliers ^b	11	67	78
Outliers ^b [%]	2.3	5.6	4.6
	p = 0.0043 ^c

^a Evaluation period: starting with trial number 131403 until 202103; ^b Outliers are definded as reported MRD levels ≥ 3× larger or smaller than the target value; i.e., MRD-level > half a log up or down in a log10-correlation (concordance margin); Target MRD levels: median of all reported values of the sample derived from all I-BFM laboratories submitting a result. ^c Statistics was performed using a 2 × 2 contingency table and Two sided Fisher’s test.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Maurer-Granofszky, M.; Schumich, A.; Buldini, B.; Gaipa, G.; Kappelmayer, J.; Mejstrikova, E.; Karawajew, L.; Rossi, J.; Suzan, A.Ç.; Agriello, E.; et al. An Extensive Quality Control and Quality Assurance (QC/QA) Program Significantly Improves Inter-Laboratory Concordance Rates of Flow-Cytometric Minimal Residual Disease Assessment in Acute Lymphoblastic Leukemia: An I-BFM-FLOW-Network Report. Cancers 2021, 13, 6148. https://doi.org/10.3390/cancers13236148

AMA Style

Maurer-Granofszky M, Schumich A, Buldini B, Gaipa G, Kappelmayer J, Mejstrikova E, Karawajew L, Rossi J, Suzan AÇ, Agriello E, et al. An Extensive Quality Control and Quality Assurance (QC/QA) Program Significantly Improves Inter-Laboratory Concordance Rates of Flow-Cytometric Minimal Residual Disease Assessment in Acute Lymphoblastic Leukemia: An I-BFM-FLOW-Network Report. Cancers. 2021; 13(23):6148. https://doi.org/10.3390/cancers13236148

Chicago/Turabian Style

Maurer-Granofszky, Margarita, Angela Schumich, Barbara Buldini, Giuseppe Gaipa, Janos Kappelmayer, Ester Mejstrikova, Leonid Karawajew, Jorge Rossi, Adın Çınar Suzan, Evangelina Agriello, and et al. 2021. "An Extensive Quality Control and Quality Assurance (QC/QA) Program Significantly Improves Inter-Laboratory Concordance Rates of Flow-Cytometric Minimal Residual Disease Assessment in Acute Lymphoblastic Leukemia: An I-BFM-FLOW-Network Report" Cancers 13, no. 23: 6148. https://doi.org/10.3390/cancers13236148

APA Style

Maurer-Granofszky, M., Schumich, A., Buldini, B., Gaipa, G., Kappelmayer, J., Mejstrikova, E., Karawajew, L., Rossi, J., Suzan, A. Ç., Agriello, E., Anastasiou-Grenzelia, T., Barcala, V., Barna, G., Batinić, D., Bourquin, J.-P., Brüggemann, M., Bukowska-Strakova, K., Burnusuzov, H., Carelli, D., ... on behalf of I-BFM-FLOW-Network. (2021). An Extensive Quality Control and Quality Assurance (QC/QA) Program Significantly Improves Inter-Laboratory Concordance Rates of Flow-Cytometric Minimal Residual Disease Assessment in Acute Lymphoblastic Leukemia: An I-BFM-FLOW-Network Report. Cancers, 13(23), 6148. https://doi.org/10.3390/cancers13236148

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

An Extensive Quality Control and Quality Assurance (QC/QA) Program Significantly Improves Inter-Laboratory Concordance Rates of Flow-Cytometric Minimal Residual Disease Assessment in Acute Lymphoblastic Leukemia: An I-BFM-FLOW-Network Report

Abstract

Simple Summary

Abstract

1. Introduction

2. Materials and Methods

2.1. Samples

Sample Preparation

2.2. I-BFM-FLOW Twinning and Maturation Program

2.3. External Quality Assessment (EQA)

2.4. List-Mode-Data File Ring Trials (LMD Ring Trial)

2.5. Independent Data Survey

2.6. Statistical Analysis

3. Results

3.1. Twinning Maturation Program

3.2. External Quality Assessment

3.3. List Mode Data File Ring Trials (LMD Ring Trial)

3.4. Independent Data Concordance: A Retrospective Quality Control Measure

4. Discussion

5. Conclusions

Supplementary Materials

Author Contributions

Funding

Institutional Review Board Statement

Informed Consent Statement

Data Availability Statement

Acknowledgments

Conflicts of Interest

References

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI