Rat feeding studies with genetically modified maize - a comparative evaluation of applied methods and risk assessment standards

A 2-year rat feeding study with genetically modified NK603 maize sparked an international scientific and public debate as well as policy responses by the European Commission. The European Food Safety Authority (EFSA) evaluated the study as defective based on conceptual and methodological shortcomings by retroactive application of the recommendations of its recent guidance on 90-day feeding studies. Our comparative analysis of the three relevant NK603 publications, including a 90-day feeding study of Monsanto, showed that all of them satisfy or fail to satisfy the EFSA evaluation criteria to a comparable extent; the rejection of only one of the papers is, thus, not scientifically justified. We also show that EFSA's criteria are not standard practice in 21 other rat feeding studies lasting at a minimum of 12 months. The review reveals critical double standards in the evaluation of feeding studies submitted as proof of safety for regulatory approval to EFSA. We specifically argue that the current approach to declare statistically significant differences between genetically modified organisms and its parents as ‘biologically irrelevant’ based on additional reference controls lacks scientific rigor and legal justification in the European Union (EU) system. Only recently, the EU authorities started building up an implementing system based on its own legislation and supportive of the EU approach to risk assessment in the context of technology assessment. Until these issues are resolved, we do not expect that neither the public nor the scientific debate will subside.

Introduction

In 2007, Séralini et al. [1] re-analyzed the data of a 90-day rat feeding trial conducted by Monsanto Company and submitted to the European Food Safety Authority (EFSA) to demonstrate the safety of their genetically modified (GM) MON863 maize for food and feed import in the European Union (EU). In their re-analysis, the authors stated that the data generated by Monsanto Company revealed signs of hepatorenal toxicity. The authors concluded - in disagreement with Monsanto Company and EFSA - that ‘Longer experiments are essential in order to indicate the real nature and extent of the possible pathology.’ and that ‘With the present data, it cannot be concluded that GM corn MON863 is a safe product.’. This work sparked a controversial debate in the public and in the political, regulatory, and scientific communities regarding the disputed safety of that particular GM crop and GM food and feed testing in general. The controversy eventually elicited a policy response of the European Commission's Directorate-General for Health and Consumer Safety. In February 2013, an implementing regulation on applications for authorization of GM food and feed was adopted, which prescribes that for future GM crop applications, 90-day feeding trials are mandatory [2].

Meanwhile, Séralini et al. [3] published a report about a 2-year toxicology study with rats testing Monsanto's NK603 glyphosate-tolerant GM maize and its Roundup co-technology. The authors stated that female rats in the treated group died earlier and more often than in the control group. Treated female rats also developed mammary tumours earlier and more often than control female rats. Treated male rats showed higher levels of liver congestion and necrosis as well as kidney nephropathies. The authors underlined that 76% of altered biochemistry parameters were kidney related. The Séralini study elicited an immediate public debate showing a wide range of opinions (which also precipitated in scientific journals, e.g. [4–6]). As a reaction to this debate, the European Commission decided to provide funding for an own 2-year study on NK603 maize [7].

In this article, we evaluate the first response [8] by the EU risk assessment body, EFSA, and the voiced criticism that focuses on the underlying methodological issues. In our analysis, we will leave out the critical issue of applied statistical evaluations of the feeding trial, as this requires other expertise.

EFSA [8] assessed the study [3] with regard to five criteria: study objectives, study design, feed and treatment formulation, statistical methods, and endpoint reporting. These criteria are derived from the recently published ‘EFSA guidance on conducting repeated-dose 90-day oral toxicity study in rodents on whole food/feed’ [9]. EFSA underlined in its accompanying press release that their analysis focused on the methodology rather than on the outcomes of the Séralini publication. EFSA concludes that the study results are inadequately presented with omission of key details of the design, conduct, analysis, and reporting. Due to its insufficient scientific quality the study would be of no relevance for risk assessments of neither glyphosate nor NK603 maize.

Due to the increasing scientific and public interest, EFSA decided to make available all documents related to the NK603 risk assessment and approval on the internet [10, 11]. Before, access to these documents was only possible upon individual request for information to EFSA. This still holds true for all other approval applications of other GM organisms (GMOs). Included in the NK603 dossier is the Monsanto Technical Report MSL-17555 [12] describing their 90-day rat feeding study and including the raw data. For its second and final response, EFSA [13] primarily added reactions of EU Member States but did not expand or change the substantive issues published previously.

In order to alleviate this result-triggered, selective application of evaluation criteria and offer an objective evaluation of the applied methodologies in other feeding trials regardless of their outcomes, in particular those on which EFSA based its safety conclusions, we carried out a comparative analysis with the following objectives:

1. 1.

   To evaluate how Monsanto's technical study [12] and the subsequent peer-reviewed publication of that study's data by Hammond et al. [14], as part of the application for EU wide regulatory approval of NK603 maize met the criteria EFSA applied so far only to the Séralini et al. study [3]. We also include the reply by Séralini et al. [15] on the voiced criticism, which gives additional information on study details.

2. 2.

   To evaluate the degree of rigor and scrutiny EFSA applied in its assessment of the Monsanto study [12] and Hammond et al. [14] as elements of the NK603 risk assessment procedure. The two scientific opinions of the EFSA Panel on Genetically Modified Organisms on NK603 [16, 17] serve as the basis for our analysis.

3. 3.

   To evaluate how 21 other long-term, peer-reviewed rat feeding studies for comparable purposes but testing other substances would fulfil EFSA's criteria. These additional studies were selected for their methodological criteria being similar to the Séralini study, e.g. a duration of 12 months or longer, using SD rats and performing toxicology sometimes in combination with carcinogenicity tests.

Comparative analysis of the application of the EFSA criteria

The three NK603 publications discussed by us were written - and hence, experiments were planned - before EFSA published its 2011 Guidance on 90-day feeding trials. While the retrospective application of new standards (i.e. study methods) as undertaken by EFSA [8] is debatable, it is broadly accepted that risk assessment informing regulatory decision-making should be based on the latest state of knowledge and standards. A re-evaluation of the applied methods of relevant, older studies is justified if applied to all such relevant studies - but meaningless or even counterproductive when applied selectively to one study or a subset of studies only triggered by their reported (here adverse) effects. We consider such a selective re-evaluation as inappropriate. We note that the main responsibility for EFSA's recent work lies with the European Commission that in its mandate to EFSA singles out the Séralini study for review [17].

EFSA criterion: study objectives

EFSA [8] notes with regard to Séralini et al. [3] that the ‘study objectives are unclear’. EFSA further clarifies that ‘If a specific guideline is chosen and followed, then the objectives are inherently defined in the guideline.’ To clarify what ‘inherently defined’ objectives entail, we analyzed the four Guidelines of the Organization for Economic Co-operation and Development [18–21] discussed in this context (OECD nos. 408, 451, 452, 453, respectively). While the three latter Guidelines revised in 2009 give clear objectives, Guideline 408 on 90-day oral toxicity originally adopted in 1981 and revised in 1998 lacks such clarity. Guideline 408 is the guideline of choice by Monsanto Company for their 90-day rat feeding trial.

Open issue according to EFSA: ‘The study objectives need to be clearly stated a priori in the study protocol.’

Séralini et al. [3] stated that ‘Guideline 408 of the Organization for Economic Co-operation and Development (OECD) was followed by some manufacturers for GMOs even if it was not designed for that purpose. We have explored more parameters and more frequently than recommended in this standard in a long-term experiment.’ The authors explained that they added biochemical and haematological measurements using 10 rats per group as described in the OECD Guideline 453. This apparent combination of elements of two OECD guidelines and the addition of more parameters did not qualify as proper description of the objectives of the work by EFSA.

In its technical 90-day study, Monsanto tested a low-dose group (LD 11% NK603) and a high-dose group (HD 33% NK603) versus two parental control groups and six non-parental, so-called reference controls for 13 weeks. Blood and urine biochemical parameters, amongst others, were measured at an interim time point after 4 weeks and at termination of the study. The statements by Monsanto [12] and later by Hammond et al. [14] to have adapted the OECD Guideline 408 were considered by EFSA to be sufficient. Neither the nature nor the consequences of the adaptations made to the requirements of OECD Guideline 408 were explained in these two publications or in the scientific opinions of EFSA Panel on Genetically Modified Organisms [16, 17].

Since OECD Guideline 408 [18] in contrast to the more recent OECD guidelines does not provide clear objectives, it is doubtful whether a mere reference to this guideline as provided by Monsanto suffices EFSA's new standards on study objectives. We conclude that regarding this criterion, the Séralini publication provides at least the same quality in terms of clarity and explanation as the Monsanto and Hammond studies.

EFSA criterion: study design

The study design applied by Séralini et al. [3] triggered two main criticisms by many commentators including EFSA [8]: (a) the number of rats per group and (b) the choice of the strain of rats.

1. A.

   EFSA [8] states: ‘[g]iven that Séralini et al. (2012) conducted a two-year study, it is unclear why an OECD guideline suitable for a 2-year chronic toxicity or carcinogenicity study (i.e. OECD 451, OECD 452, or OECD 453) was not adhered to.’ Séralini et al. [3] used 10 rats per group as recommended by OECD Guideline 453 [19] for the chronic toxicity phase. But OECD Guideline 453 also clearly states that this lower number is only justified when a carcinogenicity phase with 50 rats per group is run in parallel to support the interpretation of the toxicity data. Later, Séralini et al. [15] argued that the OECD Guideline 452 in its 1981 version [22], which was applicable for the 2-year toxicology study conducted from 2008 to 2010, advised to only use 10 rats out of the 20 per group for blood and urine sampling and analysis. The recommendation to use at least 10 rats per group for biochemistry analysis at the chosen time points is maintained in the current version of Guideline 452.

Monsanto [12] used 20 rats per sex and group, while the actual values for blood chemistry, haematology and qualitative, and quantitative urine analyses were based on samples from 10 rats per sex and group, thus, comparable to the approach of Séralini et al. [3]. Nowhere in the two versions of Guideline 452 or in the Monsanto study, a method for how to select the 10 animals out of the 20 animals per group for the biochemistry tests is specified.

1. B.

   Regarding the choice of SD rats, EFSA notes that this “strain of rats chosen is known to be prone to development of tumours over their life […] This is neither taken into account nor discussed in the Séralini et al. (2012) publication.” A discussion on the choice of rats with regard to their tendency to develop tumours is not relevant for 90-day studies. Nevertheless, EFSA recommends stating reasons for the choice of the strain in its new guidance on 90-day feeding trials.

Open issue according to EFSA: ‘The biological relevance of the rat strain used should be justified with respect to the design choices.’

Neither Séralini et al. [3] justify the choice of SD rats nor do Hammond et al. [14]. Monsanto [12] explains that ‘[t]he rat was selected for the study since this species has been traditionally used to assess the safety and wholesomeness of food.’ This pragmatic explanation underlines the customary use of SD rats in such trials which of course extends also to the Séralini study but does not give a scientifically sound justification for its use with regard to the design choice as implied by EFSA's wording [8].

In (Additional file 1: Table S1), we present information on the actual use of SD rats in peer-reviewed, long-term studies and the types of information researchers usually provide and scientific journals require on the choice of animals. According to the information available, SD rats were chosen as standard test organisms by the two largest toxicity/carcinogenicity research projects worldwide and at least 21 long-term studies:

1. 1.

   The National Toxicology Program of the US Department of Health and Human Services uses this strain in its 2-year studies, after in-depth discussions on the suitability and advantages of the SD rat over previously used strains [23].

2. 2.

   The European Ramazzini Foundation for Oncology and Environmental Sciences (Italy) uses SD rats in its Ramazzini Foundation Cancer Program since more than 40 years [24].

Contrary to many media reports based to a large extent on scientists' personal opinions posted by the Science Media Centre [25], SD rats are used routinely in long-term toxicology and carcinogenicity studies. The rational to do so is that (additional) carcinogenic effects of test substances can be detected more effectively in rats that develop tumours at a relevant rate within the time span of a 2-year trial. Actually, the prime issue of an informed debate would not be so much the choice of the strain of rats but the choice of the numbers of rats depending on the strain used. The OECD Guideline 116 [26] on the conduct of long-term carcinogenicity studies states that ‘[f]or strains with poor survival such as Sprague–Dawley rats, higher numbers of animals per group may be needed in order to maximize the duration of treatment (typically at least 65/sex/group).’ The US Food and Drug Administration (US FDA) [27] recommends for carcinogenicity tests with strains with known problems in survivorship that the researchers need to ensure that at least 25 rats from those used initially remain alive at the end of the 2-year trial. For long-term toxicity trials as conducted by Séralini et al. [3], no specific recommendations are given by the US FDA. With regard to the recommended number of 10 rats per toxicological test in OECD Guideline 452 [20], the above recommendation of FDA should be applied to ensure that in average, more than 10 rats would be available for terminal testing. The reported survival rates of the control rats in Séralini et al. [3] (70% for males, 80% for females) lie at or above the upper range presented in available literature on survival rates of SD rats in 2-year studies [28–31].

We conclude that with regard to the choice of rats, a pragmatic approach was followed by Monsanto, Hammond et al. [14] and also Séralini et al. [3]. Since the Séralini group had set out to repeat the 90-day feeding trial by Monsanto but to differ with regard to the extended testing time and parameters measured, the choice of rat was appropriate in order to allow for comparability with regard to this factor. We also conclude that the SD rat strain used is a standard organism for toxicity and carcinogenicity tests. A rejection of the validity of the Séralini study based on the choice of rats would therefore either be based on a lack of familiarity with the scientific field. Or these criteria would have to apply to all carcinogenicity trials conducted to date with this rat strain and, thus, leave us with hardly any valid carcinogenicity study. Criticism of the Séralini study based on the lack of a scientific explanation of the choice of the rat strain would similarly apply to the Monsanto and Hammond publications.

Open issue according to EFSA: ‘Suitable controls for all treatment groups are not present.’

None of the discussed OECD guidelines was developed for the testing of whole foods but rather for testing the toxicity of isolated chemicals when feeding rats with standardized, nutritionally balanced feed. In most cases, the test substance is administered separately from the normal feed or mixed with the feed in a minor volume. As long as the addition of the test substance to the feed does not alter significantly the relative concentration of the feed components, a single control with zero test substance is sufficient. If the test substance is an inherent component of the feed as in the NK603 feeding trial and the different test concentrations are produced through varying percentages of GM and non-GM maize varieties in the feed, appropriate controls are necessary according to EFSA [9]. Hammond et al. [14] and Monsanto [12] report that all test groups received feed containing 33% maize with a different percentage of GM versus non-GM maize. Séralini et al. [3] did not report whether they had adjusted the total maize content to 33% in those batches containing 11% or 22% GM maize. To clarify, Séralini et al. [15] informed that all feed batches were mixed to contain 33% maize in total.

We conclude that based on the information given by Séralini et al. [15], this open issue is closed.

Open issue according to EFSA: ‘Measures taken to reduce the risk of bias (e.g. blinding) are not reported.’

While EFSA [8] has noted that Séralini et al. [3] did not report on measures taken to reduce bias in their experiments, we note that also neither Hammond et al. [14] nor Monsanto [12] reported such measures. Blinding measures are not requested by any of the above-mentioned OECD guidelines. However, Séralini et al. [15] explained that anatomopathology and biochemical measures were indeed performed in a blinded manner.

The issue of blinded evaluation was promoted by EFSA when it presented its draft [32] of the above-mentioned Guidance on a 90-day oral toxicity study for public consultation. EFSA's approach was rejected by many commentators from pathology sciences and industry [33] with reference to the ‘Best Practices Guideline: Toxicologic Histopathology’ of the Society of Toxicologic Pathology[33]. This guideline explains that unblinded evaluation is regarded as the appropriate approach in the first round of pathological observations because it ‘allows the pathologist to intensely focus the histopathologic evaluation and to find important, and sometimes subtle, differences between the tissues of treated and untreated animals’. The guideline recommends that blinded evaluations should be applied in the re-evaluation of findings in specific tissues and when samples with a defined spectrum of lesions due to a known toxic syndrome are investigated. These recommendations to not apply blinding as a general measure were reflected in EFSA's report on their public consultations [34] and in the final EFSA Guidance on 90-day oral toxicity study. It is unclear why EFSA [8] requested blinding measures in the context of the Séralini study but did take no issue of this in its positive opinions on the NK603 dossiers including Monsanto's rat feeding studies.

We conclude that a double standard was applied with regard to the criterion of blinding. We also conclude that EFSA’s approach does not reflect the previously published opinions of various experts in the field.

EFSA criterion: feed and treatment formulation

Open issue according to EFSA: ‘The appropriateness and comparability of the diets cannot be assessed as critical information about their composition is not reported.’

Séralini et al. [3] did not present their data on the composition of the feed but informed that ‘all feed formulations consisted in balanced diets, chemically measured as substantially equivalent except for the transgene’. Similarly, Hammond et al. [14] did not present such data but mentioned that all diet preparations were analyzed to confirm that they met the supplier's specifications for certified 5002 rodent diet. Monsanto [12] presented detailed data on the composition of the feed, as it is standard procedure for technical studies for regulatory feeding trials.

We conclude that the two publications by the Hammond and Séralini groups do not fully comply with this criterion while both studies mention that the composition of the different feed was measured and found to be comparable. A rejection of only one study due to the lack of reporting of data would constitute a double standard.

Open issue according to EFSA: ‘The stability of the diets cannot be assessed as details of their storage conditions are not provided.’

EFSA [8] noted that Séralini et al. [3] did not inform about the storage conditions of the feed - likewise, neither Hammond et al. [14] nor Monsanto [12] informed about this. Consequently, the stability of the diets cannot be assessed in any of the studies submitted by the developers to EFSA, which went uncommented in EFSA’s positive opinions on NK603 maize [16, 35].

Open issue according to EFSA: ‘It is impossible to evaluate whether or not there was any contamination of the diets, e.g. by mycotoxins, as it is not reported.’

Although Séralini et al. [3] did not report about mycotoxin measurements in their original publication, later, the authors [15] stated that all measured mycotoxins showed concentrations below the recommended threshold values for food and feed. The EU maximum level is 4 ppb for e.g. the sum of Aflatoxines B1, B2, G1, and G2 in processed maize for human consumption and 20 ppb or lower for Aflatoxine B1 in feed materials. Similarly, Hammond et al. [14] informed that the levels of aflatoxins were below detection limits but did not provide the actual individual measurements or raw data. Monsanto [12] did not refer to aflatoxins alone but to the measurement of a total of 19 different mycotoxins. The author noted that there was no contamination with mycotoxins that might interfere with the results. Of the 19 analyzed mycotoxins, 4 (Deoxynivalenol, Fumonisin B1, B2, B3) could be detected in individual concentrations up 1.6 ppm in the various maize samples. The concentration did not exceed 3 ppm in the formulated diet, which Monsanto [12] regards as the No Observed Effect Level for Fumonisin B1 in rats according to Voss et al. [36]. The EU maximum level for Fumonisin B1 and B2 lies at 60 ppm for maize as feed material while the EU maximum level for maize products for human consumption is 1 ppm or lower.

Open issue according to EFSA: ‘The amount of residues of glyphosate and its metabolites in treated maize NK603 is not reported.’

Séralini et al. [3] did not present the results of their glyphosate measurements but stated that pesticide content did not exceed standard limits. The current EU maximum residue level for glyphosate in maize is 1 ppm. Hammond et al. [14] noted that the levels of glyphosate were below detection limits but also did not give any data. In contrast, Monsanto [12] informed that glyphosate was indeed measurable: ‘Glyphosate residue in the test grain (0.09 ppm) was slightly above the analytical detection limit of 0.05 ppm.’ It is very likely that the NK603 maize used as feed was treated with a glyphosate-containing herbicide during its growth. In order to address the presence of two factors relevant for risk assessment - the transgene and the complementarily used herbicide - information on the use of glyphosate and its residues should be presented in research relevant for risk assessment [37].

Open issue according to EFSA: ‘The exposure to GMO, GMO + R, and R cannot be evaluated since the food and water intakes of the GM- and R-treated groups, respectively, are not clearly reported.’

Séralini et al. [3] did not present the values of feed and water intakes (both containing the test substances) - but also Hammond et al. [14] did not present feed intake data (only feed contained the test substance). Monsanto [12] presented feed intake data, as this is standard procedure for technical studies for regulatory feeding trials. However, this data still cannot be used to quantify the exposure to the CP4 EPSPS protein because Monsanto [12] did not quantify the concentration of CP4 EPSPS protein in the maize kernels used to mix into the feed. Monsanto [12] only conducted qualitative ELISA tests to confirm the presence or absence of the CP4 EPSPS protein. The impossibility to calculate exposure data based on the information given by Monsanto [12] was not flagged by EFSA in its two positive opinions on the NK603 dossier.

While the presentation of data on concentrations of contaminants is certainly useful, a rejection of the validity of the Séralini study based on the fact that only statements on such concentrations are given is debatable. And again, we conclude that applying a criterion to only one study would constitute a double standard in all four open issues discussed above.

EFSA criterion: endpoint reporting

Open issue according to EFSA: ‘All collected endpoints should be reported openly and transparently.’

The reporting on endpoint data is an essential feature of regulatory studies. Monsanto [12] delivers all data as is required for such regulatory studies amounting to 1,180 pages supplementary information, most of them containing raw data. Obviously, such amount of data is unpublishable in any scientific journal. Therefore, Hammond et al. [14] provide means and standard deviations only and did not present the urine chemistry values. Séralini et al. [3] did not present comparable means and standard deviations summarizing the raw data, but more complex statistical evaluations of some selected data sets. Séralini et al. [15] announced that more data would be published in following publications.

The tenth open issue listed by EFSA actually shows a profound difference between the Séralini study and the two others. We conclude that the approach of the Séralini study deviates from general practice with regard to the reporting of results of such feeding studies; more information should be made available.

Conclusions of the comparison of three feeding trials testing NK603 maize

Our analyses as summarized in Table 1 revealed that the major difference between the studies of the French group and Monsanto lies in the presentation of the endpoint data, one of the contentious issues also in the public scientific debate. In January 2013, the French group has deposited their raw data at a notary and will give access to them if Monsanto's raw data on NK603 and glyphosate safety are accessible publically as well [38]. Shortly after this announcement, EFSA has enabled open access to the NK603 files at its web page [11]. Unfortunately, data on safety tests with glyphosate remain confidential, as it is the general rule for pesticide data.

For all other methodological criteria established by EFSA, we only found differences between the two peer-reviewed publications that did not exceed the normal range of differences exhibited by all published experiments that study a similar issue with similar protocols, including those considered for regulatory purposes. As expected and required, Monsanto's technical study presents more details than the two publications. These standards cannot be met by scientific publications as journals do not allow for publications exceeding several hundreds of pages of technical details. However, even in this extensive technical study, a whole range of criteria requested and applied by EFSA to the Séralini study and deplored as insufficient, were likewise absent or incomplete.

In conclusion, we note that applying the evaluation criteria to Séralini et al. [3] only but not to Monsanto [12] and Hammond et al. [14] constitutes a double standard.

Comparative analysis of other long-term rat feeding studies

In a literature search effort, we found 21 peer-reviewed long-term studies with SD rats published in the last 20 years (Belpoggi et al. [39]; Bornhard et al. [40]; Butenhoff et al. [41]; Gámez et al. [42]; Gutiérrez et al. [43]; Hack et al. [44]; Holmberg and Eckstöm [45]; Johannsen and Levinskas [46]; Klimisch et al. [47]; Lee et al. [48]; Liang et al. [49]; Minardi et al. [50]; Morcos and Camilo [51]; Perricone et al. [52]; Perri et al. [53]; Quast [54]; Soffritti et al. [55]; Soffritti et al. [56]; Soffritti et al. [57]; Soffritti et al. [58]; Voss et al. [59]). The aim of our analysis of these studies was to find out whether the set of EFSA criteria with its 10 requirements is met by generally applied standards in scientific toxicology and carcinogenesis research. Our compilation revealed that the EFSA criteria and requirements applied to the Séralini study are hardly fulfilled by any of these publications (for details, see Additional file 1: Table S1). The only requirement that is consistently fulfilled by the 21 peer-reviewed publications is the endpoint reporting in the form of means and standard deviations.

The other requirements were only met by very few studies. Only one publication mentioned the OECD guideline the researchers adhered to in their experiments [42]. The choice of the strain was never discussed in scientific publications as required by EFSA. Only Butenhoff et al. [41] deliberated on the characteristics of SD rats in the context of comparing their results with studies using other strains. Only Liang et al. [49] applied a blinded evaluation in their histological analysis as asked for by EFSA. Butenhoff et al. [41] had chosen a blinded evaluation for their re-assessment of existing histopathological probes. None of the 21 studies reported about storage conditions or presented power calculations.

An interesting point with regard to the scientific and public discourse is the number of rats used in these peer-reviewed experiments. While public and scientific commentators, including EFSA [9], obviously assume that all animals tested per group and sex are in the end also used for analysis of e.g. biochemical parameters, the applicable OECD guidelines and several of the screened publications apply a different approach. As mentioned previously, the OECD Guideline 452 recommends the use of 10 out of the 20 rats per group for biochemical analysis. As indicated in (Additional file 1: Table S1) 9 of the 12 publications that analyzed biochemical parameters in the toxicology study also used 10 or fewer rats in some or all tests.

EFSA reactions regarding the three NK603 studies

Monsanto [12] reported 77 comparisons in which statistically significant differences between the NK603 treatments and the different controls were measured (for details, see Additional file 2: Table S2). The authors considered 71% (55) of the statistically significant differences between treatments and controls as either ‘not biologically significant’ (19), ‘not biologically relevant’ (28), ‘not biologically meaningful’ (6), or ‘not toxicologically significant’ (2). Seventeen of the remaining 22 comparisons showed a significantly higher food consumption of the HD 33% NK603 maize group. Monsanto [12] noted that ‘[i]n total, some 1,050 comparisons were made and approximately 53 of these were anticipated to be significant by chance alone at the 5% significance level.’ It remains unclear if the author intended to suggest that these remaining 22 significant differences were produced by mere chance and, thus, could also be regarded as irrelevant. The author's decisions are based on conceptual and methodological arguments, no peer-reviewed science is provided to underpin the argumentation and categorization.

In its first opinion - based on Monsanto [12] - EFSA [35] did neither report about the 77 statistically significant differences nor about the explanations of Monsanto and the underlying norms and standards to declare 55 of them as irrelevant. In fact, no agreed EU norms and standards existed in 2003 that EFSA could have used to justify such a categorization. EFSA also did not comment on the remaining 22 comparisons that Monsanto did not declare explicitly as insignificant but did so implicitly by assigning them to pure chance only and, thus, did forego any further analysis as to their quality. EFSA declared that only two comparisons - average corpuscular volume and average corpuscular haemoglobin in female rats in the high-dose group - exhibiting ‘slightly elevated levels’ showed ‘consistent differences’ (Table 2). It remains unclear why only these two ‘slightly’ differing comparisons out of 77 significant differences were singled out and much more pronounced differences recorded, for example, in the group of haematology values went uncommented by EFSA.

In its second opinion, EFSA [16] based its essentially unchanged conclusions on Hammond et al. [14]. This is surprising because the authors state explicitly that there are no statistically significant differences anymore. The reason for this complete loss of the statistical significant differences still reported by Monsanto [12] is due to a new approach of dealing with the raw data. In contrast to Monsanto [12], two modifications were applied: (1) the only comparator in Hammond et al. [14] are the mean values of all six reference controls but not of the parental control and, (2) additionally, the double standard deviation is used as upper and lower threshold to determine the statistical significance. This new statistical approach - uncommented by EFSA - resulted in eliminating all statistically significant differences as still reported by Monsanto [12]. No scientifically documented justification for this approach to eliminate statistical differences was provided by Monsanto nor requested by EFSA.

In conclusion, with regard to the provisions of the EU laws on GM organism risk assessment and approval, we argue that a comparison of a GM plant only to genetically unrelated reference controls excluding the unmodified isogenic parents is not in compliance with EU law and guidelines on GMO risk assessment. Based on the above reported outcomes of our comparative analysis, we note a striking double standard in rigor applied in the evaluation of the submitted rat feeding trials conducted and submitted by the developer to EFSA in comparison to the rigor applied in the evaluation of the Séralini study.

The need for appropriate regulatory sciences

Of prime importance and long overdue is an inclusive discussion on the appropriate testable, scientific hypotheses and methodologies that are supportive of the provisions and tasks formulated by the EU legal framework on risk assessment of GM organisms. The EU law is based on the axiom that risks, which may be caused by GM organisms, cannot be a priori-deducted from experience with existing organisms. Due to their genomic and evolutionary novelty, specific laws regulate the testing in the open environment and the market approval process of GM organisms since 1990. Therefore, the EU Directive 2001/18 on the deliberate release into the environment of genetically modified organisms clearly states as a ‘General Principle’ for environmental risk assessment that ‘identified characteristics of the GMO and its use which have the potential to cause adverse effects should be compared to those presented by the non-modified organism from which it is derived and its use under corresponding situations’. Nowhere does this Directive or the Regulation (EC) No 1829/2003 on genetically modified food and feed suggest that GM organisms also need to be compared to unrelated control organisms in the context of risk assessment (see Regulation 1829/2003 Art. 4 1. and Art. 5 3. (e)). The evaluation of characteristics of a GM organism with regard to the ‘accepted limits of natural variations’ - requiring data from unrelated reference varieties - is only mentioned in Art. 5 3. (f) in the context of labelling but not risk assessment.

We argue that the concept of substantial equivalence or the concept of familiarity, originally developed in the context of the US regulatory system for GM organisms and based on opposite assumptions as the EU regulatory system [60], is not per se applicable in the EU context. One element of this US approach is the additional use of unrelated varieties as reference controls to establish the natural variation against which GM food products need to be evaluated. While this approach was taken up in a report by the World Health Organisation [61] and in a technical risk assessment guidelines of the Codex Alimentarius Commission [62], it failed to become recognized in the more general Codex Alimentarius Commission Principles 44–2003 [63] or in the international legally binding Cartagena Protocol on Biosafety [64]. In practice, adding genetically unrelated controls will increase the unspecific background variability in the data and make the statistical detection of specific differences due to the insertion of the transgenic trait more difficult or impossible.

We conclude that the actual effect of introducing the use of unrelated reference controls lies more in providing an instrument to declare technology-induced significant differences as irrelevant against the background of unspecified ‘noise’ (stemming from broad, unrelated natural variation) than in setting up a sound methodology for risk assessment under the applicable legal framework. While over the past years, no norms, standards, and scientific criteria for the concept of familiarity and its core instrument of reference controls were developed, it is still at the discretion of applicants and authorities if and how to use the respective test results.

The non-scientific character of this approach can be illustrated through its use by Monsanto [12]. Monsanto selected six maize varieties as reference controls ‘selected to represent a range of growing environments and a diversity of germplasm’. Monsanto does not prove that the selected varieties and the growing areas are indeed representative for the USA maize production and subsequent exposure of animals and humans to maize products. Such demonstrated representativeness would be essential to justify the use of reference control as means to declare biological irrelevance in regulatory science, as for example suggested by the Codex Alimentarius Commission guidelines [62]. EFSA accepted the approach by Monsanto [12] without requiring proof for representativeness or any other robust scientific justification of the use of these reference controls. In order to obtain some information about the six varieties, we performed an internet search. As shown in Table 3, no relevant information could be found for three of the used varieties. Two of them obviously had some level of importance ten or more years ago. Only for Pioneer 3394 maize, we were able to retrieve information that showed high relevance of this variety for the US maize agriculture in the 1990s. Based on this search, we conclude that it is doubtful whether the chosen six varieties were a representative sample for the US maize when the Monsanto rat study was performed. In any case, it can safely be concluded that the six varieties were of no relevance for EU maize agriculture when EFSA accepted the study in 2003 and 2009. Biological irrelevance of many statistically significant differences between NK603 maize and its unmodified parents was declared without any scientific justification or experimental proof.

The recent controversy focused on the validity of the study published by Séralini et al. [3] as adequate evidence for questioning the safety of NK603 maize and its glyphosate-based co-technology. In our comparative analysis based on four out of five criteria of EFSA, we found the studies of Monsanto [12] and Hammond et al. [14] submitted as evidence for proving the safety of NK603 maize to suffer from comparable deficiencies. We did find critical double standards in acceptance and rigor of the evaluation of feeding studies submitted as proof of safety for regulatory approval to EFSA. We further conclude that during 23 years of EU regulation for GM organisms and various GM crop risk assessments and approvals, regulatory sciences and methodology are used that were developed under the contrasting US regulatory concept. It is this mismatch between these different legal frameworks and the choice of risk assessment methodology that we see to be the basis of the often-stated public mistrust in the EU authorities. Only recently, the EU authorities started to build up an implementing system that takes its directives from the current EU legislation and is supportive of its own approach. Concerns about the intransparent and arbitrary use of reference controls as instrument to state biological irrelevance of significant differences between GM organisms and its parents must be addressed in these ongoing activities. Activities of prime concern in this regard is the GMO Risk Assessment and Communication of Evidence (GRACE) Project [65] developing protocols for 90-day rat feeding trials with GM plants and certainly influencing the suggested development of respective 2-year testing protocols with rats. But until now, the GRACE Project adheres to the intransparent use of reference varieties. Until the scientific questions with regard to appropriate risk research and assessment are resolved and the application of double standards has ceased, we do not expect that neither the public nor the scientific debate will subside.

CP4 EPSPS:

5-enolpyruvylshikimate-3-phosphate synthase of agrobacterium strain CP4

HD:

High dose

LD:

Low dose

SD:

Sprague–Dawley (rats).

The authors thank several ENSSER members for initial discussions and valuable comments.

Authors and Affiliations

1. European Network of Scientists for Social and Environmental Responsibility (ENSSER), In den Steinäckern 13, 38116, Braunschweig, Germany

   Hartmut Meyer & Angelika Hilbeck

2. Swiss Federal Institute of Technology, Institute of Integrative Biology IBZ, Universitätstrasse, 16, 8092, Zurich, Switzerland

   Angelika Hilbeck

Corresponding author

Correspondence to Hartmut Meyer.

Competing interests

The authors declare that they have no competing interests. The work of HM was financially supported by ENSSER. ENSSER did not have any role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.

Authors’ contributions

HM carried out the analysis of the documents submitted to EFSA by Monsanto with its application for approval of NK602 maize for food and feed purposes. HM and AH wrote the review. Both authors read and approved the final manuscript.

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions