Behaviour and the standardization fallacy

Behavioural phenotyping of mouse mutants has become an important tool in the search for biological roles of single gene products in complex mammalian traits, such as memory, aggression, anxiety and addiction (see page 277 for more on the genetics of addiction¹). Because specific strains of mice, husbandry practice and test protocol may influence behavioural effects of mutations in unpredictable ways, scientists strive for consensus on rigorous standards to maximize reproducibility of results across laboratories². Increasing reproducibility of results through standardization, however, accentuates and obscures the very problem—that of reporting artefacts that are idiosyncratic to particular circumstances.

Standardization serves to reduce individual differences within study populations (within-experiment variation) in order to facilitate detection of treatment effects, and to reduce differences between studies (between-experiment variation) in order to maximize reproducibility of results³. If standardization were fully effective, inter-individual variation within study populations would decrease to zero, and reproducibility of effects between experiments would increase to 100%. At the same time, however, each experiment would turn into a single-case study with zero information gain. This is what I here refer to as the standardization fallacy: the increase of reproducibility at the expense of external validity.