Abstract
Among classical bibliometric indicators, direct and relative impact measures for countries or other players in science are appealing and standard. Yet, as shown in this article, they may exhibit undesirable statistical properties, or at least ones that pose questions of interpretation in evaluation and benchmarking contexts. In this article, we address two such properties namely sensitivity to the Yule-Simpson effect, and a problem related to convexity. The Yule-Simpson effect can occur for direct impacts and, in a variant form, for relative impact, causing an apparent incoherence between field values and the aggregate (all-fields) value. For relative impacts, it may result in a severe form of ‘out-range’ of aggregate values, where a player’s relative impact shifts from ‘good’ to ‘bad’, or conversely. Out-range and lack of convexity in general are typical of relative impact indicators. Using empirical data, we suggest that, for relative impact measures, ‘out-range’ due to lack of convexity is not exceptional. The Yule-Simpson effect is less frequent, and especially occurs for small players with particular specialisation profiles.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aksnes, D., Sandström, U. (2006), National citation indicators, their methodological foundation and political interpretation, Book of Abstracts, 9th International Conference on Science & Technology Indicators, September 2006, Leuven, Belgium, pp. 3–5.
Balassa, B. (1965), Trade liberalization and ‘revealed’ comparative advantage. Manchester School, 33: 99–123.
Braun, T., Glänzel, W., Schubert, A. (1985), Scientometric Indicators. A 32 Country Comparison of Publication Productivity and Citation Impact. Singapore: World Scientific Publishing Co.
Brusoni, S., Geuna, A. (2004), Specialisation and integration: Combining patents and publications data to map the ’structure’ of specialised knowledge. In: W. Glänzel, H. Moed, U. Schmoch (Eds), Handbook of Quantitative Science and Technology Research, Dordrecht: Kluwer Academic Publishers, pp. 733–758.
CEST (2004), Les institutions du Domaine des Ecoles polytechniques fédérales, Profils de recherche et comparaisons internationales, Indicateurs bibliométriques 1981–2002, Centre de compétence scientométrie du CEST, CEST 2004/5, www.cest.ch
David, H. A., Edwards, A. F. W. (2001), Yule’s Paradox (Simpson’s paradox), In: Annotated Readings in the History of Statistics (pp. 137–140): Springer.
Egghe, L., Rousseau, R. (1996), Average and global impact of a set of journals. Scientometrics, 36: 97–107.
Egghe, L., Rousseau, R. (2002), A general frame-work for relative impact indicators. Canadian Journal of Information and Library Science — Revue canadienne des Sciences de l’Information et de Bibliothéconomie, 27: 29–48.
Garfield, E. (1967), Primordial concepts, citation indexing and historio-bibliography. Journal of Library History, 2: 235–249.
Garfield, E. (1998), Random thoughts on citationology. Its theory and practice — Comments on theories of citation? Scientometrics, 43: 69–76.
Glänzel, W., De Lange, C. (1997), Modelling and measuring multilateral co-authorship in international scientific collaboration. Part I. Development of a new model using a series expansion approach. Scientometrics, 40: 594–604. Part II. A comparative study on the extent and change of international scientific collaboration links. Scientometrics, 40: 605–626.
Grupp, H. (1998), Foundations of the Economics of Innovation: Theory, Measurement and Practice. Cheltenham: Elgar.
Luukkonen, T., Tijssen, R. J. W., Persson, O., Sivertsen, G. (1993), The measurement of international scientific collaboration. Scientometrics, 28: 15–36.
May, R. M. (1997), The scientific wealth of nations. Science, 275: 793–796.
Murugesan, P., Moravcsik, M. J. (1978), Variation of the nature of citation measures with journal and scientific specialties. Journal of the American Society for Information Science, 29: 141–155.
OST (2004), in Note Méthodologique B-5, Indicateurs de sciences et de technologies, Rapport de l’Observatoire des Sciences et des Techniques (L. Esterle, G. Filliatreau (Eds)), 2004.
Ramanana-Rahary, S., Zitt; M., Rousseau, R. (2007), Aggregation properties of relative impact and other classical indicators: convexity issues and the Yule-Simpson paradox. In: Proceedings of ISSI 2007, D. Torres-Salinas, H. F. Moed (Eds), Madrid, CINDOC-CSIC, pp. 643–654.
Schubert, A., Braun, T. (1986), Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9: 281–291.
Tijssen, R. J. W., De Leeuw, J., Van Raan, A. F. J. (1987), Quasi-correspondence analysis on scientometric transaction matrices. Scientometrics, 11: 351–366.
Van Raan, A. F. J. (1996), Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics,36: 397–420.
Van Raan, A. F. J. (2004), Measuring Science. In: W. Glänzel, H. Moed, U. Schmoch (Eds.), Handbook of Quantitative Science and Technology Research, Dordrecht: Kluwer Academic Publishers, pp. 19–50.
Vinkler, P. (2003), Relations of relative scientometric indicators. Scientometrics, 58(3): 687–694.
Yule, G. U. (1903), Notes on the theory of association of attributes in statistics. Biometrika, 2: 121–134.
Zitt, M., Bassecoulard, E., Okubo, Y. (2000), Shadows of the past in international cooperation: Collaboration profiles of the top five producers of science. Scientometrics, 47: 627–657.
Zitt, M., Ramanana-Rahary, S., Bassecoulard, E. (2003), Correcting glasses help fair comparisons in international science landscape: Country indicators as a function of ISI database delineation. Scientometrics, 56: 259–282.
Zitt, M., Ramanana-Rahary, S., Bassecoulard, E. (2005), Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation. Scientometrics, 63: 373–401.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ramanana-Rahary, S., Zitt, M. & Rousseau, R. Aggregation properties of relative impact and other classical indicators: Convexity issues and the Yule-Simpson paradox. Scientometrics 79, 311–327 (2009). https://doi.org/10.1007/s11192-009-0420-4
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-009-0420-4