More Web Proxy on the site http://driver.im/

research-article

Does monetary support increase citation impact of scholarly papers?

Authors:

Müge AkbulutAuthors Info & Claims

Scientometrics, Volume 125, Issue 2

Pages 1617 - 1641

https://doi.org/10.1007/s11192-020-03688-y

Published: 01 November 2020 Publication History

Abstract

One of the main indicators of scientific development of a given country is the number of papers published in high impact scholarly journals. Many countries introduced performance-based research funding systems to create a more competitive environment where prolific researchers get rewarded with subsidies to increase both the quantity and quality of papers. Yet, subsidies do not always function as a leverage to improve the citation impact of scholarly papers. This paper investigates the effect of the publication support system of Turkey (TR) on the citation impact of papers authored by Turkish researchers. Based on a stratified probabilistic sample of 4521 TR-addressed papers, it compares the number of citations to determine whether supported papers were cited more often than those of not supported ones and published in journals with relatively higher citation impact in terms of journal impact factors (JIF), article influence scores (AIS) and quartiles. Both supported and not supported papers received comparable number of citations per paper and were published in journals with similar citation impact values. The results of the hurdle model test showed that monetary support is related with reducing the number of uncited papers, and with slightly increasing the citation impact of papers with positive (i.e., non-zero) citations. Journal-level metrics of JIF, AIS and quartiles are not associated with papers’ getting their first citations nor with receiving higher citation counts. Findings suggest that subsidies do not seem to be an effective incentive to improve the citation impact of scholarly TR-addressed papers. Such support programs should therefore be reconsidered.

References

[1]

Abramo G, D’Angelo CA, and Di Costa F When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation Journal of Informetrics 2019 13 3 830-840

[2]

Akça S and Akbulut M Türkiye’deki yağmacı dergiler: Beall listesi üzerine bir araştırma Bilgi Dünyası 2018 19 2 255-274

[3]

Arendt, J. (2010). Are article influence scores comparable across scientific fields? Issues in Science and Technology Librarianship, 60. Retrieved September 16, 2019, from http://www.istl.org/10-winter/refereed2.html.

[4]

Article Influence Score. (2019). Retrieved December 1, 2019, from http://help.incites.clarivate.com/incitesLiveJCR/glossaryAZgroup/g4/7790-TRS.html.

[5]

Auranen O and Nieminen M University research funding and publication performance—An international comparison Research Policy 2010 39 6 822-834

[6]

Baccini A, De Nicolao G, and Petrovich E Citation gaming induced by bibliometric evaluation: A country-level comparative analysis PLoS ONE 2019 14 9 e0221212

[7]

Butler L Explaining Australia’s increased share of ISI publications—The effects of a funding formula based on publication counts Research Policy 2003 32 1 143-155

[8]

Butler L et al. Moed HF et al. What happens when funding is linked to publication counts? Handbook of quantitative science and technology research: The use of publication and patent statistics in studies of S&T systems 2004 Dordrecht Kluwer 389-405

[9]

Casadevall, A., & Fang, F. C. (2012). Causes for the persistence of impact factor mania. mBio, 5(2). Retrieved September 16, 2019, from http://mbio.asm.org/content/5/2/e00064-14.full.pdf.

[10]

Çetinsaya, G. (2014). Büyüme, kalite, uluslararasılaşma: Türkiye yükseköğretimi için bir yol haritası (2nd ed.). Ankara: Yükseköğretim Kurulu. Retrieved December 1, 2019, from https://www.yok.gov.tr/Documents/Yayinlar/Yayinlarimiz/buyume-kalite-uluslararasilasma-turkiye-yuksekogretim-icin-bir-yol-haritasi.pdf.

[11]

Checchi D, Malgarini M, and Sarlo S Do performance-based research funding systems affect research production and impact? Higher Education Quarterly 2019 73 45-69

[12]

Chen C Predictive effects of structural variation on citation counts Journal of the American Society for Information Science and Technology 2012 63 3 431-449

[13]

De Boer, H., et al. (2015). Performance-based funding and performance agreements in fourteen higher education systems. Report for the Ministry of Culture and Science (Reference: C15HdB014). Enschede: Center for Higher Education Policy Studies University of Twente. Retrieved September 16, 2019, from http://bit.ly/2DZNVWP.

[14]

De Rijcke S, Wouters P, Rushforth AD, Franssen T, and Hammarfelt BMS Evaluation practices and effects of indicator use—A literature review Research Evaluation 2016 25 2 161-169

[15]

Demir SBPredatory journals: Who publishes in them and why?Journal of Informetrics20181241296-13113786485

[16]

Demir SB Pros and cons of the new financial support policy for Turkish researchers Scientometrics 2018 116 3 2053-2068

[17]

Didegah F and Thelwall M Which factors help authors produce the highest impact research? Collaboration, journal and document properties Journal of Informetrics 2013 7 861-873

[18]

Didegah F and Thelwall M Determinants of research citation impact in nanoscience and nanotechnology Journal of the American Society for Information Science and Technology 2013 64 5 1055-1064

[19]

European Commission. (2010). Assessing Europe’s university-based research: Expert Group on Assessment of University-Based Research (EUR24187EN). Retrieved September 16, 2019, from https://ec.europa.eu/research/science-society/document_library/pdf_06/assessing-europe-university-based-research_en.pdf.

[20]

Fire M and Guestrin C Over-optimization of academic publishing metrics: Observing Goodhart’s Law in action GigaScience 2019 8 6 1-20

[21]

Fischer I and Steiger H-J Dynamics of Journal Impact Factors and limits to their inflation Journal of Scholarly Publishing 2018 50 1 26-36

[22]

Geuna A and Martin B University research evaluation and funding: An international comparison Minerva 2003 41 4 277-304

[23]

Glänzel W and Moed HF Journal impact measures in bibliometric research Scientometrics 2002 53 2 171-193

[24]

Gök A, Rigby J, and Shapira P The impact of research funding on scientific outputs: Evidence from six smaller European countries Journal of the Association for Information Science & Technology 2016 67 3 715-730

[25]

Good B, Vermeulen N, Tiefenthaler B, and Arnold E Counting quality? The Czech performance-based research funding system Research Evaluation 2015 24 2 91-105

[26]

Hammarfelt B and Haddow G Conflicting measures and values: How humanities scholars in Australia and Sweden use and react to bibliometric indicators Journal of the Association for Information Science & Technology 2018 69 7 924-935

[27]

Harley YX, Huysamen E, Hlungwani C, and Douglas T Does the DHET research output subsidy model penalise high-citation publication? A case study South African Journal of Science 2016 112 5–6 1-3

[28]

Hedding DW Payouts push professors towards predatory journals Nature 2019 565 267

[29]

Herbst M Financing public universities: The case of performance funding 2007 Dordrecht Springer

[30]

Heywood JS, Wei X, and Ye G Piece rates for professors Economics Letters 2011 113 3 285-287

[31]

Hicks D et al. Moed HF et al. The four literatures of Social Science Handbook of quantitative science and technology research: The use of publication and patent statistics in studies of S&T systems 2004 Dordrecht Kluwer 473-496

[32]

Hicks D Performance-based university research funding systems Research Policy 2012 41 2 251-261

[33]

Hongyang, L. (2017). Lancet restaurant gives medical professionals food for thought. China Daily, Retrieved September 16, 2019, from http://www.chinadaily.com.cn/china/2017-11/02/content_34013235.htm.

[34]

Jackman, S., et al. (2020). Package ‘pscl’. Retrieved June 7, 2020, from https://cran.r-project.org/web/packages/pscl/pscl.pdf.

[35]

Jonkers, K., & Zacharewicz, T. (2016). Research performance based funding systems: A comparative assessment. Luxembourg: Publications Office of the European Union. Retrieved September 16, 2019, from http://publications.jrc.ec.europa.eu/repository/bitstream/JRC101043/kj1a27837enn.pdf.

[36]

Kamalski, J., Huggett, S., Kalinaki, E., Lan, G., Lau, G., Pan, L., & Scheerooren, S. (2017). World of research 2015: Revealing patterns and archetypes in scientific research. Elsevier Analytic Services. Retrieved September 16, 2019, from http://www.doc88.com/p-2032803429898.html.

[37]

Kleiber C and Zeileis AVisualizing count data regressions using rootogramsThe American Statistician2016703296-3033535517

[38]

Koçak Z Predatory publishing and Turkey (editorial) Balkan Medical Journal 2019 36 4 199-201

[39]

Lee ATK and Simon CA Publication incentives based on journal rankings disadvantage local publications South African Journal of Science 2018 114 9/10 1-3

[40]

Liu F, Guo W, and Zuo C High impact factor journals have more publications than expected Current Science 2018 114 5 955-956

[41]

Liu W, Hu G, and Gu M The probability of publishing in first-quartile journals Scientometrics 2016 106 3 1273-1276

[42]

Lumley T, Diehr P, Emerson S, and Chen L The importance of the normality assumption in large public health data sets Annual Review of Public Health 2002 23 151-169

[43]

Mallapaty S China bans cash rewards for publishing papers Nature 2020 579 18

[44]

Marx, W., & Bornmann, L. (2013). Journal Impact Factor: “The poor man’s citation analysis” and alternative approaches. European Science Editing, 39(3), 62–63. Retrieved September 16, 2019, from http://www.ease.org.uk/sites/default/files/aug13pageslowres.pdf.

[45]

Miranda R and Garcia-Carpintero E Comparison of the share of documents and citations from different quartile journals in 25 research areas Scientometrics 2019 121 1 479-501

[46]

Moed HF and van Leeuwen TN Impact factors can mislead Nature 1996 381 6579 186

[47]

Mouton J and Valentine A The extent of South African authored articles in predatory journals South African Journal of Science 2017 113 7/8 1-9

[48]

Muller SM Academics as rent seekers: Distorted incentives in higher education, with reference to the South African case International Journal of Educational Development 2017 52 58-67

[49]

Muller JZ The tyranny of metrics 2018 Princeton, N.J. Princeton University Press

[50]

Nederhof AJ Bibliometric monitoring of research performance in the Social Sciences and Humanities: A review Scientometrics 2006 66 81-100

[51]

Nicolaisen J and Frandsen TV Zero-impact: A large scale study of uncitedness Scientometrics 2019 119 1227-1254

[52]

Ochsner M, Hug SE, and Daniel H-D Setting the stage for the assessment of research quality in humanities. Consolidating the results of four empirical studies Zeitschrift für Erziehungswissenschaft 2014 117 111-132

[53]

Osuna C, Cruz-Castro L, and Sanz-Menéndez L Overturning some assumptions about the effects of evaluation systems on publication performance Scientometrics 2011 86 3 575-592

[54]

Pajić D Globalization of the social sciences in Eastern Europe: Genuine breakthrough or a slippery slope of the research evaluation practice? Scientometrics 2014 102 3 2131-2150

[55]

Pillay TS Subject and discipline-specific publication trends in South African medical research, 1996–2011 South African Journal of Science 2013

[56]

Quan W, Chen B, and Shu F Publish or impoverish: An investigation of the monetary reward system of science in China (1999–2016) Aslib Journal of Information Management 2017 69 5 486-502

[57]

Sætnan AR, Tøndel G, and Rasmussen B Does counting change what is counted? Potential for paradigm change through performance metrics Research Evaluation 2019 28 1 73-83

[58]

Seglen PO Why the impact factor of journals should not be used for evaluating research British Medical Journal 1997 314 7079 498-502

[59]

Şengör AMC How scientometry is killing science GSA Today 2014 24 12 44-45

[60]

Shao J and Shen H Research assessment: The overemphasized impact factor in China Research Evaluation 2012 21 3 199-203

[61]

Sīle L and Vanderstraeten R Measuring changes in publication patterns in a context of performance-based research funding systems: The case of educational research in the University of Gothenburg (2005–2014) Scientometrics 2019 118 71-91

[62]

Sivertsen G Patterns of internationalization and criteria for research assessment in social sciences and humanities Scientometrics 2016 107 357-368

[63]

Sivertsen G Understanding and evaluating research and scholarly publishing in the Social Sciences and Humanities (SSH) Data and Information Management 2019 2 3 1-11

[64]

Sombatsompop N and Markpin T Making an equality of ISI impact factors for different subject fields Journal of the American Society for Information Science and Technology 2005 56 7 676-683

[65]

Teodorescu D and Andrei T An examination of “citation circles” for social sciences journals in Eastern European countries Scientometrics 2014 99 2 209-231

[66]

Tomaselli KG Perverse incentives and the political economy of South African academic journal publishing South African Journal of Science 2018 114 11/12 1-6

[67]

Tonta Y Does monetary support increase the number of scientific papers? An interrupted time series analysis Journal of Data and Information Science 2017 3 1 19-39

[68]

Tonta, Y. (2017b). TÜBİTAK Türkiye Adresli Uluslararası Bilimsel Yayınları Teşvik (UBYT) Programının değerlendirilmesi. Ankara: TÜBİTAK ULAKBİM. Retrieved March 28, 2020, from http://ulakbim.tubitak.gov.tr/sites/images/Ulakbim/tonta_ubyt.pdf.

[69]

Tonta, Y., & Akbulut, M. (2019). Does monetary support increase citation impact of scholarly papers? In G. Catalano, et al. (eds.), 17th international conference on Scientometrics & Informetrics ISSI2019 with a special STI indicators conference track, 2–5 September 2019, Sapienza University of Rome, Italy. Proceedings (pp. 1952–1963). Rome: International Society for Scientometrics and Informetrics. Retrieved September 20, 2019, from http://yunus.hacettepe.edu.tr/~tonta/Yayinlar/tonta_ISSI2019.pdf.

[70]

TÜBİTAK Türkiye Adresli Uluslararası Bilimsel Yayınları Teşvik Programı Uygulama Esasları. (2015). (237 BK-EK 1). Retrieved September 16, 2019, from http://www.tubitak.gov.tr/sites/default/files/237bk-ek1_0.pdf.

[71]

TÜBİTAK Türkiye Adresli Uluslararası Bilimsel Yayınları Teşvik Programı Uygulama Esasları. (2020). Retrieved July 5, 2020, from https://cabim.ulakbim.gov.tr/wp-content/uploads/sites/4/2020/06/2020_UBYT_Program%c4%b1_Uygulama_Usul_ve_Esaslar%c4%b1.pdf.

[72]

Van Leeuwen T Bibliometric research evaluations, Web of Science and the Social Sciences and Humanities: A problematic relationship? Bibliometrie—Praxis und Forschung 2013

[73]

Van Leeuwen TN, Moed HF, Tijssen RJW, Visser MS, and Van Raan AFJ Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance Scientometrics 2001 51 1 335-346

[74]

Wilsdon J et al. The metric tide: Report of the independent review of the role of metrics in research assessment and management 2015 London Sage

[75]

Wouters P et al. The metric tide: Literature review (supplementary report I to the independent review of the role of metrics in research assessment and management) HEFCE 2015

[76]

Yuret T Do researchers pay attention to publication subsidies? Journal of Informetrics 2017 11 2 423-434

[77]

Zeileis A, Kleiber C, and Jackman S Regression models for count data in R Journal of Statistical Software 2008 27 8 1-25

[78]

Zhang L, Rousseau R, and Sivertsen G Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen’s work on journal impact and research evaluation PLoS ONE 2017 12 3 e0174205

[79]

Zhang L and Sivertsen G The new research assessment reform in China and its implementation Scholarly Assessment Reports 2020 2 1 3

[80]

Zhang, L., & Sivertsen, G. (2020b). For China’s ambitious research reforms to be successful, they will need to be supported by new research assessment infrastructures (blog post). LSE Impact Blog. Retrieved June 20, 2020 from https://blogs.lse.ac.uk/impactofsocialsciences/2020/06/11/for-chinas-ambitious-research-reforms-to-be-successful-they-will-need-to-be-supported-by-new-research-assessment-infrastructures/.

Cited By

Öztürk OTaşkın Z(2024)How metric-based performance evaluation systems fuel the growth of questionable publications?Scientometrics10.1007/s11192-024-04991-8129:5(2729-2748)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s11192-024-04991-8
Tutuncu L(2023)All-pervading insider bias alters review time in Turkish university journalsScientometrics10.1007/s11192-023-04724-3128:6(3743-3791)Online publication date: 13-May-2023
https://dl.acm.org/doi/10.1007/s11192-023-04724-3
Santos JHorta HLi H(2022)Are the strategic research agendas of researchers in the social sciences determinants of research productivity?Scientometrics10.1007/s11192-022-04324-7127:7(3719-3747)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s11192-022-04324-7
Show More Cited By

Recommendations

What can university administrators do to increase the publication and citation scores of their faculty members?

Studies on publication and citation scores tend to focus mostly on frequently published and cited scholars. This paper contributes to advancing knowledge by simultaneously looking into both high and low performing scholars, including non-publishing ...
Normalisation of citation impact in economics
Abstract
This study is intended to facilitate fair research evaluations in economics. Field- and time-normalisation of citation impact is the standard method in bibliometrics. Since citation rates for journal papers differ substantially across publication ...
Scholarly publications beyond pay-walls: increased citation advantage for open publishing

First, we aim to determine the total amount of scholarly articles freely available on the internet. Second, we aim to prove whether there exists a citation advantage for open publishing. The total scholarly publication output of Norway is indexed in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Scientometrics

Scientometrics Volume 125, Issue 2

Nov 2020

946 pages

ISSN:0138-9130

Issue’s Table of Contents

© Akadémiai Kiadó, Budapest, Hungary 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 November 2020

Received: 28 July 2020

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Öztürk OTaşkın Z(2024)How metric-based performance evaluation systems fuel the growth of questionable publications?Scientometrics10.1007/s11192-024-04991-8129:5(2729-2748)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s11192-024-04991-8
Tutuncu L(2023)All-pervading insider bias alters review time in Turkish university journalsScientometrics10.1007/s11192-023-04724-3128:6(3743-3791)Online publication date: 13-May-2023
https://dl.acm.org/doi/10.1007/s11192-023-04724-3
Santos JHorta HLi H(2022)Are the strategic research agendas of researchers in the social sciences determinants of research productivity?Scientometrics10.1007/s11192-022-04324-7127:7(3719-3747)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1007/s11192-022-04324-7
Faria JMixon F(2021)The Marginal Impact of a Publication on Citations, and Its Effect on Academic PayScientometrics10.1007/s11192-021-04073-z126:9(8217-8226)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1007/s11192-021-04073-z
Vîiu GPăunescu M(2021)The citation impact of articles from which authors gained monetary rewards based on journal metricsScientometrics10.1007/s11192-021-03944-9126:6(4941-4974)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1007/s11192-021-03944-9

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents