An analysis of use statistics of electronic papers in a Korean scholarly information repository

Introduction

Recent advances in information technology have facilitated the sharing of and access to scholarly papers in digital format. However, researchers’ free access to scholarly papers is still limited due to commercial publishers charging for their content. The open-access initiative emerged as an alternative to overcome the paradox in the academic environment. This initiative aims to facilitate the sharing of information between researchers and users such that anyone can freely access and utilise information without legal, financial, and technical barriers (Suber, 2005). Open access refers to the removal of sharing and free access limitations posed by existing commercial publishers and the return of the academic environment to its original aspirations of sharing and communication. Specific open access strategies include gold open access, where an author often pays an article processing charge to publish their paper in an open access journal. The journal then allows everyone to freely view academic papers. Green open access is where an author publishes their paper in any journal and then self-archives a copy in a freely accessible institutional or specialist online archive (a repository or a Website).

Repositories are important sources of easy access to the scholarly information needed in academic research and practice. It is a model that allows free use of copyrighted scholarly publications (even for publications whose free access is limited by commercial publishers) and offers a new direction for scholarly information management systems (Bhardwaj, 2014). The first repository launched in 1991 was the physics repository now known as arXiv, which started as a server for articles on theoretical physics and included other areas of physics, mathematics and computer science as the Internet developed. Other early repositories include EconPapers, CogPrints and PubMed Central.

As in many countries, Korea has witnessed an increasing number of institutional repositories being developed by universities and research institutes, making a major contribution to vital scholarly exchanges. Among these, the oldest and largest repository of science and technology scholarly papers was established by the Korea Institute of Science and Technology Information (hereafter, the Institute), an institution that has provided scholarly information services since 1962. The repository is made available in Korea through the National Digital Science Library, a science and technology information service platform developed to foster scientific and technological innovations in the country by providing high-quality information to industry and academic research collaborators. The platform offers a content search function consisting of approximately 100 million records of diverse scholarly information such as scholarly papers, patent information and technical reports, as well as a specialised search service by type of content. As of March 2020, the Institute repository of scholarly papers had 100 million entries of metadata and full text of scholarly papers published inside and outside Korea. For scholarly papers published overseas, typically, only the metadata are deposited and a link to the full paper is provided. For papers published in Korea, original texts and metadata are stored in the repository, as the Institute has signed agreements with Korean academic institutions in the fields of science, technology, interdisciplinary convergence and integration. The Institute increases the quality of scholarly information in the repository through several curation processes, including author identification, institution identification, funding organisation identification, personal information processing, encryption, digital object identifier registration, reference extraction and quantitative analysis. The Institute also processes the original texts of papers provided by academic institutions, in addition to developing metadata and making them freely accessible in a timely manner. Yoon and Tibbo (2012) verified that the Institute repository has been effective according to several criteria, by enabling reuse through content curation, storing and maintaining original data, developing ways to use even more data in future projects, and enhancing the ability to verify research findings to promote the use of data in education.

However, a considerable amount of resources is necessary for the development and maintenance of a repository. In addition, continuous investment in a repository requires the assessment of its economic validity. The value of offline academic libraries handling scholarly information has been evaluated in many studies, among which Oakleaf (2010) stands out for having summarised years of work on the value of academic libraries.

With regards to the economic assessment of online-based repositories, Houghton and Beagrie’s research (2012) is a representative example. They conducted a fully developed quantified economic impact study on one of the UK Data Archives, the Economic and Social Data Service (ESDS), and found that quantifiable benefits and returns significantly exceeded the value of the funding invested in the platform. The study predicted a range of values, moving from those focusing on minimum values towards methods measuring some of the broader impacts, and, simultaneously performed qualitative analysis, such as case studies and the Keep Research Data Safe (KRDS) benefit framework. The KRDS benefit framework is intended to identify, assess and communicate the benefits of investing resources in the curation and long-term preservation of research data (Beagrie, 2011). The authors conducted a study on the Archaeology Data Service (ADS) by applying the framework used in the evaluation of ESDS (Beagrie and Houghton, 2013), and further proceeded to conduct a study which compared three data centre services, including the British Atmospheric Data Centre (BADC) (Beagrie and Houghton 2014). In these studies, qualitative assessment was conducted using the contingent value method (CVM) mainly based on surveys and interviews. The contingent value of a non-market good or service refers to the amount that users are willing to pay for it and/or willing to accept in return for it.

The value of scholarly information stored in a repository can be assessed from a data or information perspective. This valuation methodology is largely based on asset valuation methods on exchange value and use value. However, it is not easy to evaluate the exchange value of scholarly information as the market is not fully developed. Regarding the use value, it is not easy to estimate the economic effect through use. Moody and Walsh (1999) argued that information has seven unique attributes, distinguished from conventional tangible or intangible assets, and suggested the modified historical cost method, which takes the unique attributes into account. According to this method, data producers are very rational, which leads them to invest based on the assumption that the future economic value will be greater than the current investment. However, the value of the data ranges from potentially no value to enormous value. For example, sometimes data are never used and are only an overhead. At the other end of the spectrum, data can be used in crucial decision-making processes. In this sense, the method adjusts for the value of the data based on their unique attributes, such as its quality and the potential for an infinite number of users, in addition to considering data duplication. The most prominent feature of the modified historical cost method is that it relies heavily on statistical data on the current status of information use. Because statistical data can be easily obtained from information systems and the modified historical cost method is based on objective data, this method is advantageous in comparison to others in that assumptions and subjective viewpoints are removed.

This study aimed to analyse the current use of Korean scholarly papers available in the Institute repository in order to assess the economic validity – benefit-cost (B/C) ratio – of the maintenance and operation of the repository. To this end, this study was based on the modified historical cost method and analysed the use of Korean scholarly articles by year and by subject area adopting regression analysis to show the economic validity of the repository and provide basic data to be used in the development of operational strategies.

The remainder of this article is as follows: the methods section provides a description of the data used in the study and the methodological approach; the results section presents the results of the overall analysis and of the specific analysis by year and by subject area. In the discussion section, economic validity and future operational strategies are described and the analysis of the results is presented in the results section. The final section presents the conclusions of the study.

Methods

Dataset

The data used in this study were the numbers of full-text scholarly papers published in Korean academic journals deposited in the Institute repository between 2007 and 2019 and their corresponding use. For the period considered, 1,154,549 Korean scholarly papers across twenty-six subject areas were identified with their respective information on years of deposit, years of publication and subject area, as well as general information (including descriptive metadata on publication). The details of the deposited papers, that is the numbers of papers by year of deposit and year of publication, are summarised in Tables 6 and 7 in the Appendix, respectively.

The use volume through the National Digital Science Library was defined as the number of downloads of Korean scholarly articles between 2013 and November 29, 2019. Papers published outside Korea were excluded from this study because their full texts were not deposited in the repository, and this is often a gateway for overseas publishers. Prior to conducting the main analysis, abnormal use values were examined during data cleansing. Several papers showed an abnormally high number of downloads, albeit not very many. After a review of citation information and the contents of the papers, three papers were excluded from the study as it was concluded that they displayed over 10,000 downloads per year due to a system error or an abnormal phenomenon. In the final dataset, it was observed that during the study period, 996,151 articles were downloaded at least once and 26,164,642 downloads occurred in total.

Procedure

This study is based on the modified historical cost method suggested by Moody and Walsh (1999). They propose the following modifications to the historical cost method for valuing information, which incorporates the unique attributes of information.

• Cost of information collection is used as the baseline for the value of information and this cost needs to be standardised by using a standard information cost for each item.
• The cost of the information management system should be considered based on the cost of the processes used to extract the data from the information repository
• Redundant information has zero value to avoid double counting
• Unused information also has zero value.
• The number of users and number of accesses to the information are used to multiply the value of the information. The first time information is used, it is valued as the cost of information collection and maintenance. This value is added to each additional use.
  • Moody and Walsh’s (1999) key idea is that the historical cost of information should be modified in light of its use in practice and reflects the cumulative nature of its value across different users.
• The value of information depreciates based on its retention period.
• The value of inaccurate information is discounted by relative acceptability.

This method consists of three parts: the cost of data collection, the quality of the data collected and the use of the data. However, in this study, we are mainly concerned with the use of data.

First, due to the confidentiality of the cost data, it is not appropriate to provide a detailed view of the cost of data collection. Two types of cost data, however, were considered in this study: labour cost and repository system operating cost. Labour cost and system operating cost consist of acquisition cost and maintenance cost, respectively. Taking into account all these costs, we have obtained the value of each deposited product. Second, the quality of the collected data is guaranteed by the well-established Institute data acquisition and standardised data management processes.

Therefore, to apply the modified historical cost method in this paper, we analysed the use of the collected papers. The use of papers tends to decrease over time, but since a collected paper will continue to be used in the future, we estimate the total use through regression analysis of current and future use.

Results

Basic statistics

As shown in Figure 1, a Pareto analysis was performed to examine the total number of deposited papers by subject area, as classified by the Korean Institute of Science and Technology Information (full names are given in Table 5 of the Appendix). Engineering accounted for 25% of all deposited papers. Although areas such as chemical were listed separately, it is speculated that engineering still encompasses numerous subareas with many academic journals and papers produced each year. The subject area with the second-highest number of deposited papers was medicine, comprising 15.8% of the total. As many academic journals and research papers are produced in medicine, as in engineering; however, a large number of them are published in overseas academic journals and, thus, the number of papers published in Korean academic journals is smaller compared to engineering. With the addition of agriculture, the top three areas accounted for 50% of all deposited papers. In addition, the upper half of the subject areas represented approximately 90% of all deposited papers, showing a bias towards particular subject areas. This phenomenon is believed to be due more to the distinctive general features of scholarly information rather than to a feature of Korean academic policy.

Data comparing the volumes of papers deposited and published each year are presented in Figure 2.

The yearly volume of deposited papers varied considerably over the study period. In 2010, 230,000 papers were deposited, while only approximately 36,000 papers were deposited in 2019. The number of papers in 2010 was relatively high because, in that year, several scholarly databases were integrated into the repository. Thereafter, a similar number of papers were deposited each year with the exception of 2016, suggesting a steady acquisition of scholarly papers.

In comparison with article deposits, the yearly volume of publications was strongly constant. However, the number of deposited papers is decreasing in more recent years, which seems to be due to the difference between the years of publication and deposit and an embargo period on academic publishing.

Data on the use volumes of the papers are summarised in Table 1. From 2013 through 2019, annually, approximately 574,000 papers were downloaded at least once and a total of 996,151 papers were downloaded during the same period. Thus, approximately 86% of all deposited papers were downloaded at least once. This rate is considered significantly high, given that researchers can download papers not only from NDSL, but also through various other routes, such as publisher homepages and portals. Except for 2016, the number of papers downloaded did not vary greatly, which suggests the presence of a stratum of fixed users.

Nevertheless, the number of downloads fluctuated over time. With the exception of 2017, when articles were downloaded 2.9 million times, the annual number of downloads was over three million. Over the past three years, both the number of downloads and the number of deposited papers has been on an increasing trend. However, such an increase has occurred only recently, and it is therefore difficult to determine whether the number of downloads would increase in the long run.

During the study period, a paper was downloaded 26.27 times on average and a total of approximately 26 million downloads occurred. However, the number of times each article was downloaded varied considerably, as shown by the Pareto analysis presented in Figure 3. According to Figure 3, 10.91% of the papers were downloaded only once and the number of papers decreased exponentially as the number of downloads increased. Overall, however, Figure 3 reveals that the long tail principle, rather than the Pareto principle, was at work. This means that not many papers were downloaded numerous times, but such papers did exist. Due to this phenomenon, although the mean was 26.27, the first and the third quartiles were three and twenty-seven, respectively, and the median was ten; that is, 50% of all deposited papers were downloaded ten or fewer times during the study period (2013-2019), with a yearly average of 1.5 or fewer times.

Use statistics by subject area

Figure 4 shows the number of downloaded papers, the number of deposited papers, and the average number of downloads per paper for each subject area. In general, the number of deposited papers was high in subject areas with a high number of downloads. Hence, it is believed that the subject areas of the deposited papers have been properly chosen. Among the subject areas, mathematics showed a ratio of downloaded to deposited articles of 61.25%, while the corresponding ratio was 77.58% for building & construction and 79.59% for technology. With the exception of these three subject areas, the remaining twenty-three areas showed a ratio above 80%. Hence, the ratio of use was very high. Although the number of articles was on the low side, in home & family management and history, the ratios were above 95%, clearly showing that almost all of the deposited papers in these subject areas were used. This finding can be interpreted as a demonstration that the characteristics of academic areas were highly influential, and it seemed that the influence of a handful of papers raising issues from a socio-environmental perspective was significant.

However, the average number of downloads per article varied widely by subject area. In home & family management and history (subject areas with very high use ratios), the average number of downloads per article was the highest (forty-five and thirty-seven, respectively), while in the subject areas with low use ratios (such as fossils & prehistoric life and technology), the average number of downloads per paper was also low. The correlation between the use ratio and the average number of downloads per paper was positive, with a coefficient of ρ=0.48.

Based on their amount of use and the use trends, the subject areas were divided into three groups: heavy use subject area, medium use subject area, and light use subject area. The summary statistics for each group are presented in Table 2.

The heavy use group consisted of two subject areas, engineering and medicine, which showed ratios slightly over 40%. When this group and the medium use group were combined, a total of seven subject areas showed ratios higher than 70%. Given the comparability between the ratios of deposited articles and usage, it is believed that the scholarly information deposited in the repository adequately reflects the usage volume.

Use statistics by year

Table 8 in the Appendix presents the annual number of downloaded papers by year of publication. The column shows the year of download and the row shows the year of publication. For example, in 2013, 518,777 papers were downloaded, of which 330,184 were published in or before 2007, while 31,240 were published in 2008.

Although the specific values shown in the table are meaningful in themselves, the overall trend is even more significant. In each column, the values continuously increase up to the most recent one or two years, showing that the most recently published papers were downloaded more frequently. However, the most recently published papers don’t follow this trend, apparently because the volume of recently deposited papers is not very high and because more time should pass for papers to be exposed to researchers.

An examination of the values in each row showed that the number of downloaded papers decreased, except for in 2016 (the year when the number of downloads was higher compared with other years), suggesting a steady demand for the most recent papers. However, the number did not drop abruptly indicating that the demand for papers published in the past did not decrease suddenly. This finding is consistent with what Starr and Williams (2008) reported and demonstrates the need to continue managing papers deposited in the past, as well as recent publications.

Table 9 in Appendix shows the yearly volume of downloads by the paper’s year of publication. The column lists the year of the download and the rows list the years of publication. The trend in the volume of downloads is the same as the result of the analysis conducted using the data in Table 8.

For a more detailed analysis, the data in Table 9 were manipulated to obtain the results shown in Figure 5. In Figure 5, the x-axis shows the difference in years between publication and download, while the y-axis shows the ratio of number of downloads of papers published in a given year to the total number of downloads in that year. For example, in 2016, the number of downloads of papers published in 2010 was 219,404, representing 5.62 % of the total use volume in 2016 (219,404/3,902,914 = 5.62%). This data point is the fourth of the six points corresponding to the x value of 6 (2016-2010) in Figure 5.

The graph reveals the trend of downloads according to the number of years passed since publication, which also has a bearing on the lifespan of the deposited papers. That is, the trend of the data points in Figure 5 is that the ratio of downloads is gradually decreasing, and the rate of decrease appears to be non-linear, indicating that the number of downloads decreases over time. Accordingly, although it is crucial to acquire new papers to meet information needs, it is also important to manage those already deposited. Another trend that can be seen in the graph is that the volatility in the ratio of downloads decreases over time. This finding could be interpreted as convergence over time; however, there is a limitation in conducting statistical analysis because, as the number of years between publication and download increased, the size of available data was small.

In addition to the ratio of papers by year of publication, it is also crucial to examine cumulative ratios. The cumulative ratio enables the prediction of the lifespan of the papers currently deposited, which can be strategically used in the management of the information deposited in the repository. Cumulative download ratios are shown in Figure 6. For example, in 2016, the number of downloads of papers published before 2014 was 3,399,951 representing 86.88% of the total use volume in 2016 (3,399,951/3,902,914 = 86.88%). This data point is the third one of the six points corresponding to the x value of 2 (2016-2014) in Figure 6. This figure presents useful information to identify the use trends of the information currently deposited, as it provides information on future use levels. According to Figure 6, the cumulative ratios are gradually decreasing. Regression analysis was conducted to statistically confirm the observation.

The response variable, Y, was the cumulative download ratio and the independent variable, X, was the difference between use year and publication year. Cases with X≥8 were excluded due to lack of data and a total of fifty-three data points were submitted to regression analysis. The results are presented in Table 3 below.

The explanatory power of the model, R²=0.84, was sufficiently high. The equation was as follows:

Y=0.995-0.076 X

indicating that the use ratio for articles currently deposited decreases by 7.6% per year. This finding shows that the download ratio is likely to reach 0% after approximately thirteen years (that is, the lifespan of the papers currently deposited) and, thereafter, the most downloaded papers will be those deposited after the current point in time. This means that the average lifespan of deposited papers is thirteen years or more. Therefore, it is necessary to set the minimum period for the management of papers at above thirteen years. As there are wide variations in scholarly articles, it is reasonable to set the management period to be much longer than thirteen years. In addition, as stated in Kryder’s Law that information storage costs continue to decrease (Rosenthal, 2017), the costs of storing and managing already deposited articles are relatively low compared to the cost of acquiring new papers, and the unit cost continuously becomes cheaper. Therefore, it would not be difficult to plan for a management period longer than the average lifespan.

Table 4 shows the ratio of downloaded papers to deposited papers. The data after 2008 show that at least 50% of the deposited papers were downloaded. In other words, more than half of the deposited papers were downloaded at least once in a given year. The numbers in each column indicate a trend of more recent papers having higher download ratios. However, the ratios of download in the same year of publication are somewhat low because not enough time has passed for the papers to be exposed to researchers. An examination of the numbers in each row (except in 2016, a year that showed abnormally high download ratios) clearly confirms the aforementioned finding that the download ratio decreased over time, although the rate of decrease was not high.

Discussion

This study examined the current use status of the Korea Institute of Science and Technology Information repository of Korean scholarly papers by year and by subject area. The findings indicate various implications for the repository’s economic validity and efficient operation.

First, over the past eight years, approximately 50,000 papers per year were deposited in the Institute's repository and the average number of downloads per year was 3.7 million. On average, the number of downloads was approximately seventy-five times higher than the number of deposited papers, indicating that the repository has been actively used. Second, approximately 86% of the deposited papers were downloaded at least once, which demonstrates that the scholarly information stored in the repository is useful. This finding also justifies the need to continue operating the National Digital Science Library (NDSL). Third, the twenty-six subject areas identified were categorised into three use groups: heavy, medium and light. This classification was applied to the current status of both the use and deposit of papers, showing that the NDSL operational strategy adequately reflects the needs of researchers. Lastly, in general, the older the publications, the lower the use ratio. The rate of decrease was analysed in a linear relationship and the lifespan of the articles currently deposited was estimated at thirteen years. Thus, the deposited papers should be managed for at least thirteen years in order to support researchers’ academic activities.

The analysis of the current use status of the Institute's repository of Korean scholarly papers showed that, currently, the papers deposited in the repository have been used properly. This finding means that the current operational strategy is working well and suggests the need to acquire scholarly articles more proactively. In addition, the economic validity of the repository can be evaluated based on the modified historical cost method, as follows.

First, cleansed data were used in the study, which allows one to assume that problems such as data quality and duplication had been resolved. Therefore, from the perspective of the benefit to cost ratio (BCR), the value of scholarly information stored in the Institute's repository of Korean scholarly papers can be assessed by defining benefit as the total number of uses per paper and total investment cost as the sum of the costs for deposit (acquisition) and storage per paper.

By assigning a value of one to acquisition cost and assuming that the annual management cost is 10% (a conventional estimate used in Korea) with a decrease rate of 50% per year on the basis of Kryder’s Law (Rosenthal, et al., 2012), the cost per paper is estimated to be 1.2. As shown in Table 3, a paper was downloaded on average of 26.27 times and approximately 86% of the deposited papers were downloaded. Therefore, the average number of downloads per paper among the downloaded papers is 26.27 ×0.86= 22.6; accordingly, B/C=22.6/1.2=18.8. This indicates that the acquisition and operation of the repository is estimated to have a benefit approximately nineteen times greater than the investment.

Conclusion

In this study, the current use status of the Institute's repository of Korean scholarly papers was examined to provide basic data for economic validity analysis and operational strategy development for continuous operation of the repository. Specifically, the status of the papers deposited in the repository as at 2019 and the use status of deposited articles from 2013 through 2019 were analysed from a variety of viewpoints (including by subject area and by year).

According to the analysis, it was found that approximately 86% of all deposited papers have been used and that each paper was downloaded on average twenty-six times. Approximately 50,000 papers were deposited per year and the annual number of downloads was 3.5 million, a number seventy times higher than the volume of deposited papers, suggesting that the repository has been actively used. Regarding the economic validity of the repository, the benefit:cost ratio was estimated to be 18.8 on the basis of the concept of the modified historical cost method, demonstrating that benefit was much greater compared to investment.

This research was initiated to evaluate the economic value of the Institute's repository of Korean scholarly papers. However, given the intrinsic attributes of the research, there was a limitation in conducting accurate cost analysis, and the valuation was performed in a simplified manner due to a lack of holistic methods to evaluate the economic value of scholarly papers.

Hence, this study mainly focused on quantitative analysis on the use of deposited papers. However, it is important to develop an effective holistic methodology for evaluating the development and operation of institutional repositories of scholarly papers that should consider both economic and social values including qualitative analysis of the user's benefit or scenario. It is also necessary to consider the characteristics of the domestic academic situation in each subject area.

Acknowledgement

This research was supported by Korea Institute of Science and Technology Information.

About the authors

Jayhoon Kim is a Principal Researcher at the Korea Institute of Science and Technology Information, Daejeon, Korea. He is a Ph.D. candidate in Library and Information Science at the Chungnam National University. His research focuses on digital curation of scholarly information. He can be contacted at jay.kim@kisti.re.kr
Su-yeong Bae is a Research Engineer at the Korea Institute of Science and Technology Information, Daejeon, Korea. He received his MS degree in Computer Engineering from Hongik University. His research focuses on scholarly information service. He can be contacted at sybae@kisti.re.kr
Hyekyong Hwang is a Principal Researcher, Director of Content Curation Center at the Korea Institute of Science and Technology Information, Daejeon, Korea. She received her Ph.D.degree in Library and Information Science from Yonsei University, Korea. Her research areas are Content Curation of the Scholarly Communication, Open Access, Citizen Science, and AI Services. She can be contacted at hkhwang@kisti.re.kr
Sung-Seok Ko (Corresponding author) is a Professor in the Department of Industrial Engineering, Konkuk University, Seoul, Korea. He received his Ph.D. degree in Industrial and Systems Engineering from Georgia Institute of Technology, USA. His areas of research interests include Operations Research, Stochastic Process and Data Science. He can be contacted at ssko@konkuk.ac.kr

Appendices