More Web Proxy on the site http://driver.im/

research-article

Use of accounting concepts to study research: return on investment in XSEDE, a US cyberinfrastructure service

Scientometrics, Volume 128, Issue 6

Pages 3225 - 3255

https://doi.org/10.1007/s11192-022-04539-8

Published: 14 February 2023 Publication History

Abstract

This paper uses accounting concepts—particularly the concept of Return on Investment (ROI)—to reveal the quantitative value of scientific research pertaining to a major US cyberinfrastructure project (XSEDE—the eXtreme Science and Engineering Discovery Environment). XSEDE provides operational and support services for advanced information technology systems, cloud systems, and supercomputers supporting non-classified US research, with an average budget for XSEDE of US$20M+ per year over the period studied (2014–2021). To assess the financial effectiveness of these services, we calculated a proxy for ROI, and converted quantitative measures of XSEDE service delivery into financial values using costs for service from the US marketplace. We calculated two estimates of ROI: a Conservative Estimate, functioning as a lower bound and using publicly available data for a lower valuation of XSEDE services; and a Best Available Estimate, functioning as a more accurate estimate, but using some unpublished valuation data. Using the largest dataset assembled for analysis of ROI for a cyberinfrastructure project, we found a Conservative Estimate of ROI of 1.87, and a Best Available Estimate of ROI of 3.24. Through accounting methods, we show that XSEDE services offer excellent value to the US government, that the services offered uniquely by XSEDE (that is, not otherwise available for purchase) were the most valuable to the facilitation of US research activities, and that accounting-based concepts hold great value for understanding the mechanisms of scientific research generally.

References

[1]

Apon AW, Ngo LB, Payne ME, and Wilson PW Assessing the effect of high performance computing capabilities on academic research output Empirical Economics 2015 48 1 283-312

[2]

Brase J, Campbell N, Helland B, Hoang T, Parashar M, Rosenfield M, Sexton J, and Towns J The COVID-19 High-Performance Computing Consortium Computing in Science & Engineering 2022 24 1 78-85

[3]

Brazil, M., Brunson, D., Culich, A., DeStefano, L., Jennewein, D., Jolley, T., Middelkoop, T., Neeman, H., Rivera, L., Smith, J., & Wernert, J. (2019). Campus champions: Building and sustaining a thriving community of practice around research computing and data. In Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning) (pp. 1–7).

[4]

Chityala, P., Costa, C. M., Wernert, J. A., & Stewart, C. A. (2022a). Cyberinfrastructure value: A survey on perceived importance and usage. In Practice and experience in advanced research computing (PEARC ’22) (pp. 1–4). Association for Computing Machinery.

[5]

Chityala, P., Costa, C., Wernert, J., & Stewart, C. (2022b). Datasets: Cyberinfrastructure value: Perceived importance and usage. Indiana University. https://hdl.handle.net/2022/27600

[6]

Costa, C. M., Wernert, J. A., McMullen, D., Stewart, C. A., Blood, P. D., Sinkovits, R., Mehringer, S., Knepper, R., Bland, M., & Rogers, G. (2022). Dataset: XSEDE return on investment data and analysis - July 2014 to August 2021. https://hdl.handle.net/2022/27622

[7]

Coulter, E., Fischer, J., Hallock, B., Knepper, R., & Stewart, C. (2016). Implementation of simple XSEDE-like clusters: Science enabled and lessons learned. In Proceedings of the XSEDE16 conference on diversity, big data, and science at scale (pp. 1–8).

[8]

Frontier Economics Ltd. (2014). Rates of return to investment in science and innovation: A report prepared for the Department for Business, Innovation and Skills (BIS). https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/333006/bis-14-990-rates-of-return-to-investment-in-science-and-innovation-revised-final-report.pdf

[9]

Globus. (2022). https://www.globus.org

[10]

Hancock DY, Stewart CA, Vaughn M, Fischer J, Lowe JM, Turner G, Swetnam TL, Chafin TK, Afgan E, Pierce ME, and Snapp-Childs W Jetstream-Early operations performance, adoption, and impacts Concurrency and Computation: Practice and Experience 2019 31 16 4683

[11]

Hart, D. (2022). Measuring XSEDE: Usage metrics for the XSEDE federation of resources. In Practice and Experience in Advanced Research Computing (PEARC ’22), (pp. 1–12). Association for Computing Machinery, Boston, MA.

[12]

IBM. (2021). Cost of a Data Breach Report 2021. https://www.ibm.com/security/data-breach

[13]

Kee KF and Browning LD The dialectical tensions in the funding infrastructure of cyberinfrastructure Computer Supported Cooperative Work (CSCW) 2010 19 3 283-308

[14]

Kinney MR and Raiborn CA Cost Accounting: Foundations and Evolutions 2011 Boston, MA Cengage Learning

[15]

Knepper R and Börner K Comparing the consumption of CPU hours with scientific output for the eXtreme Science and Engineering Discovery Environment (XSEDE) PloS one 2016 11 6 0157628

[16]

Lane J Assessing the impact of science funding Science 2009 324 1273-1275

[17]

LIGO Caltech. (2022). Laser Interferometer Gravitational-Wave Observatory. https://www.ligo.caltech.edu/

[18]

Mahendran, S., Costa, C. M., Wernert, J. A., & Stewart, C. A. (2022). Institutional value of a Nobel Prize. In Practice and experience in advanced research computing (PEARC ’22) (pp. 1–4). Association for Computing Machinery.

[19]

National Academies of Sciences. (2020). Information technology innovation: Resurgence, confluence, and continuing impact. National Academies Press.

[20]

National Science Foundation. (2011). NSF Award#: 1053575 XSEDE: eXtreme Science and Engineering Discovery Environment. https://www.nsf.gov/awardsearch/showAward?AWD_ID=1053575

[21]

National Science Foundation. (2016). NSF Award#: 1548562 XSEDE 2.0: Integrating, enabling and enhancing national cyberinfrastructure with expanding community involvement. https://www.nsf.gov/awardsearch/showAward?AWD_ID=1548562

[22]

National Science Foundation. (2021). Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS). https://www.nsf.gov/pubs/2021/nsf21555/nsf21555.htm

[23]

Navarro, J.-P., Stewart, C. A., Knepper, R., Liming, L., Lifka, D., & Dahan, M. (2017). The community software repository from XSEDE: A resource for the national research community. In Proceedings of the practice and experience in advanced research computing 2017 on sustainability, success and impact (pp. 1–8).

[24]

O’Hara M, Yao C, and Ye M What’s not there: Odd lots and market data The Journal of Finance 2014 69 5 2199-2236

[25]

Saunders CS and Jones JW Measuring performance of the information systems function Journal of Management Information Systems 1992 8 4 63-82

[26]

Schumpeter, J. (2003). Theorie der Wirtschaftlichen Entwicklung (reprint of 1912 book). Springer.

[27]

Smith, P., & Lien Harrell, S. (2021). Research computing on Campus—Application of a production function to the value of academic High-Performance computing. In Practice and experience in advanced research computing (PEARC’21) (pp. 1–8).

[28]

Stewart, C. A., Costa, C. M., Wernert, J. A., Hancock, D. Y., McMullen, D. F., Blood, P., Sinkovits, R., Mehringer, S., Knepper, R., Fischer, J., Bland, M., Rogers, G., Couvares, P., Campbell, T., Jankowski, H., Snapp-Childs, W., & Towns, J. (2022). Metrics of financial effectiveness: Return on investment in XSEDE, a national cyberinfrastructure coordination and support organization. In Practice and experience in advanced research computing (PEARC ’22) (pp. 1–12). Association for Computing Machinery.

[29]

Stewart, C. A., Hancock, D. Y., Wernert, J., Furlani, T., Lifka, D., Sill, A., Berente, N., McMullen, D. F., Cheatham, T., Apon, A., Payne, R., & Slavin, S. D. (2019). Assessment of financial returns on investments in cyberinfrastructure facilities: A survey of current methods. In Proceedings of the practice and experience in advanced research computing on rise of the machines (learning) (pp. 1–8).

[30]

Stewart, C. A., Knepper, R., Link, M. R., Pierce, M., Wernert, E., & Wilkins-Diehr, N. (2019). Cyberinfrastructure, cloud computing, science gateways, visualization, and cyberinfrastructure ease of use. In Advanced methodologies and technologies in network architecture, mobile computing, and data analytics (pp. 157–170). IGI Global.

[31]

Stewart, C. A., Roskies, R., Knepper, R., Moore, R. L., Whitt, J., & Cockerill, T. M. (2015). XSEDE value added, cost avoidance, and return on investment. In Proceedings of the 2015 XSEDE conference: Scientific advancements enabled by enhanced cyberinfrastructure (pp. 1–8). St. Louis, MO.

[32]

Stewart, C. A., Wernert, J. A., Costa, C. M., Hancock, D. Y., Knepper, R., & Snapp-Childs, W. (2022). Return on investment in research cyberinfrastructure: State of the art. In Practice and experience in advanced research computing (PEARC ’22) (pp. 1–4). Association for Computing Machinery.

[33]

The Value Reporting Foundation. (2021). 2021 International <IR> Framework. https://www.integratedreporting.org/wp-content/uploads/2021/01/InternationalIntegratedReportingFramework.pdf

[34]

Towns J, Cockerill T, Dahan M, Foster I, Gaither K, Grimshaw A, Hazlewood V, Lathrop S, Lifka D, Peterson GD, et al. XSEDE: accelerating scientific discovery Computing in Science & Engineering 2014 16 5 62-74

[35]

von Laszewski, G., Wang, F., Fox, G. C., Hart, D. L., Furlani, T. R., DeLeon, R. L., & Gallo, S. M. (2015). Peer comparison of XSEDE and NCAR publication data. In 2015 IEEE international conference on cluster computing (pp. 531–532).

[36]

Wernert, J. A., Costa, C. M., McMullen, D. F., Stewart, C. A., Blood, P. D., Sinkovits, R., Mehringer, S. H., Knepper, R., Bland, M., & Rogers, G. (2022). Technical report: XSEDE return on investment (proxy) data and analysis methods, July 2014 to August 2021. Technical report. https://hdl.handle.net/2022/27623

[37]

Wilkins-Diehr, N., Sanielevici, S., Alameda, J., Cazes, J., Crosby, L., Pierce, M., Roskies, R. (2015). An overview of the XSEDE extended collaborative support program. pp. 3-13 In: Gitler, I. and J. Klapp (eds). High Performance Computer Applications - 6th International Conference, ISUM 2015, Revised Selected Papers. Springer Communications in Computer and Information Science, vol. 595. ISSN 1865-0929.

[38]

WK Kellogg Foundation. (2004). WK Kellogg Foundation logic model development guide. WK Kellogg Foundation. https://wkkf.issuelab.org/resource/logic-model-development-guide.html

[39]

XSEDE. (2021). Campus champions. https://hdl.handle.net/2142/114451

[40]

XSEDE. (2022a): URLs for XSEDE annual or project year reports. https://hdl.handle.net/2142/114456

[41]

XSEDE. (2022b): XSEDE Federation. https://hdl.handle.net/2142/114454

[42]

XSEDE. (2020). XSEDE Service Provider Checklist Version 4.3. http://hdl.handle.net/2142/108886

Cited By

Kee KJankowski HKnepper RSnapp-Childs WBrady R(2024)Fostering a Thriving Virtual Organizational Culture: The Development of Metrics for Ten Soft Skills in the Case of XSEDESN Computer Science10.1007/s42979-024-02980-55:5Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1007/s42979-024-02980-5
Shah DBelshe RSpeyer GYalim J(2024)Measuring the Impact of Centralized High-Performance Computing with Research Collaboration NetworksSN Computer Science10.1007/s42979-024-02890-65:5Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1007/s42979-024-02890-6
Snapp-Childs WHart DCosta CWernert JJankowski HTowns JStewart C(2024)Evaluating Return on Investment for Cyberinfrastructure Using the International Integrated Reporting FrameworkSN Computer Science10.1007/s42979-024-02889-z5:5Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1007/s42979-024-02889-z
Show More Cited By

Recommendations

Evaluating Return on Investment for Cyberinfrastructure Using the International Integrated Reporting Framework
Abstract
This paper investigates the return on investment (ROI) in cyberinfrastructure (CI) facilities and services by comparing the value of end products created to the cost of operations. We assessed the cost of a US CI facility called XSEDE and the ...
Metrics of financial effectiveness: Return On Investment in XSEDE, a national cyberinfrastructure coordination and support organization
PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

This paper explores the financial effectiveness of a national advanced computing support organization within the United States (US) called the eXtreme Science and Engineering Discovery Environment (XSEDE). XSEDE was funded by the National Science ...
XSEDE value added, cost avoidance, and return on investment
XSEDE '15: Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure

It is difficult for large research facilities to quantify a return on the investments that fund their operations. This is because there can be a time lag of years or decades between an innovation or discovery and the realization of its value through ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Scientometrics

Scientometrics Volume 128, Issue 6

Jun 2023

567 pages

ISSN:0138-9130

Issue’s Table of Contents

© The Author(s) 2023.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 14 February 2023

Accepted: 01 September 2022

Received: 28 December 2020

Author Tags

Author Tags

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kee KJankowski HKnepper RSnapp-Childs WBrady R(2024)Fostering a Thriving Virtual Organizational Culture: The Development of Metrics for Ten Soft Skills in the Case of XSEDESN Computer Science10.1007/s42979-024-02980-55:5Online publication date: 11-Jun-2024
https://dl.acm.org/doi/10.1007/s42979-024-02980-5
Shah DBelshe RSpeyer GYalim J(2024)Measuring the Impact of Centralized High-Performance Computing with Research Collaboration NetworksSN Computer Science10.1007/s42979-024-02890-65:5Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1007/s42979-024-02890-6
Snapp-Childs WHart DCosta CWernert JJankowski HTowns JStewart C(2024)Evaluating Return on Investment for Cyberinfrastructure Using the International Integrated Reporting FrameworkSN Computer Science10.1007/s42979-024-02889-z5:5Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1007/s42979-024-02889-z
Shah DBelshe RKuznia KSpeyer GYalim J(2023)Employing a Research Community Network to Assess Centralized Computing ImpactPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3593604(52-59)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3593604

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents