More Web Proxy on the site http://driver.im/

research-article

Assessment of non-financial returns on cyberinfrastructure: A survey of current methods

Authors:

Craig A. Stewart,

David Y. Hancock,

Thomas Furlani,

Nicholas Berente,

Thomas Cheatham,

Shawn D. SlavinAuthors Info & Claims

HARC '19: Proceedings of the Humans in the Loop: Enabling and Facilitating Research on Cloud Computing

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/3355738.3355749

Published: 29 July 2019 Publication History

Abstract

In recent years, considerable attention has been given to assessing the value of investments in cyberinfrastructure (CI). This paper focuses on assessment of value measured in ways other than financial benefits - what might well be termed impact or outcomes. This paper is a companion to a paper presented at the PEARC'19 conference, which focused on methods for assessing financial returns on investment. In this paper we focus on methods for assessing impacts such as effect on publication production, importance of publications, and assistance with major scientific accomplishments as signified by major awards. We in particular focus on the role of humans in the loop - humanware. This includes a brief description of the roles humans play in facilitating use of research cyberinfratructure - including clouds - and then a discussion of how those impacts have been assessed. Our conclusion overall is that there has been more progress in the past very few years in developing methods for the quantitative assessment of financial returns on investment than there has been in assessing non-quantitative impacts. There are a few clear actions that many research institutions could take to start better assessing the non-financial impacts of investment in cyberinfrastructure. However, there is a great need for assessment efforts to turn more attention to the assessment of non-financial benefits of investment in cyberinfrastructure, particularly the benefits of investing in humans and the benefits to humans who are involved in supporting and using cyberinfrastructure, including clouds.

References

[1]

[n. d.]. Asia-Pacific Grid. http://www.apgridpma.org/. Accessed: 2019-07-01.

[2]

[n. d.]. CyVerse Publications. http://www.cyverse.org/about/. Accessed: 2019-07-01.

[3]

[n. d.]. European Data Initiative. https://eudat.eu/. Accessed: 2019-07-01.

[4]

[n. d.]. European Grid Initiative. https://www.egi.eu/. Accessed: 2019-07-01.

[5]

[n. d.]. European Open Science Cloud. https://ec.europa.eu/research/openscience/index.cfm?pg=open-science-cloud. Accessed: 2019-07-01.

[6]

[n. d.]. G-Index. https://en.wikipedia.org/wiki/G-index. Accessed: 2019-07-01.

[7]

[n. d.]. Galaxy Publications. https://galaxyproject.org/publication-library/. Accessed: 2019-07-01.

[8]

[n. d.]. National Center for Atmospheric Research. https://ncar.ucar.edu. Accessed: 2019-07-01.

[9]

[n. d.]. The Nobel Prize. https://sweden.se/society/the-nobel-prize-awarding-great-minds/. Accessed: 2019-07-01.

[10]

[n. d.]. Open Science Grid. https://opensciencegrid.org/. Accessed: 2019-07-01.

[11]

[n. d.]. The OpenStack Foundation. https://www.openstack.org. Accessed: 2019-07-01.

[12]

[n. d.]. Pervasive Technology Institute Publications. https://bibbase.org/service/mendeley/42d295c0-0737-38d6-8b43-508cab6ea85d. Accessed: 2019-07-01.

[13]

[n. d.]. Scopus. https://www.scopus.com/. Accessed: 2019-07-01.

[14]

[n. d.]. UITS User Satisfaction Survey. http://www.iu.edu/~uitssur/. Accessed: 2019-07-01.

[15]

2004. W.K. Kellog Foundation Logic Model Development Guide.

[16]

2018. Colleges With the Greatest Percentage Decreases in Full-Time Undergraduates. Chronicle of Higher Education 65, 15 (12 2018). https://www.chronicle.com/article/Colleges-With-the-Greatest/245285

[17]

Amy Apon, Stanley Ahalt, Vijay Dantuluri, Constantin Gurdgiev, Moez Limayem, Linh Ngo, and Michael Stealey. 2010. High Performance Computing Instrumentation and Research Productivity in U.S. Universities. Journal of Information Technology Impact 10, 2 (9 2010), 87--98. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1679248

[18]

Amy W. Apon, Linh B. Ngo, Michael E. Payne, and Paul W. Wilson. 2015. Assessing the effect of high performance computing capabilities on academic research output. Empirical Economics 48, 1 (01 Feb 2015), 283--312. https://doi.org/10.1007/s00181-014-0833-7

[19]

Jill Barshay. 2018. College students predicted to fall by more than 15% after the year 2025. The Hechinger Report (9 2018). https://hechingerreport.org/college-students-predicted-to-fall-by-more-than-15-after-the-year-2025/

[20]

Katy Börner, Richard Klavans, Michael Patek, Angela M Zoss, Joseph R Biberstine, Robert P Light, Vincent Larivière, and Kevin W Boyack. 2012. Design and update of a classification system: The UCSD map of science. PloS one 7, 7 (2012), e39464.

[21]

Clarivate. 2019. Web of Science Kernel. https://clarivate.com/products/web-of-science/

[22]

C. Collberg and T.A. Proebsting. 2016. Repeatability in Computer Science Systems Research. Commun. ACM 59, 3 (2016), 62--69.

Digital Library

[23]

Open Geospatial Consortium. 2019. OGC Testbed-14: Federated Clouds Engineering Report. http://www.opengis.net/doc/PER/t14-D023.

[24]

Lizanne DeStefano and Lorna Rivera. 2015. Cyberinfrastructure Value Assessment Report. Technical Report.

[25]

Jeremy Fischer, Brian W Beck, Sanjana Sudarshan, George Turner, Winona SnappChilds, Craig A Stewart, and David Y Hancock. 2018. Methodologies and practices for adoption of a novel national research environment. Proceedings of the Practice and Experience on Advanced Research Computing-PEARC 18 (2018).

Digital Library

[26]

Jeremy Fischer, David Y Hancock, John Michael Lowe, George Turner, Winona Snapp-Childs, and Craig A Stewart. 2017. Jetstream: A cloud system enabling learning in higher education communities. In Proceedings of the 2017 ACM Annual Conference on SIGUCCS. ACM, 67--72.

Digital Library

[27]

David Y Hancock, Craig A Stewart, Matthew Vaughn, Jeremy Fischer, John Michael Lowe, George Turner, Tyson L Swetnam, Tyler K Chafin, Enis Afgan, Marlon E Pierce, et al. 2017. JetstreamâĂŤEarly operations performance, adoption, and impacts. Concurrency and Computation: Practice and Experience (2017), e4683.

[28]

Jorge E Hirsch. 2005. An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences 102, 46 (2005), 16569--16572.

[29]

Jeremiah P. Ostriker, Paul W. Holland, Charlotte V. Kuh, and James A. Voytuk. 2011. A Data-Based Assessment of Research-Doctorate Programs in the United States (2011). Available online at http://www.nap.edu/rdp/.

[30]

Jeffrey T. Palmer, M. Gallo, Steven, Thomas R. Furlani, Matthew D. Jones, Robert L. DeLeon, Joseph P. White, Nikolay Simakov, Abani K. Patra, Jeanette Sperhac, Thomas Yearke, Ryan Rathsam, Martins Innus, Cynthia Cornelius, James Browne, William L. Barth, and T. Evans, Richard. 2015. Open XDMoD: A Tool for the Comprehensive Management of High Performance Computing Resources. Computing in Science and Engineering 17, 4 (2015), 52--62.

Digital Library

[31]

Christopher Peebles, Craig Stewart, Brian Voss, and Sue Workman. 2001. Measuring quality, cost, and value of it services. EDUCAUSE Conference Proceedings, Indianapolis IN.

[32]

Teresa Penfield, Matthew J Baker, Rosa Scoble, and Michael C Wykes. 2014. Assessment, evaluations, and definitions of research impact: A review. Research Evaluation 23, 1 (2014), 21--32.

[33]

Olga Scrivener, Gangadeep Singh, Sara E. Bouchard, Scott C. Hutcheson, Ben Fulton, Matthew R. Link, and Katy Borner. 2018. XD Metrics on Demand Value Analytics: Visualizing the Impact of Internal Information Technology Investments on External Funding, Publications, and Collaboration Networks. Frontiers in Research Metrics and Analytics (2018). https://doi.org/10.3389/frma.2017.00010

[34]

Craig A Stewart, William K Barnett, Eric A Wernert, Julie A Wernert, Von Welch, and Richard Knepper. 2015. Sustained Software for Cyberinfrastructure: Analyses of Successful Efforts with a Focus on NSF-funded Software. In Proceedings of the 1st Workshop on The Science of Cyberinfrastructure: Research, Experience, Applications and Models. ACM, 63--72.

Digital Library

[35]

Craig A Stewart, David Y Hancock, Julie Wernert, Thomas Furlani, David Lifka, Alan Sill, Nicholas Berente, Donald F McMullen, Thomas Cheatham, Amy Apon, et al. 2019. Assessment of financial returns on investments in cyberinfrastructure facilities: A survey of current methods. (2019).

[36]

Craig A. Stewart, David Y. Hancock, Julie Wernert, Matthew R. Link, Nancy Wilkins-Diehr, Therese Miller, Kelly Gaither, and Winona Snapp-Childs. 2018. Return on Investment for Three Cyberinfrastructure Facilities: A Local Campus Supercomputer, the NSF-Funded Jetstream Cloud System, and XSEDE (the eXtreme Science and Engineering Discovery Environment). In 2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing (UCC). IEEE, 223--236. https://doi.org/10.1109/UCC.2018.00031

[37]

Craig A. Stewart, R. Knepper, M.R. Link, M. Pierce, E.A. Wernert, and N. Wilkins-Diehr. 2017. Cyberinfrastructure, Cloud Computing, Science Gateways, Visualization, and Cyberinfrastructure Ease of Use. In Encyclopedia of Information Science and Technology, Fourth Edition (fourth ed.), M. Khosrow-Pour (Ed.). IGI Global, Hershey, PA, 1063--1074.

[38]

D.Y. Hancock-T. Miller J. Fischer L. Liming G. Turner J.M. Lowe S. Gregory E. Skidmore M. Vaughn D. Stanzione N. Merchant I. Foster J. Taylor P. Rad V. Brendel E. Afgan M. Packard W. Snapp-Childs. Stewart, C.A. 2018. Jetstream - a novel cloud system for science. In Contemporary High Performance Computing: From Petascale toward Exascale, Volume Three, J. Vetter (Ed.). Oxford University Press, Oxford, 189--222.

[39]

F. Pérez B. Granger M. Bussonier J. Frederic K. Kelley J. Hamrick J. Grout S. Corlay P. Ivanov D. Avila S. Abdalla C. Willing T. Kluyver, B. Ragan-Kelley and Jupyter Development Team. 2016. Jupyter NotebooksâĂŤa publishing format for reproducible computational workflows. https://dx.doi.org/10.3233/978-1-61499-649-1-87.

[40]

PR Thomas and David S Watkins. 1998. Institutional research rankings via bibliometric analysis and direct peer review: A comparative case study with policy implications. Scientometrics 41, 3 (1998), 335--355.

[41]

John Towns, Timothy Cockerill, Maytal Dahan, Ian Foster, Kelly Gaither, Andrew Grimshaw, Victor Hazlewood, Scott Lathrop, Dave Lifka, Gregory D Peterson, et al. 2014. XSEDE: accelerating scientific discovery. Computing in Science & Engineering 16, 5 (2014), 62--74.

[42]

Viswanath Venkatesh, Michael G Morris, Gordon B Davis, and Fred D Davis. 2003. User acceptance of information technology: Toward a unified view. MIS quarterly (2003), 425--478.

[43]

Gregor von Laszewski, Fugang Wang, Geoffrey C Fox, David L Hart, Thomas R Furlani, Robert L DeLeon, and Steven M Gallo. 2015. Peer comparison of xsede and ncar publication data. In 2015 IEEE International Conference on Cluster Computing. IEEE, 531--532.

Digital Library

[44]

G. von Laszewski, F. Wang, G. C. Fox, D. L. Hart, T. R. Furlani, R. L. DeLeon, and S. M. Gallo. 2015. Peer Comparison of XSEDE and NCAR Publication Data. In 2015 IEEE International Conference on Cluster Computing. 531--532. https://doi.org/10.1109/CLUSTER.2015.98

Cited By

Swetnam TAntin PBartelme RBucksch ACamhy DChism GChoi ICooksey ACosi MCowen CCulshaw-Maurer MDavey RDavey SDevisetty UEdgin TEdmonds AFedorov DFrady JFonner JGillan JHossain IJoyce BLang KLee TLittin SMcEwen IMerchant NMicklos DNelson ARamsey ARoberts SSarando PSkidmore ESong JSprinkle MSrinivasan SStanzione DStrootman JStryeck STuteja RVaughn MWali MWall MWalls RWang LWickizer TWilliams JWregglesworth JLyons E(2024)CyVerse: Cyberinfrastructure for open sciencePLOS Computational Biology10.1371/journal.pcbi.101127020:2(e1011270)Online publication date: 7-Feb-2024
https://doi.org/10.1371/journal.pcbi.1011270
McIntosh TVerleye EBalch JCattau MIlangakoon NKorinek NNagy RSanovia JSkidmore ESwetnam TTuff TQuarderer NWessman C(2023)Cyberinfrastructure deployments on public research clouds enable accessible Environmental Data Science educationPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3597606(367-373)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3597606
Skidmore ECosi MSwetnam TMerchant NXu ZChoi IDavey SFrady JWall MYung M(2023)Cloud Computing for Research and Education Gets a Sweet Upgrade with CACAOPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3597555(251-254)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3597555
Show More Cited By

Index Terms

Assessment of non-financial returns on cyberinfrastructure: A survey of current methods

Recommendations

Assessment of financial returns on investments in cyberinfrastructure facilities: A survey of current methods
PEARC '19: Practice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)

In recent years, considerable attention has been given to assessing the value of investments in cyberinfrastructure (CI). This paper includes a survey of current methods for the assessment of financial returns on investment (ROI) in CI. Applying the ...
Financial credit risk assessment: a recent review

The assessment of financial credit risk is an important and challenging research topic in the area of accounting and finance. Numerous efforts have been devoted into this field since the first attempt last century. Today the study of financial credit ...
Training a Neural Logic Network to predict financial returns: a case study

Companies in the S&P 500 produce quarterly financial statements that are closely studied by investors who forecast the stock market performance of those companies. This paper describes a Neural Logic Network (NLN) for predicting stock market returns ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

HARC '19: Proceedings of the Humans in the Loop: Enabling and Facilitating Research on Cloud Computing

July 2019

62 pages

ISBN:9781450372794

DOI:10.1145/3355738

Program Chair:
Craig A. Stewart
Pervasive Technology Institute, Indiana University

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

ACM: Association for Computing Machinery
Indiana University: Indiana University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

HARC '19

HARC '19: Enabling and Facilitating Research on Cloud Computing

July 29, 2019

IL, Chicago, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
218
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Swetnam TAntin PBartelme RBucksch ACamhy DChism GChoi ICooksey ACosi MCowen CCulshaw-Maurer MDavey RDavey SDevisetty UEdgin TEdmonds AFedorov DFrady JFonner JGillan JHossain IJoyce BLang KLee TLittin SMcEwen IMerchant NMicklos DNelson ARamsey ARoberts SSarando PSkidmore ESong JSprinkle MSrinivasan SStanzione DStrootman JStryeck STuteja RVaughn MWali MWall MWalls RWang LWickizer TWilliams JWregglesworth JLyons E(2024)CyVerse: Cyberinfrastructure for open sciencePLOS Computational Biology10.1371/journal.pcbi.101127020:2(e1011270)Online publication date: 7-Feb-2024
https://doi.org/10.1371/journal.pcbi.1011270
McIntosh TVerleye EBalch JCattau MIlangakoon NKorinek NNagy RSanovia JSkidmore ESwetnam TTuff TQuarderer NWessman C(2023)Cyberinfrastructure deployments on public research clouds enable accessible Environmental Data Science educationPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3597606(367-373)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3597606
Skidmore ECosi MSwetnam TMerchant NXu ZChoi IDavey SFrady JWall MYung M(2023)Cloud Computing for Research and Education Gets a Sweet Upgrade with CACAOPractice and Experience in Advanced Research Computing 2023: Computing for the Common Good10.1145/3569951.3597555(251-254)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3569951.3597555
Mahendran SCosta CWernert JStewart C(2022)Institutional Value of a Nobel PrizePractice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You10.1145/3491418.3535157(1-4)Online publication date: 8-Jul-2022
https://dl.acm.org/doi/10.1145/3491418.3535157
Stewart CWernert JCosta CHancock DKnepper RSnapp-Childs W(2022)Return on Investment in Research Cyberinfrastructure: State of the ArtPractice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You10.1145/3491418.3535131(1-4)Online publication date: 8-Jul-2022
https://dl.acm.org/doi/10.1145/3491418.3535131
Broude Geva SChalker AHillegas CPetravick DSill AStewart C(2021)Results from a second longitudinal survey of academic research computing and data center usage: expenditures, utilization patterns, and approaches to return on investmentPractice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions10.1145/3437359.3465589(1-4)Online publication date: 17-Jul-2021
https://dl.acm.org/doi/10.1145/3437359.3465589
Chalker AHillegas CSill ABroude Geva SStewart C(2020)Cloud and on-premises data center usage, expenditures, and approaches to return on investment: A survey of academic research computing organizationsPractice and Experience in Advanced Research Computing 2020: Catch the Wave10.1145/3311790.3396642(26-33)Online publication date: 26-Jul-2020
https://dl.acm.org/doi/10.1145/3311790.3396642
Kovatch PGai LCho HFluder EJiang D(2020)Optimizing High-Performance Computing Systems for Biomedical Workloads2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW50202.2020.00040(183-192)Online publication date: May-2020
https://doi.org/10.1109/IPDPSW50202.2020.00040
Voss BStewart C(2019)HumanwareProceedings of the Humans in the Loop: Enabling and Facilitating Research on Cloud Computing10.1145/3355738.3357001(1-6)Online publication date: 29-Jul-2019
https://dl.acm.org/doi/10.1145/3355738.3357001
Stewart CHancock DWernert JFurlani TLifka DSill ABerente NMcMullen DCheatham TApon APayne RSlavin SFurlani T(2019)Assessment of financial returns on investments in cyberinfrastructure facilitiesPractice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)10.1145/3332186.3332228(1-8)Online publication date: 28-Jul-2019
https://dl.acm.org/doi/10.1145/3332186.3332228

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten