Article

Building a research library for the history of the web

Authors:

Lucia WalleAuthors Info & Claims

JCDL '06: Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries

Pages 95 - 102

https://doi.org/10.1145/1141753.1141771

Published: 11 June 2006 Publication History

Get Access

Abstract

This paper describes the building of a research library for studying the Web, especially research on how the structure and content of the Web change over time. The library is particularly aimed at supporting social scientists for whom the Web is both a fascinating social phenomenon and a mirror on society.The library is built on the collections of the Internet Archive, which has been preserving a crawl of the Web every two months since 1996. The technical challenges in organizing this data for research fall into two categories: high-performance computing to transfer and manage the very large amounts of data, and human-computer interfaces that empower research by non-computer specialists.

References

[1]

Arms, W., Aya, S., Dmitriev, P., Kot, B., Mitchell, R., Walle, L., A Research Library for the Web based on the Historical Collections of the Internet Archive. D-Lib Magazine. February 2006. http://www.dlib.org/dlib/february06/arms/02arms.html

Google Scholar

[2]

Bergmark, D., Collection synthesis. ACM/IEEE-CS Joint Conference on Digital Libraries, 2002.

Digital Library

Google Scholar

[3]

Brin, S., and Page. L., The anatomy of a large-scale hypertextual Web search engine. Seventh International World Wide Web Conference. Brisbane, Australia, 1998.

Digital Library

Google Scholar

[4]

Burner, M., and Kahle, B., Internet Archive ARC File Format, 1996. http://archive.org/web/researcher/ArcFileFormat.php

Google Scholar

[5]

Chakrabarti, D., Zhan, Y., and Faloutsos, C., R-MAT: recursive model for graph mining. SIAM International Conference on Data Mining, 2004.

Crossref

Google Scholar

[6]

Gerner, N., Sosa, C., Fall 2005 Semester Report for Web Lab Database Load Group. M.Eng. report, Computer Science Department, Cornell University, 2005. http://www.infosci.cornell.edu/SIN/WebLib/papers/Gerner2005.doc.

Google Scholar

[7]

Ghemawat, S., Gobioff, H. and Leung, S., The Google File System. 19th ACM Symposium on Operating Systems Principles, October 2003.

Digital Library

Google Scholar

[8]

Jeyabalan, K., Kallukalam, J., Representation of Web Graph for in Memory Computation. M.Eng. report, Computer Science Department, Cornell University, 2005. http://www.infosci.cornell.edu/SIN/WebLib/papers/JeyabalanKallukalam2005.doc.

Google Scholar

[9]

J. Kleinberg. Authoritative sources in a hyperlinked environment. Ninth ACM-SIAM Symposium on Discrete Algorithms, 1998.

Digital Library

Google Scholar

[10]

Mitchell, S., Mooney, M., Mason, J., Paynter, G., Ruscheinski, J., Kedzierski, A., Humphreys, K., iVia Open Source Virtual Library System. D-Lib Magazine, 9 (1), January 2003. http://www.dlib.org/dlib/january03/mitchell/01mitchell.html

Google Scholar

[11]

Shah, S., Generating a web graph. M.Eng. report, Computer Science Department, Cornell University, 2005. http://www.infosci.cornell.edu/SIN/WebLib/papers/Shah2005a.doc.

Google Scholar

[12]

Shah, S., Retro Browser. M.Eng. report, Computer Science Department, Cornell University, 2005. http://www.infosci.cornell.edu/SIN/WebLib/papers/Shah2005b.pdf.

Google Scholar

Cited By

View all

Alnoamany YWeigle MNelson M(2016)Detecting off-topic pages within TimeMaps in Web archivesInternational Journal on Digital Libraries10.1007/s00799-016-0183-517:3(203-221)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1007/s00799-016-0183-5
AlNoamany YWeigle MNelson M(2015)Detecting Off-Topic Pages in Web ArchivesResearch and Advanced Technology for Digital Libraries10.1007/978-3-319-24592-8_17(225-237)Online publication date: 28-Nov-2015
https://doi.org/10.1007/978-3-319-24592-8_17
Damnjanovic UHermon S(2012)Generating content for digital libraries using an interactive content management systemProceedings of the Second international conference on Theory and Practice of Digital Libraries10.1007/978-3-642-33290-6_54(474-479)Online publication date: 23-Sep-2012
https://dl.acm.org/doi/10.1007/978-3-642-33290-6_54
Show More Cited By

Index Terms

Building a research library for the history of the web
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
  2. Law, social and behavioral sciences
    1. Sociology
2. Information systems
  1. Information systems applications
    1. Digital libraries and archives

Recommendations

Web Archiving and Digital Libraries (WADL)
JCDL '15: Proceedings of the 15th ACM/IEEE-CS Joint Conference on Digital Libraries

This workshop will explore integration of Web archiving and digital libraries, so the complete life cycle involved is covered: creation/authoring, uploading/publishing in the Web (2.0), (focused) crawling, indexing, exploration (searching, browsing), ...
Building interoperable digital library services: MARIAN, open archives, and the NDLTD
SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval

In this demonstration, we present interoperable and personalized search services for the Networked Digital Library of Theses and Dissertations (NDLTD). Using standard protocols and software, including those specified by the Open Archives Initiative (OAI)...
To Re-experience the Web: A Framework for the Transformation and Replay of Archived Web Pages
When replaying an archived web page, or memento, the fundamental expectation is that the page should be viewable and function exactly as it did at the archival time. However, this expectation requires web archives upon replay to modify the page and its ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

JCDL '06: Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries

June 2006

402 pages

ISBN:1595933549

DOI:10.1145/1141753

General Chair:
Gary Marchionini
University of North Carolina, Chapel Hill
,
Program Chairs:
Michael L. Nelson
Old Dominion University
,
Catherine C. Marshall
Microsoft

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

JCDL06

Sponsor:

JCDL06: Joint Conference on Digital Libraries 2006

June 11 - 15, 2006

NC, Chapel Hill, USA

Acceptance Rates

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Upcoming Conference

JCDL '24

Sponsor:
sigir
sigir

The 2024 ACM/IEEE Joint Conference on Digital Libraries

December 16 - 20, 2024

Hong Kong , China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
1,148
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alnoamany YWeigle MNelson M(2016)Detecting off-topic pages within TimeMaps in Web archivesInternational Journal on Digital Libraries10.1007/s00799-016-0183-517:3(203-221)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1007/s00799-016-0183-5
AlNoamany YWeigle MNelson M(2015)Detecting Off-Topic Pages in Web ArchivesResearch and Advanced Technology for Digital Libraries10.1007/978-3-319-24592-8_17(225-237)Online publication date: 28-Nov-2015
https://doi.org/10.1007/978-3-319-24592-8_17
Damnjanovic UHermon S(2012)Generating content for digital libraries using an interactive content management systemProceedings of the Second international conference on Theory and Practice of Digital Libraries10.1007/978-3-642-33290-6_54(474-479)Online publication date: 23-Sep-2012
https://dl.acm.org/doi/10.1007/978-3-642-33290-6_54
Tanaka KJatowtf A(2010)Automatic knowledge acquisition from historical document archivesCulture and computing10.5555/1985559.1985575(161-172)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.5555/1985559.1985575
Wang GSalles MSowell BWang XCao TDemers AGehrke JWhite W(2010)Behavioral simulations in MapReduceProceedings of the VLDB Endowment10.14778/1920841.19209623:1-2(952-963)Online publication date: 1-Sep-2010
https://dl.acm.org/doi/10.14778/1920841.1920962
Tanaka KJatowt A(2010)Automatic Knowledge Acquisition from Historical Document Archives: Historiographical PerspectiveCulture and Computing10.1007/978-3-642-17184-0_13(161-172)Online publication date: 2010
https://doi.org/10.1007/978-3-642-17184-0_13
McCown FNelson MHeath FRice-Lively MFuruta R(2009)A framework for describing web repositoriesProceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries10.1145/1555400.1555456(341-344)Online publication date: 15-Jun-2009
https://dl.acm.org/doi/10.1145/1555400.1555456
Anand ABedathur SBerberich KSchenkel RTryfonopoulos CHeath FRice-Lively MFuruta R(2009)EverLastProceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries10.1145/1555400.1555455(331-340)Online publication date: 15-Jun-2009
https://dl.acm.org/doi/10.1145/1555400.1555455
Jatowt AKawai YTanaka K(2009)Browsing Assistant for Changing PagesIntelligent Agents in the Evolution of Web and Applications10.1007/978-3-540-88071-4_7(137-160)Online publication date: 2009
https://doi.org/10.1007/978-3-540-88071-4_7
Nelson MMccown FSmith JKlein M(2007)Using the web infrastructure to preserve web pagesInternational Journal on Digital Libraries10.5555/2794654.32699446:4(327-349)Online publication date: 1-Jul-2007
https://dl.acm.org/doi/10.5555/2794654.3269944
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Web Archiving and Digital Libraries (WADL)

Building interoperable digital library services: MARIAN, open archives, and the NDLTD

To Re-experience the Web: A Framework for the Transformation and Replay of Archived Web Pages