poster

Graph-based seed selection for web-scale crawlers

Authors:

Shuyi Zheng,

Pavel Dmitriev,

C. Lee GilesAuthors Info & Claims

CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Pages 1967 - 1970

https://doi.org/10.1145/1645953.1646277

Published: 02 November 2009 Publication History

Get Access

Abstract

One of the most important steps in web crawling is determining the starting points, or seed selection. This paper identifies and explores the problem of seed selection in web-scale incremental crawlers. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a repository with more "good" and less "bad" pages. We propose a graph-based framework for crawler seed selection, and present several algorithms within this framework. Evaluation on real web data showed significant improvements over heuristic seed selection approaches.

References

[1]

A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, and J. Wiener. Graph structure in the Web. Computer Networks, 33(1-6):309--320, 2000.

Digital Library

Google Scholar

[2]

D. Hochbaum and A. Pathria. Analysis of the Greedy Approach in Problems of Maximum k-Coverage. Naval Research Logistics, 45(6):615--627, 1998.

Crossref

Google Scholar

[3]

G. Pant, P. Srinivasan, and F. Menczer. Crawling the Web. Web Dynamics, pages 153--178, 2004.

Crossref

Google Scholar

Cited By

View all

ALANOĞLU ZAKÇAYOL M(2023)Web Tarayıcıları için Etkili Tohum URL Seçimi ve Kapsam Genişletme AlgoritmasıEffective Seed URL Selection and Scope Extension Algorithm for Web CrawlerInternational Journal of Advances in Engineering and Pure Sciences10.7240/jeps.117419335:1(27-38)Online publication date: 30-Mar-2023
https://doi.org/10.7240/jeps.1174193
Jayawardana YNwala AJayawardena GWu JJayarathna SNelson MLee Giles C(2020)Modeling Updates of Scholarly Webpages Using Archived Data2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377796(1868-1877)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377796
Sanagavarapu LSarangi SReddy Y(2018)ABC Algorithm for URL ExtractionCurrent Trends in Web Engineering10.1007/978-3-319-74433-9_12(143-148)Online publication date: 22-Feb-2018
https://doi.org/10.1007/978-3-319-74433-9_12
Show More Cited By

Index Terms

Graph-based seed selection for web-scale crawlers
1. Information systems
  1. Information retrieval

Recommendations

Graph based crawler seed selection
WWW '09: Proceedings of the 18th international conference on World wide web

This paper identifies and explores the problem of seed selection in a web-scale crawler. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can ...
A Web Mining Architectural Model of Distributed Crawler for Internet Searches Using PageRank Algorithm
APSCC '08: Proceedings of the 2008 IEEE Asia-Pacific Services Computing Conference

As the World Wide Web is growing rapidly and data in the present day scenario is stored in a distributed manner. The need to develop a search engine based architectural model for people to search through the Web. Broad web search engines as well as many ...
Web graph analyzer tool
valuetools '06: Proceedings of the 1st international conference on Performance evaluation methodolgies and tools

We present the software tool "Web Graph Analyzer". This tool is designed to perform a comprehensive analysis of the Web Graph structure. By Web Graph we mean a graph whose vertices are Web pages and whose edges are hyper-links. With the help of the Web ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

November 2009

2162 pages

ISBN:9781605585123

DOI:10.1145/1645953

General Chairs:
David Cheung
University of Hong Kong, Hong Kong
,
Il-Yeol Song
Drexel University, USA
,
Program Chairs:
Wesley Chu
UCLA, USA
,
Xiaohua Hu
Drexel University, USA
,
Jimmy Lin
University of Maryland, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

CIKM '09

Sponsor:

CIKM '09: Conference on Information and Knowledge Management

November 2 - 6, 2009

Hong Kong, China

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
255
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)3

Reflects downloads up to 04 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

ALANOĞLU ZAKÇAYOL M(2023)Web Tarayıcıları için Etkili Tohum URL Seçimi ve Kapsam Genişletme AlgoritmasıEffective Seed URL Selection and Scope Extension Algorithm for Web CrawlerInternational Journal of Advances in Engineering and Pure Sciences10.7240/jeps.117419335:1(27-38)Online publication date: 30-Mar-2023
https://doi.org/10.7240/jeps.1174193
Jayawardana YNwala AJayawardena GWu JJayarathna SNelson MLee Giles C(2020)Modeling Updates of Scholarly Webpages Using Archived Data2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377796(1868-1877)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377796
Sanagavarapu LSarangi SReddy Y(2018)ABC Algorithm for URL ExtractionCurrent Trends in Web Engineering10.1007/978-3-319-74433-9_12(143-148)Online publication date: 22-Feb-2018
https://doi.org/10.1007/978-3-319-74433-9_12
Vieira KBarbosa LSilva AFreire JMoura E(2016)Finding seeds to bootstrap focused crawlersWorld Wide Web10.1007/s11280-015-0331-719:3(449-474)Online publication date: 1-May-2016
https://dl.acm.org/doi/10.1007/s11280-015-0331-7
Samar TTraub Mvan Ossenbruggen Jde Vries A(2016)Comparing Topic Coverage in Breadth-First and Depth-First Crawls Using Anchor TextsResearch and Advanced Technology for Digital Libraries10.1007/978-3-319-43997-6_11(133-146)Online publication date: 10-Aug-2016
https://doi.org/10.1007/978-3-319-43997-6_11
Stergiou STsioutsiouliklis KCao LZhang CJoachims TWebb GMargineantu DWilliams G(2015)Set Cover at Web ScaleProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining10.1145/2783258.2783315(1125-1133)Online publication date: 10-Aug-2015
https://dl.acm.org/doi/10.1145/2783258.2783315
Pirnau M(2015)Considerations on the functions and importance of a web crawler2015 7th International Conference on Electronics, Computers and Artificial Intelligence (ECAI)10.1109/ECAI.2015.7301171(Y-17-Y-22)Online publication date: Jun-2015
https://doi.org/10.1109/ECAI.2015.7301171
Wu JTeregowda PRamírez JMitra PZheng SGiles CContractor NUzzi BMacy MNejdl W(2012)The evolution of a crawling strategy for an academic document search engineProceedings of the 4th Annual ACM Web Science Conference10.1145/2380718.2380762(340-343)Online publication date: 22-Jun-2012
https://dl.acm.org/doi/10.1145/2380718.2380762
Böhm CKny EEmde BAbedjan ZNaumann F(2011)SPRINTProceedings of the 14th International Conference on Extending Database Technology10.1145/1951365.1951437(546-549)Online publication date: 21-Mar-2011
https://dl.acm.org/doi/10.1145/1951365.1951437
Denev DMazeika ASpaniol MWeikum G(2011)The SHARC framework for data quality in Web archivingThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-011-0219-920:2(183-207)Online publication date: 1-Apr-2011
https://dl.acm.org/doi/10.1007/s00778-011-0219-9
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Graph based crawler seed selection

A Web Mining Architectural Model of Distributed Crawler for Internet Searches Using PageRank Algorithm

Web graph analyzer tool