Abstract
The tremendous growth of the Web poses many challenges for all-purpose single-process crawlers including the presence of some irrelevant answers among search results and the coverage and scaling issues regarding the enormous dimension of the World Wide Web. Meanwhile, more enhanced and convincing algorithms are on demand to yield more precise and relevant search results in an appropriate amount of time. Due to the fact that employing the link based Web page importance metrics in search engines is not an absolute solution to identify the best answer set by the overall search system and because employing such metrics within a multi-processes crawler bears a considerable communication overhead on the overall system, employing a link independent Web page importance metric is required to govern the priority rule within the queue of fetched URLs. The aim of this paper is to propose a modest weighted architecture for a focused structured parallel crawler in which the credit assignment to the discovered URLs is performed upon a combined metric based on clickstream analysis and Web page text similarity analysis to the specified mapped topic(s).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Ahmadi-Abkenari, F., Selamat, A.: Application of Clickstream Analysis in a Tailored Focused Web Crawler. Journal of Communications of SIWN,The Systemic and Informatics World Network (2010)
Bharat, K., Henzinger, M.R.: Improved Algorithms for Topic Distillation in a Hyperlinked Environment. In: Proceeding of the 21st ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 104–111 (1998)
Brin, S., Page, L.: The Anatomy of a Large-Scale Hypertextual Web Search Engine. Computer Networks 30(1-7), 107–117 (1998)
Chakrabarti, S.: Mining the Web. In: Discovering Knowledge from Hypertext Data. Morgan Kaufmann, San Francisco (2003)
Chackrabarti, S.: Integrating Document Object Model with Hyperlinks for Enhanced Topic Distillation and Information Extraction. In: Proceeding of the 13th international World Wide Web Conference (WWW 2001), pp. 211–220 (2001)
Chackrabarti, S., Dom, B., Gibson, D., Kleinberg, J., Kumar, R., Raghavan, P., Rajagopalan, S., Tomkins, A.: Mining the Link Structure of the World Wide Web. IEEE Computer 32(8), 60–67 (1999)
Chackrabarti, S., Dom, B., Raghavan, P., Rajagopalan, S., Gibson, D., Kleinberg, J.: Automatic Resource Compilation by Analyzing Hyperlink Structure and Associated Text. In: Proceeding of the 7th international World Wide Web Conference, WWW 2007 (1998)
Chakrabarti, S., Van den Berg, M., Dom, B.: Focused Crawling: A New Approach to Topic Specific Web Resource Discovery. Computer Networks 31(11-16), 1623–1640 (1999)
Cho, J., Garcia-Molina, H.: Parallel Crawlers. In: Procceding of 11th International Conference on World Wide Web. ACM Press, New York (2002)
Cho, J., Garcia-Molina, H., Page, L.: Efficient Crawling through URL Ordering. In: Procceeding of 7th international Conference on World Wide Web (1998)
Diligenti, M., Coetzee, F.M., Lawrence, S., Giles, C.L., Gori, M.: Focused Crawling using Context Graph. In: Procceeding of the 26th VLDB Conference, Cairo, Egypt, pp. 527–534 (2000)
Giudici, P.: Applied Data Mining, Web Clickstream Analysis. ch.8, pp. 229–253. Wiley Press, Chichester (2003) ISBN: 0-470-84678-X
Kleinberg, J.: Authoritative Sources in a Hyperlinked Environment. Journal of the ACM 46(5), 604–632 (1999)
Liu, B.: Web Data Mining, Information Retrieval and Web Search. ch.6, pp. 183–215. Springer Press, Heidelberg (2007) ISBN: 3-540-37881-2
McCallum, A., Nigam, K.: A Comparison of Event Models for Naïve Baes Text Classification. In: Procceeding of the AAAI-1998 Workshop on Learning for Text Categorization (1998)
Menczer, F., Pant, G., Srinivasan, P.: Topical Web Crawlers: Evaluating Adaptive Algorithms. ACM Transactions on Internet Technology 4(4), 378–419 (2004)
Selamat, A., Ahmadi-Abkenari, F.: Application of Clickstream Analysis as Web Page Importance Metric in Parallel Crawlers. In: Procceeding of the International Symposium on Information Technology (ITSIM 2010), Kuala Lumpur, Malaysia (2010)
Srivastava, A.N., Sahami, M.: Text Mining, Classification, Custering and Applications. CRC Press, Boca Raton (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Selamat, A., Ahmadi-Abkenari, F. (2011). Architecture for a Parallel Focused Crawler for Clickstream Analysis. In: Nguyen, N.T., Kim, CG., Janiak, A. (eds) Intelligent Information and Database Systems. ACIIDS 2011. Lecture Notes in Computer Science(), vol 6591. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20039-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-20039-7_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20038-0
Online ISBN: 978-3-642-20039-7
eBook Packages: Computer ScienceComputer Science (R0)