More Web Proxy on the site http://driver.im/

Article

Downloading textual hidden web content through keyword queries

Authors:

Alexandros Ntoulas,

Junghoo ChoAuthors Info & Claims

JCDL '05: Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries

Pages 100 - 109

https://doi.org/10.1145/1065385.1065407

Published: 07 June 2005 Publication History

Abstract

An ever-increasing amount of information on the Web today is available only through search interfaces: the users have to type in a set of keywords in a search form in order to access the pages from certain Web sites. These pages are often referred to as the Hidden Web or the Deep Web. Since there are no static links to the Hidden Web pages, search engines cannot discover and index such pages and thus do not return them in the results. However, according to recent studies, the content provided by many Hidden Web sites is often of very high quality and can be extremely valuable to many users.In this paper, we study how we can build an effective Hidden Web crawler that can autonomously discover and download pages from the Hidden Web. Since the only "entry point" to a Hidden Web site is a query interface, the main challenge that a Hidden Web crawler has to face is how to automatically generate meaningful queries to issue to the site. Here, we provide a theoretical framework to investigate the query generation problem for the Hidden Web and we propose effective policies for generating queries automatically. Our policies proceed iteratively, issuing a different query in every iteration. We experimentally evaluate the effectiveness of these policies on 4 real Hidden Web sites and our results are very promising. For instance, in one experiment, one of our policies downloaded more than 90% of a Hidden Web site (that contains 14 million documents) after issuing fewer than 100 queries.

References

[1]

Lexisnexis http://www.lexisnexis.com.

[2]

The Open Directory Project, http://www.dmoz.org.

[3]

E. Agichtein and L. Gravano. Querying text databases for efficient information extraction. In ICDE, 2003.

[4]

E. Agichtein, P. Ipeirotis, and L. Gravano. Modeling query-based access to text databases. In WebDB, 2003.

[5]

Article on New York Times. Old Search Engine, the Library, Tries to Fit Into a Google World. Available at: http://www.nytimes.com/2004/06/21/technology/21LIBR.html, June 2004.

[6]

L. Barbosa and J. Freire. Siphoning hidden-web data through keyword-based interfaces. In SBBD, 2004.

[7]

M. K. Bergman. The deep web: Surfacing hidden value, http://www.press.umich.edu/jep/07-01/bergman.html.

[8]

K. Bharat and A. Broder. A technique for measuring the relative size and overlap of public web search engines. In WWW, 1998.

Digital Library

[9]

A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. In WWW, 1997.

Digital Library

[10]

J. Callan, M. Connell, and A. Du. Automatic discovery of language models for text databases. In SIGMOD, 1999.

Digital Library

[11]

J. P. Callan and M. E. Connell. Query-based sampling of text databases. Information Systems, 19(2):97--130, 2001.

Digital Library

[12]

K. C.-C. Chang, B. He, C. Li, and Z. Zhang. Structured databases on the web: Observations and implications. Technical report, UIUC.

[13]

J. Cho, N. Shivakumar, and H. Garcia-Molina. Finding replicated web collections. In SIGMOD, 2000.

Digital Library

[14]

W. Cohen and Y. Singer. Learning to query the web. In AAAI Workshop on Internet-Based Information Systems, 1996.

[15]

J. Cope, N. Craswell, and D. Hawking. Automated discovery of search interfaces on the web. In 14th Australasian conference on Database technologies, 2003.

Digital Library

[16]

T. H. Cormen, C. E. Leiserson, and R. L. Rivest. Introduction to Algorithms, 2nd Edition. MIT Press/McGraw Hill, 2001.

Digital Library

[17]

D. Florescu, A. Y. Levy, and A. O. Mendelzon. Database techniques for the world-wide web: A survey. SIGMOD Record, 27(3):59--74, 1998.

Digital Library

[18]

B. He and K. C.-C. Chang. Statistical schema matching across web query interfaces. In SIGMOD Conference, 2003.

Digital Library

[19]

P. Ipeirotis and L. Gravano. Distributed search over the hidden web: Hierarchical database sampling and selection. In VLDB, 2002.

Digital Library

[20]

P. G. Ipeirotis, L. Gravano, and M. Sahami. Probe, count, and classify: Categorizing hidden web databases. In SIGMOD, 2001.

Digital Library

[21]

C. Lagoze and H. V. Sompel. The Open Archives Initiative: Building a low-barrier interoperability framework In JCDL, 2001.

Digital Library

[22]

S. Lawrence and C. L. Giles. Searching the World Wide Web. Science, 280(5360):98--100, 1998.

[23]

V. Z. Liu, J. C. Richard C. Luo~and, and W. W. Chu. Dpro: A probabilistic approach for hidden web database selection using dynamic probing. In ICDE, 2004.

[24]

X. Liu, K. Maly, M. Zubair and M. L. Nelson. DP9-An OAI Gateway Service for Web Crawlers. In JCDL, 2002.

Digital Library

[25]

B. B. Mandelbrot. Fractal Geometry of Nature. W. H. Freeman & Co.

[26]

A. Ntoulas, J. Cho, and C. Olston. What's new on the web? the evolution of the web from a search engine perspective. In WWW, 2004.

Digital Library

[27]

A. Ntoulas, P. Zerfos, and J. Cho. Downloading hidden web content. Technical report, UCLA, 2004.

[28]

S. Olsen. Does search engine's power threaten web's independence? http://news.com.com/2009-1023-963618.html.

[29]

S. Raghavan and H. Garcia-Molina. Crawling the hidden web. In VLDB, 2001.

Digital Library

[30]

G. K. Zipf. Human Behavior and the Principle of Least-Effort. Addison-Wesley, Cambridge, MA, 1949.

Cited By

Alian PNashid NShahbandeh MMesbah AChristakis MPradel M(2024)Semantic Constraint Inference for Web Form Test GenerationProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680332(932-944)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680332
Bisht AKumar K(2024)CHURN FORECASTING USING DEEP LJEARNING MODEL2024 International Conference on Intelligent Systems for Cybersecurity (ISCS)10.1109/ISCS61804.2024.10581266(01-05)Online publication date: 3-May-2024
https://doi.org/10.1109/ISCS61804.2024.10581266
Malhotra DBhatia RKumar M(2023)Automated Selection of Web Form Text Field Values Based on Bayesian InferencesInternational Journal of Information Retrieval Research10.4018/IJIRR.31839913:1(1-13)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.4018/IJIRR.318399
Show More Cited By

Index Terms

Downloading textual hidden web content through keyword queries
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
2. Information systems
  1. Information retrieval
    1. Document representation
  2. Information systems applications
    1. Digital libraries and archives

Recommendations

Focused crawling for the hidden web

A constantly growing amount of high-quality information resides in databases and is guarded behind forms that users fill out and submit. The Hidden Web comprises all these information sources that conventional web crawlers are incapable of discovering. ...
Deep Web crawling: a survey

Deep Web crawling refers to the problem of traversing the collection of pages in a deep Web site, which are dynamically generated in response to a particular query that is submitted using a search form. To achieve this, crawlers need to be endowed with ...
Processing Keyword Queries Under Access Limitations
Semantic Keyword-Based Search on Structured Data Sources
Abstract
The Deep Web is constituted by data accessible through web pages, but not readily indexable by search engines, as they are returned in dynamic pages. In this paper we propose a framework for accessing Deep Web sources, represented as relational ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

JCDL '05: Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries

June 2005

450 pages

ISBN:1581138768

DOI:10.1145/1065385

General Chair:
Mary Marlino
DLESE Program Center, University Corporation for Atmospheric Research (UCAR)
,
Program Chairs:
Tamara Sumner
University of Colorado at Boulder
,
Frank Shipman
Texas A & M University

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

JCDL05

Sponsor:

JCDL05: Joint Conference on Digital Libraries 2005

June 7 - 11, 2005

CO, Denver, USA

Acceptance Rates

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Upcoming Conference

JCDL '24

Sponsor:
sigir
sigir

The 2024 ACM/IEEE Joint Conference on Digital Libraries

December 16 - 20, 2024

Hong Kong , China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

120
Total Citations
View Citations
1,222
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alian PNashid NShahbandeh MMesbah AChristakis MPradel M(2024)Semantic Constraint Inference for Web Form Test GenerationProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680332(932-944)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680332
Bisht AKumar K(2024)CHURN FORECASTING USING DEEP LJEARNING MODEL2024 International Conference on Intelligent Systems for Cybersecurity (ISCS)10.1109/ISCS61804.2024.10581266(01-05)Online publication date: 3-May-2024
https://doi.org/10.1109/ISCS61804.2024.10581266
Malhotra DBhatia RKumar M(2023)Automated Selection of Web Form Text Field Values Based on Bayesian InferencesInternational Journal of Information Retrieval Research10.4018/IJIRR.31839913:1(1-13)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.4018/IJIRR.318399
ALANOĞLU ZAKCAYOL M(2023)Web Tarayıcılarında Tohum URL Seçimi ve Performans Analizi: Kapsamlı Bir İncelemeSeed URL Selection and Performance Analysis in Web Crawlers: A Comprehensive ReviewDüzce Üniversitesi Bilim ve Teknoloji Dergisi10.29130/dubited.109712311:3(1399-1423)Online publication date: 31-Jul-2023
https://doi.org/10.29130/dubited.1097123
ALANOĞLU ZAKCAYOL M(2023)EMACrawler: Web Arama Motoru Veritabanı Tazeliği OptimizasyonuEMACrawler: Web Search Engine Database Freshness OptimizationJournal of Polytechnic10.2339/politeknik.1347054Online publication date: 15-Nov-2023
https://doi.org/10.2339/politeknik.1347054
Cassavia NCaviglione LGuarascio MLiguori AManco GZuppelli M(2023)A federated approach for detecting data hidden in icons of mobile applications delivered via web and multiple storesSocial Network Analysis and Mining10.1007/s13278-023-01121-913:1Online publication date: 14-Sep-2023
https://doi.org/10.1007/s13278-023-01121-9
Contreras Medina FMarín A(2022)La visualidad algorítmica: una aproximación social a la visión artificial en la era post internetArte, Individuo y Sociedad10.5209/aris.7466434:2(627-647)Online publication date: 13-Jan-2022
https://doi.org/10.5209/aris.74664
Gupta SBhatia K(2021)Design of a Parallel and Scalable Crawler for the Hidden WebInternational Journal of Information Retrieval Research10.4018/IJIRR.28961212:1(1-23)Online publication date: 15-Oct-2021
https://doi.org/10.4018/IJIRR.289612
U. AThampi S(2019)Dark Web and Its Research ScopesApplying Methods of Scientific Inquiry Into Intelligence, Security, and Counterterrorism10.4018/978-1-5225-8976-1.ch010(240-268)Online publication date: 2019
https://doi.org/10.4018/978-1-5225-8976-1.ch010
Wang PShea RWang JWu EBoncz PManegold SAilamaki ADeshpande AKraska T(2019)Progressive Deep Web Crawling Through Keyword Queries For Data EnrichmentProceedings of the 2019 International Conference on Management of Data10.1145/3299869.3319899(229-246)Online publication date: 25-Jun-2019
https://dl.acm.org/doi/10.1145/3299869.3319899
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents