Abstract
We propose a search method that uses sentence-based inverted indexes to achieve high accuracy at practical speeds. The proposed method well supports the vast majority of queries entered on the web; these queries contain single words, multiple words for proximity searches, and semantically direct phrases. The existing approach, the inverted index which holds word-level position data is not efficient, because the size of index becomes extremely large. Our solution is to drop the word position data and index only the existence of each word in each sentence. We incorporate the sentence-based inverted index into a commercial search engine and evaluate it using both Japanese and English standard IR corpuses. The experiment shows that our method offers high accuracy, while index size and search processing time are greatly reduced.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Callan, J.P.: Passage-Level Evidence in Document Retrieval. In: Croftand, W.B., van Rijsbergen, C.J. (eds.) Proceedings of the Seventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland, July 1994, pp. 302–310. Springer, Heidelberg (1994)
Comer, D.: The ubiquitous b-tree. ACM Computing. Survey 11(2), 121–137 (1979)
Hatcher, E., Gospodnetic, O.: Lucene in Action (In Action series). Manning Publications Co., Greenwich (2004)
Ljosland, M.: Evaluation of web search engines and the search for better ranking algorithms. In: SIGIR 1999 Workshop on Evaluation of Web Retrieval international ACM SIGIR conference on Research and development in information retrieval (1999)
Manning, C.D., Raghavan, P., Schutze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)
Morrison, D.R.: Patricia:practical algorithm to retrieve information coded in alphanumeric. J. ACM 15(4), 514–534 (1968)
Navarro, G., Mäkinen, V.: Compressed full-text indexes. ACM Comput. Surv. 39(1), 2 (2007)
Robertson, S.E., Walker, S., Hancock-Beaulieu, M., Gull, A., Lau, M.: Okapi at TREC3. In: Text Retrieval Conference, pp. 21–30 (1992)
Salton, G.: Developments in automatic text retrieval. Science 253, 974–980 (1991)
Scholer, F., Williams, H., Yiannis, J., Zobel, J.: Compression of inverted indexes for fast query evaluation. In: 25th annual international ACM SIGIR conference on research and development in information retrieval, pp. 222–229 (2002)
Sekine, S., Isahara, H.: Irex: Ir and ie evaluation project in japanese. In: The 2nd International Conference on Language Resources and Evaluation, pp. 1475–1480 (2000)
Tao, T., Zhai, C.X.: An exploration of proximity measures in information retrieval. In: SIGIR 2007: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 295–302. ACM Press, New York (2007)
Tian, C., Tezuka, T., Oyama, S., Tajima, K., Tanaka, K.: Improving web retrieval precision based on semantic relationships and proximity of query keywords. In: Bressan, S., Küng, J., Wagner, R. (eds.) DEXA 2006. LNCS, vol. 4080, pp. 54–63. Springer, Heidelberg (2006)
Voorhees, E.M., Harman, D.: Overview of the eighth text retrieval conference (trec-8). In: TREC (1999)
Witten, I.H., Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, San Francisco (1999)
Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Computing. Survey 38(2), 6 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Uematsu, Y., Inoue, T., Fujioka, K., Kataoka, R., Ohwada, H. (2008). Proximity Scoring Using Sentence-Based Inverted Index for Practical Full-Text Search. In: Christensen-Dalsgaard, B., Castelli, D., Ammitzbøll Jurik, B., Lippincott, J. (eds) Research and Advanced Technology for Digital Libraries. ECDL 2008. Lecture Notes in Computer Science, vol 5173. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87599-4_33
Download citation
DOI: https://doi.org/10.1007/978-3-540-87599-4_33
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87598-7
Online ISBN: 978-3-540-87599-4
eBook Packages: Computer ScienceComputer Science (R0)