[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.5555/779343.779356guidebooksArticle/Chapter ViewAbstractPublication PagesBookacm-pubtype
chapter

User-oriented evaluation methods for information retrieval: a case study based on conceptual models for query expansion

Published: 01 January 2003 Publication History

Abstract

This chapter discusses evaluation methods based on the use of nondichotomous relevance judgments in information retrieval (IR) experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user's point of view in modern large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. Query expansion is based on concepts, which are selected from a conceptual model, and then expanded by semantic relationships given in the model. The test is run with a best-match retrieval system (InQuery) in a text database consisting of newspaper articles. The case study indicates the usability of domain-dependent conceptual models in query expansion for IR. The results show that expanded queries with a strong query structure are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that nondichotomous relevance assessments are applicable in IR experiments and allow harder testing of IR methods. Proposed methods are user-oriented because users' benefits and efforts--highly relevant documents and number of documents to be examined-- are taken into account.

References

[1]
Alkula, R. (2001). From plain character strings to meaningful words: Producing better full text databases for inflectional and compounding languages with morphological analysis software. Information Retrieval 4(3/4), 195-208.
[2]
Allan, J., J. Callan, B. Croft, L. Ballesteros, J. Broglio, J. Xu, and H. Shu (1997). INQUERY at TREC 5. In Information technology: The Fifth Text Retrieval Conference (TREG-5), ed. E. M. Voorhees and D. K. Harman, 119-132. National Institute of Standards and Technology.
[3]
Bechhofer, S., L. Carr, C. Goble, and W. Hall (2001). Conceptual open hypermedia = the semantic web? Position paper. In Proceeedings of the Second International Workshop on the Semantic Web--SemWeb'01.
[4]
Belkin, N. J., and W. B. Croft (1987). Retrieval techniques. In Annual Review of Information Science and Technology, Vol 22, ed. M. E. Williams. 109-145. Amsterdam: Elsevier.
[5]
Blair, D. C., and M. E. Maron (1985). An evaluation of retrieval effectiveness for a full-text document-retrieval system. Communications of the ACM 28(3), 289-299.
[6]
Borlund, P. (2000). Evaluation of interactive information retrieval systems. Ph.D. thesis, Abo Akademi University.
[7]
Borlund, P., and P. Ingwersen (1998). Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, eds. W. Croft, A. Moffat, C. van Rijsbergen, R. Wilkinson, and J. Zobel, 324-331.
[8]
Chaffee, J., and S. Gauch (2000). Personal ontologies for web navigation. In Proceedings of the Ninth International Conference on Information Knowledge Management--CIKM 2000, 227-234.
[9]
Chen, H., and S. Dumais (2000). Bringing order to the web: Automatically categorizing search results. CHI Letters 2(1), 145-152.
[10]
Conover, W. J. (1980). Practical Nonparametric Statistics, 2nd ed, New York: John Wiley & Sons.
[11]
Cooper, W. S. (1968). Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems. Journal of the American Society for Information Science 19(1), 30-41.
[12]
Cosijn, E:, and P. Ingwersen (2000). Dimensions of relevance. Information Processing and Management 36(4), 533-550.
[13]
Efthimiadis, E. N. (1996). Query expansion. In Annual Review of Information Science and Technology, Vol. 31, ed. M. E. Williams, 121-187.
[14]
Green, R. (1995). The expression of conceptual syntagmatic relationshps: A comparative survey. Journal of Documentation 51(4), 315-338.
[15]
Guarino, N., C. Masolo, and G. Vetere (1998). Ontoseek: Using large linguistic ontologies for gathering information resources from the web. Technical Report 01/98, LADSEB-CNR.
[16]
Harman, D. K. (1995). Overview of the fourth text retrieval conference (TREC-4). Available at trec.nist.gov/pubs/trec4/papers/overview.ps.
[17]
Hersh, W. R. (1996). Information Retrieval: A Health Care Perspective. Berlin: Springer-Verlag.
[18]
Hersh, W. R., and D. H. Hickam (1995). An evaluation of interactive Boolean and natural language searching with an online medical textbook. Journal of the American Society for Information Science 46(7), 478-489.
[19]
Hull, D. (1993). Using statistical testing in the evaluation of retrieval experiments. In Proceedings of the 16th International Conference on Research and Development in Information Retrieval, eds. R. Korfhage, E. M. Rasmussen, and P. Willett, 329-338.
[20]
Ingwersen, P., and P. Willett (1995). An introduction to algorithmic and cognitive approaches for information retrieval. Libri 45, 160-177.
[21]
Jacob, E. K. (1991). Classification and categorization: Drawing the line. In Advances in Classification Research. Proceedings of the 2nd ASIS SIG/CR Classification Research Workshop, Vol. 2, ed. B. H. Kwasnik and R. Fidel, 67-83.
[22]
Järvelin, K., and J. Kekäläinen (2000). IR evaluation methods for highly relevant documents. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ed. N. J. Belkin, P. Ingwersen, and M.-K, Leong, 41-48.
[23]
Järvelin, K., J. Kekäläinen, and T. Niemi (2001). Expansiontool: Concept-based query expansion and construction. Information Retrieval 4(3/4), 231-255.
[24]
Järvelin, K., J. Kristensen, E. Sormunen and H. Keskustalo (1996). A deductive data model for query expansion. In Proceedings of the 19th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, ed. H.-P. Frei, D. Harman. P. Schäuble, and R. Wilkinson, 235-249.
[25]
Keen, E. M. (1991). The use of term position devices in ranked output experiments. Journal of Documentation 47(1), 1-22.
[26]
Keen, E. M. (1992). Presenting results of experimental retrieval comparisons. Information Processing and Management 28(4), 491-501.
[27]
Kekäläinen, J. (1999). The effects of query complexity, expansion and structure on retrieval performance in probabilistic text retrieval. Ph. D. thesis, University of Tampere. Available at www.info.uta.fi/research/postscript_docs/JK1_99.pdf.
[28]
Kekäläinen, J., and K. Järvelin (1998). The impact of query structure and query expansion on retrieval performance. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ed. W. B. Croft. A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, 130-137.
[29]
Kekäläinen, J., and K. Järvelin (2000). The co-effects of query structure and expansion on retrieval performance in probabilistic text retrieval. Information Retrieval 1(4), 329-344.
[30]
Korfhage, R. (1997). Information Storage and Retrieval. New York: John Wiley & Sons.
[31]
Lancaster, F. W. (1986). Vocabulary Control for Information Retrieval, 2nd ed. Arlington, VA: Information Resources Press.
[32]
Losee, R. M. (1998). Text Retrieval and Filtering: Analytic Models of Performance. Boston: Kluwer.
[33]
Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM 38(11), 39-41.
[34]
Myaeng, S. H., and R. R. Korfhage (1990). Integration of user profiles: Models and experiments in information retrieval. Information Processing and Management 26(6), 719-738.
[35]
Over, P. (1999). TREC-7 interactive track report. Available at trec.nist.gov/pubs/trec7/papers/t7irep.pdf.gz.
[36]
Rajashekar, T. B., and W. B. Croft (1995). Combining automatic and manual index representations in probabilistic retrieval. Journal of the American Society for Information Science 46(4), 272-283.
[37]
Robertson, S. E., and N. J. Betkin (1978). Ranging in principle. Journal of Documentation 34(2), 93-100.
[38]
Rocchio, Jr., J. J. (1966). Document retrieval systems--Optimization and evaluation. Ph. D. thesis, Harvard University.
[39]
Salton, G. (1989). Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Boston: Addison-Wesley.
[40]
Salton. G., and M. J. McGill (1983). Introduction to Modern Information Retrieval. New York: McGraw-Hill.
[41]
Saracevic, T. (1996). Relevance reconsidered '96. In Proceedings of the Second International Conference on Conceptions of Library and Information Science: Integration in Perspective, ed. P. Ingwersen and N. O. Pors, 201-218. The Royal School of Librarianship.
[42]
Saracevic, T., P. Kantor, A. Chamis, and D. Trivison (1988). A study of information seeking and retrieving. I. Background and methodology. Journal of the American Society for Information Science 39(3), 161-176.
[43]
Schamber, L. (1994). Relevance and information behavior. In Annual Review of Information Science and Technology, Vol. 29, ed. M. E. Williams, 3-48.
[44]
Soergel, D. (1999). The rise of ontologies or the reinvention of classification. Journal of the American Association for Information Science 50(12), 1119-1120.
[45]
Sormunen, E. (2000). A method for measuring wide range performance of Boolean queries in full-text database. Ph. D. thesis, University of Tampere. Available at acta.uta.fi/english/teos.phtml?3786
[46]
Sormunen, E., 3. Kekäläinen, J. Koivisto, and K. Järvelin (2001). Document text characteristics affect the ranking of the most relevant documents by expanded structured queries. Journal of Documentation 57(3), 358-376.
[47]
Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28, 11-21.
[48]
Tague-Sutcliffe, J. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing and Management 28(4), 467-490.
[49]
Turtle, H. R. (1990). Inference networks for document retrieval. Ph.D. thesis, University of Massachusetts.
[50]
Uschold, M., and M. Gruninger (1996). Ontologies: Principles, methods and applications. The Knowledge Engineering Review 11(2), 93-136.
[51]
Vakkari, P., and N. Hakala (2000). Changes in relevance criteria and problem stages in task performance. Journal of Documentation 56, 540-562.
[52]
Xu, J., and W. B. Croft (1996). Query expansion using local and global document analysis. In Proceedings of the 19th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, ed. H.-P. Frei, D. Harman, P. Schäuble, and R. Wilkinson, 4-11.

Cited By

View all
  • (2019)Investigating Cognitive Effects in Session-level Search User SatisfactionProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3292500.3330981(923-931)Online publication date: 25-Jul-2019

Recommendations

Reviews

Ian Ruthven

Rosemann and Green consider some ontological constructs defined by Bunge, and made more specific in the context of information systems by Wand and Weber. These constructs provide a semantically clear and solid foundation for understanding and creating domain models of various information systems (IS)-related domains, independently of whether computer-based systems are, or are planned to be, used for automation of some processes within these domains. The authors claim that the understandability of these ontological constructs could be improved by presenting "a meta model [...] using a meta language that is familiar to many IS professionals." The language chosen in the paper is that of an extended entity-relationship (ER) model. These claims are dubious since the semantics of important constructs used in the extended ER model have not been clearly specified (often relying on the fallacy of so-called meaningful names). Second, the representation used in the ER diagrams is not based on semiotic principles (similar, but somewhat different constructs, such as relationships, have not been presented in similar, but somewhat different ways). Third, at least some ER diagrams shown in the paper are unclear. The visual familiarity of ER diagramming constructs may provide an illusion of understandability to some readers of these diagrams, thus giving "a superficial but false sense of security" (Dijkstra). Furthermore, the composition relationship (perhaps the most important one among the ontological constructs by Bunge, Wand, and Weber) has not been properly addressed in the extended ER model. A substantially better treatment of this (and other) ontological constructs was provided by Wand in "A proposal for a formal model of objects" [1], which was not included in the paper's lengthy reference list. In addition, the semantics of important constructs often used in various extended ER models have been clearly defined in international standards, such as the Reference Model of Open Distributed Processing (RMODP) and the General Relationship Model (GRM), and these definitions (also not mentioned in this paper) are close to the ones used by Bunge, Wand, and Weber. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide books
Exploring artificial intelligence in the new millennium
January 2003
414 pages
ISBN:1558608117

Publisher

Morgan Kaufmann Publishers Inc.

San Francisco, CA, United States

Publication History

Published: 01 January 2003

Qualifiers

  • Chapter

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2019)Investigating Cognitive Effects in Session-level Search User SatisfactionProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3292500.3330981(923-931)Online publication date: 25-Jul-2019

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media