research-article

Grannies, tanning beds, tattoos and NASCAR: evaluation of search tasks with varying levels of cognitive complexity

Authors:

Wan-Ching Wu,

Diane Kelly,

Ashlee Edwards,

Jaime ArguelloAuthors Info & Claims

IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

Pages 254 - 257

https://doi.org/10.1145/2362724.2362768

Published: 21 August 2012 Publication History

Get Access

Abstract

One of the most challenging aspects of designing an interactive information retrieval (IIR) study is the development of search tasks. In this paper, we present preliminary results of a study designed to evaluate a set of search tasks that were developed for use in IIR studies. We created 20 search tasks using five levels of cognitive complexity and four domains, and conducted a laboratory evaluation of these tasks with 48 undergraduate subjects. We describe preliminary results from an analysis of data from 24 subjects for 10 search tasks. Initial results show that, in general, as cognitive complexity increased, subjects issued more queries, clicked on more search results, viewed more URLs and took more time to complete the task. Subjects' expected and experienced difficulty ratings of tasks generally increased as cognitive complexity increased with some exceptions. When subjects were asked to rank tasks according to difficulty and engagement, tasks with higher cognitive complexity were rated as more difficult than tasks with lower cognitive complexity, but not necessarily as more engaging. These preliminary results suggest that behaviors and ratings are fairly consistent with the differences one might expect among the search tasks and provide initial evidence of the usefulness of these tasks in IIR studies.

References

[1]

Anderson, L. W. & Krathwohl, D. A. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman.

Google Scholar

[2]

Arguello, J., Wu, W. C., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proceedings of SIGIR '12, Portland, OR.

Digital Library

Google Scholar

[3]

Borlund, P. (2003). The IIR evaluation model: A framework for the evaluation of interactive information retrieval systems. Information Research, 8(3), paper 152.

Google Scholar

[4]

Byström, K. & Jäärvelin, K. (1995). Task complexity affects information seeking and use. IP&M, 31(2), 191--213.

Digital Library

Google Scholar

[5]

Jansen, B. J., Booth, D., & Smith, D. (2009). Using the taxonomy of cognitive learning to model online searching. IP&M, 45, 643--663.

Digital Library

Google Scholar

[6]

Li, Y. & Belkin, N. J. (2008). A faceted approach to conceptualizing tasks in information seeking. IP&M, 44, 1822--1837.

Digital Library

Google Scholar

[7]

Toms, E. G. (2011). Task-based information searching and retrieval. In I. Ruthven & D. Kelly (Eds.) Interactive Information Seeking, Behaviour and Retrieval. Facet Publishing, 43--59.

Google Scholar

[8]

Vakkari, P. (2003). Task-based information searching. ARIST, 37, 413--464.

Google Scholar

[9]

Whittaker, S., Terveen, L., & Nardi, B. (2000). Let's stop pushing the envelope and start addressing it: a reference task agenda for HCI. Human Computer Interaction, 15, 75--106.

Digital Library

Google Scholar

[10]

Wildemuth, B. M. & Freund, L. (2009) Search tasks and their role in studies of search behaviors. Paper presented at the Third Annual Workshop on HCIR, Washington D.C.

Google Scholar

Cited By

View all

Sharma TKyi LWang YBiega ABalzarotti DXu W(2024)"I'm not convinced that they don't collect more than is necessary"Proceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699057(2797-2812)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699057
Medlar AKotkov DGłowacka D(2024)Unexplored Frontiers: A Review of Empirical Studies of Exploratory SearchACM SIGIR Forum10.1145/3687273.368727858:1(1-19)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3687273.3687278
Liu SHu YTian ZJin ZRuan SMao JSakai TIshita EOhshima HHasibi FMao JJose J(2024)Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search TasksProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698406(103-113)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698406
Show More Cited By

Index Terms

Grannies, tanning beds, tattoos and NASCAR: evaluation of search tasks with varying levels of cognitive complexity
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
2. Information systems
  1. Information retrieval
    1. Information retrieval query processing

Recommendations

Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework
ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval

One of the most challenging aspects of designing interactive information retrieval (IIR) experiments with users is the development of search tasks. We describe an evaluation of 20 search tasks that were designed for use in IIR experiments and developed ...
An empirical study of interest, task complexity, and search behaviour on user engagement
Highlights
- User engagement (UE) is an important outcome measure in interactive information retrieval.
Abstract
User engagement has become an important outcome measure in interactive information retrieval (IIR) research, as commercial (e.g., search engines and e-commerce companies) and educational (e.g., libraries) enterprises focus on capturing ...
Representing Tasks with a Graph-Based Method for Supporting Users in Complex Search Tasks
CHIIR '23: Proceedings of the 2023 Conference on Human Information Interaction and Retrieval

Despite the considerable advancements in modern search systems for assisting users in search tasks of varying types, support for complex tasks that call for multi-round interactions remains challenging. Identifying users’ tasks is essential to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

August 2012

347 pages

ISBN:9781450312820

DOI:10.1145/2362724

General Chairs:
Jaap Kamps,
Wessel Kraaij,
Program Chair:
Norbert Fuhr
University of Duisburg-Essen

In-Cooperation

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IIiX'12

Sponsor:

University of Amsterdam

IIiX'12: Information Interaction in Context: 2012

August 21 - 24, 2012

Nijmegen, The Netherlands

Acceptance Rates

Overall Acceptance Rate 21 of 45 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

78
Total Citations
View Citations
352
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sharma TKyi LWang YBiega ABalzarotti DXu W(2024)"I'm not convinced that they don't collect more than is necessary"Proceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699057(2797-2812)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699057
Medlar AKotkov DGłowacka D(2024)Unexplored Frontiers: A Review of Empirical Studies of Exploratory SearchACM SIGIR Forum10.1145/3687273.368727858:1(1-19)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3687273.3687278
Liu SHu YTian ZJin ZRuan SMao JSakai TIshita EOhshima HHasibi FMao JJose J(2024)Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search TasksProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698406(103-113)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698406
Xu XPalani SMunkhbat ALee TDow S(2024)Idea-Centric Search: Four Patterns of Information Seeking During Creative IdeationProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656193(280-291)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656193
Urgo KArguello J(2024)The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning ProcessesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638348(278-290)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638348
Zendel OCulpepper JScholer FThomas P(2024)Enhancing Human Annotation: Leveraging Large Language Models and Efficient Batch ProcessingProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638322(340-345)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638322
Li AO'Brien HSinnamon L(2023)The effect of research video abstract presentation style on viewer comprehension and engagementProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578326(394-400)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578326
Urgo KArguello J(2022)Understanding the “Pathway” Towards a Searcher’s Learning ObjectiveACM Transactions on Information Systems10.1145/349522240:4(1-43)Online publication date: 11-Jan-2022
https://dl.acm.org/doi/10.1145/3495222
Song QJiang J(2022)How Misinformation Density Affects Health Information SearchProceedings of the ACM Web Conference 202210.1145/3485447.3512141(2668-2677)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512141
Urgo KArguello J(2022)Learning assessments in search-as-learningInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10282159:2Online publication date: 9-May-2022
https://dl.acm.org/doi/10.1016/j.ipm.2021.102821
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework

An empirical study of interest, task complexity, and search behaviour on user engagement

Representing Tasks with a Graph-Based Method for Supporting Users in Complex Search Tasks

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations