More Web Proxy on the site http://driver.im/

research-article

Open access

Requirements and Concepts for Interactive Media Retrieval User Interfaces

Authors:

Victor Adriel de Jesus Oliveira,

Gernot Rottermanner,

Stefanie Größbacher,

Magdalena Boucher,

Peter JudmaierAuthors Info & Claims

NordiCHI '22: Nordic Human-Computer Interaction Conference

Article No.: 4, Pages 1 - 10

https://doi.org/10.1145/3546155.3546701

Published: 08 October 2022 Publication History

All formats PDF

Abstract

Large amounts of multimedia data are constantly digitized and stored in archives. Accurate search and annotation tools are essential for the fast retrieval of archival records by archivists, scientists, and the general public. The complexity of processing and navigating large collections evidence the demand for solutions that are tailored to the needs of diverse target groups. In this paper, we investigate the requirements for multimedia search and annotation tools. After identifying examples of graphical user interfaces and visualization techniques to support navigating and annotating audiovisual content in archives, we performed an iterative user research. Based on expert interviews, focus groups, and surveys, we propose a series of requirements and concept ideas for user interfaces aimed at quality control and AI-assisted search of multimedia data. Results also show that open challenges and needs include the definition of tailored ontologies to describe archival multimedia data.

References

[1]

Hosam Al-Samarraie and Shuhaila Hurmuzan. 2018. A review of brainstorming techniques in higher education. Thinking Skills and Creativity 27 (2018), 78–91.

[2]

Giuseppe Amato, Paolo Bolettieri, Fabio Carrara, Franca Debole, Fabrizio Falchi, Claudio Gennaro, Lucia Vadicamo, and Claudio Vairo. 2021. The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval. Journal of Imaging 7, 5 (2021). https://doi.org/10.3390/jimaging7050076

[3]

Mohammadreza Babaee, Stefanos Tsoukalas, Gerhard Rigoll, and Mihai Datcu. 2015. Visualization-Based Active Learning for the Annotation of SAR Images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 8, 10 (Oct. 2015), 4687–4698. https://doi.org/10.1109/JSTARS.2015.2388496

[4]

V. Barbosa, T. Pellegrini, M. Bugalho, and I. Trancoso. 2011. Browsing videos by automatically detected audio events. In 2011 IEEE EUROCON - International Conference on Computer as a Tool. IEEE, Lisbon, 1–4. https://doi.org/10.1109/EUROCON.2011.5929358

[5]

Jurgen Bernard, Marco Hutter, Matthias Zeppelzauer, Dieter Fellner, and Michael Sedlmair. 2018. Comparing Visual-Interactive Labeling with Active Learning: An Experimental Study. IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan. 2018), 298–308. https://doi.org/10.1109/TVCG.2017.2744818

[6]

Jürgen Bernard, Matthias Zeppelzauer, Michael Sedlmair, and Wolfgang Aigner. 2018. VIAL: a unified process for visual interactive labeling. The Visual Computer 34, 9 (Sept. 2018), 1189–1207. https://doi.org/10.1007/s00371-018-1500-3

Digital Library

[7]

Yen-ning Chang, Youn-kyung Lim, and Erik Stolterman. 2008. Personas: From Theory to Practices. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction: Building Bridges (Lund, Sweden) (NordiCHI ’08). Association for Computing Machinery, New York, NY, USA, 439–442. https://doi.org/10.1145/1463160.1463214

Digital Library

[8]

Santanu Chaudhury, Anupama Mallik, and Hiranmay Ghosh. 2015. Multimedia ontology: representation and applications. CRC Press.

[9]

Mehdi Elahi, Reza Hosseini, Mohammad H. Rimaz, Farshad B. Moghaddam, and Christoph Trattner. 2020. Visually-Aware Video Recommendation in the Cold Start. In Proceedings of the 31st ACM Conference on Hypertext and Social Media. ACM, Virtual Event USA, 225–229. https://doi.org/10.1145/3372923.3404778

Digital Library

[10]

A. Endert, W. Ribarsky, C. Turkay, W. Wong, I. Nabney, I. Díaz Blanco, and Fabrice Rossi. 2017. The State of the Art in Integrating Machine Learning into Visual Analytics. Computer Graphics Forum 36, 8 (Dec. 2017), 458–486. https://doi.org/10.1111/cgf.13092 arXiv:1802.07954.

[11]

Maria Eskevich, Huynh Nguyen, Mathilde Sahuguet, and Benoit Huet. 2015. Hyper Video Browser: Search and Hyperlinking in Broadcast Media. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, Brisbane Australia, 817–818. https://doi.org/10.1145/2733373.2812618

Digital Library

[12]

Ralph Ewerth, Markus Mühling, Thilo Stadelmann, Julinda Gllavata, Manfred Grauer, and Bernd Freisleben. 2015. Videana: A Software Toolkit for Scientific Film Studies. transcript Verlag, 101–116. https://doi.org/10.1515/9783839410233-008

[13]

Alireza Fathi, Maria Florina Balcan, Xiaofeng Ren, and James M. Rehg. 2011. Combining Self Training and Active Learning for Video Segmentation. In Procedings of the British Machine Vision Conference 2011. British Machine Vision Association, Dundee, 78.1–78.11. https://doi.org/10.5244/C.25.78

[14]

Manali Gaikwad and Orland Hoeber. 2019. An Interactive Image Retrieval Approach to Searching for Images on Social Media. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. ACM, Glasgow Scotland UK, 173–181. https://doi.org/10.1145/3295750.3298930

Digital Library

[15]

Ralph Gasser, Luca Rossetto, and Heiko Schuldt. 2019. Towards an All-Purpose Content-Based Multimedia Information Retrieval System. arXiv:1902.03878 [cs] (Feb. 2019). http://arxiv.org/abs/1902.03878 arXiv:1902.03878.

[16]

Ben Green and Yiling Chen. 2019. The Principles and Limits of Algorithm-in-the-Loop Decision Making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (Nov. 2019), 1–24. https://doi.org/10.1145/3359152

Digital Library

[17]

Benjamin Hoferlin, Rudolf Netzel, Markus Hoferlin, Daniel Weiskopf, and Gunther Heidemann. 2012. Inter-active learning of ad-hoc classifiers for video visual analytics. In 2012 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, Seattle, WA, USA, 23–32. https://doi.org/10.1109/VAST.2012.6400492

Digital Library

[18]

Lulu Huang, Stan Matwin, Eder J. de Carvalho, and Rosane Minghim. 2017. Active Learning with Visualization for Text Data. In Proceedings of the 2017 ACM Workshop on Exploratory Search and Interactive Data Analytics - ESIDA ’17. ACM Press, Limassol, Cyprus, 69–74. https://doi.org/10.1145/3038462.3038469

Digital Library

[19]

Wolfgang Hurst and Rob van de Werken. 2015. Human-Based Video Browsing - Investigating Interface Design for Fast Video Browsing. In 2015 IEEE International Symposium on Multimedia (ISM). IEEE, Miami, FL, 363–368. https://doi.org/10.1109/ISM.2015.104

[20]

Ichiro Ide, Tomoyoshi Kinoshita, Tomokazu Takahashi, Hiroshi Mo, Norio Katayama, Shin’ichi Satoh, and Hiroshi Murase. 2012. Efficient Tracking of News Topics Based on Chronological Semantic Structures in a Large-Scale News Video Archive. IEICE Transactions on Information and Systems E95.D, 5(2012), 1288–1300. https://doi.org/10.1587/transinf.E95.D.1288

[21]

Masahiko Itoh, Masashi Toyoda, Cai Zhi Zhu, Shin’ichi Satoh, and Masaru Kitsuregawa. 2014. Image Flows Visualization for Inter-media Comparison. In 2014 IEEE Pacific Visualization Symposium. IEEE, Yokohama, 129–136. https://doi.org/10.1109/PacificVis.2014.49

Digital Library

[22]

Dag Johansen, Pål Halvorsen, Håvard Johansen, Håkon Riiser, Cathal Gurrin, Bjørn Olstad, Carsten Griwodz, Åge Kvalnes, Joseph Hurley, and Tomas Kupka. 2012. Search-based composition, streaming and playback of video archive content. Multimedia Tools and Applications 61, 2 (Nov. 2012), 419–445. https://doi.org/10.1007/s11042-011-0847-5

[23]

Tomas Koctur, Matus Pleva, and Jozef Juhar. 2015. Interface for smart audiovisual data archive. In 2015 25th International Conference Radioelektronika (RADIOELEKTRONIKA). IEEE, Pardubice, Czech Republic, 292–294. https://doi.org/10.1109/RADIOELEK.2015.7129036

[24]

Gregor Kovalčík, Vít Škrhak, Tomáš Souček, and Jakub Lokoč. 2020. VIRET Tool with Advanced Visual Browsing and Feedback. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. ACM, Dublin Ireland, 63–66. https://doi.org/10.1145/3379172.3391725

Digital Library

[25]

Kostiantyn Kucher, Carita Paradis, Magnus Sahlgren, and Andreas Kerren. 2017. Active Learning and Visual Analytics for Stance Classification with ALVA. ACM Transactions on Interactive Intelligent Systems 7, 3 (Oct. 2017), 1–31. https://doi.org/10.1145/3132169

Digital Library

[26]

Yaman Kumar, Agniv Sharma, Abhigyan Khaund, Akash Kumar, Ponnurangam Kumaraguru, and Rajiv Ratn Shah. 2018. IceBreaker: Solving Cold Start Problem for Video Recommendation Engines. https://doi.org/10.48550/ARXIV.1808.05636

[27]

Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.

[28]

Hongsen Liao, Li Chen, Yibo Song, and Hao Ming. 2016. Visualization-Based Active Learning for Video Annotation. IEEE Transactions on Multimedia 18, 11 (Nov. 2016), 2196–2205. https://doi.org/10.1109/TMM.2016.2614227

Digital Library

[29]

Christian Limberg, Kathrin Krieger, Heiko Wersing, and Helge Ritter. 2019. Active Learning for Image Recognition Using a Visualization-Based User Interface. In Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning, Igor V. Tetko, Věra Kůrková, Pavel Karpov, and Fabian Theis (Eds.). Vol. 11728. Springer International Publishing, Cham, 495–506. https://doi.org/10.1007/978-3-030-30484-3_40

Digital Library

[30]

Qi Liu, Ruobing Xie, Lei Chen, Shukai Liu, Ke Tu, Peng Cui, Bo Zhang, and Leyu Lin. 2020. Graph Neural Network for Tag Ranking in Tag-enhanced Video Recommendation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. ACM, Virtual Event Ireland, 2613–2620. https://doi.org/10.1145/3340531.3416021

Digital Library

[31]

Shenglan Liu, Xiang Liu, Yang Liu, Lin Feng, Hong Qiao, Jian Zhou, and Yang Wang. 2018. Perceptual Visual Interactive Learning. https://doi.org/10.48550/ARXIV.1810.10789

[32]

Jakub Lokoč, Patrik Veselý, František Mejzlík, Gregor Kovalčík, Tomáš Souček, Luca Rossetto, Klaus Schoeffmann, Werner Bailer, Cathal Gurrin, Loris Sauter, Jaeyub Song, Stefanos Vrochidis, Jiaxin Wu, and Björn þóR Jónsson. 2021. Is the Reign of Interactive Search Eternal? Findings from the Video Browser Showdown 2020. ACM Trans. Multimedia Comput. Commun. Appl. 17, 3, Article 91 (jul 2021), 26 pages. https://doi.org/10.1145/3445031

Digital Library

[33]

Yijuan Lu, Nicu Sebe, Ross Hytnen, and Qi Tian. 2011. Personalization in multimedia retrieval: A survey. Multimedia Tools and Applications 51, 1 (Jan. 2011), 247–277. https://doi.org/10.1007/s11042-010-0621-0

Digital Library

[34]

Phuong Anh Nguyen, Yi-Jie Lu, Hao Zhang, and Chong-Wah Ngo. 2018. Enhanced VIREO KIS at VBS 2018. In MultiMedia Modeling, Klaus Schoeffmann, Thanarat H. Chalidabhongse, Chong Wah Ngo, Supavadee Aramvith, Noel E. O’Connor, Yo-Sung Ho, Moncef Gabbouj, and Ahmed Elgammal (Eds.). Vol. 10705. Springer International Publishing, Cham, 407–412. https://doi.org/10.1007/978-3-319-73600-6_42

[35]

Lyndon J.B. Nixon, Shu Zhu, Fabian Fischer, Walter Rafelsberger, Max Göbel, and Arno Scharl. 2017. Video Retrieval for Multimedia Verification of Breaking News on Social Networks. In Proceedings of the First International Workshop on Multimedia Verification. ACM, Mountain View California USA, 13–21. https://doi.org/10.1145/3132384.3132386

Digital Library

[36]

Cathy Pearl. 2016. Designing voice user interfaces: Principles of conversational experiences. ” O’Reilly Media, Inc.”.

[37]

Diogo Pedrosa, Rodrigo Laiola Guimarães, Pablo Cesar, and Dick C. A. Bulterman. 2013. Designing Socially-Aware Video Exploration Interfaces: A Case Study Using School Concert Assets. In Proceedings of International Conference on Making Sense of Converging Media - AcademicMindTrek ’13. ACM Press, Tampere, Finland, 110–117. https://doi.org/10.1145/2523429.2523454

Digital Library

[38]

John Pruitt and Jonathan Grudin. 2003. Personas: practice and theory. In Proceedings of the 2003 conference on Designing for user experiences. 1–15.

Digital Library

[39]

Haolin Ren, Benjamin Renoust, Marie-Luce Viaud, Guy Melancon, and Shin’ichi Satoh. 2018. Generating “Visual Clouds” from Multiplex Networks for TV News Archive Query Visualization. In 2018 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, La Rochelle, 1–6. https://doi.org/10.1109/CBMI.2018.8516482

[40]

Luca Rossetto, Ralph Gasser, Jakub Lokoc, Werner Bailer, Klaus Schoeffmann, Bernd Muenzer, Tomas Soucek, Phuong Anh Nguyen, Paolo Bolettieri, Andreas Leibetseder, and Stefanos Vrochidis. 2021. Interactive Video Retrieval in the Age of Deep Learning – Detailed Evaluation of VBS 2019. IEEE Transactions on Multimedia 23 (2021), 243–256. https://doi.org/10.1109/TMM.2020.2980944

[41]

Yang Song and Qi Guo. 2016. Query-Less: Predicting Task Repetition for NextGen Proactive Search and Recommendation Engines. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Montréal Québec Canada, 543–553. https://doi.org/10.1145/2872427.2883020

Digital Library

[42]

Newton Spolaôr, Huei Diana Lee, Weber Shoity Resende Takaki, Leandro Augusto Ensina, Claudio Saddy Rodrigues Coy, and Feng Chung Wu. 2020. A systematic review on content-based video retrieval. Engineering Applications of Artificial Intelligence 90 (April 2020), 103557. https://doi.org/10.1016/j.engappai.2020.103557

Digital Library

[43]

Wei Xu. 2019. Toward human-centered AI: a perspective from human-computer interaction. Interactions 26, 4 (June 2019), 42–46. https://doi.org/10.1145/3328485

Digital Library

Cited By

Montagnuolo MNegro FMessina ABruccoleri AIacoviello R(2023)Who’s in My Archive? An End-to-End Framework for Automatic Annotation of TV Personalities2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386323(2061-2070)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386323
de Jesus Oliveira VRottermanner GBoucher MGrößbacher SJudmaier PBailer WThallinger GKurz TFrank JBauer CFröschl GBatlogg M(2023)Taylor – Impersonation of AI for Audiovisual Content Documentation and SearchMultiMedia Modeling10.1007/978-3-031-27818-1_63(751-757)Online publication date: 31-Mar-2023
https://doi.org/10.1007/978-3-031-27818-1_63

Index Terms

Requirements and Concepts for Interactive Media Retrieval User Interfaces
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. User centered design
2. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Requirements analysis

Recommendations

Interactive Visual User Interfaces: A Survey

Following a short survey of input data types on which to construct interactive visual user interfaces, we report on a new and recent implementation taking concept hierarchies as input data. The visual user interfaces express domain ontologies which are ...
Creating highly-interactive and graphical user interfaces by demonstration

It is very time-consuming and expensive to create the graphical, highly-interactive styles of user interfaces that are increasingly common. User Interface Management Systems (UIMSs) attempt to make the creation of user interfaces easier, but most ...
Interfaces for interactive audio-visual media browsing
MM '06: Proceedings of the 14th ACM international conference on Multimedia

In this demo, we present new interfaces for interactive navigation in continuous, time-based multimedia files, such as video recordings. In contrast to common techniques for multimedia skimming, our approaches enable users to interactively navigate ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

NordiCHI '22: Nordic Human-Computer Interaction Conference

October 2022

1091 pages

ISBN:9781450396998

DOI:10.1145/3546155

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2022

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK)

Conference

NordiCHI '22

NordiCHI '22: Nordic Human-Computer Interaction Conference

October 8 - 12, 2022

Aarhus, Denmark

Acceptance Rates

Overall Acceptance Rate 379 of 1,572 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
398
Total Downloads

Downloads (Last 12 months)223
Downloads (Last 6 weeks)25

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Montagnuolo MNegro FMessina ABruccoleri AIacoviello R(2023)Who’s in My Archive? An End-to-End Framework for Automatic Annotation of TV Personalities2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386323(2061-2070)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386323
de Jesus Oliveira VRottermanner GBoucher MGrößbacher SJudmaier PBailer WThallinger GKurz TFrank JBauer CFröschl GBatlogg M(2023)Taylor – Impersonation of AI for Audiovisual Content Documentation and SearchMultiMedia Modeling10.1007/978-3-031-27818-1_63(751-757)Online publication date: 31-Mar-2023
https://doi.org/10.1007/978-3-031-27818-1_63

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents