Critiquing-based Modeling of Subjective Preferences
Pages 234 - 242
Abstract
Applications designed for entertainment and other non-instrumental purposes are challenging to optimize because the relationships between system parameters and user experience can be unclear. Ideally, we would crowdsource these design questions, but existing approaches are geared towards evaluation or ranking discrete choices and not for optimizing over continuous parameter spaces. In addition, users are accustomed to informally expressing opinions about experiences as critiques (e.g. it’s too cold, too spicy, too big), rather than giving precise feedback as an optimization algorithm would require. Unfortunately, it can be difficult to analyze qualitative feedback, especially in the context of quantitative modeling.
In this article, we present collective criticism, a critiquing-based approach for modeling relationships between system parameters and subjective preferences. We transform critiques, such as “it was too easy/too challenging”, into censored intervals and analyze them using interval regression. Collective criticism has several advantages over other approaches: “too much/too little”-style feedback is intuitive for users and allows us to build predictive models for the optimal parameterization of the variables being critiqued. We present two studies where we model: These studies demonstrate the flexibility of our approach, and show that it produces robust results that are straightforward to interpret and inline with users’ stated preferences.
Supplementary Material
Presentation video for the UMAP'22 paper Critiquing-based Modeling of Subjective Preferences
- Download
- 63.23 MB
References
[1]
Omar Alonso, Daniel E Rose, and Benjamin Stewart. 2008. Crowdsourcing for relevance evaluation. In ACM SigIR Forum, Vol. 42. ACM New York, NY, USA, 9–15.
[2]
Takeshi Amemiya. 1973. Regression analysis when the dependent variable is truncated normal. Econometrica: Journal of the Econometric Society (1973), 997–1016.
[3]
Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing subjective tasks: the case study of understanding toxicity in online discussions. In Companion proceedings of the 2019 world wide web conference. 1100–1105.
[4]
Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. 2018. The moral machine experiment. Nature 563, 7729 (2018), 59–64.
[5]
Beina Azadgoli, Daniel J Gould, Emma Vartanian, and Ketan M Patel. 2019. The public’s perception on breast and nipple reconstruction: a crowdsourcing-based assessment. Aesthetic surgery journal 39, 9 (2019), NP370–NP376.
[6]
Rushlene Kaur Bakshi, Navneet Kaur, Ravneet Kaur, and Gurpreet Kaur. 2016. Opinion mining and sentiment analysis. In 2016 3rd international conference on computing for sustainable global development (INDIACom). IEEE, 452–455.
[7]
Sean Baron. 2012. Cognitive flow: the psychology of great game design. Gamastura (2012).
[8]
Tara S Behrend, David J Sharek, Adam W Meade, and Eric N Wiebe. 2011. The viability of crowdsourcing for survey research. Behavior research methods 43, 3 (2011), 800–813.
[9]
Regina Bernhaupt. 2010. User experience evaluation in entertainment. In Evaluating user experience in games. Springer, 3–7.
[10]
Robin Burke. 2002. Interactive critiquing forcatalog navigation in e-commerce. Artificial Intelligence Review 18, 3 (2002), 245–267.
[11]
Robin D Burke, Kristian J Hammond, and BC Yound. 1997. The FindMe approach to assisted browsing. IEEE Expert 12, 4 (1997), 32–40.
[12]
Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of thompson sampling. In Advances in neural information processing systems. 2249–2257.
[13]
Li Chen and Pearl Pu. 2012. Critiquing-based recommenders: survey and emerging trends. User Modeling and User-Adapted Interaction 22, 1 (2012), 125–150.
[14]
Matthew JC Crump, John V McDonnell, and Todd M Gureckis. 2013. Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PloS one 8, 3 (2013), e57410.
[15]
Mihaly Csikzentmihaly. 1990. Flow: The psychology of optimal experience. Vol. 1990. Harper & Row New York.
[16]
Kapil Dev, Nicolas Villar, and Manfred Lau. 2017. Polygons, points, or voxels? stimuli selection for crowdsourcing aesthetics preferences of 3D shape pairs. In Proceedings of the symposium on Computational Aesthetics. 1–7.
[17]
Most Attractive Expressions. [n. d.]. Mirror Mirror: Crowdsourcing Better Portraits. To appear in ACM TOG 33([n. d.]), 6.
[18]
Boi Faltings, Pearl Pu, Marc Torrens, and Paolo Viappiani. 2004. Designing example-critiquing interaction. In Proceedings of the 9th international conference on Intelligent user interfaces. 22–29.
[19]
Elena Filatova. 2012. Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing. In Lrec. Citeseer, 392–398.
[20]
Gianfranco Frojo, Aurora M Kareh, Kenneth X Probst, Jeffrey D Rector, Christina M Plikaitis, Herluf G Lund, and Alexander Y Lin. 2021. Defining Ideal External Female Genital Anatomy Via Crowdsourcing Analysis. Aesthetic surgery journal(2021).
[21]
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576(2015).
[22]
Marc Hassenzahl. 2008. User experience (UX) towards an experiential perspective on product quality. In Proceedings of the 20th Conference on l’Interaction Homme-Machine. 11–15.
[23]
Marc Hassenzahl and Noam Tractinsky. 2006. User experience-a research agenda. Behaviour & information technology 25, 2 (2006), 91–97.
[24]
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2020. A survey on conversational recommender systems. arXiv preprint arXiv:2004.00646(2020).
[25]
Yucheng Jin, Wanling Cai, Li Chen, Nyi Nyi Htun, and Katrien Verbert. 2019. MusicBot: Evaluating critiquing-based music recommenders with conversational interaction. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 951–960.
[26]
Yuan Jin, Mark Carman, Ye Zhu, and Yong Xiang. 2020. A technical survey on statistical modelling and design methods for crowdsourcing quality control. Artificial Intelligence(2020), 103351.
[27]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694–711.
[28]
Ron Kohavi and Roger Longbotham. 2017. Online Controlled Experiments and A/B Testing.Encyclopedia of machine learning and data mining 7, 8 (2017), 922–929.
[29]
Ron Kohavi and Stefan Thomke. 2017. The surprising power of online experiments. Harvard Business Review(2017).
[30]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.
[31]
Gitte Lindgaard and Jurek Kirakowski. 2013. The tricky landscape of developing rating scales in HCI. Interacting with Computers 25, 4 (2013), 271–277.
[32]
Jonathan P Massie, Rachita Sood, Ian T Nolan, Daniel C Sasson, Marco Swanson, Shane D Morrison, and Otto Placik. 2021. Defining aesthetic preferences for the penis: a photogrammetric and crowdsourcing analysis. Aesthetic surgery journal(2021).
[33]
Alan Medlar, Joel Pyykkö, and Dorota Glowacka. 2017. Towards fine-grained adaptation of exploration/exploitation in information retrieval. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. 623–627.
[34]
Maria Perez-Ortiz and Rafal K Mantiuk. 2017. A practical guide and software for analysing pairwise comparison experiments. arXiv preprint arXiv:1712.03686(2017).
[35]
Pearl Pu and Boi Faltings. 2004. Decision tradeoff using example-critiquing and constraint programming. Constraints 9, 4 (2004), 289–310.
[36]
José Luis González Sánchez, Francisco Luis Gutiérrez Vela, Francisco Montero Simarro, and Natalia Padilla-Zea. 2012. Playability: analysing user experience in video games. Behaviour & Information Technology 31, 10 (2012), 1033–1054.
[37]
Natasha Dow Schull. 2005. Digital gambling: The coincidence of desire and design. The Annals of the American Academy of Political and Social Science 597, 1(2005), 65–81.
[38]
Noor Shaker, Georgios N Yannakakis, and Julian Togelius. 2012. Crowdsourcing the aesthetics of platform games. IEEE Transactions on Computational Intelligence and AI in Games 5, 3(2012), 276–290.
[39]
Marissa Suchyta, Amee Azad, Ashraf A Patel, Rohit K Khosla, H Peter Lorenz, and Rahim S Nazerali. 2020. Applied online crowdsourcing in plastic and reconstructive surgery: a comparison of aesthetic outcomes in unilateral cleft lip repair techniques. Annals of plastic surgery 84, 5S (2020), S307–S310.
[40]
Steve Swink. 2008. Game feel: a game designer’s guide to virtual sensation. CRC Press.
[41]
Terry M Therneau and Thomas Lumley. 2014. Package ‘survival’. Survival analysis Published on CRAN 2 (2014), 3.
[42]
Raymond W Tse, Eugene Oh, Joseph S Gruss, Richard A Hopper, and Craig B Birgfeld. 2016. Crowdsourcing as a novel method to evaluate aesthetic outcomes of treatment for unilateral cleft lip. Plastic and reconstructive surgery 138, 4 (2016), 864–874.
[43]
Zeynep Tufekci. 2018. YouTube, the great radicalizer. The New York Times 10(2018), 2018.
[44]
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022(2016).
[45]
Niels Van Berkel, Jorge Goncalves, Danula Hettiachchi, Senuri Wijenayake, Ryan M Kelly, and Vassilis Kostakos. 2019. Crowdsourcing perceptions of fair predictors for machine learning: a recidivism case study. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–21.
[46]
Emma Vartanian, Daniel J Gould, Ziyad S Hammoudeh, Beina Azadgoli, W Grant Stevens, and Luis H Macias. 2018. The ideal thigh: a crowdsourcing-based assessment of ideal thigh aesthetic and implications for gluteal fat grafting. Aesthetic surgery journal 38, 8 (2018), 861–869.
[47]
Paolo Viappiani, Boi Faltings, and Pearl Pu. 2006. Preference-based search using example-critiquing with suggestions. Journal of artificial intelligence Research 27 (2006), 465–503.
[48]
Jesse Vig, Shilad Sen, and John Riedl. 2011. Navigating the tag genome. In Proceedings of the 16th international conference on Intelligent user interfaces. 93–102.
[49]
Ga Wu, Kai Luo, Scott Sanner, and Harold Soh. 2019. Deep language-based critiquing for recommender systems. In Proceedings of the 13th ACM Conference on Recommender Systems. 137–145.
Index Terms
- Critiquing-based Modeling of Subjective Preferences
Index terms have been assigned to the content through auto-classification.
Recommendations
Latent Linear Critiquing for Conversational Recommender Systems
WWW '20: Proceedings of The Web Conference 2020Critiquing is a method for conversational recommendation that iteratively adapts recommendations in response to user preference feedback. In this setting, a user is iteratively provided with an item recommendation and attribute description for that item;...
Deep language-based critiquing for recommender systems
RecSys '19: Proceedings of the 13th ACM Conference on Recommender SystemsCritiquing is a method for conversational recommendation that adapts recommendations in response to user preference feedback regarding item attributes. Historical critiquing methods were largely based on constraint- and utility-based methods for ...
User Experience and the Role of Personalization in Critiquing-Based Conversational Recommendation
Critiquing—where users propose directional preferences to attribute values—has historically been a highly popular method for conversational recommendation. However, with the growing size of catalogs and item attributes, it becomes increasingly difficult ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2022
360 pages
ISBN:9781450392075
DOI:10.1145/3503252
- General Chairs:
- Alejandro Bellogin,
- Ludovico Boratto,
- Olga C. Santos,
- Program Chairs:
- Liliana Ardissono,
- Bart Knijnenburg
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 July 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
UMAP '22: 30th ACM Conference on User Modeling, Adaptation and Personalization
July 4 - 7, 2022
Barcelona, Spain
Acceptance Rates
Overall Acceptance Rate 162 of 633 submissions, 26%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 223Total Downloads
- Downloads (Last 12 months)92
- Downloads (Last 6 weeks)17
Reflects downloads up to 10 Dec 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in