Abstract
Privacy policies are often lengthy and complex legal documents, and are difficult for many people to read and comprehend. Recent research efforts have explored automated assistants that process the language in policies and answer people’s privacy questions. This study documents the importance of two different types of reasoning necessary to generate accurate answers to people’s privacy questions. The first is the need to support taxonomic reasoning about related terms commonly found in privacy policies. The second is the need to reason about regulatory disclosure requirements, given the prevalence of silence in privacy policy texts. Specifically, we report on a study involving the collection of 749 sets of expert annotations to answer privacy questions in the context of 210 different policy/question pairs. The study highlights the importance of taxonomic reasoning and of reasoning about regulatory disclosure requirements when it comes to accurately answering everyday privacy questions. Next we explore to what extent current generative AI tools are able to reliably handle this type of reasoning. Our results suggest that in their current form and in the absence of additional help, current models cannot reliably support the type of reasoning about regulatory disclosure requirements necessary to accurately answer privacy questions. We proceed to introduce and evaluate different approaches to improving their performance. Through this work, we aim to provide a richer understanding of the capabilities automated systems need to have to provide accurate answers to everyday privacy questions and, in the process, outline paths for adapting AI models for this purpose.
A. Ravichander and I. Yang—The first two authors contributed equally to this work. Please refer questions about this research to the last author, Prof. N. Sadeh - sadeh@cs.cmu.edu.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Our interpretation of this pseudo-hypernym is that payment information can include Point-of-Sale data and therefore be indicative of a location visited by the user.
- 2.
Accessed Nov 30, 2023.
- 3.
We find that Llama-2 often generates responses we are unable to map to any of our categories, in these cases we do not assign a positive score to the model’s prediction.
References
Andow, B., et al.: \(\{\)PolicyLint\(\}\): investigating internal privacy policy contradictions on google play. In: 28th USENIX Security Symposium (2019)
Balaji, A., Duesterwald, L., Yang, I., Priyanshu, A., Alfieri, C., Sadeh, N.: Generating effective answers to people’s everyday cybersecurity questions: an initial study. In: International Conference on Web Information Systems Engineering (2024)
Bhatia, J., et al.: Automated extraction of regulated information types using hyponymy relations. In: 2016 IEEE 24th International Requirements Engineering Conference Workshops, pp. 19–25. IEEE (2016)
Evans, M.C., et al.: An evaluation of constituency-based hyponymy extraction from privacy policies. In: IEEE 25th International Requirements Engineering Conference (2017)
Harkous, H., et al.: Polisis: automated analysis and presentation of privacy policies using deep learning. arXiv preprint arXiv:1802.02561 (2018)
Kelley, P.G., et al.: A nutrition label for privacy. In: Proceedings of the 5th Symposium on Usable Privacy and Security. ACM (2009)
Kelley, P.G., et al.: Privacy as part of the app decision-making process. Association for Computing Machinery (2013)
Manandhar, S., et al.: Smart home privacy policies demystified: a study of availability, content, and coverage. In: 31st USENIX Security Symposium (2022)
Mysore Sathyendra, K., Wilson, S., Schaub, F., Zimmeck, S., Sadeh, N.: Identifying the provision of choices in privacy policy text. In: EMNLP (2017)
Oltramari, A., et al.: PrivOnto: a semantic framework for the analysis of privacy policies. Seman. Web 9(2), 185–203 (2017)
OpenAI: GPT-4 technical report (2023)
Ravichander, A., et al.: Question answering for privacy policies: combining computational and legal perspectives. In: EMNLP (2019)
Reidenberg, J.R., et al.: Disagreeable privacy policies: mismatches between meaning and users’ understanding. Berkeley Tech. LJ 30, 39 (2015)
Sadeh, N., et al.: The usable privacy policy project: combining crowdsourcing, machine learning and natural language processing to semi-automatically answer those privacy questions users care about, Technical Report CMU-ISR-13-119, Carnegie Mellon University, Pittsburgh, Pennsylvania (2013)
Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models (2023)
Wilson, S., et al.: The creation and analysis of a website privacy policy corpus. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2016)
Zhang, S., et al.: How usable are IOS app privacy labels. Proc. Priv. Enhancing Technol. 2022(4) (2022)
Zimmeck, S., et al.: Maps: scaling privacy compliance analysis to a million apps. Proc. Priv. Enhancing Technol. 2019(3), 66–86 (2019)
Acknowledgments
This research has been supported in part by grants from the National Science Foundation under the SaTC program (grants CNS-1914486, 1914444, 1914446) and under the REU program, the latter in part through CMU’s RE-USE Program (NSF grant 2150217).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ravichander, A., Yang, I., Chen, R., Wilson, S., Norton, T., Sadeh, N. (2025). Incorporating Taxonomic Reasoning and Regulatory Knowledge into Automated Privacy Question Answering. In: Barhamgi, M., Wang, H., Wang, X. (eds) Web Information Systems Engineering – WISE 2024. WISE 2024. Lecture Notes in Computer Science, vol 15436. Springer, Singapore. https://doi.org/10.1007/978-981-96-0579-8_31
Download citation
DOI: https://doi.org/10.1007/978-981-96-0579-8_31
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-0578-1
Online ISBN: 978-981-96-0579-8
eBook Packages: Computer ScienceComputer Science (R0)